Skip to main content
Home > “Admin Menu” > “Knowledge Bases”
With VARIOS AI, you can create powerful vector databases based on the Retrieval-Augmented Generation (RAG) approach – we call them knowledge bases. Documents are ingested through various data sources (e.g. manual upload, GitHub, WebDAV, VARIOS AI server folder, website), stored as embeddings, and made available via semantic search. Knowledge bases can be assigned to one or more assistants. Benefits:
  • Semantic search: Instead of simple keyword matching, the system searches for semantically similar content using vectorization.
  • Up-to-date knowledge: New or changed documents can be synchronized automatically or manually depending on the source type.
  • Use in assistants: Enable knowledge bases for assistants to incorporate company-specific knowledge into responses (see Global Assistants).

​
Knowledge Base Overview

On the Knowledge Bases page, you can see all created knowledge bases in a table: Actions per row:
  • Edit (pencil icon): Open the knowledge base, manage name/description and sources.
  • Delete (trash icon): Remove the knowledge base including all sources and documents.
You can filter by name using the search field. The list is paginated (“Previous” / “Next”).
When deleting a knowledge base, all associated sources and documents are permanently removed. Consider removing the knowledge base from all assistants first.

​
Create and Edit a Knowledge Base

1

Create a new knowledge base

Click New Knowledge Base. You will be redirected to the editing view.
2

Set name and description

Enter a name and optionally a description (e.g. purpose and content). Click Save to create the knowledge base.
3

Set data protection (DLP)

The Data protection (DLP) switch defaults to Pseudonymized: search results of the knowledge base are checked and pseudonymized before they reach the language model whenever the DLP check is active in the chat. Choose Unchecked only for content without personal data, for example public product documentation: results then reach the model unchecked, and every bypass is logged.
A knowledge base set to Unchecked bypasses the DLP check of its results. Review the setting whenever you add new sources.
4

Select AI Model for Generating Embeddings

Select which embedding model should be used in the knowledge database to generate embeddings.
5

Add data sources

Only after saving can you create sources under Create New Source. Select the data source type and click Create Source. Then configure the source (see section Data Sources and Source Types).
6

Assign knowledge base to an assistant

Enable the knowledge base for one or more Global Assistants. Permissions controlling which users can access which knowledge base are managed via the assistants.

​
References

The References tab in the editing view lists under Global assistants all global assistants that use the knowledge base. Clicking an entry opens the assistant. Personal assistants are not listed individually; their number appears below the list, for example “3 personal assistants”. While you edit a source, the tabs are hidden.

​
Data Sources and Source Types

A knowledge base can have multiple sources. Each source has a source type that is set when creating it and cannot be changed afterwards.
Source type is permanent: The type of a source (e.g. GitHub, WebDAV) cannot be changed after creation. For a different type, you must create a new source and optionally delete the old one.

​
Available Source Types

​
General Configuration for Each Source

  • Your name for the data source: A freely chosen display name (e.g. “GitHub 1”, “WebDAV Legal Texts”).
  • Test connection: For GitHub, WebDAV, website, and VARIOS AI server folder, this button checks credentials and reachability. If errors occur, a red “Error” status appears in the configuration. Dropbox sources are connected through New connection instead, see Dropbox.
  • Synchronization: Only for sources with external access (not for Manual Upload):
    • Interval: Manual, every hour, every day, every week, or every month.
    • Sync now: Starts an immediate sync.
After saving a source, the interface shows Configuration (Status: No Errors / Error) and Files (number of documents, status Done / Processing / Error). You can reload, delete, or edit the source using the icons.

​
Manual Upload

  • Only one source of type “Manual Upload” is allowed per knowledge base. When the limit is reached, “Manual Upload (Limit reached)” appears and the option is disabled.
  • Documents: Drag files into the drag & drop area or click Upload and select files.
  • There is no synchronization interval – content is only added by manual upload.
  • Selected documents: List of uploaded files with status:
    • Done (green): Successfully processed and embedded.
    • Processing (orange): Processing is still in progress.
    • Error (red): Processing failed (e.g. invalid format).
  • Individual documents can be removed from the source using the trash icon.
Supported formats can be found in the application (e.g. PDF, DOCX, TXT, MD). Very large files may be excluded depending on the configuration.

​
VARIOS AI server folder

Documents are read from a local folder that is mounted as a volume into the container. The share folder you specify in the UI is a subfolder within the container base directory /data/Data/KnowledgebaseDocuments.
  • Test connection checks whether the folder exists in the container and is readable.
  • Hidden files (starting with . or ~) and symlinks are automatically skipped.

Set Up VARIOS AI Server Folder

Technical guide: Configure Docker Compose volumes, mount network shares (SMB/NFS), troubleshoot common errors.

​
GitHub

Documents are loaded from a GitHub repository.
  • Test connection checks repository, token, and branch.
  • Possible errors: invalid repository format, authentication failed, insufficient permissions, repository/branch not found, API rate limit.

​
WebDAV

Documents are fetched from a WebDAV server (e.g. Nextcloud, ownCloud).
  • Test connection checks URL and authentication. Common errors: authentication failed, scheme/host not set.

​
Dropbox

Documents are fetched from a folder in Dropbox. VARIOS AI accesses your Dropbox account through your own Dropbox app. Under Access data, New connection opens the linking in three steps:
1

Dropbox App

Create an app in the Dropbox App Console and enter the App Key and App Secret. The app needs read access to files and their metadata, which in Dropbox are the permissions files.metadata.read and files.content.read.
2

Allow linking

Click Link app. Dropbox opens in a new tab. Allow access there and enter the displayed code under Access Code.
3

Complete linking

VARIOS AI completes the linking. Once the message “Authentication finished successfully” appears, close the window.
VARIOS AI then renews the access on its own. Use Edit connection to link the source again; you enter the app secret again in the process. After successful linking, the Configuration appears:
  • Dropbox has no Test connection. The connection is checked when saving and before every synchronization. If the check fails, the source shows “Error” and the previous state is kept.
  • Synchronization: New Dropbox sources are set to Manual. Files changed in Dropbox are read in again, deleted files are removed from the source.

​
Website

The pages of a public website are read in, for example your own web presence for a chatbot. VARIOS AI starts at the start URL, follows links on the same domain up to the configured depth and keeps the main content of each page without navigation, header and footer by default (switchable to the entire page content). The page address is stored with the document so the assistant can refer to the original page in its answers. In the edit view of the source, the settings are under Read website.
  • What the start URL allows: Only pages on the host of the start URL are read, where a leading www. does not matter: www.your-domain.de and your-domain.de count as the same site. Everything else is foreign and is never requested, even when linked: subdomains such as service.your-domain.de, other domains such as portal.example.org, and portals your site links to. The path of the start URL additionally limits which pages of that host are read, as long as Restrict to certain pages is empty.
  • Additional domains and subdomains widen the scope deliberately: an entry *.your-domain.de adds all subdomains of that domain and the domain itself, an entry portal.example.org exactly this host but not its subdomains. Every additional host is checked against its own robots.txt and is subject to the same check for internal addresses; the patterns from Restrict to certain pages and Exclude pages apply across hosts.
  • Patterns: Patterns starting with / are matched against the path and parameters of the address, patterns starting with http:// or https:// against the full address. A pattern without a leading /, such as print or *?print=1, matches anywhere in the path and parameters. * stands for any characters, and every pattern acts as a prefix: /contact covers this page and every address that starts with it. Separate multiple patterns with commas; matching is case-insensitive.
  • Respected: robots.txt, noindex/nofollow and rel="canonical". Excluded pages are never requested, pages with noindex are not stored, and for duplicate addresses the canonical address is used. If the canonical address lies outside the crawled scope or redirects back to the page itself (for example http:// on an https:// site), the page is stored under its own address.
  • Read: HTML pages as well as text and Markdown files (.txt, .md, for example an llms.txt); text files are taken as they are. Not read: addresses outside the scope, files such as PDF or images, XML feeds, and pages with less than 40 characters of text. Tracking and session parameters are stripped from addresses.
  • Synchronization: Changed pages get new content and are re-embedded, pages that disappeared are removed from the source. The crawler pauses briefly between requests, so a run with a few hundred pages takes a few minutes.
  • Test connection checks that the start URL (and the sitemap, if given) is reachable and returns an HTML or text page, or an XML sitemap respectively.
  • Internal addresses: Addresses pointing to internal networks are rejected on every request, including redirects, not only by Test connection. Intranet sites therefore cannot be read in.
  • Limits: In addition to the page limit, fixed limits apply per run: 2 MB per page (checked while reading, larger pages are abandoned), 100 MB in total, 15 minutes runtime and at most 4 parameters in an address. A run that hits a limit ends cleanly and Last run names the reason. Sitemaps are read up to the maximum number of pages.
  • Gentle crawling: The crawler waits 250 ms between requests (or the value from Pause between page requests) and respects a Crawl-delay in robots.txt of up to 10 seconds when it is longer. When the site answers 429 or 503, it doubles its pause, to at least 500 ms, or waits the time given in the Retry-After header if that is longer, at most 10 seconds in either case. It retries the request up to three times and keeps the slower pace for the rest of the run. Last run shows how often the site throttled and the pause the run ended with. The crawler sends the user agent Mozilla/5.0 (compatible; VARIOS-AI Knowledgebase Crawler) and applies the robots.txt rules for VARIOS-AI (or the rules for * if there is no dedicated group), so operators can attribute and control the requests. Pages on the last link level are requested conditionally via ETag/Last-Modified and only confirmed, not downloaded again, when unchanged; pages whose links are followed are always downloaded again.
  • Source outage: If a run returns no page at all, the previous state is kept and the source shows “Error”. If a run ends at a limit or with failed requests, pages it did not reach are kept as well; only pages the website reports as gone with 404 or 410 are removed. Only a complete run without failures removes pages that are no longer linked.
  • Stored content: The content read for each page is stored under Data/Persistent/WebsiteSources, so embedding processes exactly what was read during the run. Redirects are followed up to five times in a row when the target is on the host of the start URL or listed under Additional domains and subdomains; the page is stored under its final address.
  • Last run: After each run, the source shows under Last run the number of pages read, unchanged and failed, the addresses blocked by robots.txt, and whether the run was stopped at a limit.
Notes on using websites as a knowledge source: The interface shows these notes when a website source is added.
  • Permission: Only read websites whose content may be used for this purpose, and observe the website’s terms of use.
  • Data protection: Websites may contain personal or confidential information. Imported content can be transmitted to the configured AI model when answering requests. An enabled privacy filter can detect sensitive data but does not offer complete protection. It only applies to this content when the knowledge base has Data protection (DLP) set to Pseudonymized and DLP is active in the chat.
  • Trustworthiness: Only use trustworthy sources. Websites may contain manipulated content or hidden instructions that can influence the AI’s answers and actions.
  • Currency and completeness: The knowledge source reflects the state of the last synchronization. Not all pages and content can be read completely. Last run shows the result of the processing.

​
Status and Error Handling

​
Knowledge Base Status (Overview List)

  • Green: No errors, all sources are OK and documents have been processed.
  • Orange: At least one source or documents are “Processing” (e.g. sync or embedding is running).
  • Red: At least one source or documents have the status “Error”.

​
Source Status (in the Editing View)

  • No Errors: Configuration and connection are OK.
  • Error: Connection test or last sync failed (e.g. wrong credentials, folder unreachable).
  • Synchronizing: Sync is currently running.

​
Document Status (per Source)

  • Done: Document was successfully read and embedded.
  • Processing: Processing or embedding is still in progress.
  • Error: Processing failed (e.g. file format, size, read error).
1

If the knowledge base shows a red status

In the overview, click Edit and check the sources. Open the source with a red configuration status (edit icon).
2

Check the source

Run Test connection. Read the error message (e.g. wrong path, invalid token, WebDAV login). Correct the settings and click Save.
3

Trigger sync again if needed

For GitHub/WebDAV/VARIOS AI server folder/website, click Sync now. Then check the document list (filter “Error”, optionally remove individual documents or re-embed them if the UI offers that option).

​
Quick Overview: What to Keep in Mind

  • Source type: Choose before creating – it cannot be changed afterwards.
  • Manual Upload: Only one such source per knowledge base; documents are uploaded manually only.
  • VARIOS AI server folder: Check the path and read permissions on the host/container; mount volumes correctly if needed.
  • GitHub: Token with sufficient permissions (read repository); consider rate limits.
  • WebDAV: URL including scheme (e.g. https://), correct user and password; set share folder and folder depth appropriately.
  • Website: Only read websites whose content you may use; set the start URL, Follow links and Maximum number of pages appropriately and check Last run after the synchronization.
  • Synchronization: Only for GitHub, WebDAV, VARIOS AI server folder, and website – choose an interval or sync manually.
  • Permissions: Access to knowledge bases is managed via assignment to Global Assistants.
Permissions controlling which users can access which knowledge base are managed via the assistants. Only assign the knowledge base to assistants for which the content is intended.