Troubleshooting

This page is written for administrators, preferably with shell access on the server that runs VARIOS AI as a Docker installation. It describes where the logs are and which commands narrow down a fault.

All commands on this page run in the installation directory, the directory containing docker-compose.yml and .env. The installation guide recommends /docker/variosai for it, and the examples on this page use that path. Your installation may sit somewhere else, though. In that case substitute your own path throughout. Docker commands usually require sudo or membership in the docker group.

Standard checks first

Is the installation on the current version, and is the fault already fixed?

Under Administration → Settings → System information, system status shows whether an update is available. Your selected release channel is listed there as well. In the changelog, check the Bugfixes and Improvements sections of every version between your installed one and the current one. Watch for entries that require a manual adjustment in addition to the update: such notes are part of the changelog entry itself.

Take a snapshot of the virtual machine before every update, see Backup and restore. An update cannot be undone without a snapshot.

Check the system log

Administration → Logs shows system, security, and SCIM logs without logging in to the server. System.log is the first place to look when analyzing a fault.

Check the Docker logs

The Docker logs contain everything the containers report.

cd /docker/variosai   # or your installation directory
docker compose ps
docker compose logs --since 1h | grep -i -e error -e exception -e fatal

Which further logs exist and where they are is described below under Where the logs are.

Narrow down the fault

These facts decide where you have to look, and support will ask for them anyway.

Question	What it is for
Since when does the fault occur?	Narrows the time range in the logs
What was changed last?	Update, certificate, firewall rule, identity provider, model provider, network
Does it affect all users or individual ones?	Individual users point to roles, groups, or permissions; all users point to a service
Does it affect all models or one?	A single model points to the provider, all models to the installation
Is the fault reproducible?	A reproducible fault can be replayed with the log open
What is the exact wording of the message, and when exactly did it appear?	Only this makes the matching log entry findable

Reproduce the fault with a test account of your own while following sudo tail -f Data/Logs/System.log. That way you see the entries belonging to exactly this operation instead of searching the history.

Ground rules for working on the server

Take a snapshot before changes that touch configuration, database, or files.
One change at a time, checking after each whether it had an effect. Several changes at once make the result useless.
Never edit or delete log files. The audit, auth, SCIM, DLP, and connector logs are protected by checksums and refuse further writes after a modification.
Never pass on .env. It contains the database password, the client secret, and API tokens.
Check the server clock (timedatectl status). A wrong clock leads to rejected login tokens and failing certificate checks.
Enable support access only when needed and disable it afterwards, see System information.

Diagnosis after evaluating the system log

Work through these commands in order. In most cases one of them already reveals the cause.

cd /docker/variosai   # or your installation directory

# 1. Are all containers running, and does one restart constantly?
docker compose ps

# 2. What does the application report?
docker compose logs --tail=200 php

# 3. Did the reverse proxy start cleanly?
docker compose logs --tail=100 traefik

# 4. Is there free disk space and memory?
df -h
free -h
docker system df

# 5. Did the kernel kill a process because of low memory?
sudo dmesg -T | grep -i -e "out of memory" -e "killed process" | tail

Reading the `STATUS` column

Shown by `docker compose ps`	Meaning	Next step
`Up … (healthy)`	Service is running, health check succeeded	The cause is elsewhere
`Up …` without health check	The process runs, which says nothing about its function	Check the service log
`Restarting …`	The container crashes and is restarted	`docker compose logs <service>`, usually a configuration or permission error
`Exited (137)`	The process was killed, almost always low memory	`dmesg`, check memory, see Sizing
`Exited (1)`	The service terminated itself with an error message	Read the last lines of the container log
`Created`	The container was never started	`docker compose up -d`, check the image and the registry login

The exit code of an already stopped container: docker inspect --format '{{.State.ExitCode}} {{.State.Error}}' $(docker compose ps -aq php).

Where the logs are

VARIOS AI writes to three places: the Docker container logs, files below the installation directory and, if configured, a syslog target.

Container logs

Everything the services write to stdout and stderr ends up in the Docker log.

Container logs survive a restart of the container, but not a docker compose down, a recreation of the container, or a docker system prune. The update helper varios.sh runs docker system prune -f -a --volumes after every update. Save logs before an update if you still want to investigate a fault.

Application logs in the file system

These files are located on the host under ./Data/Logs and in the container under /data/Data/Logs. They are the most important source for everything that happens inside the application, and they survive container restarts and updates.

File	Content
`Audit.YYYY_MM_DD.log`	Audit log of all traceable actions, one file per day
`Auth.YYYY_MM_DD.log`	Logins and logouts, including failed ones
`Scim.YYYY_MM_DD.log`	User and group provisioning via SCIM
`Dlp.YYYY_MM_DD.log`	Findings and decisions of the DLP check
`Connector.YYYY_MM_DD.log`	Connector calls and their responses
`System*.log`	System messages. `System.log` is the log shown in the interface under Administration → Logs; the framework additionally creates files carrying the Flow context in their name
`Security*.log`	Security-relevant events of the framework
`Chats.log`	Chat processing, rotates to `Chats.log.1` at about 100 MB
`Exceptions/*.txt`	Full stack traces, one file per exception

Use ls to get an overview of the files that actually exist instead of guessing a file name:

sudo ls -la Data/Logs
sudo ls -lat Data/Logs/Exceptions | head

A message in System.log usually names the file of the matching stack trace. This is how you get from the message to the full exception:

sudo grep -i "exception" Data/Logs/System*.log | tail -5
sudo cat Data/Logs/Exceptions/<filename>.txt

If LOG_OUTPUT in .env is set to syslog, VARIOS AI writes no log files at all and sends everything to the target configured under LOG_SYSLOG_HOST and LOG_SYSLOG_PORT. For local troubleshooting set LOG_OUTPUT=both and restart the php service.

Rebuilding the application state

Run this command in the installation directory to flush and rebuild the caches, apply migrations, and restart all processes:

docker compose exec php ./update.sh

The command restarts all processes in the php container. VARIOS AI is briefly unavailable while it runs.

Getting Started

Usage

Administration

Installation

Operations

Security

Changelog

Standard checks first

Is the installation on the current version, and is the fault already fixed?

Check the system log

Check the Docker logs

Narrow down the fault

Ground rules for working on the server

Diagnosis after evaluating the system log

Reading the `STATUS` column

Where the logs are

Container logs

Application logs in the file system

Rebuilding the application state

​Standard checks first

​Is the installation on the current version, and is the fault already fixed?

​Check the system log

​Check the Docker logs

​Narrow down the fault

​Ground rules for working on the server

​Diagnosis after evaluating the system log

​Reading the STATUS column

​Where the logs are

​Container logs

​Application logs in the file system

​Rebuilding the application state

Standard checks first

Is the installation on the current version, and is the fault already fixed?

Check the system log

Check the Docker logs

Narrow down the fault

Ground rules for working on the server

Diagnosis after evaluating the system log

Reading the `STATUS` column

Where the logs are

Container logs

Application logs in the file system

Rebuilding the application state