> ## Documentation Index
> Fetch the complete documentation index at: https://docs.varios-ai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Narrow down faults of an on-premise installation: standard checks, logs, and diagnostic commands

This page is written for administrators, preferably with shell access on the server that runs VARIOS AI as a Docker installation. It describes where the logs are and which commands narrow down a fault.

<Note>
  All commands on this page run in the **installation directory**, the directory containing `docker-compose.yml` and `.env`. The [installation guide](/en/monthly/installation/guide) recommends `/docker/variosai` for it, and the examples on this page use that path. Your installation may sit somewhere else, though. In that case substitute your own path throughout. Docker commands usually require `sudo` or membership in the `docker` group.
</Note>

## Standard checks first

### Is the installation on the current version, and is the fault already fixed?

Under Administration → Settings → [System information](/en/monthly/admin/settings/system-information), **system status** shows whether an update is available. Your selected **release channel** is listed there as well.

In the [changelog](/en/monthly/changelog/changelog), check the **Bugfixes** and **Improvements** sections of every version between your installed one and the current one. Watch for entries that require a manual adjustment in addition to the update: such notes are part of the changelog entry itself.

<Warning>
  Take a snapshot of the virtual machine before every update, see [Backup and restore](/en/monthly/operations/backup). An update cannot be undone without a snapshot.
</Warning>

### Check the system log

Administration → [Logs](/en/monthly/admin/logs) shows system, security, and SCIM logs without logging in to the server. `System.log` is the first place to look when analyzing a fault.

### Check the Docker logs

The Docker logs contain everything the containers report.

```bash theme={null}
cd /docker/variosai   # or your installation directory
docker compose ps
docker compose logs --since 1h | grep -i -e error -e exception -e fatal
```

Which further logs exist and where they are is described below under [Where the logs are](#where-the-logs-are).

### Narrow down the fault

These facts decide where you have to look, and support will ask for them anyway.

| Question | What it is for |
| - | - |
| Since when does the fault occur? | Narrows the time range in the logs |
| What was changed last? | Update, certificate, firewall rule, identity provider, model provider, network |
| Does it affect all users or individual ones? | Individual users point to roles, groups, or permissions; all users point to a service |
| Does it affect all models or one? | A single model points to the provider, all models to the installation |
| Is the fault reproducible? | A reproducible fault can be replayed with the log open |
| What is the exact wording of the message, and when exactly did it appear? | Only this makes the matching log entry findable |

<Tip>
  Reproduce the fault with a test account of your own while following `sudo tail -f Data/Logs/System.log`. That way you see the entries belonging to exactly this operation instead of searching the history.
</Tip>

### Ground rules for working on the server

* **Take a snapshot before changes** that touch configuration, database, or files.
* **One change at a time**, checking after each whether it had an effect. Several changes at once make the result useless.
* **Never edit or delete log files.** The audit, auth, SCIM, DLP, and connector logs are protected by checksums and refuse further writes after a modification.
* **Never pass on `.env`.** It contains the database password, the client secret, and API tokens.
* **Check the server clock** (`timedatectl status`). A wrong clock leads to rejected login tokens and failing certificate checks.
* **Enable support access only when needed** and disable it afterwards, see [System information](/en/monthly/admin/settings/system-information).

## Diagnosis after evaluating the system log

Work through these commands in order. In most cases one of them already reveals the cause.

```bash theme={null}
cd /docker/variosai   # or your installation directory

# 1. Are all containers running, and does one restart constantly?
docker compose ps

# 2. What does the application report?
docker compose logs --tail=200 php

# 3. Did the reverse proxy start cleanly?
docker compose logs --tail=100 traefik

# 4. Is there free disk space and memory?
df -h
free -h
docker system df

# 5. Did the kernel kill a process because of low memory?
sudo dmesg -T | grep -i -e "out of memory" -e "killed process" | tail
```

### Reading the `STATUS` column

| Shown by `docker compose ps` | Meaning | Next step |
| - | - | - |
| `Up … (healthy)` | Service is running, health check succeeded | The cause is elsewhere |
| `Up …` without health check | The process runs, which says nothing about its function | Check the service log |
| `Restarting …` | The container crashes and is restarted | `docker compose logs <service>`, usually a configuration or permission error |
| `Exited (137)` | The process was killed, almost always low memory | `dmesg`, check memory, see [Sizing](/en/monthly/operations/sizing) |
| `Exited (1)` | The service terminated itself with an error message | Read the last lines of the container log |
| `Created` | The container was never started | `docker compose up -d`, check the image and the registry login |

<Tip>
  The exit code of an already stopped container: `docker inspect --format '{{.State.ExitCode}} {{.State.Error}}' $(docker compose ps -aq php)`.
</Tip>

## Where the logs are

VARIOS AI writes to three places: the Docker container logs, files below the installation directory and, if configured, a syslog target.

### Container logs

Everything the services write to `stdout` and `stderr` ends up in the Docker log.

<Warning>
  Container logs survive a restart of the container, but not a `docker compose down`, a recreation of the container, or a `docker system prune`. The update helper `varios.sh` runs `docker system prune -f -a --volumes` after every update. Save logs before an update if you still want to investigate a fault.
</Warning>

### Application logs in the file system

These files are located on the host under `./Data/Logs` and in the container under `/data/Data/Logs`. They are the most important source for everything that happens inside the application, and they survive container restarts and updates.

| File | Content |
| - | - |
| `Audit.YYYY_MM_DD.log` | Audit log of all traceable actions, one file per day |
| `Auth.YYYY_MM_DD.log` | Logins and logouts, including failed ones |
| `Scim.YYYY_MM_DD.log` | User and group provisioning via SCIM |
| `Dlp.YYYY_MM_DD.log` | Findings and decisions of the DLP check |
| `Connector.YYYY_MM_DD.log` | Connector calls and their responses |
| `System*.log` | System messages. `System.log` is the log shown in the interface under Administration → Logs; the framework additionally creates files carrying the Flow context in their name |
| `Security*.log` | Security-relevant events of the framework |
| `Chats.log` | Chat processing, rotates to `Chats.log.1` at about 100 MB |
| `Exceptions/*.txt` | Full stack traces, one file per exception |

Use `ls` to get an overview of the files that actually exist instead of guessing a file name:

```bash theme={null}
sudo ls -la Data/Logs
sudo ls -lat Data/Logs/Exceptions | head
```

A message in `System.log` usually names the file of the matching stack trace. This is how you get from the message to the full exception:

```bash theme={null}
sudo grep -i "exception" Data/Logs/System*.log | tail -5
sudo cat Data/Logs/Exceptions/<filename>.txt
```

<Note>
  If `LOG_OUTPUT` in `.env` is set to `syslog`, VARIOS AI writes **no** log files at all and sends everything to the target configured under `LOG_SYSLOG_HOST` and `LOG_SYSLOG_PORT`. For local troubleshooting set `LOG_OUTPUT=both` and restart the `php` service.
</Note>

## Rebuilding the application state

Run this command in the installation directory to flush and rebuild the caches, apply migrations, and restart all processes:

```bash theme={null}
docker compose exec php ./update.sh
```

<Warning>
  The command restarts all processes in the `php` container. VARIOS AI is briefly unavailable while it runs.
</Warning>