# Troubleshooting

> Where to find the logs, and how to fix the problems Kribble servers commonly run into

Start with the logs. Most problems leave a clear message there. The sections below are grouped by symptom and quote the messages to look for.

## Logs and health checks

:::tabs{group="install"}
::tab{label="Linux host"}
```sh
journalctl -u kribble-web -f
journalctl -u kribble-builder -f
journalctl -u kribble-web --since "1 hour ago" | grep '"level":"error"'
journalctl -u meilisearch -n 50
```
::
::tab{label="Docker Compose"}
```sh
docker compose logs -f kribble-web
docker compose logs -f kribble-builder
docker compose logs --tail 50 meilisearch db
```
::
:::

In prod mode, Kribble logs one JSON object per line on standard output, at level `info`. For more detail, set `LOG_LEVEL=debug` in a unit drop-in (`sudo systemctl edit kribble-web`, then `Environment=LOG_LEVEL=debug`) and restart. `LOG_FORMAT=console` makes the log easier to read by eye. `LOG_DIR` also writes the log to rotated files ([Configuration](../running-a-server/configuration.md)).

Health checks:

```sh
curl -i http://127.0.0.1:8080/healthz    # kribble-web is up
curl -i http://127.0.0.1:8080/readyz     # 503 while it shuts down
curl -i http://127.0.0.1:8081/healthz    # kribble-builder is up
curl -s http://127.0.0.1:8080/api/v1/config
curl -s http://127.0.0.1:7700/health     # Meilisearch
```

A healthy start logs `Connected to database postgres` and `Kribble web server starting on :8080` from `kribble-web`, and `Kribble builder running (workers=2, health :8081)` and `build queue: started` from `kribble-builder`. The **System** tab of the admin panel shows the number of builds waiting and whether Redis and Meilisearch are configured.

## A service doesn't start

| Message | Cause and fix |
| --- | --- |
| `config: failed to read …` | No config file at `KRIBBLE_CONFIG_PATH` (or `./config.toml`), or the service user can't read it. |
| `config: failed to parse …` | TOML syntax error; the message gives the line. |
| `config: [auth].registration must be "open" or "invite_only"` | Fix the value. |
| `config: [[publish_targets.…]] "<name>" missing required field …` or `publish target name "<name>" used twice` | Complete the target or rename it ([Publish targets](../running-a-server/publish-targets.md)). |
| `[database].dsn is required` | Set `[database].dsn`. |
| `Database migration failed` | PostgreSQL is unreachable, the password is wrong, or the user doesn't own the database. Test the DSN with `psql "<dsn>"`. |
| `S3 storage requires S3_ENDPOINT, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET` | `[storage].backend` is `"s3"` (the default) but `[storage.s3]` is incomplete. Fill it in, or use `backend = "disk"` with `dir`. |
| `disk storage requires STORAGE_DIR` | Set `[storage].dir`. |
| `mkdir …: read-only file system` or `permission denied` | The storage directory is outside `/opt/kribble`, which the systemd units make read-only. Move it, or add `ReadWritePaths=` in a drop-in ([Installing](../running-a-server/installing.md)). |
| `redis ping: …` | `[redis].url` is set, but Redis isn't reachable. Start Redis, or empty the setting for a single `kribble-web`. |
| `ldap source "<name>" is enabled but url is empty` (or `user_search_base`) | Complete the `[[ldap]]` block or set `enabled = false`. |
| `Failed to run web server` with `bind: address already in use` | Something else uses `[server].port`. For the builder, `Builder health server failed` means `[builder].port` is taken; the builder keeps building. |

systemd restarts a service that exits with an error every 5 seconds, so the same message repeats until you fix it.

## Signing in

**Signed out again right after signing in, or after a reload.** The session cookie is marked `Secure`, so browsers drop it on plain HTTP. Open the server through its HTTPS address. On an internal test machine without TLS you can use `mode = "dev"`, never on a real server.

**`429 Too Many Requests` when signing in.** Sign-in allows 10 requests a minute per client address. If the reverse proxy doesn't send `X-Forwarded-For`, every user shares the proxy's address and the limit is reached quickly. Fix the proxy ([Installing](../running-a-server/installing.md)). After several wrong passwords, a single person also has to wait up to 16 seconds between attempts.

**Locked out of the admin panel.** Add your email to `[admin].emails` and restart `kribble-web`. If the log says `admin: email listed in admin_emails but no user row yet`, the account with that email doesn't exist.

**Someone forgot their password and email isn't configured.** There is no password reset in the admin panel. Configure email ([Authentication](../running-a-server/authentication.md)), restart, and have them use **Forgot password** on the sign-in page.

## Email doesn't arrive

- At startup `kribble-web` logs `[email] provider=… from=…`. With `provider=none`, nothing is sent and each email is only logged as `[email] (no provider) would send to=… subject=…`. Set `[email].provider` and its settings.
- `[email] failed to send to <address>` gives the transport's error:
  - `smtp auth: … unencrypted connection`: Kribble won't send the SMTP password without TLS, except to `localhost`. Use port 587 (STARTTLS) or 465 (TLS), or set `encryption`.
  - `smtp dial: … certificate …`: the SMTP server's certificate doesn't match `host` or isn't trusted.
  - Errors from Resend usually mean the `from` domain isn't verified there.
- Links in emails pointing to `localhost:5173` mean `[server].frontend_url` isn't set.
- Mail that is sent but lands in spam needs SPF and DKIM for the `from` domain. That is set up at your mail provider.

## LDAP logins fail

Check the **Auth Sources** tab of the admin panel to see what Kribble loaded, and `LDAP: enabled sources=…` at startup. Then look for these messages after a failed login:

| Message | Meaning |
| --- | --- |
| No LDAP message, login fails | No enabled source found the user. Check `user_search_base` and `user_filter`, for example with `ldapsearch -H <url> -D <bind_dn> -W -b <user_search_base> "<filter with the username>"`. |
| `LDAP: source authentication failed` with `service bind` | `bind_dn` or `bind_password` is wrong. |
| … with `dial` | The server is unreachable, or TLS fails (certificate not trusted by the host). `tls_skip_verify` confirms a certificate problem, but don't keep it on. |
| … with `filter matched N entries (expected 1)` | The filter matches several entries. Make it stricter. |
| `LDAP auth failed, falling through to local` | All sources failed with a directory error, so the local password was tried. |
| `LDAP find/create user failed` with `missing required attributes` | The entry has no value for `attr_email` or `attr_username`. |
| `LDAP: group lookup failed` | Login worked, but the group search failed. Check `group_search_base`, `group_member_attribute` and `user_group_attribute`. |

A wrong password gives the normal "invalid identifier or password" error and no LDAP log line. If group sync adds nobody, check that the organisation and teams in `[[ldap.team_map]]` exist with exactly those names, and that `group_dn` matches the DN the directory returns. The `ldap sync:` lines at `info` level show who was added where.

## Search finds nothing, or misses notes

Saving a note doesn't depend on Meilisearch. The note is saved first and added to the search index afterwards, and if that fails, Kribble logs `search: failed to index notes` and the save still succeeds. So while Meilisearch is down, people can work as usual, but:

- full-text search fails: the search box in the app still finds note titles, and the CLI, the desktop app's search and agents get an error
- notes changed in the meantime are missing from search, or found by their old text, until they are saved again or the index is rebuilt

To fix it:

1. Check Meilisearch: `curl http://127.0.0.1:7700/health` (or `docker compose logs meilisearch`). In production mode Meilisearch needs a master key of at least 16 bytes and refuses to start without one.
2. Check that `[meili].host` and `[meili].key` match the running Meilisearch. A wrong key gives authorization errors in the log.
3. When `kribble-web` starts, it applies the index settings that search relies on, in the background. If Meilisearch isn't reachable, it logs `search: index setup failed; search may be degraded` and tries again, waiting longer each time, from 5 seconds up to 5 minutes. It keeps trying until Meilisearch answers, so nothing needs restarting.
4. If Meilisearch was down or unreachable while notes changed, rebuild the index: click **Rebuild index** under **Search** on the **System** tab of the admin panel (**Instance admin** in the user menu), or, as an instance admin, `POST /api/v1/admin/search/reindex`. The rebuild runs in the background, and search keeps working meanwhile. The log shows `search: index rebuilt` with the number of notes and the time it took, or `admin: search reindex failed`.
5. If Meilisearch lost its data, for example a deleted volume or a new installation, click **Rebuild index** (or restart `kribble-web`, which finds the index empty and rebuilds it, logging `search: rebuilding the index`). Either way the index and its settings are set up again before the notes are written.

Search results are checked against the database before they are shown, so a note that was deleted, or moved to a space the person can't open, never appears, even when the index is behind.

## PDF export fails

| Symptom | Fix |
| --- | --- |
| `503` with `chromium binary not found; install chromium or set [pdfgen].chrome_path in config.toml` | Install Chromium or Chrome on the `kribble-web` host, or set `chrome_path`. The Docker image includes Chromium; if you built an older image, rebuild it from `deploy/Dockerfile` ([Installing](../running-a-server/installing.md)). |
| `500` "failed to render PDF", log `pdfgen failed` | Chromium started but failed. Run the binary by hand as the `kribble` user (`sudo -u kribble chromium --headless --dump-dom about:blank`) to see why; often missing libraries or fonts. A Chromium installed as a snap may not run under the hardened systemd unit; use a deb or Google Chrome instead. |
| Text in a template prints in another font than in the PDF designer | **Serif (system)** and **Sans (system)** use the serif and sans-serif fonts installed on the `kribble-web` host (Georgia or Times New Roman, Helvetica or Arial when present). Install the fonts you want there, or use Inter, Fraunces or IBM Plex Mono, which are built into Kribble. |
| `504` from the proxy for long documents | A render can take up to 90 seconds. Raise the proxy's read timeout (nginx `proxy_read_timeout 300s`). |

[Exporting PDFs](../using-kribble/exporting-pdfs.md) covers templates and the export itself.

## Published sites

**Stuck at `queued`.** The builder isn't running, or can't reach the database. Check `systemctl status kribble-builder` (or `docker compose ps`) and its log. The System tab's **Build queue** counts sites `queued` or `building`. A running builder picks up queued sites within 10 seconds.

**Stuck at `building`.** The builder was stopped mid-build. It puts the site back in the queue when it starts again. After 3 attempts the builder marks the site `failed` on its next start instead of trying again.

**`failed`.** The site couldn't be generated. The builder logs `build queue: build failed` with the site's ID and the error, or `build queue: marked poison builds as failed` for the attempt limit. Fix the cause and publish or `kribble redeploy` again; that resets the attempts.

**`publish_failed`.** The site was built but couldn't be copied to its external target. The publish dialog and `kribble sites` show the error, and the builder logs `external publish failed`. Common errors:

| Error | Fix |
| --- | --- |
| `read-only file system`, `permission denied` (local target) | Allow `base_dir` in both units with `ReadWritePaths=`, and make it writable by `kribble`. |
| `known_hosts_path is required`, `knownhosts: key is unknown`, `key mismatch` | Fill `known_hosts_path` with the server's host key (`ssh-keyscan`), of type Ed25519, ECDSA or RSA. |
| `private key … is passphrase-protected` | Set `passphrase`. |
| `read private_key_path …: permission denied` | The key must be readable by `kribble`, and not under `/home`. |
| `ssh: handshake failed: ssh: unable to authenticate` | The public key isn't in the remote user's `authorized_keys`. |
| `… publish target "<name>" not found in config` | The target was renamed or removed. Put it back, or unpublish the site and publish it to another target. |
| Cloudflare errors with `403` or `Authentication error` | The API token lacks Cloudflare Pages edit permission for `account_id`. |
| `file too large for Cloudflare Pages` | A file in the site is over 25 MiB. Remove or shrink it. |

**Public site without images, icons or styles.** `[storage.s3].public_bucket_url` doesn't point at `https://<server>/files-public`. Fix it and rebuild the site.

**Site links reported by the CLI point to `localhost`.** Set `[server].frontend_url`.

## Uploads and downloads

**Uploads over 1 MB fail with `413`.** nginx's default `client_max_body_size` is 1 MB. Set it to `500m`.

**Large attachments don't download with the S3 backend.** Files over `proxy_threshold_bytes` (15 MiB) are sent as a redirect to a presigned S3 URL, so the browser must reach `[storage.s3].endpoint`. An endpoint only reachable inside your network (such as an internal MinIO address) breaks this. Use an endpoint browsers can reach, or raise `proxy_threshold_bytes` so files stream through Kribble.

## The CLI and agents

Start with `kribble status`: it shows the profile, server, account and both versions, and whether the token still works.

| Message | Fix |
| --- | --- |
| `could not reach <server>` | Wrong address, DNS, TLS or proxy. Try `curl https://<server>/api/v1/config`. |
| `<server> rejected the access token (revoked or expired?); run: kribble login` | The token was revoked, expired, or its creator was removed from the organisation. Sign in again. |
| `the access token can't use this (it needs the read scope)` | The token lacks the scope. Tokens from `kribble login` have `read` or `read,write`. Publishing tokens made in the web app (**Publish + Import**) can't read notes. See [Organisations and access](../running-a-server/organisations-and-access.md). |
| `MCP requires an org access token with 'read' scope` | MCP doesn't accept browser sessions. Use `kribble login` or a token with `read`. |
| `Note: the server runs X and this CLI is Y; run kribble update.` | Run `kribble update`. |
| `this server has no kribble-<os>-<arch> build` (installer) | The server has no `downloads/` directory, or it lacks that build ([The CLI and desktop downloads](../running-a-server/the-cli-and-desktop-downloads.md)). |
| The install command shows `http://` or an internal host name | The proxy doesn't pass `Host` and `X-Forwarded-Proto`. |

More on the CLI side is in [The kribble CLI](../using-kribble/the-kribble-cli.md) and [AI agents](../using-kribble/ai-agents.md).