Troubleshooting

Everything here is something you can act on from the product or the API. Deeper operator material — alert thresholds, database-level recovery, deploy rollback — is in docs/operations.md in the repository.

Two pages answer most questions before you go any further:

• /healthz — what this deployment is running and how the queue is doing. The public half needs no authentication: extractor adapter, reader, ingest, auth and signup modes, release, queue depth and drain rate. The operator half — model names, plugins, extension prefixes, sealer and sinks, dead letters and recovery counters — is sent with authorization: Bearer $LOUVAIN_METRICS_TOKEN, from the private network, or in dev-header mode, the same rule as /metrics.
• /readyz — database readiness plus dependency diagnostics. Only Postgres gates the status code; SpiceDB, configured sidecars and retrying boot tasks appear under degraded. Existing projections still serve reads during a SpiceDB outage. A person without a projection gets 503 permission_store_unavailable, not an empty answer.
• /livez — the load-balancer routing check, with no dependency I/O.
• GET /v1/org/queue and GET /v1/org/extraction — the same picture, fenced to your org.

Database migrations finish before the listener opens. Several instances can coordinate startup without blocking online index creation; those index migrations require the direct database endpoint. SpiceDB and embedding warm-up then retry in the background, with their state shown by /readyz. A retrying task is logged as a warning; only one still failing five minutes after boot (LOUVAIN_BOOT_REPORT_AFTER_MS) is reported to the error tracker, once, as boot.<task>. Queue initialization retries too: boot.queue remains retrying until the dispatcher and its consumers are ready. Accepted events remain durable during that wait.

Ingest is returning 429

Backpressure and monthly quota refusals need different responses. Read the error field before changing anything.

backlog_full

{ "error": "backlog_full", "scope": "org", "pending": 25000, "limit": 25000, "retryAfterSeconds": 5 }

The deployment is saying "not right now". More is arriving than extraction is draining, and refusing is the designed response — accepting into an unbounded backlog is how a busy afternoon becomes tomorrow's outage. The response carries a Retry-After header.

Check scope: org is this workspace's share, while deployment is the shared hard ceiling. A workspace at its share does not close admission for other workspaces. The ceiling includes concurrent acceptance reservations.

What to do: honour Retry-After and resend. The event is not lost; it was never accepted, so resending the same idempotencyKey is correct and safe.

If it does not clear:

1. GET /v1/org/queue — is drainedPerMinute (events that finished extraction or indexing in the last minute) above zero? If it is, you are ahead of the drain rate and it will clear on its own.
2. If drainedPerMinute is near zero while pending climbs, extraction or indexing is stalled rather than slow. That is an operator problem — the model endpoint, the embedding service, the worker processes, or an org over its token budget. Escalate with the queue numbers.
3. If you are backfilling history, slow down and use POST /v1/ingest/batch (up to 500 events per request) rather than one request per message. A connector that opens a socket per message spends its time on sockets.

Backfill batches default to bulk. Incremental runner batches can request live priority with X-Louvain-Lane: live, but only through a connection's token and within its live allowance. Overflow is demoted to bulk; retrying it does not make it live or consume extra allowance for an already accepted event.

admission_expired is also retryable: resend with the same idempotency keys after Retry-After. 503 admission_unavailable means the admission decision was busy; it is not evidence of a full backlog.

quota_exceeded

{ "error": "quota_exceeded", "reason": "monthly ingested events limit reached: 200,000 of 200,000" }

The account is saying "not this month". There is deliberately no Retry-After, because retrying changes nothing until the limit is raised or the month rolls over.

What to do: check GET /v1/org/usage for month-to-date against limits, and raise the limit or wait. Do not retry in a loop. Treating these two 429s alike is how a connector either hammers a limit it can never pass or abandons a backlog it should have waited out.

Permission changes and withdrawals are not held up by it. membership.change, membership.sync and record.withdrawn are never refused for the month and never counted, so a person removed at the source, or a message deleted there, stops being readable in Louvain while the workspace is over its limit. A batch that mixed them with content comes back quota_exceeded with accepted (the permission changes and withdrawals it applied) and refused (the rest); keep the cursor where it was and resend the batch once the limit is raised.

The same code appears on /v1/answer against the monthly answer limit. AI assistants query far harder than people do, so this is usually the one that trips first.

too_many_attempts

{ "error": "too_many_attempts", "retryAfterSeconds": 240 }

Not ingest — usually the login, signup, invite or password-reset rate limiter; a few other routes (discussions, source edits, the Briefs page) answer with the same code for their own limits. Wait out the window.

rate_limited

{ "error": "rate_limited", "bucket": "answer", "perMinute": 30, "retryAfterSeconds": 12 }

A per-person answer rate (Limits). One person's session, API tokens and every AI assistant they connected share it, so a script and an assistant running at once can exhaust it together. Wait retryAfterSeconds; an operator raises it with LOUVAIN_ANSWER_RATE_PER_MINUTE and its siblings.

An answer is missing something we know is in there

This is the important one, because there are two completely different causes and they look identical from the outside. Establish which it is before changing anything.

Cause 1: it was never extracted

The knowledge was never recorded. Check GET /v1/org/extraction:

{
  "statements": { "total": 1291, "typed": 856, "standing": 1204,
                  "asserted": 1102, "hedged": 44, "negative": 61, "retracted": 17 },
  "messages": {
    "read": 1291, "nothing": 2140,
    "gated": 5510, "quarantined": 2, "budgeted": 0, "errors": 11, "pending": 0
  },
  "steps": [{ "step": "extract.thread", "label": "Extract facts from conversations",
              "runs": 3431, "skipped": 12, "failed": 0, "ms": 2140 }]
}

steps is the step ledger, and it is where a silent failure shows: a step with failed climbing on every record is a bug, not a quiet corpus.

Also check whether it has been processed yet. GET /v1/org/queue reports behind as a human string:

{ "pending": { "live": 0, "bulk": 12400, "total": 12400 }, "active": 8, "behind": "2.4h" }

A backfill that is two hours behind is not missing data. It has not arrived.

Cause 2: it is not visible to you

The fact exists and you cannot see it. This is the system working, and it is by design that it looks the same as an absence. Anything else leaks the existence of what it is hiding.

Check, in this order:

1. Is your identity linked? A person's containers come from their source identity — the Slack, Teams or custom-source user their account is linked to. An unlinked account reads nothing, and reads nothing quietly. GET /v1/auth/session shows your id (principal); GET /v1/org/overview reports visibility.containers — the number of containers you can read. Zero there is the whole explanation.
2. Was the permission ever sent? Content and permissions travel the same ingest path. A connector that streams messages but never sends membership.change events produces a corpus nobody can see. For a custom source, check that each event carries permissions.containerTuples.
3. Is the permission subject namespaced? Tuple subjects must be user:<connector>/<sourceUserId> — user:slack/U0442, not user:U0442. An unprefixed subject is accepted by SpiceDB and then never matched by the readable-set projection, which reads exactly like a missing permission.
4. Is the tuple synced? GET /v1/org/overview, as an administrator, reports sync.unsynced and sync.oldestUnsyncedSeconds for your workspace, the age counted from when the change last became pending. Unsynced tuples fail closed. A number that stays above zero for minutes rather than seconds is an operator problem.

Do not widen visibility to close a support ticket. It is the one change that cannot be undone quietly. If someone should be able to see a container, add them to it at the source and let the permission flow through.

A third case: the statement exists but does not establish current state

Read /v1/entities/:name/profile?history=true. The profile separates current, last-known and ended intervals and their support. A proposed, hypothetical, hedged, rejected or retracted statement can remain in history without establishing a value. A value going stale remains last-known; it does not become absent. A plain asOf selects valid time. knownAt (on the profile and answer routes, the Ask page and the AI-assistant tools) also limits what had been recorded then.

Answers are slow

Typical shape: retrieval is milliseconds and the model call is seconds. A cold question that the planner cannot route costs a model call plus a retry.

1. Check /healthz. reader tells you whether the model path is even on. queue.pending tells you whether the box is busy extracting.
2. Is it every answer or one question? Consistent slowness across all questions points at the model endpoint. One slow question usually means the planner could not compile it and it fell through to retrieval plus a read.
3. /v1/answer is the faster surface. It runs no model for questions it can compile into a plan. The reader calls a model when a state answer cannot answer the question. If you are building an AI tool, /v1/answer is the one to build on — let your own model write the prose.
4. Long questions with many entities cost more. The evidence set grows and the read grows with it.

If you are seeing 502s rather than slow answers, that is not a Louvain refusal — it is a proxy or edge timeout in front of the API cutting the connection while the answer is still being composed. Report the elapsed time; every model call on an answer shares one deadline (LOUVAIN_ASK_DEADLINE_MS, 25 s by default), so an answer cannot outlive it, and the API's own request budget is the longest deadline any route runs under plus 30 s (210 s by default — see Limits). A 502 at a suspiciously round number of seconds below that is an edge configuration, not the answer path.

Operators have one immediate lever: LOUVAIN_READER=off degrades to deterministic planner answers. Coverage drops, correctness does not.

"Louvain's model isn't available right now"

An answer, an assistant reply or Louvain in a discussion says this when the model's provider refused the call: the account behind the model key is out of credit, has hit its spending limit, or the key is wrong or revoked. Asking again will not help until that account is fixed. It names no key or provider on purpose. "Louvain's model is busy right now" is the provider throttling; it clears on its own. An operator sees the refused calls in the API log (assistant.call), and a key out of credit also as spend_cap:<surface> on /readyz.

A connection stopped

Start at GET /v1/org/connections as an administrator. Each row carries status, activity, last_error, a runner state and stats.

{
  "id": "c1d2…", "connector": "slack", "name": "Acme workspace",
  "status": "error",
  "activity": "idle",
  "last_event_at": "2026-08-14T22:41:00Z",
  "last_error": "runner exited 1 after 4 attempts",
  "runner": { "running": false },
  "stats": { "events": 18420, "claims": 903, "bounded": false, "errors": 2 }
}

activity is whether the feed is receiving: none (nothing yet), recent (within the hour) or idle. It and stats.errors (failed events, a count) are connection health, sent to administrators only. last_event_at, stats.events and stats.claims are counted over what you can read through the connection, for an administrator as for anyone: a feed you read nothing through shows null and 0 however busy it is. Past the count cap stats.bounded is true and the figures are "at least".

status: "error"

The runner crashed three times in a row. The connection flips to error loudly rather than crash-looping, last_error carries the reason, and the manager keeps retrying it on a doubling schedule — 1, 2, 4 … minutes, capped at an hour — until a run stays up for five minutes or you pause it. next_attempt_at and attempts say where it is in that schedule, and the page shows it as "Error · retrying in 12 min". A transient failure (broker outage, API restart, a token the broker refreshes later) clears itself; the causes below need you. An enterprise runner's last line of error output names the class of failure — LOUVAIN_EXIT {"reason":"auth_revoked",…} — which tells a revoked sign-in (reconnect) from a spent quota or an unreachable broker (wait); the reasons are listed under Connectors.

"Reconnect needed". When that line says auth_revoked or auth_expired, the sign-in has expired or was revoked, and retrying cannot fix it: the connection is held instead — status stays error, next_attempt_at is null, attempts goes back to 0, warning is authorization_lapsed and last_error reads authorization lapsed (auth_expired, gmail proxy 400 on history invalid_grant) or similar — however the run ended, even when the system stopped it for lack of RAM after saying so. Nothing retries it, no other warning the runner raises replaces the hold, and its owner (or a workspace connection's administrators) is emailed once. Reconnect the account from Connections (an administrator re-enabling a workspace connection does the same) and it resumes.

Try now. At the top of the ladder the next attempt is an hour away, which is right for a credential a provider will fix on its own and wrong when you have just fixed it yourself. A personal connection showing "Sync interrupted" carries a Try now button that brings one attempt forward (POST /v1/me/connections/:id/retry). It resets nothing: the streak and last_error are kept, so if the run fails again the schedule continues from where it was rather than restarting at one minute. It answers {"retrying": false} when there was nothing to bring forward — the connection had already recovered.

A failure the host slept through is not counted. The manager's reconcile loop ticks every three seconds, so a tick a minute late means the process was not running — a closed laptop, a paused VM. A runner that failed around such a gap is discounted: attempts does not grow and the retry is capped at a minute, because a suspended machine says nothing about the connection. Measured on a laptop, twelve consecutive failures each landed inside a 45-second macOS dark wake and took a healthy Google Workspace connection to an hourly retry; last_error now ends "the host was asleep, not counted" for those. A server that never suspends never sees this.

Most common causes, in order:

• A revoked or expired credential. Re-enter it: PATCH /v1/org/connections/:id with new credentials. Credentials are sealed and cannot be read back, so re-entering is the only path.
• A rotated ingest token. POST /v1/org/connections/:id/rotate-token invalidates the previous one immediately. Anything still pushing with the old token gets 401 invalid_ingest_token.
• A path that no longer exists, for one-shot import connectors.

Set status back to enabled after fixing it; the manager reconciles every few seconds and will start the runner again.

status: "disabled" on a one-shot connector

Expected. An import runs once and returns to paused — not running is shown as not running rather than as a permanently "enabled" connection doing nothing.

running: true but activity is idle

activity moves only when something new is read in, so a quiet source looks the same as a stopped one. last_pass_at tells them apart: it is when the connection last finished a full check of its source, shown as Checked beside it, and it moves on a quiet day too. A recent check with old activity is a quiet source, or a runner connected to the wrong scope. For Slack specifically: the bot only sees private channels it has been invited to (/invite @louvain). Public channels are joined automatically unless auto-join was turned off at setup.

"Needs attention"

A connection says what its source refused rather than going quiet (warning on the connection; a code, never content):

• scope_missing_files — the source refused the files the connection reads, because the grant lacks the access (Slack: files:read, which canvases need). Add it to the integration if it is missing, then Reconnect the connection (Connections → the connection → Reconnect). Reconnecting restarts the connection's reader, and the warning clears after its next full check.
• privacy_unknown — Louvain couldn't tell from the source's sharing who may see some of its items (a meeting with no sharing setting it could read, a setting it does not know, or a team meeting from a team it cannot list), so they are kept from everyone rather than guessed at. The connection checks again on every pass; if it persists, check the item's sharing in the source.
• team_list_untrusted — a workspace connection (Fireflies) got a list of the team's members that may be incomplete: empty, with a member it cannot identify, or without the key's own user. It adds nothing rather than share a meeting with the wrong people, and asks again every half hour. Check the team's members in the source (a removed or invited-but-unjoined user is the usual cause), or reconnect with an admin's key.
• certificate_expiring — a SharePoint connection's app registration certificate expires within 30 days. When it expires, SharePoint site groups stop granting their people. Upload a new certificate to the app registration and paste it and its private key on the connection; the warning clears after the next full check.
• certificate_invalid — a SharePoint connection's certificate and private key can't be used together (half of a pair, or a key that isn't the certificate's), so site groups are counted and grant nobody; everything else keeps being read. Paste the certificate and its own private key again on the connection.
• audience_not_offered — the connection is shared more widely than its way of connecting allows: it was saved with Everyone before Louvain stopped offering that for a sign-in or key that reads as one account (Confluence, Jira, a Zendesk or Salesforce sign-in). Nobody can read what it brings in until you choose again: Connections → the connection → Access, then the source's own permissions (where offered) or the people you choose. The warning clears with that choice.
• stalled — an enabled connection that reports its checks has not completed a full check of its source in over six hours. Part of it may have stopped (its last error says which), or it is not running. Pause and resume it on Connections to start a fresh one. Gmail, Google Calendar and Google Drive (personal and company-wide), Dropbox, Box, Microsoft 365 and the structured sources report their checks, so a quiet source that is still being checked shows a recent Last checked rather than this warning.

Administrators see the warning and its fix on Connections and in the store's view in the Library; a member sees only that some items from it can't be read yet. Operators are alerted on the first: LouvainConnectionScopeMissing.

An http inbound connection that shows no runner

Correct — mode: "inbound" connectors have no runner by design. They are a credential, an org fence and a rate limit; your system pushes to /v1/ingest. runner: { "running": false } is not a fault here. Look at activity and stats.errors instead, and at whatever is doing the pushing.

Events accepted but nothing appearing

Check GET /v1/org/activity, or the Manage workspace → Processing activity page. It shows recent ingest in rooms you can read, with its outcome and statement count — metadata only, never message text. An event with a terminal outcome and zero statements was processed and yielded nothing typed; an event still pending has not been processed yet. An event in a room nobody has granted you is not listed at all — an administrator sees a count of those by outcome (GET /v1/org/activity/outside), and the feed's own connection card shows its totals. If an event you sent was refused outright, the ingest response said so: a 400 names the field (invalid_event, event_too_large) and nothing was stored.

To see what a record actually produced, expand its row on the Processing activity page (or call GET /v1/org/activity/:id): the statements it was understood to make and the excerpts behind it, fenced by the same container permissions as the Browse page. A member who cannot read the record's room is told so rather than shown an empty list, so "nothing under this row" and "nothing you may see" are never the same message.

Events that failed extraction

curl https://louvain.example.com/api/v1/org/queue/failures \
  -H "authorization: Bearer louvain_EXAMPLE_TOKEN_NOT_REAL"

Returns up to 100 events that exhausted their retries in rooms you can read (failures), each with its container, actor, error, a 140-character preview, attempts and date, so you can tell which message it was. Failures in rooms you cannot read are not listed — which records they were and when is a read of those rooms — only counted (unreadable: { "count": 3, "bounded": false }): the admin role operates the queue without reading its rooms, and a retry still includes them. Personal and delegated mailboxes are never included.

The usual reason a batch of these exists is an outage that has since been fixed. Put them back on the queue:

curl -X POST https://louvain.example.com/api/v1/org/queue/retry \
  -H "authorization: Bearer louvain_EXAMPLE_TOKEN_NOT_REAL"

{ "requeued": 214 }

Both routes require owner or admin. The retry is org-scoped — it never touches another tenant's events.

API errors that are configuration, not faults

404 sso_not_configured on any /v1/auth/sso/* route means this deployment has no identity provider configured, not that your provider is broken.

Something changed after a deploy

/healthz reports release and environment, so "is the fix deployed" is answered by asking the process rather than by reading a dashboard. It also reports extractor, reader and signup, and — with the metrics token — sealer and any loaded extensionRoutes: an enterprise module that silently failed to load would otherwise look exactly like everything being fine.

If extractor reads rule, that deployment is running the test fixture, not a model. It captures roughly one fact in eight and reports no error at all.

While the API restarts — during a deploy, or a local dev-stack.sh update that changed the API — opening the workspace waits and asks again for about twenty seconds before it says Couldn't load your workspace. Seeing that message means the API was still unreachable after that; Try again asks once more.

Extraction is using a model you did not configure

/healthz reports model (with the metrics token, or from the private network): the model actually extracting, read from the running process rather than from any file. Compare it with what you believe you set.

A process keeps the environment it started with, and a shell export beats the env file — so a long-lived process can be running a model that no file on disk mentions. Restarting is what makes an edited env file take effect; until then the file describes an intention and /healthz describes reality.

This matters beyond tidiness. Different models extract differently, so a corpus ingested across a model change is not one corpus — comparing a before against an after measures the model swap as much as whatever you were trying to measure. GET /v1/org/extraction breaks usage down per model, which is how you tell whether a workspace was ingested by one model or several.

Extraction is paused but answers work (or the reverse)

Expected, and the point. A provider key that runs out of credits pauses only the surfaces that use that key. GET /readyz lists each as spend_cap:<surface> under degraded, and GET /v1/org/extraction → governor.paused[] names the surfaces on every paused key with the time to its next probe. Extraction and checking share LOUVAIN_EXTRACTOR_API_KEY; the reader uses LOUVAIN_ANSWER_API_KEY when it is set and the extractor's otherwise — so if answers are refused too, the two share a key. Top the account up: the next completed call on that key lifts the pause and re-runs what dead-lettered during it (Limits).

GET /v1/org/usage → bySurface says which surface is spending, in requests (calls) and tokens, and estimatedCalls how many requests the provider did not itemise — a hosted rerank or a web search returns no token counts, so those are a floor.

The sign-in page does not load

On a deployment that signs people in through a managed sign-in service, the sign-in form is loaded from that service. If something between the browser and it blocks the script — an ad-blocker, a privacy extension, a corporate proxy or firewall, or an outage at the service — there is no form to show. After a few seconds /login says the sign-in is taking longer than it should, or that it could not load, and offers Try again; the site's Sign in link always stays usable, so it never waits on the service.

The workspace needs the same service when it opens. A signed-in session carries the service's own short-lived token beside Louvain's, and after a while away the browser fetches a fresh one before the first page shows. If the service can't be reached — often just after a laptop wakes, before the network is back — the page reloads itself once, which is what brings the service back. If it still can't be reached, the workspace says Reconnecting… — you are still signed in — and reloads on its own when the network comes back, when you return to the tab, and every half minute; Try again now reloads it at once. It reloads on its own at most once every 30 seconds, so a script that is blocked for good never keeps reloading the tab; the checks below apply to it. A sign-in service that can't be reached is never read as being signed out, and never offers to sign you out.

The same is true of a tab left open in the background: browsers slow a hidden tab's timers, so the token can lapse while you're away. Coming back to the tab fetches a fresh one before anything is asked, and a request refused for a lapsed token is sent again with the new one. A tab from before a deploy catches up by reloading itself while it is hidden, but never while the browser is offline and never in the middle of an answer. Your session ended appears only when the session really is over: you signed out, an administrator removed you, or the sign-in service ended your session.

Check, in order:

• Extensions. Try a private window with extensions off, or allow the deployment's sign-in domain in the blocker.
• The network. A proxy or firewall that filters third-party scripts blocks the service everywhere on that network; try another network to confirm, then ask whoever runs it to allow the domain.
• The service. If every browser on every network fails, the operator should check the sign-in provider's status and the deployment's configured publishable key (GET /v1/auth/config names the provider).

A deployment that uses Louvain's own sign-in loads nothing from elsewhere and is not affected.

Still stuck

Collect these before asking for help. They contain no message content, so they are safe to paste:

• the full /healthz body (fetched with the metrics token, so it carries the operator half),
• GET /v1/org/queue and GET /v1/org/extraction,
• the status, last_error and runner fields of the affected connection,
• for an answer problem: the question, the plan and via fields from /v1/answer, and whether retrievalDegraded was present.

If the conclusion is "this person cannot see it", that is an answer, not a bug.