Part VII · Operations
Troubleshooting¶
Foundry's operating principle is fail loudly: there are no silent fallbacks. When something breaks you get a persisted status, a visible banner, and a log trail — this page maps each failure surface to its cause and fix.
Copilot¶
The Foundry Copilot (the one generic assistant behind the copilot panel, the onboarding interview, and the prompt-editor assist) never degrades silently. If its configured provider is missing, disabled, or unsupported, the backend answers 503 {error: "copilot_disconnected"} and every assist surface shows the same banner.
| Symptom | Cause | Fix |
|---|---|---|
| Banner: "Copilot disconnected — check Instance Settings → Copilot" | The provider configured for the Foundry Copilot is missing, disabled, or unsupported | Follow the banner's link (it goes straight to /settings?tab=copilot — the Copilot tab). Fix the provider/model selection, then click Test with a hello-turn — success shows ✓ ok · <model> · <elapsed>ms; failure shows a humanized error telling you what to change |
| "Copilot request failed." (no disconnected banner) | A transient request error, not a configuration problem | Retry; if it persists, check the backend logs (below) |
Providers¶
Every provider config has a live Test button (Instance Settings → Providers; the edit drawer's primary action is Save & Test). For a config with a stored key, the test runs a real connection probe against that key with a 5-second timeout. For keyless configs it does something else — see the second table below. Either way the result is persisted on the provider row (last_tested_at / last_test_status / last_test_error), so the card keeps showing the state — status pill, "tested X ago", and on failure the error text verbatim in a red panel under the card.
| Symptom | Cause | Fix |
|---|---|---|
| Status pill failed + error text under the card | The probe reached the provider and was rejected — the panel shows the provider's own error (e.g. an auth rejection) | Read the verbatim error; typically re-enter the API key or correct the endpoint, then Test again |
| Error mentions a timeout / probe failure with no provider message | Endpoint unreachable within 5 s (wrong endpoint, network egress, provider outage) | Verify the endpoint field (empty = the schema's default endpoint) and connectivity from the backend host |
Error: unknown provider type: <name> |
The config's API schema isn't a registered adapter | Recreate the provider with a supported API schema — the schema is fixed at creation |
| Error: "Failed to decrypt stored API key" | The stored key can't be decrypted (e.g. ENCRYPTION_KEY changed since the key was saved) |
Re-enter the API key; if this appears broadly, the instance's ENCRYPTION_KEY was rotated or lost |
Keyless configs: subscriptions and ADC¶
Some configs store no credential at all. Testing them cannot mean "probe with the key", so it means something specific to each — and in every case an answer Foundry cannot positively establish is reported as a failure, never as a silent pass.
| Symptom | Cause | Fix |
|---|---|---|
| Error: "This config has no stored key to test — use 'Test this subscription' instead." | The row stores no credential — an unbound subscription row, or a Vertex ADC config authenticating from the environment | Nothing is broken by itself. For ADC, verify credentials on the backend host directly. Note the action this message names is not on the screen yet — the message means "there is nothing stored here to probe" |
| Error: "The Claude Code session that powers this subscription is signed out, so every turn on this provider will fail." | The container's Claude Code OAuth session has lapsed — this is the credential for a Subscription (Claude Pro/Max) config | Re-login inside the backend container: docker compose exec -it backend claude auth login, complete the browser flow, then run Test again |
| Error starting "Could not determine the Claude Code session status —" (mentions the CLI not installed, or a timeout) | The probe could not run at all: the Claude Code CLI is missing from the container image, or the child process was killed | Signing in again will not fix this. Run docker compose exec -it backend claude auth status in the container to diagnose |
| Error starting "Could not read the Claude Code session status —" followed by quoted output | The CLI answered with something that is not the expected JSON — a different CLI version, a wrapper script, a captive portal | Read the quoted output; run docker compose exec -it backend claude auth status in the container to diagnose |
| Error: the config "uses a container Claude session but no session endpoint resolves for it, so it is dropped from the runtime configuration" | The config is on the session auth method but has no catalog binding, so no loopback endpoint resolves — the runtime silently omits the provider | Edit the config, pick its provider from the catalog (catalog binding), and save again |
Execution runs¶
Runs are designed never to hang silently. Every run persists a terminal status (completed / failed / stopped) on its run row, first-write-wins — and at boot the backend reconciles orphans: any run still marked running after a backend restart is flipped to failed with the detail "server restarted mid-run", so the cockpit never renders a dead run as forever-running.
| Symptom (banner in the run cockpit) | Cause | Fix |
|---|---|---|
| "Execution failed — \<error>" | The run reached a terminal failure; the banner carries the persisted error | Read the error; check backend logs for the full trail; dispatch a fresh run (runs are disposable and comparable — a failed run costs you nothing but its ledger entry) |
| "The run didn't start — please try again." | Dispatch failed before a run existed | Retry; if it repeats, check backend logs and provider health |
| "An agent step is failing — the run may be stuck." | A step is failing repeatedly mid-run | Watch the live stream for the failing step; stop the run if it doesn't recover |
| "Stopped by you — branch kept" | You stopped the run | Nothing to fix — the work branch is preserved; discard or resume from a fresh run |
| A run shows failed with "server restarted mid-run" | The backend restarted (deploy, crash) while the run was live | Dispatch a fresh run |
Per-worker outcomes (a finished/failed engineer's cost, commit, worktree) are shown on a terminal banner when you select that worker in the roster.
Product onboarding: the verify stage¶
The verification suite after the setup PR (one row per dev-environment check) keeps full evidence per check — exit code and both output streams, stored separately so a chatty build can't bury its own error.
| Symptom | Cause | Fix |
|---|---|---|
| Red check row | That command failed in the cloned repo | Open View log on the row: stdout and stderr are shown separately with an exit <code> chip. "This command produced no output." means exactly that — the command failed silently |
| The whole run failed before any check ran | Job-level failure (e.g. clone/setup died) — rendered as a job-failure card, not a fake check | Read the job-failure card; use the recovery footer |
| You need the complete evidence | — | Download full log exports the whole run, both streams for every check |
| Stuck after a failure | — | The footer offers real recovery: Re-run verification (re-runs the whole suite), Back to Confirm (revise the profile), Re-detect (re-derive the dev profile) |
| "Could not load this log." | The log fetch itself failed | Not the same as an empty log — retry; if persistent, check backend logs |
A setup PR that was closed without merging also routes the product to the failed state, with its own alert explaining that distinct root cause.
Where the logs live¶
Local rig — standard compose logs from the repo root:
Cloud — containers log through the Docker gcplogs driver to Google Cloud Logging (durable, searchable, no SSH needed); dual logging keeps plain compose logs working on the VM too.
# from anywhere with gcloud auth (message is in jsonPayload.message):
gcloud logging read 'logName="projects/swisper-489418/logs/gcplogs-docker-driver" AND jsonPayload.container.metadata."com.docker.compose.service"="backend"' \
--project=swisper-489418 --limit=50 --freshness=15m \
--format="value(timestamp.date('%H:%M:%S'), jsonPayload.message)"
# on the VM:
docker compose -f deploy/docker-compose.cloud.yml logs -f backend
Backend won't boot¶
Startup failures are fatal by design: if migrations, seeds, or Polis behaviour registration fail at boot, the backend logs Schema migrate / push / seed / ... failed: plus the error and exits, instead of running with a broken chat path. The first error lines in the backend log tell you which boot step failed. In the cloud, a boot-looping backend also shows up as the deploy health gate timing out ("backend did not report healthy within 240s") — the deploy job prints the last 60 backend log lines when that happens. One classic cause after a fresh deploy: Polis migrations were not applied before backend start (deploy.sh does this; see Deployment).
Health signal: the backend container's Docker healthcheck curls /health inside the container — docker inspect --format '{{.State.Health.Status}}' <container> is the authoritative readiness check (the port is not published on the cloud host).