Error log and resolution

Summary#

The console captures production failures in two independent places. Grouped issues on the System Health screen collapses repeated premium AI tool failures into one row per distinct fault, with a count and a last-seen time. Workflow Failures lists individual failed steps from the dashboard's multi-step workflows, one row per occurrence, with the customer's input and the raw error text. Both have a resolve action. Neither is a full application error tracker, and knowing exactly what each one does and does not see is what stops you chasing a fault in the wrong table.

Purpose#

Two different questions needed answering. "Is something broken across the platform right now, and how often?" is a grouping question: a hundred identical timeouts should read as one problem, not a hundred. "Why did this customer's run fail?" is the opposite: you need the individual event, their input and the exact message. Grouped issues answers the first, Workflow Failures answers the second, and they are stored and populated separately because they are fed by different code paths.

Requirements#

• An active admin session. See Getting admin access.
• MONITOR_DB bound on the Pages project. Both tables live in D1 and are created on demand.
• Nothing else. Both screens are read paths plus a single-field update.

Permissions#

Error resolution is the one mutation an admin can perform. Everything else that changes state is owner-only. Both endpoints reply 403 {"error":"Unauthorized"} without a valid admin token.

Navigation Path#

Admin console → Operations → System Health → Grouped issues

Admin console → Main → Workflow Failures

What each surface actually records#

Grouped issues#

Rows come from the premium AI route. When a run through that route returns a non-OK response, the worker records the failure off the request path so telemetry can never slow down or break a customer's run.

A fingerprint groups similar failures. The message is normalised first: lower-cased, URLs replaced with <url>, email addresses with <email>, long hexadecimal ids with <id>, every run of digits with <n>, whitespace collapsed, then truncated to 200 characters. That normalised string is hashed together with the tool id. Two failures that differ only by a domain, an id or a number therefore land on the same row and increment its count instead of creating a second entry.

Each row stores the tool, a title (the first 180 characters of the message), the count, first-seen and last-seen timestamps, and a sample (the first 500 characters of the most recent message). The sample is not shown in the table body: it is the title attribute of the Issue cell, so hovering the issue text reveals it.

regressed is the most important state on the screen. It means a fault you believed fixed has returned, and it appears without anybody having to notice.

Workflow Failures#

Rows come from the dashboard's multi-step workflows. Each step runs with a hard per-step timeout of 120,000 ms and retries transient faults (429, 502, 503, 504 and anything flagged transient) up to six attempts with backoff. When a step still fails, the browser posts one row, fire-and-forget, so a broken step never blocks the rest of the workflow from rendering.

Each row carries the step key and label, the workflow key, the customer email, the input that seeded the step, the error message, and whether the failure was a timeout. Fields are truncated on write: email 200 characters, workflow key 80, step key 80, step label 120, input seed 400, error message 1,200.

Step-by-Step Guide#

Triage a grouped issue#

1. Select System Health in the Operations group.
2. Set the range selector to Last 24 hours or Last 7 days. The range affects the metric tiles and charts only. The Grouped issues table always returns the 500 most recent issues by last-seen time, whatever the range says.
3. Scan the Status column first. Take regressed rows before unresolved ones.
4. Sort your attention by Count and Last seen together. A count of 400 last seen three days ago is history. A count of 12 last seen 40 seconds ago is live.
5. Hover the Issue cell to read the full sample message.
6. Note the Tool value, then confirm the blast radius on the Tool health table above: runs, error percentage and average latency for that same tool. See Metrics and usage analytics.
7. Fix the cause. Only then press Resolve.

Resolve or reopen an issue#

1. Press Resolve on an unresolved or regressed row, or Reopen on a resolved one.
2. The screen reloads its metrics and issues. The row's badge changes.
3. The action is written to the audit log as issue.resolved or issue.unresolved, with the fingerprint as the target. See Audit log.

Resolving does not delete the row, reset its count, or stop new occurrences being recorded. It marks the issue as handled and arms the regressed transition. That is the whole effect.

Triage a workflow failure#

1. Select Workflow Failures in the Main group. The screen also loads on its own about a second after the console opens, so the sidebar badge is filled in before you go looking.
2. The filter defaults to Open only. The other options are All, Resolved only and Timeouts only.
3. Use Search failures... to filter across step key, step label, input seed, error message and customer email. The search runs in the browser over rows already loaded.
4. Read the Most Failing Steps card. It shows up to five step keys with counts and is the fastest route to a systemic break.
5. Open the failing card: amber border and a TIMEOUT pill means the step exceeded its time budget, a red border and ERROR means it threw. The monospace block holds the raw provider or worker message.
6. Reproduce with the value in Input: before you dismiss anything.
7. Press Mark resolved once handled. The label changes to Resolving…, and to Retry if the call fails.

Watch for new failures live#

The screen polls every 25 seconds while the console is open. The first poll is skipped to establish a baseline, so only genuinely new rows trigger anything. New rows pulse the sidebar badge and slide in up to three toasts, 250 ms apart, each auto-dismissing after 10 seconds. Clicking a toast jumps to the screen.

Reading the counts correctly#

The four tiles on Workflow Failures (Total Failures, Open, Last 24h, Timeouts) are computed by the server over the rows it returned, not over the whole table. This matters:

So Total Failures can drop shortly after you open the screen. That is the poll narrowing the sample, not failures disappearing. Press Refresh to recompute against the full page again.

Retention#

Nothing prunes either table. There is no scheduled cleanup job, no retention window and no delete endpoint for issues or workflow failures. Rows accumulate until someone removes them directly from the database. Resolving is the only housekeeping the console offers, and it changes a status rather than removing data.

Troubleshooting#

FAQs#

Does resolving an issue notify anyone? No. There is no email, no in-app message and no customer-facing effect. It writes an audit row and changes a badge.

Can I undo a resolve? Yes. Press Reopen, which sets the status back to unresolved and writes issue.unresolved to the audit log.

Why do two clearly different errors share one row? The fingerprint strips numbers, ids, URLs and emails before hashing. Two messages that differ only in those places are treated as the same fault by design. If the distinction matters, use the sample text on the tooltip, or find the individual events in Workflow Failures.

Is the resolve action on a workflow failure audited? No. Every /api/admin/* mutation is audited, but the workflow-failure resolve endpoint sits outside that path and writes no audit row. It does set resolved_at, so the timestamp survives even though the actor does not.

Can an admin who is not an owner resolve issues? Yes, both surfaces. It is the only mutation available at that role.

How far back does the issue list go? The endpoint returns the 500 most recently seen issues. Because issues are grouped rather than per-event, 500 rows covers a long period in practice. There is no paging control on this table.