Maintenance and health

Summary#

This page covers two things: the routine that tells you whether the platform is healthy, and the levers available when it is not. The most important fact on the page is a limitation. The Maintenance mode card writes a message that is only ever displayed inside the admin console. Customers never see it. If you need to change what a customer experiences during an incident, the working levers are turning a tool off, suspending an account, and clearing cached provider data.

Purpose#

Two jobs sit behind one screen. Day to day, someone needs to be able to look at the platform for two minutes and know whether anything is wrong. During an incident, someone needs to be able to stop the bleeding: take a failing premium tool out of service rather than let every customer burn credits on it, or force fresh data after a provider returns garbage.

Both are deliberately blunt instruments. There is no partial rollout, no per customer flag, and no scheduled window. What exists takes effect within a minute and can be reversed as fast.

Requirements#

• An active admin session. See Getting admin access.
• The owner role for every lever on this page. An admin can run the health check but cannot change anything.
• MONITOR_DB bound. Every setting is stored in D1.
• For the diagnostic endpoints, the platform's internal secret.

Permissions#

Owner-only endpoints reply 403 with their own message, for example {"error":"Only an owner can change config."}

Navigation Path#

Admin console → Operations → System Health

Admin console → Operations → Config → Maintenance mode

The daily health check#

Five screens, in this order. It takes about two minutes.

1. Dashboard → System pulse. Five tiles refresh on their own every 45 seconds while the tab is visible. Error rate (24h) turning red at 10% or above and a non-zero Open issues are the two that demand action. See Metrics and usage analytics.
2. Workflow Failures. Check the sidebar badge before you click. It carries the open count and pulses when new failures arrive. Most Failing Steps turns a list of individual failures into a pattern in one glance.
3. System Health → Tool health. Any tool badged down or degraded needs a decision. Read the Runs count next to the badge: a 100% error rate over two runs is noise.
4. System Health → Grouped issues. Take regressed rows first. They are faults you believed fixed. See Error log and resolution.
5. Jobs. Confirm nothing is stuck at Due now across successive checks. Background work is driven by an external scheduler roughly every 15 minutes, so a job that stays due is a signal that the scheduler is not reaching the worker. See Background jobs console.

There is no single status endpoint to poll. The console is the health check.

Step-by-Step Guide#

Take a failing premium AI tool out of service#

This is the strongest customer-facing lever the console has, and it works.

1. Select Config in the Operations group.
2. In Set a config value, enter the key tool_off:<type>, using the tool's internal id. The helper line on the card states the format: Known keys: set tool_off:<type> to 1 to disable a premium AI tool; maintenance_message for a banner. Blank/removed = default.
3. Enter 1 as the value and press Save. The message line reads Saved.
4. Every request for that tool now returns 503 with the body {"error":"This tool is temporarily unavailable. Please try again shortly."} and the customer sees that text.
5. To restore the tool, press Delete on its row in the Configuration table, or set the value to 0.

Both the set and the delete are audited as config.set and config.delete. See Audit log.

Stop one account#

Suspension is per customer and immediate.

1. Select Customers, load the address, and press Suspend in the Actions card.
2. The account is blocked at the paid-AI gate with 403 and the body {"error":"This account is suspended. Please contact support.","code":"account_suspended"}
3. Reactivate removes the flag. Both actions bust the settings cache, so they take effect at once rather than after the normal cache interval.

The footnote on the card states the scope precisely: Suspend blocks the customer from running paid AI tools. It is not a full lockout. See Customer lookup and billing.

Force fresh provider data#

If a provider returned bad data and it has been cached, clearing the cache makes the next runs fetch fresh.

1. Select Tools & Cache.
2. To clear one tool, press Clear on its row. The message reads Cleared <n> cached row(s) for <Tool>.
3. To clear everything, press Clear all cached data. The browser confirms with Clear ALL cached provider data now? The next runs will fetch fresh (higher provider cost).

Clearing is safe for correctness and expensive for us: every subsequent run pays a provider call. Prefer the per-tool clear. Both are audited as cache.clear. See Tools and cache management.

Set the maintenance message#

1. Select Config. The Maintenance mode card is visible to owners only.
2. Tick Show maintenance banner and type a message into Banner message (e.g. Back at 5pm UTC).
3. Press Save. Leaving the text empty with the box ticked writes the default Site is undergoing maintenance. Please check back soon.
4. Unticking the box and pressing Save deletes the key and removes the strip.

The banner appears in two places: inside the admin console as a strip reading Maintenance: <your message>, and at the top of the customer dashboard as a dismissible banner. The dashboard reads it from GET /api/status/banner and a new message can take up to a minute to show because of the config cache. A customer can dismiss it for their browser session; a changed message reappears.

How quickly a change takes effect#

Settings are read through a cache that holds the whole table for 60 seconds, and each running worker instance keeps its own copy.

When you disable a tool, wait a minute and confirm with a real request before declaring it done.

The settings cache also fails open. If the database cannot be read, every default applies and tools stay on. That is the deliberate choice: a database problem must not silently disable the product. It also means a tool-off key cannot be relied on as a safety control during a database incident.

Diagnostics#

The worker exposes a set of diagnostic probes under /api/diag/. Every one of them requires the platform's internal secret as a key query parameter, and the guard fails closed: with no secret configured, all of them return 403 {"ok":false,"error":"forbidden"} The probes hit real upstream providers, so they cost money and are not something to run casually. See Diagnostic endpoints.

The same internal secret protects the endpoint the external scheduler calls to run background jobs, and the AI article writer's internal handoff. If article generation starts returning 503 across the board, a missing or mismatched secret is the first thing to check. See Environment variables and secrets.

Troubleshooting#

FAQs#

Is there a real maintenance mode that takes the site offline? No. Nothing in the product serves a maintenance page or blocks customer traffic. The closest available controls are per-tool disable and per-customer suspension.

Can I schedule a maintenance window in advance? No. Every lever is immediate and manual, and there is no scheduling control.

Does disabling a tool refund customers who already paid for a failing run? No. Refunds are a separate manual action on the Customers screen, and they adjust recorded usage rather than money.

Will suspending an account stop their scheduled reports and alerts? Suspension blocks paid AI tool runs. It is not a global stop on background work. If automation must stop for an account, that has to be handled on the customer's own automation settings.

Who is notified when I change one of these settings? Nobody automatically. Every mutation writes an audit row, so other admins can see what happened, but no email or in-app message is sent to anyone.

Can a bad configuration value break the platform? Not the cache lifetime keys, which fall back to their built-in default on anything unparseable. A tool-off key set on the wrong tool will take a working tool out of service, which is why the key needs verifying against the catalog before you save it.