Background jobs and scheduling

Summary#

_worker.js exports a scheduled() handler, and it never runs. Cloudflare only invokes scheduled() on a project deployed as a Worker with a [triggers] crons block, and this project deploys to Pages. Background work is therefore driven from outside: a separate, tiny Cloudflare Worker called metricvault-cron fires every 15 minutes and POSTs to /api/cron/run on the Pages deployment, and a GitHub Actions workflow does the same thing hourly as a backup.

That endpoint runs eight registered jobs. Every one of them decides for itself what is due by reading a timestamp out of D1, so calling the endpoint more often than necessary is harmless. Running both schedulers at once is deliberate, not an accident.

Purpose#

The product needs work that happens without a user: competitor pages must be re-fetched, rank positions re-checked, AI-visibility measurements taken, Stripe entitlements reconciled, saved results purged at 90 days. None of that can hang off a request.

The obvious answer, a cron trigger, is unavailable. Rather than move the whole site to a Workers deploy for one feature, the design splits the clock from the work: the clock is a Worker (12 lines of real logic, no bindings, no database), and the work stays in the Pages worker where the data and the code already are. The cost of that split is one extra deploy and one shared secret. The benefit is that the site keeps its Pages deployment model, and the jobs are reachable by anything that can make an authenticated HTTP request, which makes them testable by hand.

Architecture#

  metricvault-cron  (a real Cloudflare Worker)
    [triggers] crons = ["*/15 * * * *"]        PRIMARY
        │  POST {TARGET_URL}/api/cron/run
        │  header: x-mv-internal-secret
        ▼
  .github/workflows/cron.yml                    BACKUP
    schedule: "17 * * * *"                      (hourly, minute 17)
        │  curl -X POST "$TARGET/api/cron/run"
        ▼
  ┌───────────────────────────────────────────────────────┐
  │  _worker.js  →  handleCronRun                         │
  │    mvVerifyInternalSecret   (503 unset / 401 mismatch)│
  │    globalThis.__mvEnv = env                           │
  │    ?job=<key>  or  all nineteen                       │
  │    ctx.waitUntil(Promise.all(jobs))                   │
  │    return 200 { ok, triggered[], at } IMMEDIATELY     │
  └───────────────────────────────────────────────────────┘
        │
        ▼  each job reads its own "due" predicate from D1
  monitor · schedules · rank_alerts · tier2 · crawl · sweep_orphans
  purge_results · purge_watchtower · sweep_caches · higgsfield · social_schedule
  blog_schedule · cms_schedule · billing_sync · google_sync · ga4_sync
  gsc_inspect · gsc_vitals · backlink_watch

Why Pages never calls scheduled()#

The handler exists in the default export alongside fetch, and it runs four jobs inside ctx.waitUntil. The comment directly above it records why it is dead code that is kept anyway: Cloudflare only calls it when the project is deployed as a Worker with a [triggers] crons block, and this repository deploys to Pages. It stays so that the same source would work unchanged if the project were ever redeployed as a Worker.

Three facts back this up, and each is worth knowing before someone tries to "fix" it:

• wrangler.toml carries pages_build_output_dir = "dist", which is the Pages format. Changing it to the Workers format breaks the bindings. See System architecture.
• The header comment in wrangler.toml records that a previous version declared a six-hourly cron (0 */6 * * *) which never took effect.
• The deploy workflow runs wrangler pages deploy.

The primary clock: cron-worker/#

A standalone Worker with its own wrangler.toml, its own deploy, and no storage bindings at all. That is deliberate: it holds no business logic, so it can never be the reason a job is wrong.

It also exports a fetch handler, so a plain GET to the worker runs the same trigger and returns the JSON result. That is the manual run and health check. The response is { ok, status, url, body, at }, where body is the first 500 characters of the site's reply; status is 200 when ok and 502 otherwise.

If the secret is missing it refuses to call the site at all and returns MV_INTERNAL_SECRET is not set on this cron worker. Run: wrangler secret put MV_INTERNAL_SECRET.

Changing the schedule means editing crons in cron-worker/wrangler.toml and redeploying that worker. It is not affected by a Pages deploy, and a Pages deploy does not update it. See Deployment.

The backup: GitHub Actions#

.github/workflows/cron.yml, named Scheduled jobs (competitor monitor, reports, rank alerts, AI tracking), runs at 17 * * * *. Its concurrency group is scheduled-jobs with cancel-in-progress: false, so two runs never overlap.

It targets ${{ vars.CRON_TARGET_URL || 'https://metricvaultai.com' }} and uses ${{ secrets.MV_INTERNAL_SECRET }}. It fails loudly rather than silently:

workflow_dispatch accepts an optional job input for a manual single-job run. Its help text reads Single job to run (blank = all): monitor | schedules | rank_alerts | tier2 | crawl, which lists only four of the eight valid keys. The other four still work if you type them.

The shared secret#

MV_INTERNAL_SECRET must hold the same value in three places:

1. The Cloudflare Pages project metricvaultai, under Settings, Environment variables.
2. The cron-worker Worker, via wrangler secret put MV_INTERNAL_SECRET.
3. GitHub repository secrets, for the backup workflow.

The verifier mvVerifyInternalSecret fails closed:

The secret is read from the x-mv-internal-secret header, or from a ?key= query parameter when the header is absent. There is deliberately no guessable fallback: an older mv-internal-job-<day> scheme was removed because anyone could derive it. The same verifier also guards /api/billing/sync, which is why a missing secret also stops plans reconciling against Stripe. Full variable inventory in Environment variables and secrets.

Components#

POST or GET /api/cron/run reaches handleCronRun. The job registry:

Behavior of the handler:

1. Stashes env on globalThis.__mvEnv, because dispatchAlerts needs RESEND_API_KEY and is called deep inside checkSingleUrl with no env parameter.
2. Reads ?job=. Blank runs everything.
3. An unknown key returns HTTP 400 with {"error":"Unknown job: <name>","valid":[...]}.
4. Each job is wrapped so one failure cannot abort the others. A throw is logged as [cron] <name> failed: <message> to console.error.
5. The work is handed to ctx.waitUntil, so it keeps running after the response.
6. The response is {"ok":true,"triggered":[...],"at":"<ISO 8601>"}, returned before the jobs finish.

The endpoint is declared in MV_ACT_ROUTES as { cat:'system', action:'cron.ran', title:'Scheduled jobs ran', module:'Automation', bucket:'hour' }, so it appears in the Library as Scheduled jobs ran, bucketed hourly.

Data flow#

Idempotency#

This is the property that makes a 15-minute tick safe. No job runs simply because it was called; each one selects only the rows that are due.

Two of these deserve a closer look because their safety is not just a WHERE clause.

social_schedule claims each post with an UPDATE whose WHERE includes the old status, then checks meta.changes. If another run already claimed the row, changes is 0 and the loop skips it. A post is marked sent if any channel succeeded.

billing_sync grants first and revokes second, and a revoke only ever touches a row this system wrote (set_by = 'stripe'), so manually comped accounts and admin overrides are never clobbered. It pages through Stripe subscriptions with MAX_PAGES = 40, and if pagination did not complete it grants only and reports revoke_skipped: 'incomplete scan — granted only, no revocations'.

The failure path per job#

checkSingleUrl stamps last_checked_at even when the fetch throws, so a permanently broken URL is not retried every tick. runDueSchedules writes a schedule_runs row with status = 'error' and the message. runDueRankAlerts pushes { id, error } into its results rather than aborting the batch.

Manual and operator triggers#

The admin console exposes monitor, schedules, rank_alerts and tier2 through POST /api/admin/jobs/run. A non-owner admin gets {"error":"Only an owner can run jobs."}. It calls the same functions the cron runner uses, so it is not a forced re-run: the job still processes only what is due, and tier-2 still respects its 22-hour throttle. Every use writes an audit row. See Background jobs console.

Failure modes#

Two behaviors are documented here because they surprise people, not because they are bugs to fix casually:

• Scheduled reports do not execute a workflow. runDueSchedules inserts a queued row and optionally sends an email; the comment explains that running AI workflows on the cron would mean large CPU cost, so the frontend shows the run rows and the user clicks to execute. The email copy nonetheless reads as though a report was produced.
• computeNextRun matches cadence exactly and case-sensitively. Only daily, weekly and monthly map; everything else falls through to weekly.