Months of Green Ticks over a Cleanup Job That Never Deleted a File

The scheduled cleanup for Free Stencil Maker reported success on every run for months. No alerts fired. No one looked twice. Then on 12 September 2026, a routine check of the storage bucket showed 96,159 objects sitting past their 48-hour TTL, the oldest 147 days stale. The cleanup had never deleted a single file.

What the Workflow Actually Did

The GitHub Actions step called the cleanup endpoint, got a 403 back, then hit a fallback line:

echo 'Cleanup request failed (non-fatal)'

That echo exits zero. A zero exit code means a green tick. The workflow had been reporting success by lying, not by doing anything useful.

The fallback was written defensively, presumably so a transient network hiccup would not block a deploy. The problem is that "non-fatal" is only safe if the failure actually is transient. A 403 that fires every single run is not transient. It is a permanent misconfiguration dressed up as a warning.

Why the Endpoint Returned 403 Every Time

The cleanup endpoint reads process.env.CLEANUP_SECRET at request time and compares it against the value in the incoming request header. If the server has no secret, the guard fails closed and returns a flat 403.

The server had no secret. Not because the secret was missing from the droplet's .env file, but because pm2 never read that file.

Checking /proc/<pid>/environ on the live pm2 process showed 24 environment variables. Zero of them were application secrets. The process had started with only the system environment, which on a DigitalOcean droplet contains nothing from your project's .env.

Why Most of the App Still Worked

This is the part that makes the bug hard to spot. Nuxt bakes build-time environment variables into the compiled output at the moment you run nuxt build. If a secret is referenced inside a plugin, a composable, or any code that runs at build time, it gets embedded in the bundle. The running process does not need to read it from process.env again.

Only secrets read at request time, inside a server route handler, are fetched live from process.env. CLEANUP_SECRET lives inside a server route. It was undefined every time a request arrived. Everything else looked fine because everything else was baked in.

This is a subtle and genuinely dangerous property of SSR frameworks. The app appears healthy. Pages render. API calls succeed. One quiet background job silently fails.

The Measured Backlog

Before running any deletions, the backlog was audited:

• 96,159 objects past the 48-hour TTL
• Oldest object: 147 days
• Cross-referenced against: 16,425 projects and 39,118 layers
• Zero live references found among the candidates
• Bucket size before purge: 98,068 files
• Bucket size after purge: 2,032 files

The cross-reference step matters. Deleting by age alone risks removing files that are still referenced if the TTL logic ever misfired or if a project was created with an unusual timestamp. Checking against actual database records is slower but safe.

The Three Fixes

Fix one: ecosystem.config.cjs now parses the .env file.

The env block for each pm2 app entry now includes a parsed spread of the .env file. Any key set explicitly after the spread, such as RUN_SCHEDULED_EMAIL_TASKS, overrides the parsed value. pm2 does not do this automatically. You have to wire it up yourself, and the documentation does not shout about it.

Fix two: the workflow step exits non-zero on anything but 200.

The echo fallback is gone. The step now checks the HTTP status code and calls exit 1 if it is not 200. A 403 will fail the workflow run. A 500 will fail the workflow run. The only way to get a green tick is for the endpoint to return 200.

This is the fix that would have caught the bug on day one. A green tick should mean the thing worked, not that the thing ran.

Fix three: the endpoint returns 500 when the server has no secret.

Previously, a missing CLEANUP_SECRET on the server produced a 403. That is misleading. A 403 implies the caller sent the wrong credentials. A 500 says the server is broken. The distinction matters when you are reading logs at midnight trying to work out why a job is not running.

The endpoint now checks for the presence of the secret before comparing it. If the secret is absent, it returns 500 with a message that names the missing variable. Loud failure, not quiet failure.

How to Verify pm2 Is Actually Loading Your Secrets

Before shipping any pm2-managed process that depends on application secrets, do this check:

• Find the pid: pm2 list or pm2 show <app-name>
• Read its environment: cat /proc/<pid>/environ | tr '\0' '\n'
• Count your secrets: grep for a key you know should be present

If your secret is not in that output, pm2 did not load it. Fix the ecosystem config before assuming the .env file is enough.

The 1,965 tests that passed after the fix confirmed nothing else broke. The cleanup ran, the bucket dropped from 98,068 to 2,032 files, and the next scheduled run returned 200.

What I'd do differently

• Treat every scheduled job as untrusted from day one. A green tick from a workflow that has an echo fallback is not evidence the job ran. It is evidence the workflow completed.
• Add a canary check. After a cleanup run, assert that the object count in the bucket dropped by at least some minimum number. If the count does not move, the job did nothing and the check should fail.
• Verify pm2 env loading in the deploy script. A one-line grep against /proc/<pid>/environ right after pm2 restart would have caught this on the first deploy.
• Distinguish 403 from 500 at the endpoint level from the start. The original guard returned 403 for both "wrong secret" and "no secret". Those are different problems and deserve different status codes.
• Document the pm2 .env gap explicitly. It is not obvious that pm2 ignores .env files. A comment in the ecosystem config explaining why the parse step exists would save the next person an hour of confusion.