# Months of Green Ticks over a Cleanup Job That Never Deleted a File > A scheduled cleanup reported success for months while doing nothing. pm2 never read the droplet .env, so CLEANUP_SECRET was undefined, every call got a 403, and an echo fallback swallowed the error. - Author: Dil Ahdan (https://dil.dev/) - Published: 2026-10-19 - Updated: 2026-10-19 - Tags: pm2, devops, node, debugging - Canonical: https://dil.dev/notes/pm2-runtime-secret-silent-cleanup/ ## Key takeaways - pm2 does not read a .env file automatically; secrets must be parsed into the ecosystem config's env block. - A non-zero exit code from a workflow step is the only reliable signal of failure; echo fallbacks silently bury errors. - Nuxt bakes build-time secrets in at compile time, so most secrets survived, but any secret read at request time was undefined. - 96,159 objects accumulated past their 48-hour TTL, the oldest 147 days, before the bug was caught. - The endpoint now returns an explicit 500 when the server has no secret configured, so the failure is loud rather than quiet. The scheduled cleanup for Free Stencil Maker reported success on every run for months. No alerts fired. No one looked twice. Then on 12 September 2026, a routine check of the storage bucket showed 96,159 objects sitting past their 48-hour TTL, the oldest 147 days stale. The cleanup had never deleted a single file. ## What the Workflow Actually Did The GitHub Actions step called the cleanup endpoint, got a 403 back, then hit a fallback line: ``` echo 'Cleanup request failed (non-fatal)' ``` That echo exits zero. A zero exit code means a green tick. The workflow had been reporting success by lying, not by doing anything useful. The fallback was written defensively, presumably so a transient network hiccup would not block a deploy. The problem is that "non-fatal" is only safe if the failure actually is transient. A 403 that fires every single run is not transient. It is a permanent misconfiguration dressed up as a warning. ## Why the Endpoint Returned 403 Every Time The cleanup endpoint reads `process.env.CLEANUP_SECRET` at request time and compares it against the value in the incoming request header. If the server has no secret, the guard fails closed and returns a flat 403. The server had no secret. Not because the secret was missing from the droplet's `.env` file, but because pm2 never read that file. Checking `/proc//environ` on the live pm2 process showed 24 environment variables. Zero of them were application secrets. The process had started with only the system environment, which on a DigitalOcean droplet contains nothing from your project's `.env`. ## Why Most of the App Still Worked This is the part that makes the bug hard to spot. Nuxt bakes build-time environment variables into the compiled output at the moment you run `nuxt build`. If a secret is referenced inside a plugin, a composable, or any code that runs at build time, it gets embedded in the bundle. The running process does not need to read it from `process.env` again. Only secrets read at request time, inside a server route handler, are fetched live from `process.env`. `CLEANUP_SECRET` lives inside a server route. It was undefined every time a request arrived. Everything else looked fine because everything else was baked in. This is a subtle and genuinely dangerous property of SSR frameworks. The app appears healthy. Pages render. API calls succeed. One quiet background job silently fails. ## The Measured Backlog Before running any deletions, the backlog was audited: - **96,159 objects** past the 48-hour TTL - **Oldest object:** 147 days - **Cross-referenced against:** 16,425 projects and 39,118 layers - **Zero live references** found among the candidates - **Bucket size before purge:** 98,068 files - **Bucket size after purge:** 2,032 files The cross-reference step matters. Deleting by age alone risks removing files that are still referenced if the TTL logic ever misfired or if a project was created with an unusual timestamp. Checking against actual database records is slower but safe. ## The Three Fixes **Fix one: ecosystem.config.cjs now parses the .env file.** The env block for each pm2 app entry now includes a parsed spread of the `.env` file. Any key set explicitly after the spread, such as `RUN_SCHEDULED_EMAIL_TASKS`, overrides the parsed value. pm2 does not do this automatically. You have to wire it up yourself, and the documentation does not shout about it. **Fix two: the workflow step exits non-zero on anything but 200.** The echo fallback is gone. The step now checks the HTTP status code and calls `exit 1` if it is not 200. A 403 will fail the workflow run. A 500 will fail the workflow run. The only way to get a green tick is for the endpoint to return 200. This is the fix that would have caught the bug on day one. A green tick should mean the thing worked, not that the thing ran. **Fix three: the endpoint returns 500 when the server has no secret.** Previously, a missing `CLEANUP_SECRET` on the server produced a 403. That is misleading. A 403 implies the caller sent the wrong credentials. A 500 says the server is broken. The distinction matters when you are reading logs at midnight trying to work out why a job is not running. The endpoint now checks for the presence of the secret before comparing it. If the secret is absent, it returns 500 with a message that names the missing variable. Loud failure, not quiet failure. ## How to Verify pm2 Is Actually Loading Your Secrets Before shipping any pm2-managed process that depends on application secrets, do this check: - **Find the pid:** `pm2 list` or `pm2 show ` - **Read its environment:** `cat /proc//environ | tr '\0' '\n'` - **Count your secrets:** grep for a key you know should be present If your secret is not in that output, pm2 did not load it. Fix the ecosystem config before assuming the `.env` file is enough. The 1,965 tests that passed after the fix confirmed nothing else broke. The cleanup ran, the bucket dropped from 98,068 to 2,032 files, and the next scheduled run returned 200. ## What I'd do differently - **Treat every scheduled job as untrusted from day one.** A green tick from a workflow that has an echo fallback is not evidence the job ran. It is evidence the workflow completed. - **Add a canary check.** After a cleanup run, assert that the object count in the bucket dropped by at least some minimum number. If the count does not move, the job did nothing and the check should fail. - **Verify pm2 env loading in the deploy script.** A one-line grep against `/proc//environ` right after pm2 restart would have caught this on the first deploy. - **Distinguish 403 from 500 at the endpoint level from the start.** The original guard returned 403 for both "wrong secret" and "no secret". Those are different problems and deserve different status codes. - **Document the pm2 .env gap explicitly.** It is not obvious that pm2 ignores `.env` files. A comment in the ecosystem config explaining why the parse step exists would save the next person an hour of confusion. ## Quick answers **Why did pm2 not pick up my .env file?** pm2 does not automatically source a .env file from the project root. Unless you explicitly parse the file and spread its contents into the env block of ecosystem.config.cjs, the process starts with only the system environment, which on a fresh droplet contains no application secrets. **How do I pass .env variables to a pm2 process?** Parse the .env file inside ecosystem.config.cjs using dotenv or a manual fs.readFileSync approach, then spread the result into the env block for each app entry. Any key you set after the spread will override the parsed values, which is useful for flags like RUN_SCHEDULED_EMAIL_TASKS. **Why did most of the app still work if pm2 never read the .env?** Nuxt bakes build-time environment variables into the compiled output at the point you run the build command. Only secrets that are read at request time, like CLEANUP_SECRET inside a server route handler, are fetched from process.env live and were therefore undefined. **How do I stop a GitHub Actions workflow step from swallowing a non-200 response?** Check the HTTP status code explicitly in the step and exit with a non-zero code if it is anything other than 200. An echo fallback that always exits zero will mark the step green regardless of what the server returned. **How do I safely purge a large backlog of orphaned files without deleting live assets?** Cross-reference every object key against your database before deleting. In this case 96,159 candidate objects were checked against 16,425 projects and 39,118 layers before any deletion ran. **What should a cleanup endpoint return when it has no secret configured?** Return an explicit 500, not a 403. A 403 implies the caller sent the wrong secret; a 500 signals a server-side misconfiguration. The distinction matters when you are reading logs to diagnose why a scheduled job is not working.