So today I discovered that there’s a cron job that holds non-reproducible state that died, and now our system is fucked.

The cron job doesn’t live inside any source control. This morning it entered a terminal state, and because it overwrites its state there’s no way to revert it.

I’m currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.

  • qaz@lemmy.worldEnglish
    2·
    21 hours ago
    1. They’re all gone when you restart
    2. It doesn’t properly deal with logging
    3. You can’t set up dependencies between services but that doesn’t matter due to point 1

    I recommend using systemd services and/or docker compose instead. systemd services are files that describe which program / script to run and when (like after networking is active or after a certain other service is loaded).