My Most Important Learning from Cron Jobs

Over the course of a year, I have written and implemented a few cron jobs often with great trepidation. This fear comes from the inherent danger of a cron job — since many of our cron jobs run every minute, a single bug can be run over and over again and its disastrous effect compounded quite quickly. A cron job deployed to send emails about up-coming appointments contained a bug which led to an email being sent to a single person every 60 seconds until the bug was discovered. Such events have made me increasingly nervous and cautious about designing, deploying, and logging of the cron job and have led me to extreme caution as to prevent mistakes such as the run away emails being sent. Here is the most important finding I’ve had about the design of a cron job. But first a quick overview of cron jobs.

Cron jobs have a very basic structure:

1. Some SQL statement which looks at specific triggers to recognize what action is required
2. Logic to filter which action (if at all) is required (ie. what email to send a person)
3. Action to carry out the desired goal of the cron job (ie. an api request to send an email)
4. SQL update statement to mark the action is a completed (ie. mark email as sent so they don’t appear as deserving an email in the next run)
5. Log the action took place

The fundamental flaw comes between step 3 and 4. What happens if between step 3 and 4, the code fails? The update statement would never be run, and when the cron runs again, the same user will appear as not having received the email. This is a huge issue — that the action can continue to be run over and over again and cannot be stopped.

Preventing this is quite simple. Yes, you could somehow wrap things in try-excepts, so that even on exception raising the update statement can be run. But, even simpler and more readable, is switching the order of the action and the update statement. This small ordering change can have a massive impact, ensuring that the action is never trigged even if the code crashes elsewhere.

The new cron job structure is:

1. Some SQL statement which looks at specific triggers to recognize what action is required
2. Logic to filter which action (if at all) is required (ie. what email to send a person)
3. SQL update statement to mark the action is a completed (ie. mark email as sent so they don’t appear as deserving an email in the next run)
4. Action to carry out the desired goal of the cron job (ie. an api request to send an email)
5. Log the action took place

This simple re-ordering makes a massive difference and is cleaner than a try-except. But most importantly, it forces you to think about why/how you write code and consequence of every line.