My Most Important Learning from Cron Jobs

Over the course of a year, I have written and implemented a few cron jobs often with great trepidation. This fear comes from the inherent danger of a cron job — since many of our cron jobs run every minute, a single bug can be run over and over again and its disastrous effect compounded quite quickly. A cron job deployed to send emails about up-coming appointments contained a bug which led to an email being sent to a single person every 60 seconds until the bug was discovered. Such events have made me increasingly nervous and cautious about designing, deploying, and logging of the cron job and have led me to extreme caution as to prevent mistakes such as the run away emails being sent. Here is the most important finding I’ve had about the design of a cron job. But first a quick overview of cron jobs.

Cron jobs have a very basic structure:

  1. Some SQL statement which looks at specific triggers to recognize what action is required
  2. Logic to filter which action (if at all) is required (ie. what email to send a person)
  3. Action to carry out the desired goal of the cron job (ie. an api request to send an email)
  4. SQL update statement to mark the action is a completed (ie. mark email as sent so they don’t appear as deserving an email in the next run)
  5. Log the action took place

The fundamental flaw comes between step 3 and 4. What happens if between step 3 and 4, the code fails? The update statement would never be run, and when the cron runs again, the same user will appear as not having received the email. This is a huge issue — that the action can continue to be run over and over again and cannot be stopped.

Preventing this is quite simple. Yes, you could somehow wrap things in try-excepts, so that even on exception raising the update statement can be run. But, even simpler and more readable, is switching the order of the action and the update statement. This small ordering change can have a massive impact, ensuring that the action is never trigged even if the code crashes elsewhere.

The new cron job structure is:

  1. Some SQL statement which looks at specific triggers to recognize what action is required
  2. Logic to filter which action (if at all) is required (ie. what email to send a person)
  3. SQL update statement to mark the action is a completed (ie. mark email as sent so they don’t appear as deserving an email in the next run)
  4. Action to carry out the desired goal of the cron job (ie. an api request to send an email)
  5. Log the action took place

This simple re-ordering makes a massive difference and is cleaner than a try-except. But most importantly, it forces you to think about why/how you write code and consequence of every line.

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Pytest in Python

Translating mobile apps, automate all the things!

Release Wrap Up — August 2020

Backup & Restore data in ‘etcd’

Galaxy Blitz INO Whitelisting Result

The Composite Pattern

Scope Methods In Rails: What Are They And When Should You Use Them?

Reusing angular component— The RIGHT way (from scratch)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Naveen Mohideen

Naveen Mohideen

More from Medium

My Experience at CES 2022

Smartphone cameras explained

Tech Game Designer’s Scrapbook #1

A “Good Job” goes a long way!