Understanding Magento 2's Cron Functionality

Magento cron is one system crontab entry that runs bin/magento cron:run every minute. Each run reads every module's etc/crontab.xml, writes upcoming jobs into the cron_schedule table as pending, then executes the ones that are due. You never add module jobs to the server crontab; you add them to crontab.xml.

That single line hides every scheduled job in every module, and that is where most cron confusion starts. Let's go over the terms first, then how the pieces fit together inside Magento's framework.

What is the difference between a cron, crontab, and cron job?

Three words get mixed up constantly.

A cron is a daemon service that runs on the server and executes scheduled tasks. You can think of it as a background app that runs jobs from a crontab file.

A crontab is a configuration file, and its name is short for cron table. A crontab file contains a list of jobs that run on a defined schedule. This schedule is written as a cron expression (e.g., * * * * *), and you can learn more about cron expressions from crontab guru.

A cron job is an individual scheduled task listed in a crontab file. There can be one or many cron jobs defined within a single crontab file.

How does cron work with Magento?

In a plain PHP application, you would typically define many cron jobs within a single crontab file. Each job then executes on the schedule specified by its cron expression.

For example, here is a user crontab (edited with crontab -e) that contains two cron jobs:

# Run a PHP script every day at 3:00 AM
0 3 * * * /usr/bin/php /var/www/html/daily_backup.php
 
# Run another PHP script every 30 minutes
*/30 * * * * /usr/bin/php /var/www/html/check_notifications.php

Things work a bit differently in Magento. Defining many cron jobs for multiple modules within a crontab file would be extremely difficult to manage, because this file lives outside of the Magento project. It isn't in version control, and it's hard to keep in sync between environments.

So instead, a Magento instance gets exactly one cron job, which is the same for every environment. It belongs in the crontab of the user that owns the Magento files (the web server user, e.g. crontab -u www-data -e), not in /etc/crontab, which expects an extra user column:

* * * * * /usr/local/bin/php /var/www/html/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /var/www/html/var/log/magento.cron.log

The command above executes bin/magento cron:run every minute, as defined with the * * * * * cron expression. Everything after the command filters out the "Ran jobs by schedule" noise and appends the rest of the output to var/log/magento.cron.log inside the Magento root (/var/www/html/var/log, not the system /var/log).

This means scheduled tasks in Magento are completely managed by Magento's cron:run command. If you're new to the Magento CLI, you can learn how to execute CLI commands in Magento to manage cron and other system functions.

How do I install Magento cron?

You don't need to hand-write that line. Magento ships a command that writes it for you:

bin/magento cron:install

Run it as the user that owns the Magento files. It adds this block to that user's crontab:

#~ MAGENTO START 69dd2b02e1f3a65918182048ea4e29979a849d8942e8f53ed20a4bf10e529b36
* * * * * /usr/local/bin/php /var/www/html/bin/magento cron:run >> /var/www/html/var/log/magento.cron.log 2>&1
#~ MAGENTO END 69dd2b02e1f3a65918182048ea4e29979a849d8942e8f53ed20a4bf10e529b36

The PHP binary is whatever ran the command, and the hash after MAGENTO START is a SHA-256 of the Magento root path, so several installs on one server each get their own block. If a block already exists, cron:install refuses to touch it unless you pass --force. To take it back out, run bin/magento cron:remove. Check what got written with crontab -l.

Defining cron jobs in Magento

With this single cron job in place, Magento runs its own scheduler internally, and any module can plug into it.

Cron jobs are an essential part of Magento's framework. They handle everything from reindexing products to sending out customer emails, and you can use the same system to define your own scheduled tasks.

Just like we can define a crontab file on a Unix server, we can define our own crontab file within Magento, but in standard Magento fashion… it's gotta be an XML file.

What is the purpose of crontab.xml?

The crontab.xml file specifies the following details for each cron job:

• Job name: An identifier for the cron job, which must be unique across all other cron jobs in Magento.
• Schedule: How often the cron job should run, written in cron expression syntax.
• Run model: The class and method that execute when the cron job runs.

This file lives at etc/crontab.xml in any Magento module and looks like this:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Cron:etc/crontab.xsd">
    <group id="default">
        <job name="daily_backup" instance="Vendor\Module\Cron\DailyBackup" method="execute">
            <schedule>0 3 * * *</schedule>
        </job>
        <job name="check_notifications" instance="Vendor\Module\Cron\CheckNotifications" method="execute">
            <schedule>*/30 * * * *</schedule>
        </job>
    </group>
</config>

You'll notice that daily_backup and check_notifications are the same cron jobs as the earlier example, but this time in Magento's crontab XML format. We are simply defining the class responsible for each cron job and the method to execute.

If you want admins to change the schedule without a deploy, swap <schedule> for <config_path>, which points at a config value holding the cron expression. Core does this for currency rate updates:

<job name="currency_rates_update" instance="Magento\Directory\Model\Observer" method="scheduledUpdateCurrencyRates">
    <config_path>crontab/default/jobs/currency_rates_update/schedule/cron_expr</config_path>
</job>

That value resolves through the normal Magento config fallback, so it can also be pinned in env.php per environment.

Cron groups

Looking at the example above and just about any other crontab.xml file in Magento, you'll notice that the group ID is usually default. This is the default group used to organize cron jobs. Core also ships an index group for indexers and a consumers group for message queue consumers.

You can define your own custom group to better organize your custom cron jobs. For example, setting <group id="foo"> places your cron jobs into a new foo group. While you'll usually use the default group, a custom group can be handy, especially during development, as Magento lets you execute cron jobs per group with:

bin/magento cron:run --group=foo

This line will only execute cron jobs associated with the foo group.

This approach lets you focus on your custom cron jobs during development without triggering all the other cron jobs in Magento. It can be a real time-saver, especially when you are knee-deep in development and debugging your code.

A custom group also needs its own etc/cron_groups.xml. Without it, the group has no scheduling settings, and this is the most common reason a job in a custom group never runs:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Cron:etc/cron_groups.xsd">
    <group id="foo">
        <schedule_generate_every>15</schedule_generate_every>
        <schedule_ahead_for>20</schedule_ahead_for>
        <schedule_lifetime>15</schedule_lifetime>
        <history_cleanup_every>10</history_cleanup_every>
        <history_success_lifetime>60</history_success_lifetime>
        <history_failure_lifetime>4320</history_failure_lifetime>
        <use_separate_process>1</use_separate_process>
    </group>
</config>

These are the same values core uses for the default group (in Magento_Cron/etc/cron_groups.xml), except use_separate_process, which runs the group in its own PHP process so one slow job doesn't hold up the rest. All of the times are in minutes:

• schedule_generate_every: How often new pending rows get written for the group.
• schedule_ahead_for: How far into the future those rows are scheduled.
• schedule_lifetime: How late a job can start before it's marked missed.
• history_cleanup_every, history_success_lifetime, history_failure_lifetime: How often old rows are deleted, and how long successful and failed rows are kept.

crontab.xml works with cron_schedule

The system cron fires cron:run every minute, but that doesn't mean every job runs every minute. Each run does two things for each group:

1. It reads the cron expressions from every crontab.xml and generates future pending rows in the cron_schedule table (every schedule_generate_every minutes, covering the next schedule_ahead_for minutes).
2. It executes the pending rows whose scheduled_at time has arrived.

A row that sits unrun for longer than schedule_lifetime becomes missed.

The cron_schedule database table

This table keeps track of every scheduled run and its result. In core Magento (Magento_Cron/etc/db_schema.xml), it has exactly eight columns:

• schedule_id: The auto-increment primary key.
• job_code: The cron job's identifier, from the name attribute of the <job> node in crontab.xml.
• status: The current status of the run: pending, running, success, missed or error.
• messages: Output for the run, such as the exception message and trace when a job fails.
• created_at: When the row was created.
• scheduled_at: When the run is scheduled to start.
• executed_at: When the run actually started.
• finished_at: When the run finished.

There is no group, hostname, duration or PID column in core. If you see those, they come from a third-party cron manager or your hosting layer.

Why is my Magento cron job not running?

When a job doesn't fire, check these in order:

1. Is the system cron installed? Run crontab -l as the Magento file owner and look for the #~ MAGENTO START block. Without it, none of the jobs defined in crontab.xml files will ever execute.
2. Is there a row for it? Query cron_schedule for your job_code. No rows usually means the group has no cron_groups.xml, the module isn't enabled, or the cache still has the old cron config (bin/magento cache:clean config).
3. What status did it end up in? Rows piling up as missed mean cron isn't running often enough, or a long job in the same group is blocking it, which is exactly what use_separate_process is for. Rows in error have the exception in the messages column.
4. How long does it take? Compare executed_at and finished_at. A job that takes longer than its own schedule interval will keep stepping on itself.

You don't need to clean this table by hand. Magento deletes old rows on its own using the history settings above. If the table grows too large, tune them per group at Stores > Configuration > Advanced > System > Cron (Scheduled Tasks) with "History Cleanup Every", "Success History Lifetime" and "Failure History Lifetime".

Recap

• Don't add module jobs to the system crontab. Install the single cron:run entry with bin/magento cron:install, and define jobs in crontab.xml.
• Use cron groups to organize and selectively run specific sets of jobs, and give every custom group a cron_groups.xml.
• Read the cron_schedule table's status and messages columns when a job misbehaves.
• Let Magento's history settings clean up cron_schedule; tune them instead of deleting rows.

Cron is one of two ways Magento runs work outside the request. The other is message queue consumers, which you can learn in the Message Queues course.