Orchestration & reliability · rule dag-catchup-unbounded
Airflow catchup left on by default
A Metacenta review checks this under the rule A deploy does not trigger a backfill. Everything below applies whether or not you ever commission one.
What this rule checks
This rule flags scheduled Airflow DAGs that set catchup=True, or do not set it at all. catchup=False passes, and so does a DAG with schedule=None or no schedule we can read. Dagster and Prefect have no such default, so they are not judged.
Why it matters
Catchup replays every interval between the start date and now. Pause a DAG for a fortnight, or deploy one with an old start date, and resuming it queues hundreds of runs at once. The warehouse bill and the load spike arrive together.
How to fix it
Stop a redeploy from replaying the schedule. Set catchup=False unless the backfill is wanted. Where it is wanted, bound it with max_active_runs.
Before:
with DAG("daily_orders", schedule="@daily", start_date=datetime(2024, 1, 1)):
...
After:
with DAG(
"daily_orders",
schedule="@daily",
start_date=datetime(2024, 1, 1),
catchup=False,
):
...
When it is fine to leave
A DAG meant to backfill, such as one loading partitions by date, is working as intended. We read an undeclared catchup as on, which is Airflow 2's default. Airflow 3 changed the default to off, so tell us your version.
What we need to check it
The Airflow DAG files you name to us one at a time. We rate catchup=True written out at high confidence, and an undeclared catchup at medium.