Orchestration & reliability · rule dag-sensor-no-timeout
Airflow sensors with no timeout
A Metacenta review checks this under the rule A sensor gives up in bounded time. Everything below applies whether or not you ever commission one.
What this rule checks
This rule flags DAG files with a sensor that sets no timeout, no mode="reschedule" and no deferrable=True. Any one of the three passes. A file that builds no sensor is not judged.
Why it matters
Airflow's default sensor timeout is seven days, and a sensor in the default poke mode holds a worker slot for all of it. A missing upstream file can then stop other jobs from starting.
How to fix it
Give every sensor a timeout. Set timeout to how late the data can reasonably be. Use mode="reschedule" or a deferrable sensor, so the wait does not hold a slot.
Before:
FileSensor(
task_id="wait_for_orders",
filepath="orders.csv",
)
After:
FileSensor(
task_id="wait_for_orders",
filepath="orders.csv",
timeout=6 * 60 * 60,
mode="reschedule",
)
When it is fine to leave
A short poke-mode sensor on a cluster with spare slots does little harm, and poke mode reacts faster. Set a timeout even then, so a file that never lands fails the run.
What we need to check it
The DAG files you name to us one at a time. We count sensors by their constructor, so the finding is medium confidence.