A sanitised reference implementation. Drop it into Claude Code, point it at your own scheduled jobs, adapt the windows. Generic on purpose, with no business specifics and no real schedules.
The problem
A scheduled job can fail two different ways. It can stop running at all, which is the easy case to catch. Or it can keep running, keep reporting success, and quietly stop doing the one thing it exists to do, storing something. A watcher that only asks "did it run" misses the second kind completely, and that kind is the one that does real damage, because everything looks fine right up until someone needs the data that was never actually saved.
The idea
Give every watched job two separate clocks, not one. The first tracks the last time it succeeded at all. The second, usually a longer window, tracks the last time it actually produced something, a stored row, a written file. A job can be well within its first window and still be flagged, because the second one has run out. Every watched job also carries its own short hint, written in advance, about what to actually go and check if it goes quiet, so the first response to an alert is a look in the right place, not a cold start.
for each job: last success ──> within window? ──yes──> last output ──> within
│no quiet window? │no
v │yes v
STALE OK QUIET
Minimal reference implementation
# dead_mans_switch.py, reference only. Swap in your own job list and log source.
from datetime import timedelta
# name, success window, quiet window, hint. The quiet window is usually longer
# than the success window, for a job that only expects new data every so often.
WATCHED_JOBS = (
("example_job", timedelta(hours=2), timedelta(days=3),
"check the job's own log for an error line before assuming it is dead"),
)
def judge(last_success, window, now, woke_recently=False):
"""A job that has never once succeeded is always stale. A job waking from
sleep gets one cycle of grace before it is called stale, never longer."""
if last_success is not None and now - last_success <= window:
return "ok"
if last_success is not None and woke_recently:
return "grace"
return "stale"
def check(job_state, now, woke_recently=False):
"""job_state: {name: (last_success, last_output)}. Returns one finding per
watched job. Never raises; a job it cannot read is its own finding."""
findings = []
for name, window, quiet_window, hint in WATCHED_JOBS:
last_success, last_output = job_state.get(name, (None, None))
state = judge(last_success, window, now, woke_recently)
quiet = judge(last_output, quiet_window, now, woke_recently) == "stale"
findings.append({
"name": name, "state": state, "quiet": quiet,
"flagged": state == "stale" or quiet,
"hint": hint if quiet else None,
})
return findings
Why it works
- Two clocks, not one. Ran recently and produced something recently are two different facts. Checking only the first is how a job that "succeeds" while storing nothing goes unnoticed for weeks.
- Never once succeeded is always stale. A job with no history gets no benefit of the doubt, however new it is.
- A short grace window after a wake, never longer. A laptop that was asleep overnight should not fire a false alarm the moment it wakes, but the grace has to end after one cycle or a genuinely dead job hides behind it forever.
- A hint per job, written in advance. "Something is wrong" sends a person hunting. "Check the job's own log for an error line" sends them straight there.
- Fails soft, on a budget. The check itself is time bounded and never raises. A slow read or a missing source becomes its own separate finding, never a crash that takes the whole check down with it.
Adapt it
- Set the quiet window per job, not globally. A job that only expects new data every few weeks needs a much longer quiet window than one that runs every fifteen minutes.
- Watch a log file the same way as a database table. The two clock idea does not care where the timestamp comes from.
- Keep the hint text specific and short. It is read once, under pressure, at the moment something has actually gone quiet.
Built for my own set of scheduled jobs, runs at the start of every working session, and this is the generalised version. Take it, point it at your own job list, tell me what you would add.