# Dead Man's Switch, the watcher that speaks up when a scheduled job goes quiet

A sanitised reference implementation. Drop it into Claude Code, point it at your own
scheduled jobs, adapt the windows. Generic on purpose, with no business specifics
and no real schedules.

## The problem

A scheduled job can fail two different ways. It can stop running at all, which is
the easy case to catch. Or it can keep running, keep reporting success, and quietly
stop doing the one thing it exists to do, storing something. A watcher that only
asks "did it run" misses the second kind completely, and that kind is the one that
does real damage, because everything looks fine right up until someone needs the
data that was never actually saved.

## The idea

Give every watched job two separate clocks, not one. The first tracks the last time
it succeeded at all. The second, usually a longer window, tracks the last time it
actually produced something, a stored row, a written file. A job can be well within
its first window and still be flagged, because the second one has run out. Every
watched job also carries its own short hint, written in advance, about what to
actually go and check if it goes quiet, so the first response to an alert is a look
in the right place, not a cold start.

```
for each job: last success ──> within window? ──yes──> last output ──> within
                  │no                                   quiet window?  │no
                  v                                          │yes      v
                STALE                                        OK      QUIET
```

## Minimal reference implementation

```python
# dead_mans_switch.py, reference only. Swap in your own job list and log source.
from datetime import timedelta

# name, success window, quiet window, hint. The quiet window is usually longer
# than the success window, for a job that only expects new data every so often.
WATCHED_JOBS = (
    ("example_job", timedelta(hours=2), timedelta(days=3),
     "check the job's own log for an error line before assuming it is dead"),
)

def judge(last_success, window, now, woke_recently=False):
    """A job that has never once succeeded is always stale. A job waking from
    sleep gets one cycle of grace before it is called stale, never longer."""
    if last_success is not None and now - last_success <= window:
        return "ok"
    if last_success is not None and woke_recently:
        return "grace"
    return "stale"

def check(job_state, now, woke_recently=False):
    """job_state: {name: (last_success, last_output)}. Returns one finding per
    watched job. Never raises; a job it cannot read is its own finding."""
    findings = []
    for name, window, quiet_window, hint in WATCHED_JOBS:
        last_success, last_output = job_state.get(name, (None, None))
        state = judge(last_success, window, now, woke_recently)
        quiet = judge(last_output, quiet_window, now, woke_recently) == "stale"
        findings.append({
            "name": name, "state": state, "quiet": quiet,
            "flagged": state == "stale" or quiet,
            "hint": hint if quiet else None,
        })
    return findings
```

## Why it works

- **Two clocks, not one.** Ran recently and produced something recently are two
  different facts. Checking only the first is how a job that "succeeds" while
  storing nothing goes unnoticed for weeks.
- **Never once succeeded is always stale.** A job with no history gets no benefit
  of the doubt, however new it is.
- **A short grace window after a wake, never longer.** A laptop that was asleep
  overnight should not fire a false alarm the moment it wakes, but the grace has to
  end after one cycle or a genuinely dead job hides behind it forever.
- **A hint per job, written in advance.** "Something is wrong" sends a person
  hunting. "Check the job's own log for an error line" sends them straight there.
- **Fails soft, on a budget.** The check itself is time bounded and never raises. A
  slow read or a missing source becomes its own separate finding, never a crash
  that takes the whole check down with it.

## Adapt it

- Set the quiet window per job, not globally. A job that only expects new data
  every few weeks needs a much longer quiet window than one that runs every
  fifteen minutes.
- Watch a log file the same way as a database table. The two clock idea does not
  care where the timestamp comes from.
- Keep the hint text specific and short. It is read once, under pressure, at the
  moment something has actually gone quiet.

Built for my own set of scheduled jobs, runs at the start of every working session,
and this is the generalised version. Take it, point it at your own job list, tell me
what you would add.