AI judgment
OIS — opportunity gate
As of 2026-07-24, OIS read public complaint sources around the clock and handed me a short daily list of opportunities worth a closer look. Most of what it found, it threw away on purpose — and the throwing away was the part I had to think hardest about.
My role
Sole author and operator. I designed the pipeline, wrote the gate prompt and the scoring rubric, built the workflows, and I ran it on my own infrastructure. When it broke, I was the one who fixed it — see below, because it broke.
The decision
The gate was the first stage of a two-stage pipeline: a cheap LLM call deciding whether a raw candidate was worth spending a real scoring call on. Its job was coarse sieving, not quality ranking — and the metric that matters is false-drop rate, not accuracy.
The split is the design. Sieving and ranking are different questions with different tolerances for being wrong, and one scoring pass would have to serve both with a single threshold. So the gate answered only could this be worth a closer look, and a separate, more expensive stage did the ranking on whatever survived.
That split is what makes the metric choice obvious. Accuracy is the default and it is wrong here, because the two errors are neither symmetric nor equally visible. A false pass costs a few cents and one row in a queue the second stage would kill anyway — loud, cheap, self-correcting. A false drop is gone permanently and generates no complaint. Accuracy averages the two into one number that improves as the gate gets stricter, which is exactly the failure I need to detect. So the gate was deliberately over-permissive, and ranking was a separate stage with its own budget.
Proof
As of 2026-07-24 I had eight OIS workflows on my n8n instance, all active and scheduled: capture jobs pulling Reddit hourly and Shopify app reviews six-hourly, the gate, the scorer, a tier-two queue and its scorer, a gated-sample job that pulls a limited pool of opportunities back out for review, and a daily digest.
The most inspectable thing I have is an incident. The gate began erroring on 2026-07-23; I found it the next day during an audit, not from an alert. The cause was a timeout chain, not the gate logic: GPU contention on my self-hosted inference host pushed calls to 133–180 seconds against 90-second timeouts downstream — provable because the router in front kept succeeding at those same durations while the gate aborted. Timeouts raised, service restarted, all four stranded items re-gated: three gated out, one passed.
Code is private; walkthrough available on request.
What it shows I reason about which failure mode matters before optimising.
Honest limitations
There is no eval number, and I am not going to substitute accuracy for one. I have never measured this gate's false-drop rate. The argument above establishes reasoning and architecture, not a measurement — treat "the gate works" as unproven.
The measurement is specified rather than done: a harness that takes a case set, a run function, a scoring function and a metrics config, and returns a confusion matrix plus a false-drop view. Step one is a hand-labelled golden set of items the gate discarded, which does not exist yet. When the number arrives it ships with its test set and at least one failure mode it caught, not as a bare percentage.
And nothing paged me when the gate broke: it failed for roughly seventeen hours before an audit surfaced it — for a pipeline whose defining risk is silent loss, the wrong kind of irony. Alerting isn't built.
Stack
n8n on my own VPS, scheduled triggers, a router workflow in front of the LLM calls, Postgres/Supabase for the opportunity store. Built and exercised; today only the Reddit and Shopify capture jobs and the daily digest are running, the gate and scorer are built, not running.