/ Local-first LLM agent · safety-event triage

Which event needs a human in the next 24 hours?

A config-driven agent that reads any tabular safety-event feed, classifies severity with deterministic rules plus a tool-calling local LLM, and flags what likely needs regulatory reporting. No byte of data ever leaves your machine.

Run the demo Open the review workbench

false negatives end-to-end on 60 hand-labeled events

95%

rules-only agreement with human labels (57/60)

domains (NHTSA, OSHA), one codebase, config only

100%

local. Ollama on your machine, no API key

In one sentence

Regulators give operators as little as 24 hours to report a qualifying safety event, and deciding which of hundreds of incoming events qualifies still happens in spreadsheets. This agent does the first pass: rules set the floor, a local LLM reads the narrative, and a human reviews a short, explained queue.

01 · Problem

Hard clocks, soft tooling

NHTSA's Standing General Order gives AV operators as little as one day to report a qualifying crash; OSHA gives employers 24 hours for an in-patient hospitalization. Missing a report risks fines and consent orders; flagging everything buries reviewers until they rubber-stamp.

The failure mode that matters most is the false negative: a reportable event that never reaches a human. The design goal is a floor that never misses on structured signals, plus an escalation path for what only the narrative reveals.

02 · How it works

A floor you can audit, an escalation you can read

Normalize via config

A YAML file maps your dataset's columns onto a small canonical schema. New domain, new config, zero code changes. The repo ships NHTSA AV crash reports and OSHA severe-injury reports driving identical code.

Deterministic rules set the floor

Keyword and threshold rules on structured fields (injury severity, crash partner, pre-crash speed) produce a severity and a reportability call with stated reasons. Auditable, fast, and never silently wrong.

A local LLM reads the narrative

A tool-calling model via Ollama drafts a plain-language summary and can escalate over the floor when the free-text narrative reveals what structured fields miss. It can raise, never lower: escalation only adds to the review queue.

Why fully local: safety-event data is often sensitive (PII, confidential business information, regulatory exposure). Local-only inference is not a preference here, it is the constraint that shaped the architecture.

03 · Mini demo

Watch one event get triaged

Four real events from NHTSA's public SGO crash data (pre-redacted at the source). Pick one, press run, and watch the floor and the escalation do their separate jobs.

Choose an event to begin.

1 · Normalize (YAML config)

Waiting…

2 · Rules floor

Waiting…

3 · Local LLM (qwen2.5:14b via Ollama)

Waiting…

This demo replays recorded pipeline output for four real events, exactly what the agent produced, no live model call in the browser. The review workbench has all 25 demo events with filtering, labeling, and rules-vs-LLM reasoning side by side.

04 · Proof

Measured against hand-labeled ground truth

Sixty real NHTSA events were hand-labeled (LLM-drafted, then human-reviewed event by event, including deliberate disagreements with the rules engine). The eval harness scores the agent against those labels:

Configuration	Agreement	False negatives	False positives
Rules only	95% (57/60)	3	0
Rules + LLM escalation	32% (19/60)	0	41

The three events the rules missed are visible only in the narrative (an alleged minor injury, a dog struck at 28 mph, an AV driving under a closing railroad gate) and the LLM escalation caught all three. The cost is 41 extra events in the review queue, a deliberate trade: in this product, a missed report costs more than a longer queue, and the numbers that prove the trade are published, not hidden.

05 · Generality

Nothing is hard-coded to NHTSA

config/nhtsa_sgo_ads.yaml

NHTSA AV crash reports: keyword rules on injury and crash-partner fields, a numeric pre-crash-speed threshold.

config/osha_severe_injury.yaml

OSHA workplace severe-injury reports: numeric rules mirroring the statutory reporting triggers (amputation, hospitalization, loss of an eye).

Your dataset

Any CSV with structured fields plus a free-text narrative. Write a config, point the CLI at it, 57 tests keep the core honest.

06 · Reproduce

Runs on your machine, against real public data

ollama pull qwen2.5:14b
pip install -e '.[dev]'
python fetch_data.py            # real NHTSA SGO CSVs
python main.py --config config/nhtsa_sgo_ads.yaml
python eval/run_eval.py         # score against the 60 hand labels

MIT licensed. Built AI-assisted under the staged process documented in AI-SDLC.