SGO-Audit
GitHub

/ Data-quality audit · NHTSA SGO crash reports

Does the reported automation label match the rest of the record?

An open-source data-quality tool for public NHTSA crash reports. It flags ADS vs Level 2 ADAS labels that look inconsistent, so a human can review them.

Try the mini demo See how it works

In one sentence

Manufacturers report crashes as either self-driving (ADS) or driver-assist (Level 2 ADAS). SGO-Audit checks whether that label looks consistent with the same public record, using rules, an ML model, and a local AI reader, then queues mismatches for people to inspect.

01 · Problem

Self-reported labels in a high-stakes public dataset

NHTSA’s Standing General Order (SGO) crash data is used by researchers, journalists, and policymakers. The ADS vs Level 2 ADAS label is self-reported, and NHTSA has said some records get corrected over time.

If you take every label at face value, you can misread the dataset. SGO-Audit asks a narrower question: does this record’s label look consistent with its own fields and narrative?

02 · Approach

Four checks. Plain English.

Think of it as three independent reviewers plus a ranking desk. None of them “decide the truth.” They raise their hand when something looks off.

    03 · Operating point

    Prefer a short, high-confidence queue

    The ML model could flag thousands of maybe-wrong labels. Instead we choose a precision-first threshold: keep synthetic precision at least 80%, then maximize recall inside that constraint.

    04 · Boundary

    What this is not

    • Not a verdict. A flag means “worth human review,” never “this label is wrong.”
    • Not a safety score. No manufacturer rankings, no crash rates, no “who is safer.”
    • Not exposure-normalized. SGO data lacks a reliable miles-driven denominator.
    • Not affiliated with NHTSA or any automaker.

    05 · Mini demo

    Watch one record get audited

    Pick a case. Press run. See what the rule checker, ML model, and narrative reader each say, then whether it lands in the review queue.

    Choose a case to begin.

    1 · Rule checker

    Waiting…

    2 · ML model

    Waiting…

    3 · LLM narrative reader

    Waiting…

    06 · Proof

    Tracked results on Archive-2021-2025

    Loading…

    Phase 1 × Phase 2 sample

    ODI investigation cross-check

      Live review queue

      Browse the tracked consensus queue. Filter by which checker disagreed.

      Report ID Entity Reported Checks Why it was queued

      07 · Reproduce

      Same pipeline. Fresh data. Your machine.

      Re-run on NHTSA’s archived 2021–2025 snapshot (canonical demo) or the current third-amended CSVs. To adapt to another similarly structured crash dataset, map columns in src/fields.py and keep the same three-signal pattern.

      Canonical Archive run

      pip install -r requirements.txt
python scripts/download_data.py
python scripts/run_all.py --llm-backend transformers --llm-sample 50
python scripts/build_showcase_data.py
python scripts/serve_showcase.py

      Current NHTSA CSVs

      python scripts/download_data.py --current --overwrite
python scripts/run_all.py --current --llm-backend transformers --llm-sample 50

      Unit tests use tiny fixtures and need no download: python -m pytest