/ Perception evaluation case study
See where detectors miss the people they should protect.
Off-the-shelf detectors can look fine on aggregate metrics while missing exactly the road users who matter most: small, distant, or poorly lit pedestrians and cyclists. VRU-Detect measures that failure honestly on BDD100K road scenes, fine-tunes, and chooses operating thresholds with an explicit miss-cost argument.
Fine-tuning changed the failure shape, not just the score
Same 300-image validation split, confidence 0.25. The baseline's pathology was rider false positives (COCO confuses person and bike); fine-tuning collapsed those while lifting person and bike recall. Aggregate numbers improved, and the residual risk moved somewhere specific.
Precision by class
conf = 0.25
Recall by class
conf = 0.25
Person precision dips slightly (.754 → .689) while person recall rises, an acceptable trade when a missed pedestrian costs more than a false alarm. That judgment, not the score, is the product decision.
Where the risk still lives
Stratified recall under the conditions that matter for urban safety. Every condition improved. None of them is solved, and small objects barely moved. This chart is the reason the project exists: aggregate mAP hides exactly these rows.
Hard-condition recall, baseline → fine-tuned
validation split, conf = 0.25
Small-object recall of 0.079 means a distant pedestrian is still missed more than nine times out of ten. Publishing that number is the point: it is what deciding not to ship on aggregate mAP looks like in practice.
Thresholds are product decisions
A confidence threshold is not a tuning detail; it encodes whose error costs more. Sweeping thresholds on the fine-tuned model makes the trade explicit, then each class gets an operating point chosen against a stated miss-cost story.
Precision / recall vs. confidence threshold
fine-tuned model, all classes
Try it: move the operating point yourself
Live numbers from the committed threshold sweep on the validation split (1,059 true VRU instances, all classes). No simulation, these are the measured counts.
| Class | Chosen threshold | P / R at that point | Why |
|---|---|---|---|
| person | 0.05 | 0.291 / 0.605 | Recall-first: within 95% of max recall, then highest precision. A missed pedestrian outweighs a false alarm. |
| rider | 0.05 | 0.219 / 0.432 | Same recall-first rule; riders share the person miss-cost profile. |
| bike | 0.15 | 0.424 / 0.209 | Best F1: false bike triggers were noisy relative to the recall they bought. |
Method
Packaged so another engineer can reproduce the claim, not just admire it.
Normalize. Pull a BDD100K road-scene subset and convert labels into a small VRU taxonomy (person, rider, bike) with lighting, weather, scene, and size metadata attached.
Measure the baseline. Score a COCO-pretrained YOLO26n with metrics stratified by condition, never just mean AP.
Fine-tune. Train on a VRU-preferring local split with oversampling of the rare classes.
Re-measure identically. Same split, same strata, so every delta is attributable.
Choose operating points. Sweep thresholds per class and pick against an explicit miss-vs-false-positive cost story, documented in the repo.
What this is not
Not a deployment claim
A CPU-trained nano model on a subsample. The claim is methodological: expose condition-specific risk and choose thresholds deliberately.
Not solved
Night-person recall 0.262 and small-object recall 0.079 are published as open product risk, not buried in an appendix.
Not license-blind
No BDD100K pixels or restricted weights are committed anywhere in the repo or its history. Test fixtures are synthetic.
Not SOTA theater
No leaderboard claims from a subsample. The model card states intended use and eight limitations.