ObjectModel-v1: Betting on a Fixed Latent Memory

Everything Bench Labs has shipped so far generates something: pixels, voxels, tokenizer output. ObjectModel-v1 is the first thing that looks at an image and tells you what's in it. It's a from-scratch, NMS-free object detector, 40.8M parameters, trained end to end on COCO2017. No pretraining, no hyperparameter search across seeds, none of the planned ablations run yet.

Peak validation AP is 0.358, at epoch 95 of 100. That's below the 40-50 range we set as a competitiveness bar going in, and we're saying so up front rather than burying it. This post is the honest version of that number: what the model is testing, what it actually does on real footage, and what the gap is telling us to try next.

The bet

Most transformer detectors do global reasoning by attending over the full feature map, every pixel token, every layer. ObjectModel-v1 compresses that down first: a small convolutional backbone builds a multi-scale pyramid, that pyramid gets pooled into a handful of tokens, and those tokens are packed into a fixed latent memory. Object queries cross-attend to that memory for global context.

The obvious risk is that compressing global context loses the geometric precision a detector needs for tight boxes. ObjectModel-v1's answer is to get precision back a different way: each query also runs a local sampler that reads directly from the full-resolution pyramid, centered on and scaled to the query's current box estimate. Early decoder layers can sample broadly since the box guess is still rough; later layers narrow in as the box refines.

image
  -> compact convolutional backbone (strides 8/16/32)
  -> top-down pyramid fusion
  -> pooled multi-scale tokens
  -> fixed latent memory (global semantics)
  -> learned object queries
       -> query self-attention
       -> cross-attention to latent memory
       -> local sampling around the current query box
       -> iterative class and box prediction
  -> object set (no anchors, no NMS)

The model is NMS-free. It predicts a fixed set of objects directly and is trained with Hungarian bipartite matching against the ground truth, the same set-prediction idea DETR introduced. There's also an optional dense auxiliary head active only during training, adding one-to-many spatial supervision on top of the one-to-one matched loss. It's discarded at inference and, like everything else here, needs to be run as an ablation rather than assumed to help.

Numbers

Train loss over the same run, log-shaped like most detectors trained from scratch: a sharp drop in the first few epochs, then a long gradual decline from around 6 down to 4.1:

AP50 (a loose IoU 0.50 localization threshold) against AP75 (strict, IoU 0.75). Both climb together early, then AP75 plateaus lower. Coarse localization got easy faster than precise localization did:

And AP broken out by object size. Large objects were detected reliably well before small ones caught up, and small ones never really did:

Honesty section

0.358 AP is a real number from a real, single-seed, un-pretrained run, not a benchmark claim. It's below the 40-50 range that would put it in the neighborhood of 20-40M-parameter real-time detectors, which was the bar we set for "competitive" before training started. We're not rounding that up.

The size-broken-out AP is the more useful signal than the headline number: 0.188 on small objects against 0.493 on large ones is a wider gap than the usual small-vs-large spread for this class of detector. That points somewhere specific, the fixed latent memory or the local sampler's pyramid resolution, rather than just "needs more epochs." More on that below.

Getting the run to actually finish in reasonable time took real debugging, separate from the architecture question. Two things cost the most: per-step GPU-to-CPU syncs (calling .item() on loss tensors every step) were quietly stalling torch.compile's CUDA-graph replay, and the loss computation had several nested Python loops over images and decoder layers building cost matrices and dense targets one at a time. Vectorizing those, batched Hungarian cost-matrix construction, batched dense-target assignment, batched set-prediction loss targets, took throughput from around 48 img/s to the ~100 img/s the numbers above report, on the same 5090, mid-run, without changing what the loss actually computes.

Examples

Detections from the epoch-95 checkpoint on COCO val2017 images, confidence ≥ 0.35. Picked for variety, not cherry-picked for perfection:

Street market: person at 0.94, market umbrella correctly boxed.

A crowd at a rodeo, most people correctly boxed, one animal still mislabeled, left in rather than cropped out.

A kite at 0.80 confidence, people along the shore, and a distant boat, three very different scales in one frame.

From detector to tracker

ObjectModel-v1 has no temporal component of its own, every frame is detected independently. To show what that looks like on video, we built a small from-scratch SORT-style tracker on top of it (src/objectmodel_v1/tracking.py): a constant-velocity Kalman motion model plus IoU/Hungarian frame-to-frame association, giving each box a persistent id and a short motion trail.

Full-length video (20s, MP4) →

Source footage is vtest.avi, OpenCV's standard pedestrian test clip (BSD-3, ships with OpenCV), genuine video the model never trained on. We ran the same clip through the same unchanged tracker at three points in training, so the only thing that changed between rows is the detector's checkpoint:

Id count issued doesn't fall much across checkpoints, new people keep entering frame throughout the clip and each earns a new id, which is correct behavior. What actually improved is persistence: ids for people already in frame survive longer as the detector gets better. A separate test on a genuinely different scene, an eye-level warehouse clip not shown here, surfaced a distinct limitation instead: the detector still occasionally hallucinates objects on plain background surfaces, reading a support pillar as "refrigerator", even at this peak checkpoint. Tracking quality rides directly on detection quality, and detection quality on footage that doesn't look like COCO's own photography, different camera angles, lighting, compression, is visibly weaker than on COCO's own validation images.

Restricted zone detection

One layer further: a polygon zone monitor (src/objectmodel_v1/zones.py) that fires an event only on the transition into or out of a region, not on every frame a track spends inside it. It keys off each box's bottom-center point, roughly where feet touch the ground, rather than the box centroid, since a zone drawn on a floor plane should care where someone is standing, not where their torso is.

Full-length video (20s, MP4) →

We checked this on a synthetic sequence before trusting it on video: a track walking through a rectangle produces exactly one entered and one exited event, none of the frames spent inside re-fire, a track that never enters never fires, and a fresh id gets independent state. That test lives with the module.

On the plaza clip above, the zone fired 91 enter/exit events over 20 seconds. Read that carefully: most of it is not 91 different real crossings. It's the same tracker id churn from the section above, the same person's track resetting and re-entering the zone as a "new" id, plus a handful of misclassified objects (a backpack or handbag read as its own tracked object) that shouldn't have triggered an event at all. The event mechanics are verified correct. The real-world event count is exactly as reliable as the detector and tracker underneath it, which right now is usable for a demo, not for anything where a false alert has a real cost.

from objectmodel_v1.zones import RestrictedZoneMonitor

zone = [(300, 150), (650, 150), (650, 320), (300, 320)]  # pixel-space polygon
monitor = RestrictedZoneMonitor(zone)

for frame_index, frame in enumerate(video_frames):
    boxes, labels, scores = detect(frame)
    tracks = tracker.update(boxes, labels, scores)
    for event in monitor.update(tracks, frame_index):
        print(event.track_id, event.kind, event.position)  # "entered" or "exited"

Neither of these two layers retrains the detector. Persistent ids from the tracker are enough on their own to build most counting and monitoring logic on top: unique counting, movement heatmaps from the same trail data the demos already compute, dwell-time timers, line-crossing counters, class-based counting across any of the 80 COCO categories. Behavior/anomaly detection and multi-camera re-identification are real extensions but not close ones, they need motion modeling beyond a Kalman filter and cross-camera identity matching, neither of which this repository does yet.

What v2 needs

Three things came out of this run as specific findings, not a generic "train it bigger":

• The small-vs-large AP gap (0.188 vs 0.493) is wider than usual for this class of detector, and points at the fixed latent memory or the local sampler's pyramid resolution as the first places to look. The latent-layer-count and latent-count ablations in the repo's validation protocol exist specifically to test that.
• Detection quality drops visibly on footage that doesn't look like COCO's own photography, the warehouse clip above is the clearest example, which caps every extension built on top of it. Broader training data or augmentation is the direct lever, and it's not currently in the required ablation list, it should be.
• v1 trained from random initialization only. A pretrained comparison is already called for in the validation protocol, and based on how much pretraining typically moves the needle at this parameter count in the baselines we're citing, it's the single change most likely to move AP the most.

Try it

git clone https://huggingface.co/bench-labs/objectmodel-v1
cd objectmodel-v1
python3.11 -m venv .venv
.venv/bin/pip install -e '.[coco,export,dev]'

objectmodel-eval \
  --config configs/objectmodel_v1.yaml \
  --checkpoint objectmodel_v1_best.pt \
  --data-root /path/to/coco

Full architecture notes, the required-ablations list, and the research this builds on (DETR, Conditional/Deformable DETR, RT-DETR, D-FINE, DEIM, LW-DETR) are in the ObjectModel-v1 repository.

First detector out of Bench Labs. It works well on the data it was trained on, worse off that data, and the README says exactly where. That's what v1 is for.