High-Speed Object Detection
& Inference Optimization

Overview

Group project at Télécom Paris (DSAI, 6 members) that benchmarks 24 object detectors from five architectural families — two-stage CNNs, YOLO, DETR, RT-DETR, and dense one-stage models — on a single NVIDIA Tesla T4 GPU at batch size 1 in FP32. Rather than producing another accuracy/speed ranking, the goal was to explain why one detector is slower than another by attributing end-to-end latency to individual operators (convolution, normalization, activation, attention, pooling, post-processing).

The core hypothesis: at batch size 1, latency depends less on raw arithmetic (FLOPs) than on which operators run, whether they are fused, how many kernels are launched, and how much data moves around. Every model was profiled under a strict shared protocol — 640×640 COCO val2017 inputs, 50 warm-up runs, 1000 timed passes with CUDA events, median latency, and pycocotools mAP at 50:95 — then pushed through the same optimization stack: AMP FP16, Conv-BN fusion, torch.compile, and TensorRT.

My Contribution

I was responsible for the DETR family (DETR R50, Conditional DETR, DAB-DETR and Deformable DETR), and I also wrote shared tooling used by the whole team: the latency harness, the COCO loader and the evaluation path. I proposed the common experimental protocol (FP32, 640×640, batch size 1) that every member adopted so that per-family results would be comparable.

• Operator-level analysis of the DETR backbone: identified that 28–32% of backbone time is lost to a FrozenBatchNorm block that is never folded into the preceding convolutions, because a custom module makes it invisible to PyTorch’s standard fusion pass. Confirmed the finding with both torch.profiler traces and an NVIDIA Nsight Systems pass.
• Optimizations: AMP FP16, BatchNorm folding, and torch.compile. My DETR models reached speed-ups of 2.34× (DETR R50), 2.31× (Conditional DETR), 2.13× (DAB-DETR) and 1.44× (Deformable DETR) with mAP preserved.
• TensorRT: implemented TensorRT engines for all of my models, plus additional optimizations not included in the report because they could not be compared against models from other team members within the deadline.
• Team coordination: organized the weekly meetings and kept the workload balanced across members — an organizational contribution I consider as important as the code itself.

Key Findings

• Convolutions dominate backbones across all families, taking 56–90% of backbone compute time — which makes them the easiest target for Tensor Core acceleration.
• The DETR family is backbone-bound and loses 28–32% of backbone time to an unfused FrozenBatchNorm; Deformable DETR is the slowest model at 263 ms because its deformable attention falls back to a Python loop.
• TensorRT FP16 gave the largest gains (e.g. 5.57× on RT-DETR R101) by fusing convolution, GEMM, residual and activation work into fewer kernels — beyond what precision change alone achieves.
• Two-stage models split roughly half backbone / half head, with the head cost driven by RoIAlign and NMS rather than feature extraction.

Takeaways

The project taught me that agreeing on a shared measurement protocol from day one is the single most important decision in a benchmarking study — we lost one to two weeks converging on a common harness. It also showed that the real bottlenecks in GPU inference are memory movement and kernel launch overhead, not FLOPs, and that every optimization must be validated at the operator level to know where the speed-up actually comes from.