u/fapa64

r/computervision r/learnmachinelearning r/cpp r/CUDA r/pytorch

RF-DETR C++ is a production-grade TensorRT inference engine for RF-DETR

Built a C++ inference library for RF-DETR, Roboflow's transformer-based object detection model. The library runs the full pipeline in C++: a fused CUDA kernel for preprocessing, async H2D transfers via pinned memory, and CUDA Graph capture for low-overhead inference dispatch.

This release covers both object detection and instance segmentation. Segmentation masks are decoded on the GPU through a custom kernel that upsamples and thresholds all detections in parallel.

github.com

u/fapa64 — 4 hours ago

▲ 31 r/CUDA+3 crossposts

RF-DETR CPP: TensorRT inference library for RF-DETR with GPU mask decode, CUDA Graph, and zero Python at runtime

Built a C++ inference library for RF-DETR, Roboflow's transformer-based object detection model. The motivation was simple, every RF-DETR deployment I came across was running Python and PyTorch at inference time, which is fine for experimentation but not great when you need consistent low latency in production.

The library runs the full pipeline in C++: a fused CUDA kernel for preprocessing, async H2D transfers via pinned memory, and CUDA Graph capture for low-overhead inference dispatch. On an RTX 5070 Ti at FP16 it hits around 2ms per frame for detection and 6ms for instance segmentation.

This release covers both object detection and instance segmentation. Segmentation masks are decoded on the GPU through a custom kernel that upsamples and thresholds all detections in parallel. 

Happy to get any feedback or ideas.

u/fapa64 — 4 hours ago