PiVoT

Detecting and tracking many objects from noisy point clouds remains challenging in data-scarce radar applications. Current Bayesian trackers based on Poisson measurement models offer a training-free solution, but struggle to achieve accuracy and efficiency under real-world challenges such as severe clutter, large object populations, and full-resolution Doppler point clouds.

We address these challenges with PiVoT (Poisson Measurements-based Variational Multi-object Detection and Tracking). PiVoT is a fast, training-free, and clutter-resilient method for large-scale multi-object detection and tracking, applicable to both positional and Doppler point-cloud measurements. It processes raw point clouds directly, without external detection or clustering, and handles an unknown and time-varying number of objects.

Method in Brief

PiVoT is built around a two-stage variational inference framework, with its workflow shown in Figure 1(e), tailored to Poisson measurement models to address several inherent inference challenges. Its efficiency and strong empirical performance are further enabled by several methodological innovations, including: 1) theoretically justified early removal of ineffective birth objects that would otherwise converge to having no associated measurements; 2) an exact global optimiser for object existence and detectability probabilities that reduces the standard quadratic-time update to linear time; and 3) a Doppler-augmented NHPP model that, to our knowledge, is the first to incorporate Doppler information while preserving a linear-Gaussian per-object likelihood. This model also provides a simple, reusable building block for efficient Doppler processing that can be incorporated into other trackers.

A detailed methodological exposition is provided in our paper, and a short introduction to Poisson measurement models is available here.

Demonstrated Capabilities

PiVoT substantially outperforms existing Poisson-measurement-based trackers in challenging settings, as shown by the quantitative results reported in the paper. More importantly, PiVoT achieves the following capabilities, illustrated in the videos below, which, to our knowledge, have not been demonstrated by existing Bayesian trackers:

1. Robustness in dense clutter, even when objects are difficult to distinguish by human inspection.

2. Detecting and tracking a thousand objects in under a second on a standard laptop without gating.

3. To our knowledge, PiVoT is the first fully training-free (i.e., no supervised learning is used in either detection or tracking) detector and tracker to run in real time on full-scale modern automotive radar datasets, with performance comparable to a deep-learning detection benchmark. At the same time, PiVoT additionally provides estimates of tracks, shapes, data association, detectability and existence probabilities.

Demonstrations of PiVoT

The following four groups of demonstrations highlight PiVoT’s capabilities: two on real radar data — automotive and maritime — and two on challenging simulated scenarios — dense clutter and large-scale tracking of a thousand objects.

Automotive Radar Applications

We apply PiVoT to real-world automotive radar data from the RadarScenes dataset (Schumann et al., 2021); full implementation details are provided in our paper. We note that PiVoT is applied directly to raw radar point clouds without any external detection or clustering step. It also satisfies the average real-time processing requirement, achieving an average processing time of less than 0.25 s per frame across all evaluated sequences (system: 8-core Intel Core i9, 32 GB RAM).

Below, we present demonstrations of PiVoT on three selected RadarScenes clips.

PiVoT on real driving scenes

The camera image is shown only as a visual reference. Each radar point is annotated with an arrow visualising its observed Doppler velocity after ego-motion compensation. Coloured points denote ground-truth radar returns from moving objects provided by the dataset, while coloured curves and ellipses denote PiVoT’s estimated tracks and shapes, with colour indicating object identity. Videos are played at 2x real time.

These scenes are challenging because complex real-world environments generate cluttered measurements from diverse road users, stationary infrastructure, and other sources. Even for a human observer, distinguishing genuinely moving objects from the background is often difficult. PiVoT jointly tracks moving and stationary objects while reporting only those confidently identified as moving. Together with its efficient inference mechanisms, this allows PiVoT to achieve reliable detection and tracking across diverse and cluttered driving environments.

Comparison with RadarGNN

We present frame-by-frame video comparisons of PiVoT and RadarGNN, a deep-learning-based detection method (Fent et al., 2023), on three complete radar sequences played at 2× real time. For implementation details and further comparisons, see our paper. Coloured points indicate ground-truth radar returns from moving objects, while grey points indicate clutter or returns from stationary objects. Coloured curves and ellipses show PiVoT’s estimated tracks and shapes, whereas black bounding boxes show RadarGNN detections. PiVoT also displays ghost-target estimates using cyan curves and ellipses.

In the “moderately busy, sunny, moving ego, vehicles” demo, PiVoT consistently suppresses ghost targets caused by multipath reflections, which RadarGNN sometimes falsely detects as moving objects. This highlights the benefit of retaining stationary structures as evidence for ghost-target rejection, as detailed in our paper.

PiVoT also directly produces multi-object tracks, as shown in the video, along with estimates such as object kinematics and existence probabilities. By contrast, the outputs of a deep-learning-based detector require additional post-processing to form tracks.

Maritime Radar Applications

This demo presents PiVoT applied to real radar data of ships navigating near an anchorage area along a river. Dozens of vessels of varying sizes enter and exit the scene, alongside static structures such as buoys, anchored ships, and a bridge spanning the river on the left. Blue points denote radar measurements, which appear densely packed — approximately 4000 measurements per time step. Red lines and ellipses show PiVoT’s estimated trajectories and shapes for all objects, including stationary structures such as the bridge.

PiVoT takes 0.171 seconds on average per time step on an Apple M4 Pro system (24GB RAM), comfortably meeting real-time requirements despite this heavy measurement load. Interestingly, in the absence of a clutter map, the bridge is interpreted as two connected, elongated elliptical objects — such spurious detections can easily be filtered out during post-processing by examining velocity or shape, as was done in the automotive radar examples.

Simulated Heavy Clutter Scenes

In this demo, we showcase PiVoT’s capability of joint detection, tracking, and estimation of shape and measurement rates under dense clutter (Poisson rate 1575). Here, the number of objects increases from 42 to 99 and then decreases back to 44, with objects appearing and disappearing throughout the surveillance region. Two object types are present: smaller circular objects with a Poisson rate of 5 and larger circular ones with a rate of 8.

The measurements over 50 time steps are shown below. The primary challenge in this setting is the dense clutter—true objects are difficult to distinguish, even to the human eye.

We now apply PiVoT to this task for detection, tracking and estimation of object shapes and rates. Below are the results.

Each ground-truth trajectory and large or small circle is closely matched by a red or green track and ellipse, demonstrating accurate detection, tracking, and shape estimation. The average processing time per time step is just 0.28 seconds, highlighting PiVoT’s efficiency.

Simulated Thousand-Object Tracking Scenes

In this demo, we showcase PiVoT’s real-time detection and tracking performance in large-scale tracking. For simplicity, we assume relatively low clutter (Poisson rate 15.75), and known objects shape and rates. Here, the object number starts from 100, then with four sudden increases to a number up to 1036 objects, and finally decreases to a number around 450. All objects are with Poisson rate 5.

The measurements over 50 time steps are shown below. Key challenges in this scenario include efficiently handling massive data, promptly detecting numerous object births and deaths, and resolving frequent track coalescence.

Each ground-truth trajectory is closely matched by a red or green track, demonstrating ability to track a thousand objects simultaneously without excessive track loss and timely detection of object births and deaths. The average processing time per time step is only 0.67 seconds on an Apple M4 Pro system (24GB RAM). This is particularly impressive considering that no gating is used, with up to 1036 objects and more than 5,000 measurements.

Further speed improvements could be expected through optimised implementation using CPU/GPU parallelisation and gating for a large surveillance area, making PiVoT a highly promising solution for real-time multi-object tracking in large-scale, cluttered environments.

Outlook

PiVoT demonstrates that model-based detection and tracking can remain accurate, robust, and computationally practical in challenging real-world settings. There are many promising directions for extending PiVoT, both methodologically and in its applications. We hope PiVoT will motivate a broader class of real-time, practically deployable, and probabilistically interpretable model-based methods for modern radar and point-cloud-based perception systems, which are particularly valuable in many surveillance applications where large labelled datasets are difficult to obtain.