Abstract

Fundamental to any state estimation problem is the concept of estimation error. In both autonomous robotics and tracking research, the ability to assess the performance of robotic mapping and target tracking algorithms is of crucial importance. This article focusses on metrics for the automatic evaluation of target tracking and feature map estimation algorithms, in the presence of both detection and spatial uncertainty. In such realistic cases, many metrics fail to provide a meaningful and intuitive assessment of robotic map estimates. Recently the Optimal Sub-pattern Assignment (OSPA) metric provided a solution, as it was shown to provide more meaningful assessments of target tracking algorithm performance than its predecessors. This article will demonstrate that the OSPA metric still suffers various disadvantages under realistic mapping scenarios. These include its saturation to a limiting value, irrespective of the cardinality error of different estimators, and its inability to distinguish between repetitions of balanced estimates, in which single ground truth features are estimated with multiple false alarms. The Cardinalized Optimal Linear Assignment (COLA) metric is therefore introduced as a complement to the OSPA metric, and their combination is analysed in order to gauge target tracking and map estimation errors in an intuitive and meaningful manner.