Markerless 3D model-based tracker module overview

In ViSP we propose a 3D model-based tracker that allows simultaneously the tracking of a markerless object using the knowledge of its CAD model while providing its 3D localization (i.e., the object pose expressed in the camera frame) when a calibrated camera is used. Considered objects should be modeled by lines, circles or cylinders.

The tracking method is divided in two steps. The first one consists in tracking the features of an object. They are in the image the moving-edges along the normal to the projection of the CAD model, or some keypoints on visible object faces. Then, the second step consists in determining the pose of the camera in order to match the tracked features with the projection of the 3D model. To compute the pose, a robust virtual visual servoing method is used.

In ViSP, depending on the visual features that are used three trackers are available:

• a tracker that consider moving-edges behind the visible lines of the model. This tracker is appropriate to track untextured objects.
• an other tracker that consider KLT keypoints that are detected and tracked on each visible face of the model. This tracker is more designed to track textured objects with edges that are not really visible.
• an hybrid version that is able to consider moving-edges and KLT keypoints. This tracker is appropriate to track textured objects with visible edges.

The source code documentation shows which are the corresponding classes part of model-based tracker module.

This module uses some optional third-party libraries to enable some capabilities.

To be used, the tracker requires two files : a 3D CAD model and an XML configuration file. If you initialize the tracking thanks to mouse clicks, you will need another initialisation file.

The following video illustrates the capabilities of the model-based tracker on a 3D object where the CAD model corresponds to object edges modeled by a set of segments.

Object tracking using the 3D model-based tracker. [source code] A tutorial is available here.	3D CAD model and configuration file

Below, other videos show some use cases that illustrate the tracking of a planar object, as well as more complex 3D objects such a tabasco box, a castel or a satellite.

This video illustrates the markerless model-based tracking of a postcard, modeled as a 2D planar object with two simple rectangles.  The post card frame is displayed in the image in order to check if the computed pose is correct. [source code] [data] The result could be typically used for augmented reality.	This videos presents an augmented reality application using markerless model-based tracker. A rough 3D model of the castle (in red) is used to compute the pose. Green dots represent the moving edges features used by the tracker.

This video shows how use of position based visual servoing and visual tracking approaches can be mixed to grasp a box tracked by the markerless hybrid model-based tracker and deliver it to a human.	This video simulates a satellite target tracking moving on a geosynchronous orbit using markerless hybrid model-based tracker. The tracker is able to handle visible faces changes and large displacement in successive images. The use case is here a remove debris application.