Video Tracking Without Clicking

The Progressive Point Tracker

Posted by Falk Mielke on 1st Jan 2020

Video Tracking Tool

Since my undergrad projects, I required a tool for tracking points on videos. I have been tracking videos of falling objects, swimming brine shrimp, rat whiskers during social touch, squirrels, ungulate hooves and joints... the number of frames I clicked is immense (which is a burden to efficiency, see below).

The unabated need to track videos in my PhD project has led me to overhaul my tracking tool. I thought it might be useful for others, so here you go.

The Cost of Clicking

I originally started with commonly used tools, but found myself being inefficient due to the need for clicking on each frame of the video. Although clicking seems to be a tiny investment, it accumulates and gets annoying, especially when the video frame rate is sufficient for scientific analysis. Automated tracking algorithms exist, they certainly can facilitate this sort of task. However, for many of my applications, they failed - I guess because I was rarely blessed with markers that provide enough contrast for these algorithms to work reliably, or because my accuracy requirements were too high. So shortly after I got into learning Python, I started programming a tool myself.

Progressive Tracking

The unique feature of my tool: it allows the user to simply follow a point on the video with the mouse cursor. Playback speed can be adjusted (by mouse wheel and keyboard shortcut). Playing backwards is possible (press "control" key while playing). The cursor position is recorded frame by frame. No clicking!

This is what it looks like:

As you see, the interface is minimalistic. I tried to reduce dependencies to a minimum, for example by using matplotlib instead of more involved image libraries (such as opencv or scikit-image; these are very useful, for example for contrast enhancement, and can be included easily when needed).

• The default matplotlib/TkAgg navigation toolbar stays usable and is extended by a menu and status indicator.
• The video frame is central and has maximal size.
• At the bottom, there is a time axis (clickable) that indicates tracking progress.
• Many features are accessible via keyboard shortcuts (see help overlay and comments in the python file).
• The image sequence is loaded by a background process and stored in memory, enabling quick playback.

So in summary, this is a lightweight, efficient video tracking tool that has saved me lots of hours.