In microrobotics, visual servoing is often the only way to track the position of a robot or manipulator. To allow for automated high-speed micromanipulation, reliable high-speed visual servoing systems are needed. This paper presents a smart camera implementation that is capable of tracking multiple microrobots with an accuracy below 2 µm inside a working range of 120X80mm² using a state of the art CMOS vision sensor. The implementation of the tracking algorithm was partly carried out in hardware to allow for predictable signal latency and high update rate. The system is capable of tracking a single robot with an update rate of over 200 Hz.