The book starts with some basic motivations for minimally invasive surgery and states the various distinctions between robotic and non-robotic (conventional) versions of this procedure. Common components of this type of operation are presented with a review of the literature addressing the automation aspects of such a setup. Examples of tracking results are shown for both motion and gesture recognition of surgical tools, which can be used as part of the surgeon–computer interface. In the case of marker-less tracking, where no special visual markers can be added to the surgical tools, the tracking results are divided into two types of methodology, depending on the nature and the estimate of the visual noise. Details of the tracking methods are presented using standard Kalman filters and particle filters. The last part of the book provides approaches for tracking a region on the surgical scene defined by the surgeon. Examples of how these tracking approaches can be used as part of image-guided navigation are demonstrated. This book is designed for control engineers interested in visual tracking, computer vision researchers and system designers involved with surgical automation, as well as surgeons, biomedical engineers, and robotic researchers.