Model-independent vision-based indirect manipulation of unknown deformable objects using the da Vinci surgical robotic system




Nalamwar, Kunal

Journal Title

Journal ISSN

Volume Title



Despite a lot of progress in rigid body manipulation in the field of robotics, autonomous deformable object manipulation has been for long an unexplored and untouched area of research. The deformable object manipulation problem has recently come into the spotlight because of the potential economically important applications in the food, medical, home as well the manufacturing industry. Most of the research in this direction assumed homogeneous physical properties and ignored the existence of any disturbances or considered some a priori knowledge of the manipulation environment. Also, the parametric identification of the deformation model is difficult to perform beforehand since it requires several physical interactions testing and might not completely capture viscoelastic and time variant nature of many deformable objects. Hence, we present here a new optimization-based feedback method to automatically servo-control the shape of soft objects with robotic manipulators. This vision-based feedback helps in online estimation of the deformable object and does not require any prior knowledge about the object. To evaluate the performance of the proposed method, da Vinci Research Kit was used and the experiment used a rectangular silicone slab as an unknown deformable object. Setting up the robot, development of scripts along with software upgrades for the experiment and teleoperation were performed. Experiments for homogeneous 2-D manipulation demonstrated successful real-time estimation and performance of the algorithm while manipulating the silicone to desired points. Chapter 2 talks about the utilized robotic hardware that is the da Vinci Research Kit (dVRK). The now retired first generation da Vinci robot along with its arms has been described. This chapter also talks about the controller stack, surgeon console as well as the vision stack. Different tools which have been used are also described. Chapter 3 talks about the software stack, which was developed for autonomous manipulation. It includes description about writing JSON scripts for making the robot work. We also talk about the changes made in calibration files for compatibility with dVRK 2.0. Teleoperation capabilities were also set up. Explanation of setting up the video feed from endoscope has been explained. Vision based algorithms for object tracking and corner detection have also been mentioned. We have also briefly explained the robot control algorithm used. Chapter 4 talks about the experiments performed and their results. Successful teleoperation has been described along with the task which was completed. Results also have been presented for successful one arm and two arm autonomous manipulations. Chapter 5 talks about the Conclusion and describes the Future Work about using motion capture system which can act as a replacement to the existing endoscope video feed.


LCSH Subject Headings