Endoscopy and Computer Vision

Project FLEXABLE is a Seventh Framework Programme funded by the European Research Council (ERC). The project is conducted by the EnCoV group.

Contact: flexable@udamail.fr

Description

Project FLEXABLE lies in the scientific field of Computer Vision which studies the computational interpretation of images. In particular, 3D Computer Vision seeks to recover depth or the 3D shape of the observed environment from images. One of the most successful and mature techniques in 3D Computer Vision is Shape-from-Motion which is based on the well-established theory of Multiple-View Geometry. This uses multiple images and assumes that the observed environment is rigid.

The world is however made of objects which move and undergo deformations rather than remaining rigid. Researchers have tried to extend Shape-from-Motion to a deformable environment for about a decade, yet with only very limited success to date. We believe that there are two main reasons for this. Firstly there is still a lack of a solid underlying theory for Deformable Shape-from-Motion. Fundamental questions, such as what kinds of deformation can facilitate unambiguous 3D reconstruction, are not yet answered. Secondly practical solutions have not yet come about because the Motion cue has been mostly used on its own. For accurate and dense 3D shape results, it must be combined with other visual cues, since it is certainly weaker in the deformable than in the rigid case. Furthermore, strong priors are needed, requiring one to bridge the gap with object recognition.

This project focuses on developing these two key areas and in doing so will include three main lines of research: theory, its computational implementation, and its real-world application. Deformable Multiple-View Geometry will generalize the existing rigid theory to the deformable case and will provide researchers with a rigorous mathematical framework that underpins the use of Motion as a proper visual cue for Deformable 3D Reconstruction. Our theory will require us to introduce new mathematical tools from differentiable projective manifolds to Computer Vision. Our implementation will study and develop new computational means for solving the difficult inference problems formulated in our theory. Because we strongly believe that Motion is not sufficient for solving the general Deformable 3D Reconstruction problem alone, it will be combined with other visual cues including Shading. Finally, we will develop cutting-edge applications of our framework specific to Minimally Invasive Surgery, with a focus in interventional laparoscopy, for which there is a very high need for 3D Computer Vision.

Members

• Principal Investigator: Pr Adrien Bartoli
• Project Manager: Natalia Alves (since October 2014)

Collaborators

• Pr Michel Canis - MD University Hospital (CHU) and Université d’Auvergne
• Dr Nicolas Bourdel – MD University Hospital (CHU) and Université d’Auvergne
• Toby Collins – Researcher at IRCAD
• Dr Daniel Pizarro – Colleague and associate member since September 1, 2014

Alumni

• Project Manager: Julia Allirot (January 2013 – September 2014)

Postdocs

• Shaifali Parashar (October 2017 - September 2018) – Research topic: NRSfM with projective manifolds
• Dr Katrin Skerl (July 2016 – July 2017) – Research topic: On the Well-posedness of SfT in the Isometric and Non-Isometric Cases, with Common Types of Inputs
• Dr Darshan Venkatrayappa (April 2016 – April 2017) – Research topic: 3D Reconstruction of Deformable Objects from Monocular Images
• Dr Rindra Rantoson (April 2015 – April 2017) – Research topic: Shape-from-Template Meets Object Recognition
• Dr Rahat Khan (November 2013 – October 2015) – Research topic: 3D Reconstruction for Deformable Surfaces from Mutiple Visual Cues with Application to Laparoscopy

PhD students

• Mathias Gallardo (September 2014 - August 2017) – Research topic: 3D Monocular Reconstruction in an Uncontrolled Environment and Medical Applications
• Shaifali Parashar (September 2014 – August 2017) – Research topic: 3D Modelling of deformable materials
• Ajad Chhatkuli (September 2013 – August 2016) – Research topic: The Multiple-View Geometry of Deformable Surfaces

Interns

• Armine Vardazaryan (January – August 2017) Research topic: SfT-from-CNN – Using Deep Learning to Aid Deformable 3D Reconstruction
• Yogesh Langhe (January – June 2017) Research topic: Does elastography imaging support 3D reconstruction in laparoscopy?
• Ronan Bureau (March – September 2016) Research topic: Creating digital and physical models to validate deformable 3D reconstruction and aid endoscopic training
• Shrinivasan Sankar (January – June 2016) Research topic: Does detection and recognition help seeing deformable objects in 3D?
• Tom François (February – July 2015) Research topic: 3D reconstruction of multiple deformable objects by memory-based Shape-from-Template
• Héla Hadhri (February – July 2015) Research topic: Development of a real-time prototype for 3D reconstruction by Shape-from-Template, and its application in interventional laparoscopy
• Jules Gorny (March – August 2014)
• Kristina Prokopetc (February – June 2014) Research topic: Building a Fallopian Tube Detector for Computer Assisted Intervention
• Shaifali Parashar (February – June, 2014) Research topic: Modeling Multiple Images of a Deformable Surface Using Differential Projective Geometry
• Pierre Bruyer (April – August, 2013) Master thesis title: Motion based segmentation and 3D reconstruction of the uterus

Collaborators

• Dr Abed Malti - Engineer

Publications by year