Endoscopy and Computer Vision

Back to index

Project FLEXABLE (ERC Starting Consolidator Grant, 2013-2018)

Project FLEXABLE is a Seventh Framework Programme funded by the European Research Council (ERC). The project is conducted by the EnCoV group.

Contact: flexable@uca.fr

Description

Project FLEXABLE lies in the scientific field of Computer Vision which studies the computational interpretation of images. In particular, 3D Computer Vision seeks to recover depth or the 3D shape of the observed environment from images. One of the most successful and mature techniques in 3D Computer Vision is Shape-from-Motion which is based on the well-established theory of Multiple-View Geometry. This uses multiple images and assumes that the observed environment is rigid.

The world is however made of objects which move and undergo deformations rather than remaining rigid. Researchers have tried to extend Shape-from-Motion to a deformable environment for about a decade, yet with only very limited success to date. We believe that there are two main reasons for this. Firstly there is still a lack of a solid underlying theory for Deformable Shape-from-Motion. Fundamental questions, such as what kinds of deformation can facilitate unambiguous 3D reconstruction, are not yet answered. Secondly practical solutions have not yet come about because the Motion cue has been mostly used on its own. For accurate and dense 3D shape results, it must be combined with other visual cues, since it is certainly weaker in the deformable than in the rigid case. Furthermore, strong priors are needed, requiring one to bridge the gap with object recognition.

This project focuses on developing these two key areas and in doing so will include three main lines of research: theory, its computational implementation, and its real-world application. Deformable Multiple-View Geometry will generalize the existing rigid theory to the deformable case and will provide researchers with a rigorous mathematical framework that underpins the use of Motion as a proper visual cue for Deformable 3D Reconstruction. Our theory will require us to introduce new mathematical tools from differentiable projective manifolds to Computer Vision. Our implementation will study and develop new computational means for solving the difficult inference problems formulated in our theory. Because we strongly believe that Motion is not sufficient for solving the general Deformable 3D Reconstruction problem alone, it will be combined with other visual cues including Shading. Finally, we will develop cutting-edge applications of our framework specific to Minimally Invasive Surgery, with a focus in interventional laparoscopy, for which there is a very high need for 3D Computer Vision.

Members

• Principal Investigator: Pr Adrien Bartoli
• Project Manager: Natalia Alves (since October 2014)

Collaborators

• Pr Michel Canis - MD University Hospital (CHU) and Université d’Auvergne
• Dr Nicolas Bourdel – MD University Hospital (CHU) and Université d’Auvergne
• Toby Collins – Researcher at IRCAD
• Dr Daniel Pizarro – Colleague and associate member since September 1, 2014

Alumni

• Project Manager: Julia Allirot (January 2013 – September 2014)

Postdocs

• Shaifali Parashar (October 2017 - September 2018) – Research topic: NRSfM with projective manifolds
• Dr Katrin Skerl (July 2016 – July 2017) – Research topic: On the Well-posedness of SfT in the Isometric and Non-Isometric Cases, with Common Types of Inputs
• Dr Darshan Venkatrayappa (April 2016 – April 2017) – Research topic: 3D Reconstruction of Deformable Objects from Monocular Images
• Dr Rindra Rantoson (April 2015 – April 2017) – Research topic: Shape-from-Template Meets Object Recognition
• Dr Rahat Khan (November 2013 – October 2015) – Research topic: 3D Reconstruction for Deformable Surfaces from Mutiple Visual Cues with Application to Laparoscopy

PhD students

• Mathias Gallardo (September 2014 - August 2017) – Research topic: 3D Monocular Reconstruction in an Uncontrolled Environment and Medical Applications
• Shaifali Parashar (September 2014 – August 2017) – Research topic: 3D Modelling of deformable materials
• Ajad Chhatkuli (September 2013 – August 2016) – Research topic: The Multiple-View Geometry of Deformable Surfaces

Interns

• Armine Vardazaryan (January – August 2017) Research topic: SfT-from-CNN – Using Deep Learning to Aid Deformable 3D Reconstruction
• Yogesh Langhe (January – June 2017) Research topic: Does elastography imaging support 3D reconstruction in laparoscopy?
• Ronan Bureau (March – September 2016) Research topic: Creating digital and physical models to validate deformable 3D reconstruction and aid endoscopic training
• Shrinivasan Sankar (January – June 2016) Research topic: Does detection and recognition help seeing deformable objects in 3D?
• Tom François (February – July 2015) Research topic: 3D reconstruction of multiple deformable objects by memory-based Shape-from-Template
• Héla Hadhri (February – July 2015) Research topic: Development of a real-time prototype for 3D reconstruction by Shape-from-Template, and its application in interventional laparoscopy
• Jules Gorny (March – August 2014)
• Kristina Prokopetc (February – June 2014) Research topic: Building a Fallopian Tube Detector for Computer Assisted Intervention
• Shaifali Parashar (February – June, 2014) Research topic: Modeling Multiple Images of a Deformable Surface Using Differential Projective Geometry
• Pierre Bruyer (April – August, 2013) Master thesis title: Motion based segmentation and 3D reconstruction of the uterus

Collaborators

• Dr Abed Malti - Engineer

Publications by year

Accepted (found 2 in 7)
Shape-from-Template with Curves M. Gallardo, D. Pizarro, T. Collins and A. Bartoli International Journal of Computer Vision, accepted August 2019	[paper]
Local Deformable 3D Reconstruction with Cartan’s Connections S. Parashar, D. Pizarro and A. Bartoli IEEE Transactions on Pattern Analysis and Machine Intelligence, accepted May 2019	[paper]
2019 (found 4 in 15)
A Differential-Algebraic Projective Framework for the Deformable Single-View Geometry of the 1D Perspective Camera A. Bartoli Journal of Mathematical Imaging and Vision, 61(7):1051-1068, September 2019	[paper]
The Highlight Ovals A. Bartoli Journal of Mathematical Imaging and Vision, 61(7):919-943, September 2019	[paper]
Equiareal Shape-from-Template D. Casillas-Perez, D. Pizarro, D. Fuentes-Jimenez, M. Mazo and A. Bartoli Journal of Mathematical Imaging and Vision, 61(5):607-626, June 2019	[paper]
3DVFX: 3D Video Editing using Non-Rigid Structure-from-Motion S. Parashar and A. Bartoli EG - Eurographics, 2019	[paper] [video]
2018 (found 12 in 17)
Isometric Non-Rigid Shape-from-Motion with Riemannian Geometry Solved in Linear Time S. Parashar, D. Pizarro and A. Bartoli IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10):2442-2454, October 2018	[paper]
Inextensible Non-Rigid Structure-from-Motion by Second-Order Cone Programming A. Chhatkuli, D. Pizarro, T. Collins and A. Bartoli IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10):2428-2441, October 2018	[paper]
Plane-Based Resection for Metric Affine Cameras A. Bartoli and T. Collins Journal of Mathematical Imaging and Vision, 60(7):1037-1064, September 2018	[paper]
Contributions to Monocular Deformable 3D Reconstruction: Curvilinear Objects and Multiple Visual Cues M. Gallardo PhD thesis (thèse de doctorat), Université Clermont Auvergne, September 2018 [supervision: T. Collins and A. Bartoli (director)] [document in English]	[thesis]
Image-Based Models for Specularity Propagation in Diminished Reality S. Hadj Said, M. Tamaazousti and A. Bartoli IEEE Transactions on Visualization and Computer Graphics, 24(7):2140-2152, July 2018	[paper]
Model-Based Active Learning to Detect an Isometric Deformable Object in the Wild with a Deep Architecture S. Sankar and A. Bartoli Computer Vision and Image Understanding, 171:69-82, June 2018	[paper]
A Geometric Model for Specularity Prediction on Planar Surfaces with Multiple Light Sources A. Morgand, M. Tamaazousti and A. Bartoli IEEE Transactions on Visualization and Computer Graphics, 24(5):1691-1704, May 2018	[paper]
Augmented Reality in a Tumor Resection Model P. Chauvet, T. Collins, C. Debize, L. Novais-Gameiro, B. Pereira, A. Bartoli, M. Canis and N. Bourdel Surgical Endoscopy, 32(3):1192-1201, March 2018 - Junior researcher prize from Clermont-Ferrand's university hospital (prix jeune chercheur du CHU de Clermont-Ferrand), 2017 - Computer-aided surgery innovation prize from the National Academy of Surgery (prix de l’innovation médico-chirurgicale assistée par ordinateur du Labex CAMI décerné par l'Académie nationale de Chirurgie), 2017	[paper]
A 3D Deformable Model-Based Framework for the Retrieval of Near-Isometric Flattenable Objects using Bag-of-Visual-Words R. Rantoson and A. Bartoli Computer Vision and Image Understanding, 167:89-108, February 2018	[paper]
Fast Shape-from-Template Using Local Features M. Famouri, A. Bartoli and Z. Azimifar Machine Vision and Applications, 29(1):73-93, January 2018	[paper]
Self-Calibrating Isometric Non-Rigid Structure-from-Motion S. Parashar, A. Bartoli and D. Pizarro ECCV - European Conference on Computer Vision, 2018	[paper]
Rolling Shutter Pose and Ego-motion Estimation using Shape-from-Template Y. Lao, O. Ait-Aider and A. Bartoli ECCV - European Conference on Computer Vision, 2018	[paper]
2017 (found 11 in 25)
A Multiple-View Geometric Model of Specularities on Non-Planar Shapes with Application to Dynamic Retexturing A. Morgand, M. Tamaazousti and A. Bartoli IEEE Transactions on Visualization and Computer Graphics, special issue: selected papers from ISMAR 2017, 23(11):2485-2493, November 2017	[paper]
Image-based Deformable 3D Reconstruction using Differential Geometry and Cartan's Connections S. Parashar PhD thesis (thèse de doctorat), Université Clermont Auvergne, November 2017 [supervision: D. Pizarro and A. Bartoli (director)] [document in English] Université Clermont Auvergne PhD prize (prix de thèse de la fondation de l'Université Clermont Auvergne), 2018	[thesis]
Particle-SfT: a Provably-Convergent, Fast Shape-from-Template Algorithm E. Özgür and A. Bartoli International Journal of Computer Vision, 123(2):184-205, June 2017	[paper]
Planar Structure-from-Motion with Affine Camera Models: Closed-Form Solutions, Ambiguities and Degeneracy Analysis T. Collins and A. Bartoli IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1237-1255, June 2017	[paper]
A Stable Analytical Framework for Isometric Shape-from-Template by Surface Integration A. Chhatkuli, D. Pizarro, A. Bartoli and T. Collins IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(5):833-850, May 2017	[paper]
Generalizing the Prediction Sum of Squares Statistic and Formula, Application to Linear Fractional Image Warp and Surface Fitting A. Bartoli International Journal of Computer Vision, 122(1):61-83, March 2017	[paper]
EasyFlow: Increasing the Convergence Basin of Variational Image Matching with a Feature-Based Cost J. Braux-Zin, R. Dupont, A. Bartoli and M. Tamaazousti IET Computer Vision, 11(2):122-134, March 2017	[paper]
A Multiple-View Geometric Model of Specularities on Non-Uniformly Curved Surfaces A. Morgand, M. Tamaazousti and A. Bartoli VRST - ACM Symposium on Virtual Reality Software and Technology, 2017	[paper]
Dense Non-Rigid Structure-from-Motion and Shading with Unknown Albedos M. Gallardo, T. Collins and A. Bartoli ICCV - IEEE International Conference on Computer Vision, 2017	[paper]
Visualizing In-Organ Tumors in Augmented Monocular Laparoscopy E. Özgür, A. Lafont and A. Bartoli ISMAR - IEEE International Symposium on Mixed and Augmented Reality, 2017	[paper] [video]
A Multiple-View Geometric Model of Specularities on Non-Planar Shapes with Application to Dynamic Retexturing A. Morgand, M. Tamaazousti and A. Bartoli ISMAR - IEEE International Symposium on Mixed and Augmented Reality, 2017	[paper (TVCG17)]
2016 (found 9 in 24)
Local Analytic and Global Convex Methods for the 3D Reconstruction of Isometric Deformable Surfaces A. Chhatkuli PhD thesis (thèse de doctorat), Université d'Auvergne, December 2016 [supervision: D. Pizarro and A. Bartoli (director)] [document in English]	[thesis]
Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives D. Pizarro, R. Khan and A. Bartoli International Journal of Computer Vision, 119(2):93-109, September 2016	[paper]
3D Conformation of a Flexible Fiber in a Turbulent Flow G. Verhille and A. Bartoli Experiments in Fluids, 57(7):117-127, July 2016	[paper]
Using Shading and a 3D Template to Reconstruct Complex Surface Deformations M. Gallardo, T. Collins and A. Bartoli BMVC - British Machine Vision Conference, 2016	[paper]
Can we Jointly Register and Reconstruct Creased Surfaces by Shape-from-Template Accurately? M. Gallardo, T. Collins and A. Bartoli ECCV - European Conference on Computer Vision, 2016	[paper]
A Perspective on Non-Isometric Shape-from-Template A. Bartoli and E. Özgür ISMAR - IEEE International Symposium on Mixed and Augmented Reality, 2016	[paper]
An Empirical Model for Specularity Prediction with Application to Dynamic Retexturing A. Morgand, M. Tamaazousti and A. Bartoli ISMAR - IEEE International Symposium on Mixed and Augmented Reality, 2016	[paper]
Isometric Non-Rigid Shape-from-Motion in Linear Time S. Parashar, D. Pizarro and A. Bartoli CVPR - IEEE International Conference on Computer Vision and Pattern Recognition, 2016	[paper]
Inextensible Non-Rigid Shape-from-Motion by Second Order Cone Programming A. Chhatkuli, D. Pizarro, T. Collins and A. Bartoli CVPR - IEEE International Conference on Computer Vision and Pattern Recognition, 2016	[paper]
2015 (found 5 in 19)
Shape-from-Template A. Bartoli, Y. Gérard, F. Chadebecq, T. Collins and D. Pizarro IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(10):2099-2118, October 2015 Université d'Auvergne research prize (prix recherche de l'Université d'Auvergne), 2016	[paper]
Metric Corrections of the Affine Camera A. Bartoli, T. Collins and D. Pizarro Computer Vision and Image Understanding, 135:141-156, June 2015	[paper]
As-Rigid-As-Possible Volumetric Shape-from-Template S. Parashar, D. Pizarro, A. Bartoli and T. Collins ICCV - IEEE International Conference on Computer Vision, 2015	[paper]
Shape-from-Template in Flatland M. Gallardo, D. Pizarro, A. Bartoli and T. Collins CVPR - IEEE International Conference on Computer Vision and Pattern Recognition, 2015	[paper]
A Linear Least-Squares Solution to Elastic Shape-from-Template A. Malti, A. Bartoli and R. Hartley CVPR - IEEE International Conference on Computer Vision and Pattern Recognition, 2015	[paper]
2014 (found 9 in 17)
Infinitesimal Plane-Based Pose Estimation T. Collins and A. Bartoli International Journal of Computer Vision, 109(3):252-286, September 2014	[paper]
Monocular Template-Based 3D Surface Reconstruction: Convex Inextensible and Nonconvex Isometric Methods F. Brunet, A. Bartoli and R. Hartley Computer Vision and Image Understanding, 125:138-154, August 2014	[paper]
Combining Conformal Deformation and Cook-Torrance Shading for 3D Reconstruction in Laparoscopy A. Malti and A. Bartoli IEEE Transactions on Biomedical Engineering, 61(6):1684-1692, June 2014	[paper]
Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity A. Chhatkuli, D. Pizarro and A. Bartoli BMVC - British Machine Vision Conference, 2014	[paper]
Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives R. Khan, D. Pizarro and A. Bartoli ECCV - European Conference on Computer Vision, 2014	[paper (IJCV16)]
Using Isometry to Classify Correct/Incorrect 3D-2D Correspondences T. Collins and A. Bartoli ECCV - European Conference on Computer Vision, 2014	[paper]
An Analysis of Errors in Graph-based Keypoint Matching and Proposed Solutions T. Collins, P. Mesejo and A. Bartoli ECCV - European Conference on Computer Vision, 2014	[paper]
Stable Template-Based Isometric 3D Reconstruction in All Imaging Conditions by Linear Least-Squares A. Chhatkuli, D. Pizarro and A. Bartoli CVPR - IEEE International Conference on Computer Vision and Pattern Recognition, 2014	[paper (PAMI16)]
Live Image Parsing in Uterine Laparoscopy A. Chhatkuli, A. Bartoli, A. Malti and T. Collins ISBI - IEEE International Symposium on Biomedical Imaging, 2014	[paper]
2013 (found 6 in 22)
A Robust Analytical Solution to Isometric Shape-from-Template with Focal Length Calibration A. Bartoli, D. Pizarro and T. Collins ICCV - IEEE International Conference on Computer Vision, 2013	[paper]
A General Dense Image Matching Framework Combining Direct and Feature-based Costs J. Braux-Zin, R. Dupont and A. Bartoli ICCV - IEEE International Conference on Computer Vision, 2013	[paper (IETCV16)]
Combining Features and Intensity for Wide-Baseline Non-Rigid Surface Registration J. Braux-Zin, R. Dupont and A. Bartoli BMVC - British Machine Vision Conference, 2013	[paper (IETCV16)]
Isowarp and Conwarp: Warps that Exactly Comply with Weak-Perspective Projection of Deforming Objects D. Pizarro, A. Bartoli and T. Collins BMVC - British Machine Vision Conference, 2013	[paper]
Monocular Template-Based 3D Reconstruction of Extensible Surfaces with Local Linear Elasticity A. Malti, R. Hartley, A. Bartoli and J.-H. Kim CVPR - IEEE International Conference on Computer Vision and Pattern Recognition, 2013	[paper]
Template-Based Isometric Deformable 3D Reconstruction with Sampling-Based Focal Length Self-Calibration A. Bartoli and T. Collins CVPR - IEEE International Conference on Computer Vision and Pattern Recognition, 2013	[paper]