|
Combining
photo-consistency and silhouette constraints
|
alt="optimizing photoconsistency while preserving exact silhouettes">
We have developed an approach to compute a watertight 3D surface that trades
off photo-consistency and shape priors while exactly enforcing silhouette
constraints. A special graph is constructed for which each valid cut
separating source from sink yields a watertight surface that exactly
satisfies the silhouette constraints. By translating costs for
photo-consistency and shape priors to edge capacities, the minimal cut can be
made to correspond to the optimal surface.
S. Sinha, M. Pollefeys, Multi-view Reconstruction using
Photo-consistency and Exact Silhouette Constraints: A Maximum-Flow
Formulation, Proc. International Conference on Computer Vision, 2005. [pdf]
|
|
Camera network
calibration and synchronization
|
alt="Camera network">
We have developed a practical approach to automatically calibrate a network
of cameras without the need for recording specific calibration data.
Our approach is robust and can deal with unsynchronized video streams and
widely separated cameras. The approach relies on the information
provided by silhouettes of moving objects.
S. Sinha, M. Pollefeys. Synchronization
and Calibration of Camera Networks from Silhouettes, Proc. International
Conference on Pattern Recognition 2004. [pdf]
S. Sinha, M. Pollefeys. Camera
Network Calibration from Dynamic Silhouettes, Proc. of IEEE Conf. on
Computer Vision and Pattern Recognition, 2004. [pdf]
|
|
Calibration of omnidirectional sensors
|
We have developed a simple linear approach that can calibrate wide-angle
lenses from three views of a plane or when the camera performs pure
rotation. The key concept for this is the radial trifocal tensor.
We are now working on the use a non-parametric model of distortion, to deal
with cata-dioptric cameras and some non-central
cameras and to use four views in general configuration (using the radial quadrifocal tensor).
S. Thirthala, M. Pollefeys,
Multi-view geometry of 1D radial cameras and its application to omnidirectional camera calibration, Proc. International
Conference on Computer Vision 2005. [pdf]
S. Thirthala, M. Pollefeys,
The Radial Trifocal Tensor: A Tool for Calibrating Radial Disortion of Wide-Angle Cameras, IEEE Conf. on
Computer Vision and Pattern Recognition, 2005. [pdf]
S. Thirthala, M. Pollefeys,
Trifocal Tensor for Heterogeneous Cameras,
OMNIVIS Workshop, 2005. [pdf]
|
|
Articulated and
Non-Rigid Motion Analysis
|
 
We are working on a factorization-based approach for analysing
articulated and non-rigid motion. An important result that we have
obtained is that the linear motion subspaces of articulated bodies intersect
and that their intersection corresponds to the motion subspace of the
joints.
J. Yan, M. Pollefeys, A
Factorization-Based Approach for Articulated Non-rigid Shape, Motion
and Kinematic Chain Recovery From Video, IEEE Trans. on Pattern
Analysis and Machine Intelligence, to appear.
J. Yan, M. Pollefeys, Recovering
Articulated Non-rigid Shapes, Motions and Kinematic
Chains From Video, Proc. IV Conference on Articulated Motion and
Deformable Objects, 2006.
J. Yan, M. Pollefeys, Automatic
Kinematic Chain Building
from Feature Trajectories of Articulated Objects, Proc. CVPR’06
(IEEE Conf. on Computer Vision and Pattern Recognition), 2006. [pdf]
J. Yan, M. Pollefeys, A
General Framework for Motion Segmentation: Independent, Articulated, Rigid,
Non-rigid, Degenerate and Non-degenerate , European Conference on
Computer Vision, 2006. [pdf]
J. Yan, M. Pollefeys, A
factorization approach to articulated motion recovery, IEEE Conf. on
Computer Vision and Pattern Recognition, 2005.[pdf]
J. Yan, M. Pollefeys, Articulated
Motion Segmentation Using RANSACwith Priors,
ICCV Workshop on Dynamical Vision 2005. [pdf]
|
|
Real-time computer
vision on commodity graphics hardware
|
|
Todays Graphic Processing Units (GPU) have tremendous processing power and are
programmable. Since the GPU is build to process images it is
particularly well suited to perform some computer vision and image processing
algorithms very efficiently. We developed a real-time stereo algorithm
that runs on the GPU and is several times faster than most CPU-based
implementations.
[more
details]
R. Yang, M. Pollefeys, A Versatile Stereo
Implementation on Commodity Graphics Hardware, Journal of Real-Time
Imaging, 2005 (to appear). [pdf]
R. Yang, M. Pollefeys, H. Yang, G. Welch, A
Unified Approach to Real-Time, Multi-Resolution, Multi-Baseline 2D View
Synthesis and 3D Depth Estimation using Commodity Graphics Hardware,
International Journal of Image and Graphics (to appear). [pdf]
R. Yang, M. Pollefeys, and S. Li, Improved
Real-Time Stereo on Commodity Graphics Hardware, Workshop on Real Time 3D
Sensors and Their Use 2004 (in conjunction with CVPR) (to appear). [pdf]
R. Yang and M. Pollefeys. Multi-Resolution
Real-Time Stereo on Commodity Graphics Hardware, Proc. IEEE Conf. on
Computer Vision and Pattern Recognition, pp. 211-218, 2003. [pdf]
Besides stereo, we have also implemented other computer vision algorithms on
the GPU, such as the KLT feature tracker and the SIFT feature detector.
More details of our KLT implementation, including the code area available
through the following link.
Sudipta N Sinha,
Jan-Michael Frahm, Marc Pollefeys
and Yakup Genc,
"GPU-Based Video Feature Tracking and Matching", EDGE 2006,
workshop on Edge Computing Using New Commodity Architectures, Chapel Hill,
May 2006. [pdf]
|
|
3D-from-Video
|
|
 [mpg]
We have developed one of the first systems to automatically extract detailed
3D models from photo or video sequences. Our first implementation was
ready in 1997. Since then we have continued to improve our image-to-3D
pipeline, removing limitations and increasing accuracy and robustness.
M. Pollefeys, L. Van Gool,
M. Vergauwen, F. Verbiest,
K. Cornelis, J. Tops, R. Koch, Visual modeling
with a hand-held camera, International Journal of Computer Vision 59(3),
207-232, 2004. [IJCV][pdf]
M. Pollefeys and L. Van Gool.
Visual modeling: from images to images, The Journal of Visualization
and Computer Animation, 13: 199-209, 2002. [pdf]
M. Pollefeys and L. Van Gool.
From Images to 3D Models, Communications of the ACM, July 2002/Vol.
45, No. 7, pp.50-55. [pdf]
M. Pollefeys, L. Van Gool,
M. Vergauwen, K. Cornelis,
F. Verbiest, J. Tops, Video-to-3D, Proc. Photogrammetric Computer Vision 2002 (ISPRS Commission
III Symposium), International Archive of Photogrammetry
and Remote Sensing. [pdf]
M. Pollefeys, R. Koch, M. Vergauwen,
L. Van Gool. Automated reconstruction of 3D
scenes from sequences of images, ISPRS Journal Of Photogrammetry
And Remote Sensing (55)4 (2000) pp. 251-267. [pdf]
M. Pollefeys, R. Koch, M. Vergauwen
and L. Van Gool, Hand-held acquisition of 3D
models with a video camera, Proc. 3DIM'99 (Second International
Conference on 3-D Digital Imaging and Modeling), IEEE Computer Society Press,
pp.14-23, 1999. [pdf]
M. Pollefeys, R. Koch, M. Vergauwen
and L. Van Gool, Metric 3D Surface
Reconstruction from Uncalibrated Image Sequences, Proc. SMILE Workshop (post-ECCV'98), LNCS 1506,
pp.138-153, Springer-Verlag, 1998. [pdf]
|
|
Polar rectification
and stereo geometry
|
|
 
Our polar rectification scheme allows to rectify
image pairs for any relative camera motion (including when the epipole is in the image) and guarantees a minimal image
size. The price to pay is that co-linearity is in general not preserved, only epipolar lines
are guaranteed to remain straight. The iso-disparity
surfaces corresponding to polar rectification have some interesting
properties and reveal more fundamental properties of stereo
configurations.
M. Pollefeys, S. Sinha. Iso-disparity surfaces for general stereo
configurations, T. Pajdla and J. Matas (Eds.), Computer Vision - ECCV 2004 (European
Conference on Computer Vision), LNCS, Vol. 3023, pp. 509-520, Springer-Verlag, 2004. [pdf]
M. Pollefeys, R. Koch and L. Van Gool, A simple and efficient rectification method for
general motion, Proc. International Conference on Computer Vision,
pp.496-501, Corfu (Greece),
1999. [pdf]
|
|
Pan-Tilt-Zoom
cameras and active camera networks
|
|
[camera]
Pan-tilt-zoom cameras allow to construct far more
flexible vision systems than static cameras. We are exploring the
possibility to build active networks of pan-tit-zoom cameras to observe,
track and reconstruct events in 3D. As a first step we have developed
an approach to automatically calibrate the pan-tilt-zoom camera over its full
zoom range and to build very high-resolution panoramas.
S.
Sinha, M. Pollefeys, Pan-Tilt-Zoom Camera Calibration and
High-Resolution Mosaic Generation, Computer Vision and Image
Understanding, 103(3), pp. 170-183, 2006.
S. Sinha, M. Pollefeys,
S.J. Kim, High-Resolution Multiscale Panoramic
Mosaics from Pan-Tilt-Zoom Cameras, 4th Indian Conference on Computer
Vision, Graphics and Image Processing (accepted).
S. Sinha, M. Pollefeys. Towards
Calibrating a Pan-Tilt-Zoom Cameras Network, OMNIVIS 2004, ECCV
Conference Workshop CD-rom proceedings, 2004. [pdf]
|
|
Multi-projector
displays
|
|
Very high-resolution displays can be constructed by combining multiple
projectors. We have developed a fully automated method to calibrate a
system consisting of casually alligned projectors.
Compared to previous approaches, we have removed the need for 4 or more fiducial points to establish a metric reference frame on
the display.
A. Raij, M. Pollefeys. Auto-Calibration
of Multi-Projector Display Walls, International Conference on Pattern Recogniton 2004. [pdf]
[Wide Area
Visuals (WAV) project]
[PixelFlex project]
|
|
Multi-view
reconstruction from specular surfaces
|
|
 
By extending the photo-consistency criterion to include specular
effects we are able to compute the 3D geometry of specular
surfaces. Our approach does not require to know
the light geometry and can deal with multiple lightsources,
including area lightsources.
R. Yang, M. Pollefeys, and G. Welch. Dealing
with Textureless Regions and Specular
Highlight: A Progressive Space Carving Scheme Using a Novel Photo-consistency
Measure, Proc. of the International Conference on Computer Vision, pp.
576-584, 2003. [pdf]
|
|
Photometric
calibration
|
|
We have extended previous work on radiometric/photometric calibratioin
and high-dynamic range (HDR) imaging to a moving camera. This mainly
consisted of developing a robust approach to estimate the brightness transfer
function in the presence of occlusion (due to wrong correspondences and
view-dependent effects). One of the applications consist
of dealing with auto-exposure while constructing a textured 3D model from a
video sequence. In the image shown above the image on the left
was not radiometrically aligned, while the image on
the right was. We are now working on also dealing with vignetting.
S.J. Kim, M. Pollefeys. Radiometric
Self-Alignment of Image Sequences, Proc. of IEEE Conf. on Computer Vision
and Pattern Recognition, 2004. [pdf]
|
|
Self-calibration
|
|
We
have developed algorithms to automatically calibrate the camera from video
sequences. This allows to obtain metric
measurements with an uncalibrated camera.
Among other contributions we have proposed the first practical algorithm to
deal with a varying focal length.
M. Pollefeys, F. Verbiest,
L. Van Gool, Surviving dominant planes in uncalibrated structure and motion recovery, A. Heyden, G. Sparr, M. Nielsen, P.
Johansen (Eds.) Computer Vision - ECCV 2002, 7th European Conference on
Computer Vision, Lecture Notes in Computer Science, Vol.2351, pp. 837-851. [pdf]
M. Pollefeys, L. Van Gool.
Do ambiguous reconstructions always give ambiguous images?, Proc.
International Conference on Computer Vision, 2001, pp.187-192. [pdf]
M. Pollefeys and L. Van Gool.
Some Geometric Insight in Self-Calibration and Critical Motion Sequences,
Technical Report Nr. KUL/ESAT/PSI/0001, PSI-ESAT, K.U.Leuven, Belgium, 2000. [pdf]
M. Pollefeys, R. Koch and L. Van Gool. Self-Calibration and Metric Reconstruction in
spite of Varying and Unknown Internal Camera Parameters, International
Journal of Computer Vision, 32(1), 7-25, 1999. [pdf]
M. Pollefeys and L. Van Gool,
Stratified Self-Calibration with the Modulus Constraint, IEEE
Transactions on Pattern Analysis and Machine Intelligence, Vol 21, No.8, pp.707-724, 1999. [pdf]
M. Pollefeys, R. Koch and L. Van Gool. Self-Calibration and Metric Reconstruction in
spite of Varying and Unknown Internal Camera Parameters, Proc.ICCV'98
(international Conference on Computer Vision), pp.90-95, Bombay, 1998. joint
winner of the David Marr prize (best paper). [pdf]
|
|
Image-based
rendering
|
|
 [avi]
We have proposed the first unstructured lightfield
rendering algorithm that can render directly from original images acquired
with a hand-held camera. Our approach combines view-dependent geometry
and textures. This allow to circumvent the
difficult problem of having to reconstruct a consistent 3D representation of
a recorded scene.
M. Pollefeys and L. Van Gool.
Visual modeling: from images to images, The Journal of Visualization
and Computer Animation, 13: 199-209, 2002. [pdf]
R. Koch, B. Heigl, and M. Pollefeys,
Image-Based Rendering from Uncalibrated Lightfields with Scalable Geometry, In R. Klette, T. Huang, G. Gimel'farb
(Eds.), Multi-Image Analysis, Lecture Notes in Computer Science, Vol. 2032,
pp.51-66, Springer-Verlag, 2001. [pdf]
B. Heigl, R. Koch, M. Pollefeys,
J. Denzler and L. Van Gool,
Plenoptic Modeling and Rendering from
Image Sequences taken by Hand-held Camera, Proc. DAGM'99, pp.94-101,
joint best paper. [pdf]
R. Koch, M. Pollefeys, B. Heigl,
L. Van Gool and H. Niemann.
Calibration of Hand-held Camera Sequences for Plenoptic
Modeling, Proc. International Conference on Computer Vision, pp.585-591, Corfu (Greece), 1999. [pdf]
R. Koch, B. Heigl, M. Pollefeys,
L. Van Gool and H. Niemann,
A Geometric Approach to Lightfield Calibration,
Proc. CAIP99, LNCS 1689, Springer-Verlag,
pp.596-603, 1999. [pdf]
|
