CS395/495(26)  Image Based Modeling and Rendering

Northwestern University Spring 2002  Jack Tumblin

Grading:   5 Self-Contained Programming Projects,
                  1 Written Midterm Exam.
                  (I changed my mind: no final exam):

--Please don't skip class to work on your project:
    instead, come to class and ask/learn how to do the hard parts. It's far more efficient.

    Project 1: 'Image Grid Viewer' 

    In all the later projects I expect you to do your own work.  However, on Project 1 I encourage you to collaborate, learn from each other and share good ideas about organizing your program, getting good GUI widgets to work, and so on.  Be sure that everyone that helps you is properly credited in the comment header at the start of al of your source files. A good project 1 solution will make the later projects easier.
    1) Get familiar with the PLY file format (3D polygonal mesh w/ texture maps)
        a) read about it, find documentation and source code at Stanford or at Greg Turk's site,
        b) Download and install this very nice 3D viewer for PLY files: (freeware).
        c) Look around on the web and find a few good 3D PLY files to look at; I recommend 
        --the Stanford 3D scan repository , where PLY file format was devised, 
        --a great collection at University of Virginia (Dave Luebke)  
        --Greg Turk's Large Geometric Models Archive

Note that many PLY files need to have normals reversed for proper display in the Quick3D viewer. On dropdown menu 'Mesh'-->invert Normals.  Also, push the 'spin' button on lower right so you flick the model to set it spinning on-screen. Cool, eh?

    2) Download, compile, and get this program for 3D manipulation of a single textured polygon using quaternion-based 'ArcBall' manipulation. This is an OpenGL, Visual C++ project; if you have trouble, you're probably missing default directories or including libraries for OpenGL. (see source code here;  Say thank you to Dan Hazen and Pierre ALLIEZ)

    3) Now stop playing around and do some homework.   The demo program shows you just one textured rectangular polygon viewed from a movable camera position.  Add a GUI control widget (dialog box, checkbox, button, menu item, edit box, whatever you like) that: 
    a) displays text values for the homogeneous projection matrix that defines how the textured polygon coordinates are changed to viewing coordinates, and 
    b) lets you set those numbers somehow--from a file or a dialog box, etc., without re-starting the program (something other than command-line switches).

  4) The demo program shows you just one texture-mapped rectangular polygon. Your next mission is to replace that single texture-mapped polygon with a texture-mapped mesh of the same size made up of a grid of tiny equal-sized squares.  After you're done it won't LOOK like you made any changes to the viewer; the same texture-mapped picture will whirl around in the same way as before (maybe slower), but now it is a texture mapped quad. mesh instead of a single polygon.  The vertices of this quad mesh will act as surrogate 'pixels', but it is OK to use a coarser than the resolution of the input texture image if the viewer is too slow.   Be sure to use FLOAT or DOUBLE for mesh vertex coordinates. 

    5) Next, add GUI controls and file read/write routines to the viewer that allows you to read and/or write mesh and it's associated texture as a PLY file.  (read/write all three coordinates of the mesh vertices--remember you'll be moving them around later!). Test your files using the freeware PLY viewer program of step 1).  Extra credit: make your PLY file reader accept non-mesh shapes such as a bunny or a dinosaur.

    6) Add another control widget  that allows you to switch from textured-polygon display to wireframe only.  
            (Extra Credit: See if you can color the wires by mesh vertex colors)
            (Extra Credit: try some some simple 2D image warps by moving mesh vertices: unequal radial expansion/contraction, sine-wave displacement (in x,y), etc.)

    Project 2: 2D Projective Image Warper

   DRAFT:

 Make image grid mesh coincide with viewing plane (e.g. textured grid is a plane, centered and parallel to the image viewing plane, so grid points equally spaced in display image).  Keeping the grid points planar (e.g. keep the image mesh flat), move the grid points to construct a different camera viewpoint. (don't move the camera, but instead move the 2D grid points within the grid image plane).  Turn off texture-mapping and view your grid as wire-frame, and look at how the mesh shape changes.  Move the camera to the new viewpoint you're simulating and look again at the mesh.

Project 3: 3D Projective Warper 

DRAFT:

    Now we will add some depth data to our mesh image.
1) Use this image of a shaded cube (or tetrahedron, or make your own) as the texture map for a mesh image.  
2) Set some depth values.  Choose few mesh points at the corners of the cube in the image, and manually and give them some plausible non-zero depth values. Use linear interpolation between these depths to find depth for all the in-between image mesh points. Set large background depth values; for example, you might set them all to a value twice as large as the initial distance from the camera to the cube center.
3) Set the camera to view the mesh image as we did in Project 1: 'head-on', so the mesh image = screen image.
4) Without moving the camera, move the mesh image vertices in 2D (no change in depth), so as to give the appearance of viewing the image from a different camera position (position B).
5) As with Project 1, repeat step 4, but now move the camera around to a different position (position B will be interesting!) so you can see what your warping does from a different angle.

Project 4: Single Image Camera Finder

DRAFT:

Read in a mesh image of a cube again,  (or anything with at least three large non-coplanar polygons with visible vertices).  Manually mark mesh image the corners of each visible cube side.  From this data, find the camera matrix that was used to make the image!  Extra credit: estimate error bounds (error ellipsoids, etc.).

Project 5: Epipolar Geometry: 
                    Find Depth, Calibrate Cameras

DRAFT:

 Now read in TWO mesh images A,B that are photos of the same 3D scene taken from mildly different camera positions (most of the things visible in image A are visible in image B).  Mark a few corresponding points in the images. Find the two camera positions and fundamental matrix, and display them as objects in 3D.  Extra Credit: Move the image meshes to camera positions in 3D, set depth values for the vertex correspondences. Show epipolar planes. Search along epipolar lines in the mesh images to find more correspondences.