4.3  Occlusion-compatible scanning order

This section presents a method that automatically prevents that background pixels overlap or hide the visibility of foreground pixels. Let us explain the overlapping problem in more detail. By construction, the presented rendering techniques define a mapping between source and destination pixels. However, one algorithmic issue is that multiple foreground and background source pixels can be mapped at the same position in the destination image. To determine the visibility of each pixel in the rendered view, a Z-buffer can be used that stores the depth of each pixel in the rendered view. When two pixels are warped at the same position, a depth comparison of the two pixels is performed and the foreground pixel with the smallest depth value is selected for rendering. However, this technique involves the usage of a memory buffer and a depth comparison for each warped pixel. An alternative technique is the occlusion-compatible scanning order [56] that we present in this section.

4.3.1  Occlusion-compatible scanning for rectified images

Let us first introduce the basic principle of the occlusion-compatible scanning order, using the simplified case of rectified images. To determine the visibility of each pixel in the rendered view, a practical method is to scan the original view such that eventually occluded background pixels are obtained prior to the occluding foreground pixels. Assuming that the views are rectified and that the intrinsic camera parameters are identical, the image rendering simply consists of a right-to-left pixel-shift operation, where the shift corresponds to the per-pixel disparity value [68]. Therefore, the rendering procedure is performed by scanning the disparity image in a left-to-right fashion such that foreground pixels overwrite background pixels, so that foreground pixels are always preserved. Additionally, to determine the positions of undefined pixels, a per-pixel flag is set when the interpolated pixel position is written. The collection of these per-pixel flags finally provides a map of occluded pixels. The procedure is illustrated by Figure 4.7.


Figure 4.7 A proper scanning order of the original view enables that foreground pixels, p3,p4,p5,p6 overwrite the background pixels, i.e., p1 and p2. Visible background pixels p8,p9,p10,p11 are preserved. The arrows indicate a simple copy operation. The symbol X represents the occluded pixels.

4.3.2  Occlusion-compatible scanning for non-rectified images

The problem of scanning background pixels prior to foreground pixels using non-rectified images can be addressed using the occlusion-compatible scanning order [56]. Let us consider two 3D scene points P1 and P2, which are projected onto a target image (virtual view) at the same pixel position p, and onto the reference view at pixel positions p1′ and p2′ (see Figure 4.8). To perform an occlusion-compatible scanning order of the reference image, it is necessary to scan first the 3D point P1 and then P2. Using the framework of oriented projective geometry [94], such a scanning can be obtained by considering the projection of the virtual/rendered camera center C onto the reference image plane, i.e., the epipole λe′e′ = M′C. Specifically, two cases that depend on the sign of the scaling factor λe′ can be distinguished.

Whereas the first case follows from the discussion in Figure 4.7, the understanding of the second case is more complicated. We have worked on a proof of that case, for which a summary is presented in Appendix 9.1.

Figure 4.8 To perform the warping of background p1′ prior to foreground p2′, the reference image is scanned from the border toward the epipole e′.