1.

INTRODUCTION

As a graduate student, I had enormous challenges when started my research on high-speed 3D imaging with fringe projection techniques. I got stuck for more than three years without much progress. It was Professor Wyant successes in both both academy and industry that kept me believe that I could pursue a career in this field. Even though I have never met Professor Wyant in person, but his generous help and kind support were critical to reach my current career status.

I took advantage of the modern digital technology for 3D imaging to address the challenges in high-speed, highresolution 3D imaging and optical information processing. As a graduate student, I have developed a system that simultaneously captures, processes and displays 3D geometries at 30 Hz with over 300,000 measurement points per frame, which was unprecedented at that time (a decade ago). My current research focuses on achieving speed breakthroughs by developing the binary defocusing techniques; and exploring novel means to store enormously large 3D geometric data by innovating geometry/video compression methods. The binary defocusing methods coincide with the inherent operation mechanism of the digital-light-processing (DLP) technology, permitting tens of kHz 3D measurement speed at camera pixel spatial resolution. The novel methods of converting 3D data to regular 2D counterparts offer us the opportunity to leverage mature 2D data compression platform, achieving extremely high compression ratios without reinventing the whole data compression infrastructure.

This paper summarizes what we have done to achieve speed breakthroughs in the field of 3D optical metrology using digital fringe projection techniques, to achieve real-time 3D video streaming by compressing enormous 3D video data size, and to solve some practical problems using the techniques we have developed including biomechanics, robotics, forensics, and entertainment.

2.

HIGH-SPEED 3D IMAGING

Figure 1 shows the basic schematics of a digital fringe projection (DFP) system. A projector projects the computergenerated fringe patterns onto an object surface, the object surface geometry deforms the fringe patterns, a camera captures the deformed fringe patterns from a different perspective, and the software algorithm analyzes the captured fringe patterns for 3D reconstruction. The fringe deformation is reflected in carrier phase as well. Phase-shifting algorithm can be used to analyze the phase information for 3D reconstruction after system calibration.

Figure 1:

Schematics of a digital fringe projection (DFP) system.²

2.1

Three-step phase-shifting algorithm

Phase-shifting algorithms are extensively employed partially because they are robust to noise, and simultaneously achieve high measurement speed and accuracy. Three-step phase-shifting algorithm is often used for high-speed 3D imaging because it only requires three fringe patterns for phase extraction,

where I’(x, y) is the average intensity, I”(x, y) is the intensity modulation and ϕ (x, y) is the phase to be solved for. The ratio of I"(x, y) and I’(x, y) reflects the fringe contrast to environmental light. Solving these equations simultaneously gives

here φ(x, y) is the wrapped phase whose value ranges from -π and π with 2π discontinuities, and a spatial or temporal phase algorithm is required to obtain a continuous phase map by removing 2π discontinuities,

where k(x, y) is an integer number for each pixel. Zhang¹ summarized the state-of-art absolute phase unwrapping methods for DFP systems.

In addition, averaging these three images washes out fringe stripes and gives a texture image, a photograph of the object

2.2

DFP system calibration

To convert phase to 3D informtion, structured light system calibration is required. The calibration process could be simple or complex, and the achievable accuracy level could be different.² The digital fringe projection system consists of a camera and a projector. The mathematical model for the camera and the projector is identical and straightforward,³

describing the transformation from the world coordinate system (x^w,y^w, z^w) to the lens coordinate system and the projection from the lens coordinate system to the image plane (u, v). Here ^T denotes matrix transpose; A is a 3 × 3 intrinsic matrix that models the focal length and the principle point (intrinsic parameters matrix); R is a 3 × 3 rotation matrix and t is a 3 × 1 translation vector, and matrix [R, t] describes the transformation from the world coordinate system to the lens coordinate system (extrinsic parameters matrix).

The camera calibration was well studied at that time, and but DFP system calibration was difficult since calibrating projector was not easy. The projector calibration is difficult because the projector cannot capture images like a camera. Zhang and Huang⁴ published a method that enables projector to capture images like a camera. Essentially, we project two sets of fringe patterns, and use the carrier phase to establish the corresponding point between camera and the projector. The known correspondence provides a mapping from the camera image to the projector pixel by pixel. Once the projector can capture images, its calibration problem becomes well-studied camera calibration problem, and the DFP system calibration is a standard stereo vision system calibration. This is one of the most extensively adopted methods in the field of structured light 3D imaging.

2.3

Real-time 3D imaging with digital-light-processing (DLP) projector

Started in graduate school, my research works on high-speed 3D imaging. I first tried the single-chip DLP projector for real-time 3D imaging. It took me 4 years to develop hardware system and software algorithms. The hardware system development involves with modifying the projector, encoding three phase-shifted patterns into three color channels, and precisely capturing these channel images in grayscale. The software algorithms includes fast phase wrapping,⁵ rapid phase unwrapping,⁶ and 3D reconstruction and visualization on a single personal computer (PC). We took advantage of the single-chip DLP projector’s unique projection mechanism to naturally three phase-shifted fringe images at a high speed.⁷

Figure 2 shows an image of the real-time 3D imaging system we developed. We were able to achieve 25 Hz 3D imaging on a single computer with dual central processing units (CPUs). The right image shows the subject being captured and the left right shows the real-time 3D image created on the computer screen.The quality of data may not be the best in today’s standard, but it was done in 2004.⁸

Figure 2:

Results captured by one of the earliest real-time 3D imaging system based on the digital fringe projection technique.⁸

2.4

Superfast 3D imaging with binary defocusing technique

We then spent many years to achieve speed breakthrough without drastically increasing the hardware cost. We ultimately found that the DLP technology had the potential. DLP technology creates grayscale images by flipping the micro mirror ON/OFF at an extremely high speed.

We came out the binary defocusing technology⁹ to fully take advantage of the binary image generation nature of DLP technology. As the computer generated binary structured patterns being more and more blurred by defocusing the lens, pseudo sinusoidal patterns can be generated. This pseudo sinusoidal patterns can be analyzed using phase-shifting algorithms for 3D imaging.

DLP Discovery 4100 can refreshes 8-bit grayscale images at approximately 300 Hz, but binary images at over 30 kHz. Since only three images are required for each 3D reconstruction, this platform allows for 3D imaging at a speed over 10 kHz. Zhang et al.¹⁰ developed a 3D imaging system that achieved 667 Hz with a three-step phase-shifting algorithm and a DLP Discovery Development kit; Gong and Zhang¹¹ achieved 4,000 Hz rate by using a commercial DLP projector; Li et al.¹² achieved 5 kHz with a modified Fourier transform algorithm; and Zuo et al.¹³ achieved 10 kHz with a modified Fourier transform algorithm.

Figure ?? shows the 3D imaging results of a flying bird robot bird. The robotic bird (Model: XTIM Bionic Bird Avitron V2.0) that we used in this research has a beat frequency of approximately 25 cycles per second with both wings made of inextensible thin membranes. The total span of a single wing is about 150 mm (L) × 70 mm (W). Precisely synchronized with the fringe projection, the camera captures images also at a rate of 5,000 Hz with an image resolution of 800 × 600 pixels.

Figure 3:

3D imaging results of a flying bird robot captured at 5,000 Hz.¹² (a) - (c) Three sample frames of 2D images; (d) - (f) three sample frames of reconstructed 3D geometries.

3.

3D IMAGE/VIDEO COMPRESSION AND COMMUNICATIONS

When 3D imaging speed increases, recording highly dynamic events becomes easier and easier. At the same time, storing 3D image data becomes increasingly challenging. As such, we had to find ways to compress 3D data to save storage space.

The approach we use is to create a virtual fringe projection system that virtually projects fringe patterns onto the 3D model and the rendered image can be “see” as the camera image.¹⁴ Since the virtually created fringe projection system can be precisely controlled, we can encode all 3D and 2D texture information into a standard RGB 2D image. The difference between the encoded and original image is pretty much random noise.

Streaming 3D videos is very interesting but is even more challenging. For example, it is impossible to stream relatively low-resolution 3D video (800 × 600) through even 5G networks. Fortunately, our compression algorithm converts 3D images to standard 2D images. We can further leverage 2D video compression methods for 3D video compression.¹⁵ Leveraging standard video compression methods such as H.264, we could stream 3D video through standard low-bandwidth networks if the median quality 3D video is acceptable.

Figure 4 shows some examples decoded from the data stored using various H.264 qualities. The lossless encoding achieves a compression ratio of 129:1. When using H.264 to lossy encode video at various levels, higher compression ratios can be achieved, requiring slower networks for real-time streaming. This figure future shows that even at the very low bitrate of 4.8 Mbps the decoded 3D images are still of great quality in both geometry and color texture.

Figure 4:

Compressing encoded 3D data with the H.264 video codec at different qualities.¹⁶ First row shows the decoded images from the video stored with lossless H.264 (compression ratio 129:1); second row shows the decoded images from H.264 video using 4:2:0 subsampling and a constant rate factor of 6 (compression ratio 551:1); and third row shows the decoded images using the same encoding instead with a constant rate factor of 12 (compression ratio 1,602:1).

We developed a 3D video streaming technology called Holostream that enables real-time 3D video capture, compression, streaming, decompression, and visualization.¹⁶ This was probably the first technology that enables high-fidelity 3D video streaming using standard cellphone networks. Figure 5 shows the successful implementation of our holostream system. The left image shows the Acquisition and Compression Modules. The right image shows multiple users on their mobile devices receiving and interacting with the live 3D video stream delivered across a standard wireless network.

Figure 5:

Successful implementation of the Holostream system.¹⁶

4.

SOME APPLICATIONS

3D imaging has been numerous applications ranging from healthcare for overall diagnosis offered by the third dimension, aerospace and defense for mapping, imaging and quality control of complex part manufacturing, along with others. The rapid growth of 3D sensor industry was driven by rising demands in consumer electronics for devices enhanced by 3D sensors (e.g., smartphones, tablets). In this section, we only pick a few surprising applications that we are luckily involved with.

The forensics community is interested in the high-resolution 3D imaging technology for crime scene documentation. Our prior research has demonstrated that high-resolution 3D imaging outperforms the current practices (i.e., 2D photography and casting).²⁴ This community needs a fully automated 3D camera that does not require any training for examiners and practitioners to operate. It remains a challenge to develop such smart 3D imaging technology for this field.

Figure 6 shows the test of shoe impression in clay with grain size being smaller than 0.01 mm. The red circles outline locations that correspond to test cuts in the shoe outsole and the orange circles outline locations that correspond to small rockholds in the shoe outsole. Figure 6(a) shows the evidence quality 2D photograph: resolved 10 test cuts and the two small rockholds; Figure 6(b) photograph of the conventional physical cast: resolved two small rockholds and all 10 test cuts, but the small bubbles produced in the casting process made visualization difficult for several of the cuts; Figure 6(c) shows the low-resolution 3D image (approximately 137 dots per inch, dpi) of the impression that resolved 9 out of the 10 test cuts, and two small rockholds; Figure 6(d) shows high-resolution 3D image(approximately 400 dpi) of the impression that resolved all 10 test cuts and two small rockholds.

Figure 6:

Test of shoe impression in clay. The red circles outline locations that correspond to test cuts in the shoe outsole and the orange circles outline locations that correspond to small rockholds in the shoe outsole.²⁴ (a) Evidence quality 2D photograph; (b) photograph of the conventional physical cast; (c) low-resolution 3D image of the impression; (d) high-resolution 3D 3D image of the impression.

The real-time 3D imaging we developed caught some attentions. But the most surprising early interest came from the entertainment industry. Radiohead, and English rock band used our technology the created this 3D music video “house of cards”.²⁵

The superfast 3D imaging was very useful in biomedical engineering field. For example, working with our collaborators, we developed a multimodal imaging system that can simultaneously record 3D geometry, 2D texture image, and 2D fluorescence images at kHz range.²⁶ The recorded information can be used to analyze the mechanics of heart (such as strain field on the bottom left video), as well as diagnose heard diseases.

5.

SUMMARY

Thank you, Professor Jim Wyant, for what you have done for the community and being my hero. I am so grateful for your generous support to someone like me whom you never met. Your successes in both academia and industry inspired me to pursue a career in 3D imaging and still motivate me to work hard every day.