1.1.

Hologram and the Replay Field

In a classical imaging system, Fig. 2, focusing optics are used to focus light scattered from a point of an object onto a corresponding point on a sensor (the recording device). In such a system, any different origin point on the object leads to a corresponding change in the position on the recording plane of the sensor. The loss of a portion of the sensor data will result in a corresponding loss in the image.

Fig. 2

A classical optical imaging system.

In a holographic imaging system, Fig. 3, scattered light is collected without the use of a focusing optics, instead of interfering the scattered light with a reference beam. Replicating recording conditions allows for replication of the light field and the resulting image, as depicted in Fig. 4. The image is stored across all parts of the recording device leading and loss of a portion of the recording only causes a loss in the quality of the image. The entire image can still be reproduced.

Fig. 3

A holographic imaging system for hologram recording.

Fig. 4

A holographic projection system for hologram reconstruction.

Traditional analog holography follows two steps known as recording and reconstruction:^1,2,17

CGH goes further than this by using a known target or scene to generate the reconstruction image, thus eliminating the requirement of a recording step.

1.2.

Limitations of CGH

CGH promises a great deal. However, in practice, a number of key limitations exist:

2.1.

Platform for Preliminary Verification of Algorithms

Most hologram generation algorithms were developed on conventional computers, utilizing the power of the latest CPU chip families. Software-based algorithms run on CPUs to efficiently minimize the development time and reduce the computational burden by exploiting advanced computation libraries, software packages, and other utilities.

Reported work based purely on CPUs form the preliminary analysis of various proposed computer hologram generation algorithms. Researchers tend to focus more on theoretical development rather than code optimization since conventional CPUs are not used for acceleration purposes.

Due to their commonality and the ease-of-use, the majority of work that incorporates CPUs often uses them as the comparison baseline for algorithm implementations on other hardware platforms that utilize dedicated accelerators.

2.2.

Available Tools and Utilities for CPUs

Since CPUs are the core components within a modern personal computer (PC) and workstation, the vast majority of software packages and development suites are readily available. Code and programs can be written in many high-level languages, and low-level application programming interfaces (APIs) and frameworks, such as OpenMP and OpenCV, are also widely available. As an API for shared memory multiprocessing, OpenMP is dedicated to high-level parallelism in Fortran and C/C++ programs.²⁹ Compiler directives, library routines as well as environment variables can be used to optimize for multiprocessing by, for example, distributing workloads among the available threads and physical cores.

We endeavor to conclude the tools and utilities that have been reported in previously published papers since 2008, as shown in Table 1. The most commonly used software application is MATLAB due to its simplicity and numerous package supports. No strict understandings in terms of hardware architectures and memory management are necessary when developing algorithms over MATLAB, as compared to other realization methods. C/C++ tends to be the most popular programming language used for algorithm implementations. C/C++ programming libraries and functions, such as FFTW, cvDFT from OpenCV, and custom library CWO++,³⁷ offer strong support for improved hologram generation performances.

Table 1

Tools and utilities employed for CPU implementations since 2008.

2.3.

Advantages and Disadvantages of Using CPUs

The most significant advantage of using CPUs for algorithm implementation is the short development time and sophisticated software toolchain support. Nearly, all the software packages that can be found on other hardware accelerator platforms have the same or equivalent toolkits, which are available on CPU-based PCs. These ranges from programming languages, such as C/C++ and Python, compile time and run-time libraries to software packages.

Other merits of using CPUs are as follows:

The disadvantages are also apparent. CPUs are optimized for sequential operations and consequently, full parallelism cannot be achieved. Although state-of-the-art CPUs at the time of writing feature a higher level of parallelism than older CPUs, with tens of cores being available in a single package, this pales in comparison to the massively parallel architectures of GPUs and FPGAs, which feature thousands of parallel execution units. Moreover, the software libraries and APIs to support parallelism, such as OpenMP, which help shorten the developing time needed for multithread and multiprocessing applications, exist but require an advanced level of skills to utilize effectively.

2.4.

Reported Work Using CPUs

Most of the reported work covering CPU-based applications are for either algorithm developments or, more commonly, for establishing baselines for cross-platform performance comparisons.

The most common method to optimize the performance of hologram generation using a computer is to combine both CPUs and GPUs together.

Shimobaba et al. reported on the development of a C++ library CWO++, which is used for diffraction calculations.³⁷ This library has been developed to run on both CPU (CWO class) and GPU (GWO class, GPU-based wave optics), and has been used in various algorithm developments.^{24,38–40,44,48–50}

We aim not to thoroughly review the work that reports on CPU-based platform performance, as in the majority of cases the CPU results are used to provide a baseline performance reference. However, the baselines are subsequently encountered in many throughout the survey.

2.5.

Summary of CPUs

Contemporary CPUs offer insufficient performance for “real-time” CGH, and hence it is not recommended to build a real-time holographic system based solely on them. Moreover, the ready availability of hardware accelerators, such as GPUs and FPGAs, provides further rationale for hologram generation algorithms not to be implemented purely on a CPU-only platform.

Algorithm developments in the initial phase, however, are one exception for purely CPU simulations and implementations, e.g., with MATLAB and Simulink, in order to significantly cut down the development time and improve the efficiency of research outputs. Moreover, this approach also encourages collaboration, lowering the skills and knowledge barriers for other research groups to replicate and improve the corresponding algorithms.

3.1.

Parallelism in GPUs

Architecturally, a GPU is significantly different from a CPU. The major difference being that GPUs exploit massive parallelism at the hardware level. A single mainstream contemporary GPU incorporates thousands of dedicated processor cores, whereas even the highest-end CPUs typically contain less than 24 cores.⁵⁴ It is this inherent parallelism that provides high-performance computation capability for highly parallel problem spaces.

3.2.

GPU Performance Trends

Over the years, NVIDIA brought out a range of core microarchitectures in their GPU series, targeting both the professional high-performance uses as well as individual consumer level applications. The last decade has seen a large increase in the performance capability of GPUs, as summarized in Table 2.^56,55 While the rated power consumption has remained relatively constant (at around 200 to 300 W), we have seen a significant increase in processing power. Ever since Fermi was regarded as “the first complete GPU computing architecture,”⁵⁷ 10 years has seen NVIDIA working on incorporating advanced shaders, hardware ray tracing, and many high-performance functionalities, with larger and faster processing capability and speed for not only graphics rendering but more general-purpose usages.⁵⁸ Consider the floating-point operations per second (FLOPS) performance of a top-end GPU. Between 2008 and 2018, the performance increased from 432 GFLOPS to 16,312 GFLOPS, an improvement of more than $37\times$.

Table 2

Microarchitectures since 2008 and their representative flagship GPU products (SP: single-precision floating point).55

GPU designs vary between different microarchitectures and production families. Figure 5 shows a typical structural overview of an NVIDIA GPU with numerous streaming multiprocessors consisting of shared memories, L1/L2 caches, CUDA cores, arithmetic units (e.g., double precision unit), load/store units, etc. This architectural setup reveals the high level of inherent hardware parallelism within modern GPU devices.

Fig. 5

A typical parallel pipeline overview of an NVIDIA GPU consisting of streaming multiprocessors each containing a number of cores and functional units.

3.3.

Available Tools and Utilities for GPUs

Two utilities are widely used: CUDA platform and open computing language (OpenCL) framework.

In 2007, a parallel computing platform and API model, CUDA, was released by NVIDIA. Prior to the introduction of CUDA, graphics and GPU programming skills for use in tools, such as Direct3D, DirectX, and OpenGL, with a good understanding of high-level shader language (HLSL) were required in order to take advantage of the high computational performance of graphics cards.⁵⁹ CUDA, however, only required standard C/C++ or Fortran programming language skills as the bare minimum.

As of the writing of this review, CUDA has iterated to its tenth generation (10.1) and comes with both compile-time and run-time libraries.⁵² In particular, the CUDA FFT library enables high-performance FFT and inverse FFT computations similar to the FFTW library.⁶⁰ Other useful libraries provided include, but are not limited to a basic linear algebra subroutine library, cuBLAS, useful for linear algebraic operations; a random number generation library, cuRAND, useful for random phase generations; and a parallel algorithms and data structures library, Thrust, to accelerate operations, such as sum and average, as well as boundary (maximum and minimum) search algorithms in parallel.

Developing programs over CUDA is straightforward with the support of a modified C programming language dedicated to the CUDA framework.

Additionally, vendors, such as NVIDIA and AMD, have provided full support and have released the implementations of OpenCL for their GPUs. OpenCL is a framework with low-level APIs for cross-platform computing. Developers can use the provided APIs from OpenCL to write programs that run across CPUs, GPUs, etc., with C programming language. However, it is worth noting that a study (not related to holography) conducted by Memeti et al. in 2017 suggested that CUDA outperforms OpenCL in terms of productivity, requiring two times less programming effort for a specific benchmark suite.⁶¹

MATLAB, in the meantime, provides a parallel computing toolbox for GPU computing. Despite some limitations, it is argued that combining both CUDA kernels and MATLAB support (using the parallel computing toolbox) can further improve and smooth the programming process.^33,62 No knowledge in CUDA is needed while exploiting the parallel computing capabilities for CGH-related computation speed-ups.

3.4.

Advantages and Disadvantages of Using GPUs

GPUs are used for accelerating the processing of images and videos at birth. The hardware architectures are specially designed for this purpose by highly optimizing the parallel characteristics in both hardware and software.

The key advantage of GPGPUs is that they can be programmed using high-level programming languages, such as C/C++, making code development and corresponding debug processes faster and easier than in other platforms, such as FPGAs.

As shown in Table 2, one of the major disadvantages of using GPUs for algorithm implementation is their high power consumption. The thermal design power, which is the maximum amount of heat generated by the chip during operation and which serves as a basic indicator of power consumption, is typically around 200 to 300 W.

A good understanding of GPU microarchitectures and, in particular, memory management is required for speed optimization, although dedicated utilities tend to offer modest support for managing the memory.

More importantly, most of the GPUs cannot work as a standalone platform. A system incorporating CPUs and other essential hardware devices tends to create limits on data throughput during read, fetch, and write operations and would increase the overall power consumption. Additionally, this level of integration introduces a data transfer bottleneck, which downgrades the overall performance of the platform. The speed for data transfers between the host PC and the GPU or GPU cluster would even slow down when the implementation has not been properly optimized.

3.5.

Development Time Using GPUs

The use of CUDA makes GPU implementations simpler. The majority of the development time will be spent on software coding using C/C++ programming language.

The major difficulty in the development of GPU hologram generation application is to optimize the codes for the potentially high-throughput and heavy computational requirements. This requires a good understanding of the GPU architectures as well as hardware and software optimization techniques. However, since most of the fast algorithms implemented, such as in Refs. 45^,63^–65, require less sophisticated operations, the optimization can be based purely on increasing the data throughput and improving the computational power with parallel processing.

3.6.

Reported Work Using GPUs

Lucente and Galyean demonstrated the first published result of CGH generation using a computer graphics workstation.⁶⁶ The achieved performance was calculated using eight $128\times 60\text{pixels}$ full-color images, which leads to a replay field of a 3D object with different viewing angles. At that time, the calculation time of 2 s over the graphics workstation was 100 times faster than a conventional computer.

Later, in 2003, Petz and Magnor used an NVIDIA Geforce 4600Ti to generate the interference fringes for holograms.⁶⁷ In their work, the authors assessed both the GPU performance and the computational time dependency based on the resolution of holograms. For an object that contains 1024 light source points, it takes 0.96 and 3.86 s to calculate the corresponding holograms of the resolutions of $512\times 512$ and $1024\times 1024$, respectively.⁶⁷

Before the introduction of CUDA, the graphical API OpenGL was used to compute holograms, as was reported in 2006⁶⁸ and 2009.²¹ However, the performance was not promising. Additionally, a real-time reconstruction system for an $800\times 600$ 64-point based 3D object CGH was reported in 2006 using HLSL and DirectX API, achieving a calculation speed $47\times$ faster than a Pentium 4 CPU.⁵⁹

The use of OpenCL for parallel computing to generate holograms with an AMD HD5000 was reported by Shimobaba et al. in 2010.⁶⁹

Since the release of CUDA, there has been a surging interest in the generation of computer holograms utilizing the full credibility and computational power of GPUs.

The GPU microarchitectures have changed remarkably throughout the past decade, and the increased computational power produced an improvement of at least 10 times. This performance improvement can also be seen in the reported literature.

Shiraki et al.⁷⁰ in 2009 reported a 1000 point light source (3D object) real-time holographic video generation system using an NVIDIA GTX 280 utilizing Tesla microarchitecture. The generated hologram resolution was $1400\times 1050\text{pixels}$. The performance was later surpassed by the introduction of GTX 1080Ti with Pascal microarchitecture in 2018.⁷¹ The system reported by Kim et al. can produce real-time high definition holographic generation and projection using 10,000 points of light, a 10 times increment in terms of object-point counts than that of the system reported in Ref. 70.

Table 3 provides a summary of some of the hardware implementations using different hologram generation algorithms in recent literature.

Table 3

A summary of CGH implemented on GPUs since 2008.

3.7.

Summary of GPUs

Traditionally, GPU vendors design their line of products in order to carry out single-precision floating point operations effectively.⁸² Throughout the years, these vendors have worked to redesign their products to allow for the use of half-precision numbers and fixed points.

GPUs are by design powerful single- and double-precision floating point hardware accelerators, recent trends have led to the use of half-precision and fixed-point arithmetic, which further enhanced the computational speed while making a trade-off in terms of precision.

Due to the hardware and manufacturing constraints, the number of streaming (CUDA) cores that can be embedded within a single GPU is limited. Therefore, in order to speed up the hologram generation process, one practical solution is to form a GPU cluster using multiple GPUs. This can be done either in a single stand-alone system⁷⁴ or over a dedicated network.⁴⁴

In general, GPU offers a strong candidate for CGH systems.

4.1.

Highly Configurable Hardware Platform

The key strength of FPGAs is its highly configurable and hardware-programmable nature. The applications can be developed using computer-based hardware description languages (HDLs), such as Verilog/SystemVerilog, VHDL, etc.⁸³ These language-based designs are portable and usually independent of technology, with the exception of applying IP cores and other chip-specific configurations. The designers are able to repeatedly program and reconfigure a given chip to affect changes at the hardware level and reuse designs across different FPGA chips that are normally from the same vendor.

The ability for FPGAs to support HDLs provides a significant benefit in that almost all on-chip cells are highly configurable and can be used to synthesize any possible hardware implementations as long as the designs can potentially be fitted into the available logic cells and hardware units.

4.2.

Implementations Based on Fixed and Floating Point

Since FPGA platforms are highly reconfigurable, the use of either fixed point or floating-point arithmetic becomes one of the most important design considerations. According to a report produced by Xilinx,⁸² FPGA applications will benefit from the conversion from floating-point to fixed-point arithmetic for certain applications requiring less power but higher speed.

Floating-point precision typically includes IEEE 754 half-precision (16-bit), single-precision (32-bit), and double-precision (64-bit) configurations, whereas fixed points are more flexible and usually range from several bits to 32-bit in width.

Devices, such as GPUs, which are used in computationally heavy applications, have traditionally been designed architecturally so that they are more efficient when supporting floating-point operations. When implemented on FPGAs at the hardware level, however, conventional floating-point operations, e.g., based on GPUs or CPUs, are slower than fixed point alternatives. This is due to the need and difficulty, which arises when controlling the mantissa and exponents of IEEE 754 floating points during the calculations.⁸⁴

4.3.

Available Tools and Utilities for FPGAs

FPGA vendors typically provide their own proprietary tools. Intel’s Quartus Prime is widely used among the community to facilitate development for Intel-based FPGAs. As for those devices offered by Xilinx, there are development tools, such as the Vivado design suite, which has replaced the Xilinx integrated synthesis environment (ISE).

ModelSim is a functional simulation software package from Mentor Graphics. It can be used independently to simulate hardware based on HDL descriptions as well as being compatible with Intel Quartus Prime, Xilinx ISE, and Vivado.

In addition, many hardware implementations use IP cores provided by the vendors to perform certain operations on FPGAs.

Due to their unique nature, the FPGA development process is very distinct from traditional CPUs and GPUs. A simplified overview is summarized as follows:⁸³

Table 4

Tools and utilities reported for FPGA implementations in recent years.

4.4.

Development Time Using FPGAs

Generally, depending on the level of hardware complexity and the use of IP cores, the development time might vary. For example, reported by Sugie et al.,⁹³ the group took over 3 years to develop and implement their algorithms into a custom-made bespoke FPGA platform consisting of eight high-end FPGA chips.

The average development time for a project based on FPGA hardware implementations will typically be significantly longer than an equivalent CPU or GPU project. Although not being reported for CGH applications, a study conducted in 2012 estimated that developing algorithms on a GPU-based hardware platform for dense optical flow, stereo, and local image extraction features takes $\sim 2\text{months}$ for one full-time postdoctoral employee, whereas developing the same algorithms and functionalities over an FPGA platform will likely take 12 months for two postdoctoral employees.⁹⁴ Overall, the development time for FPGA-based applications is likely to take longer than the equivalent for an algorithm to be implemented on a GPU platform.

4.5.

Advantages and Disadvantages of Using FPGAs

One of the merits of FPGA implementation is the potential to migrate a given FPGA register-transfer level (RTL) design into ASICs. ASICs are dedicated chipsets specifically designed for a certain application. They are inflexible and require significant one-off tooling costs; once designed, they represent an optimal combination of performance, power, and cost for a given hardware accelerator. The performance can be optimized for the generation of computer holograms with the use of ASIC technology. A recent work in 2017⁹⁵ demonstrated that an FPGA-based implementation can be migrated into a very-large-scale integration (VLSI) without the need for vast modifications.

The potential for high performance at moderate power consumption along with the ability to migrate a given design to an ASIC provides a strong argument for the use of FPGAs in CGH applications.

As pointed out in Sec. 4.4, the most significant drawback for FPGA implementation is the relatively long development time. FPGA-based hologram generation projects often require years of work by a group of researchers. Moreover, the required knowledge in terms of understanding of hardware architecture and FPGA technology for the developers sets up a high entry barrier.

4.6.

Reported Work Using FPGAs

Table 5 summarizes recent implementations on FPGA platforms. Most of the hardware models used in the lines of work are high-end FPGAs from Xilinx.

Table 5

A summary of CGH implemented on FPGAs since 2008.

Holographic reconstruction (HORN) computers, which have been in active development by Ito et al. since 1992,¹⁰⁴ have provided the research community with many insights into the field of CGH hardware implementation, notably the use of FPGAs for real-time hologram generation. So far, there are, in total, eight generations of devices being produced by this group, ranging from low-speed devices to high-speed special-purpose computers. The first four generations of HORN use DSP or small-scale FPGA chips for real-time computation tasks.^86,104–106 The later four generations of devices consist of large-scale FPGA chips embedded on delicate custom printed circuit boards.^{46,50,93,96,100} The latest product within this line of work is HORN-8, which comprises seven powerful FPGA chips for calculation and one FPGA chip for communication. As reported in Refs. 50 and 93, the HORN-8 special computer can generate a hologram for a 3D object of 10,000 points within 0.019 s with a peak performance of 0.5 tera floating-point operations per second (TFLOPS) running at a 0.25 GHz clock cycle. At the time of writing, the team’s outlook is to further develop an ASIC design based on the HORN-8 structure.¹⁰⁷

Seo et al. proposed a hardware architecture based on pixel-by-pixel calculation scheme.^87,88,92,95 In this line of work, the authors efficiently reduced the number of memory accesses by utilizing the pixel-by-pixel method, which is different from the conventional light source-by-source calculation method. The authors also demonstrated a VLSI chip, based on the proposed FPGA architecture.⁹⁵ The work reported by Seo et al.⁹⁵ demonstrated that it is relatively simple to migrate an FPGA system into an ASIC design.

4.7.

Summary of FPGAs

Benefiting from its highly configurable architecture, FPGAs are to date the most flexible hardware accelerators for use in hologram generation applications. The required calculations in hologram generation algorithms can take advantage of the high degree of parallelism within an FPGA chip. However, the development time to implement optimized algorithms on FPGAs is typically significant and requires an advanced skill set of HDLs, digital logic design, and hardware architecture, skills not typically present in traditional optics groups researching holography.

5.1.

Digital Signal Processors

DSPs are dedicated hardware platforms for signal processing applications. The microprocessors have architectures that are tuned for analog and digital signal processing tasks with the ability to support single instruction multiple data (SIMD).

Nishikawa et al. reported the use of a DSP to generate holographic images in the late 1990s.¹⁰⁸ A multi-DSP system consisting of $3\times 4$ i860 DSPs was proposed to generate 3D objects for the application. The 3D object consists of 200 points and is $640\times 480$ pixels in size. The multi-DSP system takes 68 s to generate the object as opposed to a reference workstation (SPARCstation 10), which generates the object in 291 s.

The most recent work was reported by Oi et al.¹⁰⁹ Twenty TMS320C6727 DSPs running floating-point arithmetic was used to form the DSP block in the proposed system. These DSPs were dedicated to the conversion of integral photography images to holograms in the Fresnel diffraction domain. With a $1.5\times$ redundancy design, a real-time performance of 50 FPS was achieved.

The current highest-end DSP products are those from Analog Devices and Texas Instruments. A TI TMS320C6678 eight-core floating-point DSP runs at a clock rate of 1 GHz to 1.4 GHz, with a maximum computational performance of 20 GFLOPS per core for single-precision floating-point operations.¹¹⁰ It is unlikely that these DSPs will be capable of performing complicated hologram generation algorithms due to the hardware specifications and limited computational power. However, it is still worthwhile to regard DSP as a valuable candidate to implement less complicated algorithms due to their low power profile and ease of programming. DSPs are typically programmed using C language and assemblies. The toolchain support is considered mature and time-proven, further minimizing the development time and difficulty.⁵³

5.2.

Xeon Phi Coprocessor and ClearSpeed Accelerator Board

Xeon Phi is a family of coprocessors with x86 manycore architecture designed and produced by Intel.¹¹¹ It is to-date one of the few fairly powerful many-core coprocessors that are intended for use in hardware acceleration applications.¹¹² This line of products supports the use of OpenMP.¹¹³ As was introduced in Sec. 2.2, OpenMP is an API that is optimized for shared memory multiprocessing programming and exploits multithread parallelism.

Murano et al. in 2014 reported on the use of a Xeon Phi coprocessor unit (Xeon Phi 5110P) for computer hologram generation.¹¹⁴ The authors used the Intel MKL for the calculation of FFTs along with the OpenMP functionalities to make use of the available cores present in the coprocessor. Their results show that in all their test cases, GPU outperforms the Xeon Phi accelerator by a significant margin. However, when using Xeon Phi coprocessor, the amount of existing code that needs to be rewritten in order to port software-based algorithms into the hardware accelerator, as compared to that in the GPU case, can be minimized.

Another hardware acceleration board, ClearSpeed Advance Dual CSX600, was demonstrated in 2009.¹¹⁵ The authors were able to speed up the point cloud hologram calculation $56\times$ faster than an Intel Xeon CPU performing calculation in single core. Unfortunately, as of the writing of this survey, the production of ClearSpeed accelerator boards is no longer active.

5.3.

System-on-Chip Hybrid CPU and FPGA

There is a growing need for hologram generation systems to become compact and low in power consumption. A trend toward system-on-chip (SoC) utilizing the heterogeneous system architecture (HSA) has been rapidly growing over the years. The general idea behind SoC is to incorporate different devices and peripherals on a single chip to reduce the overall die area and to minimize power consumption.¹¹⁶ One hybrid product is to have both FPGA and microprocessors or CPUs onboard one chip. A further discussion is present in Sec. 6.5.

In one of the most recent studies conducted by Yamamoto et al., the authors developed a compact holographic computer using a Xilinx Zynq UltraScale+ MPSoC consisting of an ARM CPU and an FPGA on one single chip.¹¹⁷ The reported system was able to reproduce $1920\times 1080\text{pixels}$ 3D video at a rate of 15 frames/s.¹¹⁷ They also compared the result to the performance of a Jetson TX1 platform,¹¹⁸ the calculation time of $1920\times 1080\text{pixels}$ with 6500 points on the SoC platform took 0.066 s, whereas the Jetson TX1 took 1.294 s.

The development time for these SoC hardware implementations would be even longer than that of pure FPGA developments since the incorporation of both CPU, which requires multithread programming, and FPGA, which uses HDLs, adds another level of complexity when highly optimized codes and algorithms are needed. However, the power efficiency, die area, and package size scale-down can bring about other benefits that mitigate the increased programming workload.

6.1.

Toolchain Support

One of the most important aspects to consider is the full-cycle development toolchain support. CPU and GPU platforms are likely to be less affected by the lack of available software package and library supports, as discussed in the previous sections. However, FPGAs and other accelerators, such as DSPs and coprocessors, might suffer from the lack of active development support and will, in turn, affect the overall development process. In general, the availability of tools and utilities to support the dedicated hardware creates a resource barrier toward the successful implementation.

6.2.

Choice of Algorithms and Parallel/Sequential Processing

Many algorithms exist for 2D/3D hologram generation. Different algorithms would require different hardware resources in practice, e.g., triangular-mesh based algorithms can take advantage of being compatible with modern computer graphics technologies utilizing polygon meshes for object computations.¹³ Regardless of the algorithm used, the size, e.g., hologram resolution size and number of points/polygons, is an important consideration in all cases, and more importantly, increasingly complex holograms demands larger and better hardware.

GPUs and other specialized hardware accelerators are useful to the speed enhancement of hologram calculation by utilizing parallelism and optimizing for sequential processing. For example, in the point-cloud-based calculation, the hologram patterns are calculated using the same mathematical formula, and more importantly, the calculation of these patterns for each object point is independent of other object points.¹³ The independent calculation of object points can potentially make use of parallel processing for hardware platforms.

Moreover, for CGH algorithms involving FFT operations and depending on the hardware utilized, the FFT operations can be parallelized through different cores or pipelines at the processor level, as shown in Fig. 8.

It is also of great importance, though being algorithm-dependent, to be aware of the number of sequential processes required and optimize for performance while exploiting concurrency and parallelism within the specified hardware. For example, iterative algorithms, such as the Gerchberg–Saxton algorithm,¹¹⁹ require sequential processing that cannot or tend to be difficult to parallelize and multitask. It is then of the developer’s responsibility to select a platform that does not only fulfill the need for parallelism but also has options to optimize for the sequential operations when implementing the desired algorithm.

6.3.

Portability of Software

It is essential to consider the possibility of transferring the developed software and firmware from one system to another while keeping in mind the trade-offs between portability and performance. This transfer would likely be required when upgrades toward newer generations of hardware are expected or performance comparisons between different devices are needed.

The most straightforward transfer comes when the CPU platform, which is usually based on a PC, is used. Upgrading between different operating systems and software platforms is comparatively simple due to the abundant software support. In comparison, porting from one NVIDIA GPU to another would sometimes require more work, although CUDA provides a unified development environment. This is mainly due to the upgrades in hardware between different generations of GPU products. As for intrageneration code transplant, it is usually not challenging, as long as the memory and computational power limitations have been taken into account by the developer.

Code transfer among different FPGA platforms, on the contrary, would be slightly difficult, especially when target chip IP cores are used for the application. With the above noted, transferring from a lower performance FPGA to an FPGA with higher performance can be relatively simple, this will likely be the case when HDL descriptions are used.

6.4.

Hologram Generation Quality Assessment

An end-to-end CGH hardware implementation assessment should include fast generation, hardware performance, and generated quality assessment of the holograms.

There currently is a lack of available unified criterion to assess the quality of computer holograms generated from different platforms. Kim et al.⁹⁰ use a modulation transfer function to compare the image quality of different holograms. Structural similarities have also been used in the Ref. 49 to evaluate the quality of the generated images. Another widely used metrics is to measure the mean square error and peak signal-to-noise ratio.¹²⁰ Blinder et al.¹²¹ provided a more detailed review of the quality assessment for computer-generated holograms.

6.5.

Heterogeneous Computing and its Related Hardware

There is a growing trend in the embedded systems, image and video processing communities to incorporate the state-of-the-art heterogeneous computing systems into their applications.

Heterogeneous computing systems typically refer to systems that fuse more than one type of processors or cores together,¹²² it could also refer to systems that combine a large number of processor cores with the same instruction set architecture (ISA),⁵³ e.g., Intel Xeon Phi, or a small number of cores with different execution performances, e.g., ARM big. Little platform.¹²³ In this section, we focus mainly on the development and trend in heterogeneous hardware accelerators that incorporate different types of ISA devices.

These hardware systems take advantage of conventional multicore hardware accelerators while, in the meantime, bypassing some of the limitations and disadvantages of using a single hardware accelerator architecture.¹¹⁶ The aims of having the HSA are to reduce the communication latency between different computing devices and to improve the parallel execution performance.¹¹⁶

The level of heterogeneity in a computing system gradually increases with more and more SoC platforms being produced. Among various of heterogeneous hardware platforms, the combination of CPUs and FPGAs, usually in the form of hard ARM processors embedded in an FPGA, as well as CPUs with DSPs, are potentially good candidates for low-cost low-power hologram generation platforms due to their inherent merits that balance the pros and cons of conventional system architectures.

Another heterogeneous computing platform worth mentioning is the Jetson module. Only the Jetson TX1¹¹⁸ was evaluated in the work.¹¹⁷ Its upgraded version TX2¹²⁴ and the most recent AGX Xavier,¹²⁵ both with boosted performance and power efficiency, are also of implementation interest. A low-cost variant of the Jetson family, Jetson NANO, has also become available in the market recently.¹²⁶ A list of the Jetson family is shown in Table 7.

Table 7

NVIDIA Jetson module products family.

6.6.

Future Trend in Embedded Systems and High-Performance Hardware Platforms

ARM developed the Neon technology for its Cortex-A series and R52 processors as an advanced SIMD architecture extension for image and video as well as general signal processing purposes.¹²⁸ No reported work to-date has exploited the possibility of integrating an ARM-based SoC embedded platform for the generation of computer holograms while utilizing technologies such as Neon.

NVIDIA recently announced its plan to bring CUDA acceleration to the ARM ecosystem.¹²⁹ This will potentially bring the power and accessibility of CUDA to platforms, such as ARM-based SoCs. This is also accompanied by the introduction of CUDA-X high-performance computing libraries, which can potentially further exploit parallelism and provide improved processing performance.¹³⁰

From another perspective, the recently announced Vitis Unified Software Platform from Xilinx provides another degree of flexibility to use high-level synthesis FPGA languages in order to help reduce the development overhead of FPGA applications.¹³¹ This unified platform is envisioned to shorten the overall development time with higher level implementations without the need of fully incorporating RTL-level development and to provide better programmability for the FPGA hardware.