Full Content is available to subscribers

Subscribe/Learn More  >
Proceedings Article

Efficient data storage of astronomical data using HDF5 and PEC compression

[+] Author Affiliations
Jordi Portell de Mora, Javier Castañeda, Marcial Clotet

Institute for Space Studies of Catalonia (Spain) and Univ. de Barcelona (Spain) and Institut de Ciencies del Cosmos (Spain)

Enrique García-Berro, Carlos Estepa

Institute for Space Studies of Catalonia (Spain) and Univ. Politècnica de Catalunya (Spain)

Proc. SPIE 8183, High-Performance Computing in Remote Sensing, 818305 (October 11, 2011); doi:10.1117/12.898203
Text Size: A A A
From Conference Volume 8183

  • High-Performance Computing in Remote Sensing
  • Bormin Huang; Antonio J. Plaza
  • Prague, Czech Republic | September 19, 2011

abstract

Future space missions are based on a new generation of instruments that often generate vast amounts of data. Transferring this data to ground, and once there, between different computing facilities is not an easy task whatsoever. A clear example of these missions is Gaia, a space astrometry mission of ESA. To carry out the data reduction tasks on ground, an international consortium has been set up. Among its tasks perhaps the most demanding one is the Intermediate Data Updating, which will have to repeatedly re-process nearly 100 TB of raw data received from the satellite using the latest instrument calibrations available. On the other hand, one of the best data compression solutions is the Prediction Error Coder, a highly optimized entropy coder that performs very well with data following realitic statistics. Regarding file formats, HDF5 provides a completely indexed, easily customizable file with a quick and parallel access. Moreover, HDF5 has a friendly presentation format and multi-platform compatibility. Thus, it is a powerful environment to store data compressed using the above mentioned coder. Here we show the integration of both systems for the storage of Gaia raw data. However, this integration can be applied to the efficient storage of any kind of data. Moreover, we show that the file sizes obtained using this solution are similar to those obtained using other compression algorithms that require more computing power.

© (2011) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Citation

Jordi Portell de Mora ; Enrique García-Berro ; Carlos Estepa ; Javier Castañeda and Marcial Clotet
"Efficient data storage of astronomical data using HDF5 and PEC compression", Proc. SPIE 8183, High-Performance Computing in Remote Sensing, 818305 (October 11, 2011); doi:10.1117/12.898203; http://dx.doi.org/10.1117/12.898203


Access This Proceeding
Sign in or Create a personal account to Buy this proceeding ($15 for members, $18 for non-members).

Figures

Tables

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Proceeding
Sign in or Create a personal account to Buy this proceeding ($15 for members, $18 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.