Efficient data storage of astronomical data using HDF5 and PEC compression

Jordi Portell de Mora; Enrique García-Berro; Carlos Estepa; Javier Castañeda; Marcial Clotet

doi:10.1117/12.898203

12 October 2011 Efficient data storage of astronomical data using HDF5 and PEC compression

Jordi Portell de Mora, Enrique García-Berro, Carlos Estepa, Javier Castañeda, Marcial Clotet

Proceedings Volume 8183, High-Performance Computing in Remote Sensing; 818305 (2011) https://doi.org/10.1117/12.898203
Event: SPIE Remote Sensing, 2011, Prague, Czech Republic

Abstract

Future space missions are based on a new generation of instruments that often generate vast amounts of data. Transferring this data to ground, and once there, between different computing facilities is not an easy task whatsoever. A clear example of these missions is Gaia, a space astrometry mission of ESA. To carry out the data reduction tasks on ground, an international consortium has been set up. Among its tasks perhaps the most demanding one is the Intermediate Data Updating, which will have to repeatedly re-process nearly 100 TB of raw data received from the satellite using the latest instrument calibrations available. On the other hand, one of the best data compression solutions is the Prediction Error Coder, a highly optimized entropy coder that performs very well with data following realitic statistics. Regarding file formats, HDF5 provides a completely indexed, easily customizable file with a quick and parallel access. Moreover, HDF5 has a friendly presentation format and multi-platform compatibility. Thus, it is a powerful environment to store data compressed using the above mentioned coder. Here we show the integration of both systems for the storage of Gaia raw data. However, this integration can be applied to the efficient storage of any kind of data. Moreover, we show that the file sizes obtained using this solution are similar to those obtained using other compression algorithms that require more computing power.

Citation Download Citation

Jordi Portell de Mora, Enrique García-Berro, Carlos Estepa, Javier Castañeda, and Marcial Clotet "Efficient data storage of astronomical data using HDF5 and PEC compression", Proc. SPIE 8183, High-Performance Computing in Remote Sensing, 818305 (12 October 2011); https://doi.org/10.1117/12.898203

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available