J-PAS data management pipeline and archiving

D. Cristóbal-Hornillos; N. Gruel; J. Varela; A. López-Sainz; A. Ederoclite; M. Moles; A. J. Cenarro; A. Marín-Franch; J. Hernández-Fuertes; A. Yanes-Díaz; S. Chueca; S. Rueda-Teruel; F. Rueda-Teruel; R. Luis-Simoes

doi:10.1117/12.925431

24 September 2012 J-PAS data management pipeline and archiving

D. Cristóbal-Hornillos, N. Gruel, J. Varela, A. López-Sainz, A. Ederoclite, M. Moles, A. J. Cenarro, A. Marín-Franch, J. Hernández-Fuertes, A. Yanes-Díaz, S. Chueca, S. Rueda-Teruel, F. Rueda-Teruel, R. Luis-Simoes

Author Affiliations +

Proceedings Volume 8451, Software and Cyberinfrastructure for Astronomy II; 845116 (2012) https://doi.org/10.1117/12.925431
Event: SPIE Astronomical Telescopes + Instrumentation, 2012, Amsterdam, Netherlands

Abstract

J-PAS survey is going to observe 8000 deg² in 54 optical narrow band filters plus 3 broad band ones. The survey will produce 1.2 PB of raw data in six years of observations. The treatment of about 1.5 TB per night, coming from the 14 detectors of the JPCam camera in the JST/T250, plus one detector in the JAST/T80 camera, shall be performed during the day after the operations. This contribution presents the software and hardware architecture designed to process and validate the data. The processing of the images is divided in two main stages. The first one, which deals with instrumental correction and data validation, is run daily. The second stage is run when a tile is completed and do the combination of the individual corrected frames and weight maps. To perform the astrometric calibration, image coadding and source extraction the data management pipeline uses software from the community which is integrated through Python. The software uses a database to control the process by storing the operations performed, parameters used and quality checks. This allows fast reprocessing to retrieve intermediate stages of the treatment from the raw data for any data release. This approach saves disk space by avoiding the storage of the processed individual frames. The data archiving and processing will be done in a data center 30 km away from the observatory. It will be equipped with ~2.5 PB of storage capacity to store the raw data and the final mosaics of the 57 filters, and processing power to deal with the incoming data.

Citation Download Citation

D. Cristóbal-Hornillos, N. Gruel, J. Varela, A. López-Sainz, A. Ederoclite, M. Moles, A. J. Cenarro, A. Marín-Franch, J. Hernández-Fuertes, A. Yanes-Díaz, S. Chueca, S. Rueda-Teruel, F. Rueda-Teruel, and R. Luis-Simoes "J-PAS data management pipeline and archiving", Proc. SPIE 8451, Software and Cyberinfrastructure for Astronomy II, 845116 (24 September 2012); https://doi.org/10.1117/12.925431

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available