Full Content is available to subscribers

Subscribe/Learn More  >
Proceedings Article

Integrating the ODI-PPA scientific gateway with the QuickReduce pipeline for on-demand processing

[+] Author Affiliations
Michael D. Young, Arvind Gopu

Indiana Univ. (United States)

Ralf Kotulla

Univ. of Wisconsin-Milwaukee (United States)

Wilson Liu

WIYN Observatory (United States)

Proc. SPIE 9152, Software and Cyberinfrastructure for Astronomy III, 91522U (July 18, 2014); doi:10.1117/12.2056763
Text Size: A A A
From Conference Volume 9152

  • Software and Cyberinfrastructure for Astronomy III
  • Gianluca Chiozzi; Nicole M. Radziwill
  • Montréal, Quebec, Canada | June 22, 2014

abstract

As imaging systems improve, the size of astronomical data has continued to grow, making the transfer and processing of data a significant burden. To solve this problem for the WIYN Observatory One Degree Imager (ODI), we developed the ODI-Portal, Pipeline, and Archive (ODI-PPA) science gateway, integrating the data archive, data reduction pipelines, and a user portal. In this paper, we discuss the integration of the QuickReduce (QR) pipeline into PPA's Tier 2 processing framework. QR is a set of parallelized, stand-alone Python routines accessible to all users, and operators who can create master calibration products and produce standardized calibrated data, with a short turn-around time. Upon completion, the data are ingested into the archive and portal, and made available to authorized users. Quality metrics and diagnostic plots are generated and presented via the portal for operator approval and user perusal. Additionally, users can tailor the calibration process to their specific science objective(s) by selecting custom datasets, applying preferred master calibrations or generating their own, and selecting pipeline options. Submission of a QuickReduce job initiates data staging, pipeline execution, and ingestion of output data products all while allowing the user to monitor the process status, and to download or further process/analyze the output within the portal. User-generated data products are placed into a private user-space within the portal. ODI-PPA leverages cyberinfrastructure at Indiana University including the Big Red II supercomputer, the Scholarly Data Archive tape system and the Data Capacitor shared file system. © (2014) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Citation

Michael D. Young ; Ralf Kotulla ; Arvind Gopu and Wilson Liu
" Integrating the ODI-PPA scientific gateway with the QuickReduce pipeline for on-demand processing ", Proc. SPIE 9152, Software and Cyberinfrastructure for Astronomy III, 91522U (July 18, 2014); doi:10.1117/12.2056763; http://dx.doi.org/10.1117/12.2056763


Access This Proceeding
Sign in or Create a personal account to Buy this proceeding ($15 for members, $18 for non-members).

Figures

Tables

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Proceeding
Sign in or Create a personal account to Buy this proceeding ($15 for members, $18 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.