High-performance data processing using distributed computing on the SOLIS project

Stephen Wampler

doi:10.1117/12.460917

13 December 2002 High-performance data processing using distributed computing on the SOLIS project

Stephen Wampler

Proceedings Volume 4848, Advanced Telescope and Instrumentation Control Software II; (2002) https://doi.org/10.1117/12.460917
Event: Astronomical Telescopes and Instrumentation, 2002, Waikoloa, Hawai'i, United States

Abstract

The SOLIS solar telescope collects data at a high rate, resulting in 500 GB of raw data each day. The SOLIS Data Handling System (DHS) has been designed to quickly process this data down to 156 GB of reduced data. The DHS design uses pools of distributed reduction processes that are allocated to different observations as needed. A farm of 10 dual-cpu Linux boxes contains the pools of reduction processes. Control is through CORBA and data is stored on a fibre channel storage area network (SAN). Three other Linux boxes are responsible for pulling data from the instruments using SAN-based ringbuffers. Control applications are Java-based while the reduction processes are written in C++. This paper presents the overall design of the SOLIS DHS and provides details on the approach used to control the pooled reduction processes. The various strategies used to manage the high data rates are also covered.

Citation Download Citation

Stephen Wampler "High-performance data processing using distributed computing on the SOLIS project", Proc. SPIE 4848, Advanced Telescope and Instrumentation Control Software II, (13 December 2002); https://doi.org/10.1117/12.460917

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available