The LOFAR radio telescope produces petabytes of data every year. Radio Astronomers use complex multi-step pipelines to process this data and produce scientific images. In this thesis, we describe... Show moreThe LOFAR radio telescope produces petabytes of data every year. Radio Astronomers use complex multi-step pipelines to process this data and produce scientific images. In this thesis, we describe methods for astronomers to efficiently process massive amounts of data on high-throughput clusters. We gain insights into efficient acceleration of complex scientific pipelines, as well as understanding of the scalability of the underlying software. These insights ensure the scientific output of LOFAR keeps up with the pace of the data the telescope produces. Moreover, our results can be applied to arbitrary scientific studies, greatly increasing the potential of the LOFAR telescope. Finally, our discoveries can be applied to current and future telescopes. These lessons will help astronomers efficiently producing science in the big data era. Show less
The LOFAR radio telescope is a low-frequency aperture synthesis radio telescope with headquarters in the Netherlands and stations across Europe. As a general purpose telescope, LOFAR produces... Show moreThe LOFAR radio telescope is a low-frequency aperture synthesis radio telescope with headquarters in the Netherlands and stations across Europe. As a general purpose telescope, LOFAR produces petabytes of data each year serving a wide range of science cases. The data volumes produced are difficult or impossible to process on a single machine or even a small cluster at a scientific institute. We provide a layout for serving LOFAR processing to the astronomical community by providing access to LOFAR pipelines accelerated on a high throughput platform. We build this on our previous success with parallelizing the LOFAR Surveys pipeline and with creating automated LOFAR workflows on a distributed architecture. The LOFAR As A Service platform will serve the LOFAR Key Science Projects (KSPs), specifically the LOFAR Surveys KSP, which aims to provide science ready products to the scientific community. Additionally, this system will provide a robust method to re-process LOFAR data with a single click. Show less
Rowlinson, A.; Gourdji, K.; Meulen, K. van der; Meyers, Z.S.; Shimwell, T.W.; Veen, S.; ... ; Williams, W.L. 2019
We present a modular framework, the Workload Characterisation Framework (WCF), that is developed to obtain, store and compare key characteristics of radio astronomy processing software in a... Show moreWe present a modular framework, the Workload Characterisation Framework (WCF), that is developed to obtain, store and compare key characteristics of radio astronomy processing software in a reproducible way. As a demonstration, we discuss the experiences using the framework to characterise a LOFAR calibration and imaging pipeline. Show less
Cassano, R.; Botteon, A.; Di Gennaro, G.; Brunetti, G.; Sereno, M.; Shimwell, T.W.; ... ; Tasse, C. 2019
The Low Frequency Array (LOFAR) radio telescope is an international aperture synthesis radio telescope used to study the Universe at low frequencies. One of the goals of the LOFAR telescope is to... Show moreThe Low Frequency Array (LOFAR) radio telescope is an international aperture synthesis radio telescope used to study the Universe at low frequencies. One of the goals of the LOFAR telescope is to conduct deep wide-field surveys. Here we will discuss a framework for the processing of the LOFAR Two Meter Sky Survey (LoTSS). This survey will produce close to 50 PB of data within five years. These data rates require processing at locations with high-speed access to the archived data. To complete the LoTSS project, the processing software needs to be made portable and moved to clusters with a high bandwidth connection to the data archive. This work presents a framework that makes the LOFAR software portable, and is used to scale out LOFAR data reduction. Previous work was successful in pre-processing LOFAR data on a cluster of isolated nodes. This framework builds upon it and and is currently operational. It is designed to be portable, scalable, automated and general. This paper describes its design and high level operation and the initial results processing LoTSS data. Show less
