In this thesis, topics relating to the optimization of high-throughput pipelines used for imaging are discussed. In particular, different levels of implementation, i.e., conceptual, software, and... Show moreIn this thesis, topics relating to the optimization of high-throughput pipelines used for imaging are discussed. In particular, different levels of implementation, i.e., conceptual, software, and hardware, are discussed and the thesis outlines how advances on each level need to be made to make gains in computationally demanding imaging applications. Chapter 2 concerns the implementation of real-time segmentation of X-ray computed tomography (CT). Chapter 3 concerns the end-to-end optimization of various CT workflows by using auto-differentiation frameworks. Chapter 4 concerns a novel pruning method for neural network to significantly increase the speed of convolutional neural networks (CNNs). Chapter 5 comprises a benchmark study of optimization algorithms for tuning GPU kernels, and introduces a novel graph-based approach to quantify search space difficulty. Chapter 6 introduces a novel model to improve the energy efficiency of GPUs. Show less
Dreuning, H.; Bal, H.E.; Nieuwpoort, R.V. van 2023
Deep Learning (DL) model sizes are increasing at a rapid pace, as larger models typically offer better statistical performance. Modern Large Language Models (LLMs) and image processing models... Show moreDeep Learning (DL) model sizes are increasing at a rapid pace, as larger models typically offer better statistical performance. Modern Large Language Models (LLMs) and image processing models contain billions of trainable parameters. Training such massive neural networks incurs significant memory requirements and financial cost. Hybrid-parallel training approaches have emerged that combine pipelining with data and tensor parallelism to facilitate the training of large DL models on distributed hardware setups. However, existing approaches to design a hybrid-parallel partitioning and parallelization plan for DL models focus on achieving high throughput and not on minimizing memory usage and financial cost. We introduce CAPTURE, a partitioning and parallelization approach for hybrid parallelism that minimizes peak memory usage. CAPTURE combines a profiling-based approach with statistical modeling to recommend a partitioning and parallelization plan that minimizes the peak memory usage across all the Graphics Processing Units (GPUs) in the hardware setup. Our results show a reduction in memory usage of up to 43.9% compared to partitioners in state-of-the-art hybridparallel training systems. The reduced memory footprint enables the training of larger DL models on the same hardware resources and training with larger batch sizes. CAPTURE can also train a given model on a smaller hardware setup than other approaches, reducing the financial cost of training massive DL models. Show less
Atoms and molecules are the basic units of matter. If we keep dividing a bar of gold or a glass of water into smaller parts, at the end we are left with a single gold atom or a water molecule. We... Show moreAtoms and molecules are the basic units of matter. If we keep dividing a bar of gold or a glass of water into smaller parts, at the end we are left with a single gold atom or a water molecule. We could not divide them further without them losing their identity. Molecular electronics is the study of how electrons, which are fundamental particles in nature, flow through these basic units of the matter once connected between two electrical leads. At such small dimensions matter starts to lose its macroscopic properties and its behaviour is governed by the rules of quantum mechanics. A molecule can have various electronic and mechanical degrees of freedom (or eigenstates) with discrete energies. The electrons flowing through the molecule can interact with these degrees of freedom making single molecule devices. Better control and understanding of these interactions and study of how the atomic structure of the macroscopic electrical leads affect the electronic-transport forms the main focus of this thesis. Controlled manipulation of single atoms and molecules using a low-temperature scanning tunnelling microscope, probing charge transport using high-frequency shot-noise measurement and the use of graphene as possible electrodes are the three directions which are investigated here. Show less
Polymers are the main building blocks of many biological systems, and thus polymer models are important tools for our understanding. One such biological system is the large scale organisation of... Show morePolymers are the main building blocks of many biological systems, and thus polymer models are important tools for our understanding. One such biological system is the large scale organisation of chromatin. A key question here, is how during cell division the chromosomes can separate without entanglement and knotting. One proposal is that this achieved by a specific spatial organisation of the chromosomes, known as the "fractal globule". Using Monte Carlo simulations, we found that fractal globules are unstable and thus cannot represent the biological system without further ingredients. Another proposal is that topological effects cause spatial separation of the chromosomes. These topological effects can be studied using simulations of nonconcatenated ring polymers. Using a compute device called the Graphics Processing Unit, very detailed and long simulations were carried out. From these a picture emerged in which ring polymers behave much slower than was found in previous studies. A second biological system studied here is the folded state of the protein. This is modeled by the Hamiltonian walk. Here, instead of simulations, we exactly enumerated all Hamiltonian walks of the 4x4x4 cube. Interestingly, simulations show that for larger systems many more walks exist than previously estimated. Show less
The increased availability of accelerator technology in modern supercomputers forces users to redesign their algorithms. These accelerators are specifically designed to offer huge amounts of... Show moreThe increased availability of accelerator technology in modern supercomputers forces users to redesign their algorithms. These accelerators are specifically designed to offer huge amounts of parallel compute power. In this thesis I show how to harness the power of these parallel processors for astrophysical simulations. I start with an introduction that presents the developments in astrophysical algorithms and used hardware since the 1960__s till today. In the following scientific chapters I discuss the use of GPU accelerator technology for direct N-body methods and for the more advanced hierarchical algorithms. These advanced algorithms are more complex to implement on large parallel architectures, but by redesigning the algorithms it is possible to take advantage of the GPU. The developed algorithms are applied to simulate galaxy mergers to explain discrepancies in observational results. In the simulations we test different merger configurations and try to match the results with observational data. The final chapter shows how to scale the developed software code to thousands of GPUs as available in the Titan supercomputer. The in this thesis developed and presented algorithms allow astronomers to take advantage of the new GPU technology and thereby run simulations that contain thousand times more particles than was possible before Show less
Heterogeneous computing platforms support the traditional types of parallelism, such as e.g., instruction-level, data, task, and pipeline parallelism, and provide the opportunity to exploit a... Show moreHeterogeneous computing platforms support the traditional types of parallelism, such as e.g., instruction-level, data, task, and pipeline parallelism, and provide the opportunity to exploit a combination of different types of parallelism at different platform levels. The architectural diversity of platform components makes tapping into the platform potential a challenging programming task. This thesis makes an important step in this direction by introducing a novel methodology for automatic generation of structured, multi-level parallel programs from sequential applications. We introduce a novel hierarchical intermediate program representation (HiPRDG) that captures the notions of structure and hierarchy in the polyhedral model used for compile-time program transformation and code generation. Using the HiPRDG as the starting point, we present a novel method for generation of multi-level programs (MLPs) featuring different types of parallelism, such as task, data, and pipeline parallelism. Moreover, we introduce concepts and techniques for data parallelism identification, GPU code generation, and asynchronous data-driven execution on heterogeneous platforms with efficient overlapping of host-accelerator communication and computation. By enabling the modular, hybrid parallelization of program model components via HiPRDG, this thesis opens the door for highly efficient tailor-made parallel program generation and auto-tuning for next generations of multi-level heterogeneous platforms with diverse accelerators. Show less
