This thesis mainly focuses on multimodal understanding and Visual Question Answering (VQA) via deep learning methods. For technical contributions, this thesis first focuses on improving multimodal... Show moreThis thesis mainly focuses on multimodal understanding and Visual Question Answering (VQA) via deep learning methods. For technical contributions, this thesis first focuses on improving multimodal fusion schemes via multi-stage vision-language interactions. Then, the thesis seeks to overcome the language bias challenges to build robust VQA models, and also extend the bias problem into the more complex audio-visual-textual question answering tasks. Furthermore, this thesis explores the open-world applicability of VQA algorithms from the aspects of lifelong learning and federated learning, thereby expanding the continuous and distributed training ability. The efficacy of the proposed methods in this thesis is verified by extensive experiments. This thesis also gives an overview of challenges, benchmarks and strategies for robust VQA algorithms. Show less
Liu, X.; Ye, K.; Vlijmen, H. van; IJzerman, A.P.; Westen, G.J.P. van 2023
Rational drug design often starts from specific scaffolds to which side chains/substituents are added or modified due to the large drug-like chemical space available to search for novel drug-like... Show moreRational drug design often starts from specific scaffolds to which side chains/substituents are added or modified due to the large drug-like chemical space available to search for novel drug-like molecules. With the rapid growth of deep learning in drug discovery, a variety of effective approaches have been developed for de novo drug design. In previous work we proposed a method named DrugEx, which can be applied in polypharmacology based on multi-objective deep reinforcement learning. However, the previous version is trained under fixed objectives and does not allow users to input any prior information (i.e. a desired scaffold). In order to improve the general applicability, we updated DrugEx to design drug molecules based on scaffolds which consist of multiple fragments provided by users. Here, a Transformer model was employed to generate molecular structures. The Transformer is a multi-head self-attention deep learning model containing an encoder to receive scaffolds as input and a decoder to generate molecules as output. In order to deal with the graph representation of molecules a novel positional encoding for each atom and bond based on an adjacency matrix was proposed, extending the architecture of the Transformer. The graph Transformer model contains growing and connecting procedures for molecule generation starting from a given scaffold based on fragments. Moreover, the generator was trained under a reinforcement learning framework to increase the number of desired ligands. As a proof of concept, the method was applied to design ligands for the adenosine A2A receptor (A2AAR) and compared with SMILES-based methods. The results show that 100% of the generated molecules are valid and most of them had a high predicted affinity value towards A2AAR with given scaffolds. Show less
Sbrollini, A.; Barocci, M.; Mancinelli, M.; Paris, M.; Raffaelli, S.; Marcantoni, I.; ... ; Burattini, L. 2023
Heart failure (HF) diagnosis, typically visually performed by serial electrocardiography, may be supported by machine-learning approaches. Repeated structuring & learning procedure (RS & LP... Show moreHeart failure (HF) diagnosis, typically visually performed by serial electrocardiography, may be supported by machine-learning approaches. Repeated structuring & learning procedure (RS & LP) is a constructive algorithm able to automatically create artificial neural networks (ANN); it relies on three parameters, namely maximal number of hidden layers (MNL), initializations (MNI) and confirmations (MNC), arbitrarily set by the user. The aim of this study is to evaluate RS & LP robustness to varying values of parameters and to identify an optimized combination of parameter values for HF diagnosis. To this aim, the Leiden University Medical Center HF data-base was used. The database is constituted by 129 serial ECG pairs acquired in patients who experienced myocardial infarction; 48 patients developed HF at follow-up (cases), while 81 remained clinically stable (controls). Overall, 15 ANNs were created by considering 13 serial ECG features as inputs (extracted from each serial ECG pair), 2 classes as outputs (cases/controls), and varying values of MNL (1, 2, 3, 4 and 10), MNI (50, 250, 500, 1000 and 1500) and MNC (2, 5, 10, 20 and 50). The area under the curve (AUC) of the receiver operating characteristic did not significantly vary with varying parameter values (P >= 0.09). The optimized combination of parameter values, identified as the one showing the highest AUC, was obtained for MNL = 3, MNI = 500 and MNC = 50 (AUC = 86 %; ANN structure: 3 hidden layers of 14, 14 and 13 neurons, respectively). Thus, RS & LP is robust, and the optimized ANN represents a potentially useful clinical tool for a reliable auto-matic HF diagnosis. Show less
Dreuning, H.; Bal, H.E.; Nieuwpoort, R.V. van 2023
Deep Learning (DL) model sizes are increasing at a rapid pace, as larger models typically offer better statistical performance. Modern Large Language Models (LLMs) and image processing models... Show moreDeep Learning (DL) model sizes are increasing at a rapid pace, as larger models typically offer better statistical performance. Modern Large Language Models (LLMs) and image processing models contain billions of trainable parameters. Training such massive neural networks incurs significant memory requirements and financial cost. Hybrid-parallel training approaches have emerged that combine pipelining with data and tensor parallelism to facilitate the training of large DL models on distributed hardware setups. However, existing approaches to design a hybrid-parallel partitioning and parallelization plan for DL models focus on achieving high throughput and not on minimizing memory usage and financial cost. We introduce CAPTURE, a partitioning and parallelization approach for hybrid parallelism that minimizes peak memory usage. CAPTURE combines a profiling-based approach with statistical modeling to recommend a partitioning and parallelization plan that minimizes the peak memory usage across all the Graphics Processing Units (GPUs) in the hardware setup. Our results show a reduction in memory usage of up to 43.9% compared to partitioners in state-of-the-art hybridparallel training systems. The reduced memory footprint enables the training of larger DL models on the same hardware resources and training with larger batch sizes. CAPTURE can also train a given model on a smaller hardware setup than other approaches, reducing the financial cost of training massive DL models. Show less
