Exploring deep learning for multimodal understanding

Lao, M. (2023)

Doctoral Thesis

This thesis mainly focuses on multimodal understanding and Visual Question Answering (VQA) via deep learning methods. For technical contributions, this thesis first focuses on improving multimodal fusion schemes via multi-stage vision-language interactions. Then, the thesis seeks to overcome the language bias challenges to build robust VQA models, and also extend the bias problem into the more complex audio-visual-textual question answering tasks. Furthermore, this thesis explores the open-world applicability of VQA algorithms from the aspects of lifelong learning and federated learning, thereby expanding the continuous and distributed training ability. The efficacy of the proposed methods in this thesis is verified by extensive experiments. This thesis also gives an overview of challenges, benchmarks and strategies for robust VQA algorithms.

• Multimodal Understanding
• Visual Question Answering
• Deep Learning

• All authors: Lao, M.
• Supervisor: Lew, M.S.; Plaat, A.
• Committee: Bäck, T.H.W.; Batenburg, J.; Wolstencroft, K.J.; Lelieveldt, B.P.F.; Snoek, C.G.M.
• Qualification: Doctor (dr.)
• Awarding Institution: Leiden Institute of Advanced Computer Science (LIACS), Faculty of Science, Leiden University
• Date: 2023-11-28
• ISBN (print): 9789464199826