Synthetic4Health: generating annotated synthetic clinical letters

Abstract

Since clinical letters contain sensitive information, clinical-related datasets cannot be widely applied in model training, medical research, and education. This work aims to generate reliable, diverse, and de-identified synthetic clinical letters. We explored various pre-trained language models (PLMs) for text masking and generation.

After that, we worked on Bio\_ClinicalBERT, a high-performing model, and experimented with different masking strategies. Both qualitative and quantitative methods were used for evaluation. A downstream task, Named Entity Recognition (NER), was further implemented to assess the usability of these synthetic letters. We also applied clinically focused evaluation methods - including BioGPT and GPT-3.5-turbo - to evaluate the clinical performance of the synthetic texts.

The results indicate that 1) encoder-only models outperform encoder-decoder models. 2) Among encoder-only models, those trained on general corpora perform comparably to those trained on clinical data when clinical information is preserved. 3) Preserving clinical entities and document structure aligns more closely with our objectives than simply fine-tuning the model. 4) Masking strategies have a noticeable impact on the quality of synthetic clinical letters: masking stopwords has a positive impact, while masking nouns or verbs has a negative effect. 5) BERTScore should be the primary quantitative evaluation metric, with other metrics serving as supplementary references. 6) Contextual information has only a limited effect on the models' understanding, suggesting that synthetic letters can effectively substitute real ones in downstream NER tasks. 7) Although the model occasionally generates hallucinated content, it appears to have little effect on overall clinical performance. Unlike previous research, which primarily focuses on reconstructing the original letters by training language models, this paper provides a foundational framework for generating diverse, de-identified clinical letters. It offers a direction for utilising the model to process real-world clinical letters, thereby helping to expand datasets in the clinical domain. Our codes and trained models are available at \url{https://github.com/HECTA-UoM/Synthetic4Health}