BioGPT: generative pre-trained transformer for biomedical text generation and mining

🤖 Plain-English Summary

Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record.

🔑 Key Findings

• Among the two main branches of pre-trained language models in the general language domain, i.e.
• BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT.
• While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope.

🔗 Related Resources