Exploring the application of biomedical knowledge-bearing foundation model to veterinary downstream classification task

Abstract

With the growth of global pet ownership and spending, the veterinary industry faces the challenge of a critical shortage of competent veterinarians to provide timely and quality services for companion animals. While amplifying the number of veterinarians can be seen as an alternative, this measure alone is inadequate to fill the voids of veterinarian competency in the coming years. One of the major reasons is proficiency in veterinary medicine needs several years of experience beyond the mandatory eight years of education. Recently, artificial intelligence (AI) technology is widely believed to have the potential to assist human intelligence. Nevertheless, if such a technology is to be implemented in veterinary medicine as a means to alleviate veterinarian shortage, it needs to offer more capability than just learning the pattern in veterinary data, to catch up with the level of veterinarian knowledge. For this reason, we see a need to leverage the power of the AI model that has already been embedded with knowledge relevant to veterinary medicine as a technology that can assist the current veterinary capacity to handle the increasing clinical load. We assess PubMedBERT as an example of a model that bears the knowledge of biomedical science including those of veterinary healthcare. We leverage the use case of classifying pet disease given the laboratory test results which are presented in the form of structured data. Throughout our experiments, we observed that this kind of model indeed yields better performance than data-driven models without knowledge. We also found out that such superiority in performance comes from the knowledge specific that resides in the model complemented by the text serialization technique that properly projects the structured data into an unstructured type that can be well handled by the model. Additionally, we show that the knowledge embedded in this model is transferrable to a lightweight model through the knowledge distillation technique. This enables a smaller and more practical model (35 times smaller and 25 times faster than PubMedBERT when executed using CPU) that does not have prior knowledge about veterinary medicine to properly comprehend the contextual understanding of semantic features in the data. Further, we also show that models embedded with veterinary-relevant knowledge are generally proficient in handling tasks from other domains.