UTILIZATION OF MACHINE LEARNING TO IMPROVE PRODUCTIVITY OF INTERVENTIONS CLASSIFICATION FOR A UNIVERSITY TEACHING HOSPITAL IN SINGAPORE

UTILIZATION OF MACHINE LEARNING TO IMPROVE PRODUCTIVITY OF INTERVENTIONS CLASSIFICATION FOR A UNIVERSITY TEACHING HOSPITAL IN SINGAPORE

Beng Yi Sia *, C. W. Yap, Y. R. Chen, L. T. Tan, K. T. Ng, T. M. Ng, J. W. Lin, S. L. Yap and H. Y. Lim

Division of Pharmacy, Tan Tock Seng Hospital, Singapore.

ABSTRACT: Introduction: Pharmacists have an integral role in the medication use process in performing interventions. In TTSH, accurate classification of interventions is important for supporting a learning culture within the hospital. Pharmacists spend at least 80 hours to look through 10,000 interventions monthly to ensure accurate classification. In recent years, artificial intelligence (AI) has been increasingly used in healthcare and may potentially be used to perform interventions classification. Aim: We aim to develop an intervention classification algorithm which can assist pharmacists in accurate classification of interventions. Pharmacists’ time spent can be reduced substantially and be better channeled to other higher leveled tasks to improve patient’s care. Methods: In designing the model for intervention classification algorithm, at least 80 000 pharmacist-checked intervention categories (PIC) were sent to Health Services & Outcomes Research, a division under the National Healthcare Group of Singapore (NHG). AI-predicted intervention categories (AIC) with a precision >90% matched PIC at least 95% of the time and exempted from manual pharmacists’ checking. Results: After utilization of the ML algorithm, 28% of inpatient interventions were exempted from pharmacists’ checking, corresponding to average time savings of 15.7 hours monthly and cost savings of $485.27. 33% of outpatient interventions were exempted from pharmacists’ checking, achieving an average time savings of 9.7 hours monthly and cost savings of $299.30. Conclusion: A real-world implementation of a ML model to classify interventions had led to significant savings on pharmacists’ time which could be channeled to perform clinical tasks to optimize patient outcomes.

Keywords: Machine learning, interventions, classification, artificial intelligence, pharmacy

INTRODUCTION: The medication use process is complicated and relies on the support and input from various professional disciplines to optimize patient safety. Pharmacists hold pivotal roles inmediation-related responsibilities such as ensuring access to medication, evaluating appropriateness of medication and medication management ¹.

Studies have shown that prescribing errors account for 39% of errors in the medication use process and can be due to several reasons such as prescriber’s lack of knowledge of the drug, slips and lapses ^{2, 3}. These errors are identified and resolved by pharmacists and are described as interventions.

Pharmacy interventions have been proven to reduce medication errors during hospitalization ⁴, reduce medication errors by 37% at transitions of care, and reduce frequency of emergency department visits after discharge ⁵. Tan Tock Seng Hospital (TTSH) is a university teaching hospital in the central of Singapore, with a capacity of 1700 beds and 16 specialist centers. Pharmacists work alongside physicians and initiate interventions to prevent prescribing errors. Interventions are documented and classified by the pharmacist performing the intervention according to an in-house definition of 25 different categories within the electronic medical records system – Electronic Inpatient Medication Record System (eIMR) for admitted patients, and Integrated Pharmacy (iPharm) for outpatient prescriptions. This data is used as recording of pharmacy workload and is further reported to the medication safety committee for analysis and creation of education materials and workflow changes to improve prescribing behaviors that leads to these interventions. It is important to have appropriate categorization to ensure accurate reports that can be used to inform practice change. As such, a team of 18 pharmacists spend at least 80 hours per month to review and re-classify at least 10,000 interventions monthly. Ensuring the accuracy of intervention categories is time and labor intensive, and may not be the best use of pharmacists’ time. Artificial intelligence (AI) could potentially be used in various parts of the medication management system, such as verification of orders, processing of prescriptions, procurement and monitoring ⁷, thereby improving patient safety outcomes ⁸ and quality of patient care ⁹. Several machine learning (ML) tools have been developed which can accurately predict outcomes based on learning patterns from vast healthcare data ^{10, 11}.

We aim to develop and implement a ML-based classification model which can be used to improve the productivity of interventions categories checking. Through the utilisation of ML to perform manual and routine tasks of reclassifying interventions, we hypothesized that pharmacists’ time spent on checking of interventions classification accuracy can be reduced substantially ¹³.

MATERIALS AND METHODS:

Model Development: To design the ML model, previously checked inpatient (Jun 2018 to Dec 2018) and outpatient (Apr 2018 to Dec 2018) interventions were used. The interventions were split into a training set and testing set. The training set contained 62757 inpatient interventions and 24626 outpatient interventions, and the testing set contained 5654 inpatient interventions and 2910 outpatient interventions (Supplementary Table 1). Separate models were developed for inpatient and outpatient interventions because of the variations in interventions across each category due to different practice settings. The interventions were first pre-processed to prepare it for the ML algorithm. Rules using regular expressions were used for standardization of the interventions. For example, commas were removed from numbers, and abbreviations such as Mg were converted to magnesium while mg were converted to milligram. The interventions were then converted to lower case and then tokenized into individual terms. Plural terms were converted to their singular form and wrong spellings were corrected. Some individual terms were then replaced with trigrams (e.g. “liver function test”) or bigrams (e.g. “salicylic acid”). Lemmatization of the individual terms were then performed using Word Net Lemmatizer in the Python library NLTK. Word embedding were performed on the tokens (individual terms, trigrams and bigrams) from the training set using GloVealgorithm implemented in the Python library genism ¹⁴.

Briefly, the GloVe algorithm calculated a vector with 300 dimensions for each token in the training set. The tokens in each intervention in the training set were then converted to these word embedded tokens for the subsequent ML algorithm. Only the first 60 tokens were retained for each intervention. Interventions with more than 60 tokens were truncated and interventions with fewer than 60 tokens were post-padded with zeros. Word embedding were also applied on the testing set using the learned GloVe algorithm. Y hot encoding was performed on the 25 intervention categories. Attention-based bidirectional long short-term memory ¹⁵ were then used to develop the ML-based classification model using the training set. The optimal value for batch size (128, 256, or 512), learning rate (10-6 to 10-1), regularizer (0 to 0.1), dropout (0 to 0.7) and optimizer (ADAM, SGD, or RMSProp) were determined using hyper parameter optimization with categorical cross entropy.

Utilisation of ML-based Classification Model: Monthly interventions were sent to the ML model, where it would return with the AI-predicted intervention category (AIC) with an associated precision, ranging from 0 – 100%, where precision is derived by number of correctly classified interventions in that category divided by the total number of interventions in that category. From January to June 2019, pharmacists continued to manually check intervention categories and returned the checked interventions for further refinement of the ML model. Thereafter, pharmacy used the interventions from the refined ML model and compared AIC to pharmacist-checked intervention categories (PIC). The percentage of match between AIC and PIC for each corresponding precision value was computed by the pharmacy team and defined as accuracy. Seeing that AIC matched PIC more than 95% of the time for precision > 90%, pharmacy decided to adopt all AIC with a precision of > 90%, over-riding original pharmacist intervention categories.  Interventions where AIC was =<90% were returned to the pharmacists for manual checking.

Evaluation of Improvement in Productivity: With the use of ML model, there was reduction in the need to check for accuracy of interventions classification, corresponding to pharmacists’ time savings, as each inpatient pharmacist took an average of 25 seconds to check the accuracy of classification for an intervention, while each outpatient pharmacist took an average of 36 seconds. To evaluate the improvement in productivity, pharmacists’ time savings were translated to cost using pharmacist hourly salary estimated from published local data in June 2020¹⁶; the median monthly gross wage of full-time pharmacist was S$5440, corresponding to an hourly pay of S$30.91, based on up to 44 hours weekly of contractual work hours for a full-time employee ¹⁷.

RESULTS:

Precision of Initial ML-based Classification Model: The precision of the ML-based classification model on the inpatient training set ranges from 23.2% to 100% for the 25 intervention categories, with an overall precision of 62.4%. On the other hand, the precision of the ML algorithm on the outpatient training set ranges from 8.3% to 100% with an overall precision of 69.3%. There was variation across the different categories for both inpatient and outpatient models, with some categories unable to consistently return a high precision (Table 1).

TABLE 1: PRECISION OF ML MODEL ON TRAINING SET (INPATIENT AND OUTPATIENT)

NA – data is not available because there are no interventions under that category. Precision is derived by number of data points of that category which are classified correctly by the model divided by the total number of data points classified by the model into that category.

Adoption of Refined ML-based Classification Model: Using the refined ML model and computing the percent of match between AIC and PIC, we found that accepting AIC with a precision of >90%matched PIC at least 95% of the time (Table 2). Hence, pharmacy accepted a 5% error in classification and accepted all AIC with precision >90% to override PIC for both inpatient and outpatient settings.  All other interventions were returned for pharmacists’ checking.

TABLE 2: ACCURACY OF AIC OVER 6 MONTHS FOR INTERVENTIONS WITH PRECISION > 90% (INPATIENT AND OUTPATIENT)

Accuracy is defined as number of Artificial intelligence predicted intervention categories (AIC) which match with Pharmacist-checked intervention categories (PIC).

Time and Cost Savings: After utilization of the ML-based classification model, inpatient pharmacy managed to adopt an average of 2256 AIC monthly (28% of total interventions),corresponding to average time savings of 15.7 hours monthly which translates to cost savings of SGD $485.27. Encouraging results were also replicated for outpatient pharmacy, with an average of 33% of AIC utilized, achieving an average time savings of 9.7 hours monthly and cost savings of SGD $299.30 (Table 3).

TABLE 3: TIME SAVINGS FOR TTSH PHARMACISTS

^aAverage of 25 seconds taken to check each intervention, ^bAverage of 36 seconds taken to check each intervention, ^cMissing data of April to June due to COVID-19 situation outbreak in TTSH, and interventions were substantially lower than average, ^dBased on gross median monthly wage of $5440 of a full-time pharmacist working up to 44 hours / week.

DISCUSSION: To our knowledge, this is the first time AI has been utilized for the classification of pharmacy interventions. We presented a real-world use case common in pharmacy practices ^{5, 18, 19}. The implementation of AI in our hospital has shown utility is its ability to reliably predict intervention category based on picking up of commonly used words in the pharmacists’ interventions. AI has been increasing adopted in healthcare systems, such as medical devices ²⁰, diagnosis ¹², medication management process ⁶, analysis of big datasets ²¹, robotically-assisted surgeries ²² and mitigating threats to patient safety ²³. By utilizing AI and ML to perform routine tasks, evaluating large complex datasets, pharmacists time can be directed towards providing clinical care and expanding their job scopes. Across the different intervention categories, AI was able to more reliably predict certain categories (E.g. categories 8, 15 and 18) because the documentation of certain interventions was more consistent and increases the classification precision of the AI algorithm. The lack of sufficient data for certain intervention categories (Table 1, Intervention categories 9, 11, 24 and 25 for inpatient and intervention categories 6-9, 11, 17, 19 and 20 for outpatient) affected the robustness of the AI algorithm development.

Limitations: Extracting clinical information from free text is a major challenge faced in the derivation and implementation of the ML algorithm, as shared by other authors ²⁴.

Training on intervention documentation has to be done to reduce the variations in documentation which affects model prediction. Extensive data is required in the development of the ML algorithm, and pharmacy has to invest resources in implementing and validating the AI model.

Future Plans: TTSH will be transitioning from our current electronic systems to a harmonized system, next generation electronic medication record (NGEMR). In preparation for the smooth transition, tremendous effort and time had been put in to harmonize the intervention categories across various hospitals within Singapore ²⁵.

To facilitate the development of a robust algorithm and automate the classification of interventions, pharmacists and technicians will be encouraged to utilize the SBAR ²⁶ format: situation, background, assessment and recommendation for documentation of interventions, as well as using common phrases which are associated with the intervention categories. It is hoped that a single ML-based classification model can be developed and used across the hospitals sharing the NGEMR system, to provide greater time savings among pharmacists, and ensure consistency in categorizing an intervention.

CONCLUSION: A real-world implementation of a machine learning model to classify interventions in TTSH had led to significant reduction in pharmacists’ time averaging 15.7 hours monthly which could be channeled to perform other value-added work.

As we digitalize and unify our medical data, it is hoped that the ML-based classification model can progress beyond TTSH and be shared among other institutions who are using a harmonized system of documenting pharmacy interventions. We envision that in future, ML can be used to predict accurately all pharmacy interventions, and take away the routine duty of classifying and checking the interventions for accuracy.

ACKNOWLEDGEMENTS: Nil

CONFLICTS OF INTEREST: Nil

REFERENCES:

1. American pharmacists association. Pharmacists’ impact on patient safety: A joint project of the American Pharmacists Association Academy of Pharmacy practice and management and academy of pharmaceutical research and science. https://www.pharmacist.com/pharmacists-impact-patient-safety. (Accessed 2021 July 1)
2. Schneider P: Opportunities for pharmacy. American Journal of Health-system Pharmacy 2007; 64(9): S10-6.
3. van den Bemt PMLA: Drug-related problems in hospitalized patients. Drug Safety 2000; 22(4): 321-333.
4. Ng TM, Teo CJ and Heng ST: Impact of round the clock pharmacist inpatient medication chart review on medication errors. Journal of the American College of Clinical Pharmacy 2020; 3: 1437-1443.
5. De Oliveira GS, Castro-Alves LJ, Kendall MC and McCarthy R: Effectiveness of pharmacist intervention to reduce medication errors and healthcare resources utilization after transitions of care: a meta-analysis of randomized controlled trials. J of Patie Safety 2017; epub.
6. Nelson SD, Walsh CG and Olsen CA: Demystifying artificial intelligence in pharmacy. American Journal of Health-system Pharmacy 2020; 77(19): 1556–1570.
7. Stead WW. Clinical implications and challenges of artificial intelligence and deep learning. The Journal of the American Medical Association 2018; 320(11):1107-1108.
8. Macrae C: Governing the safety of artificial intelligence in healthcare. BMJ Quality & Safety 2019; 28(6): 495-498.
9. Grossman L, Choi S and Collins S: Implementation of acute care patient portals: recommendations on utility and use from six early adopters. Journal of the American Medical Informatics Association 2018; 25(4): 370-379.
10. Murphy KP: Machine learning: a probabilistic perspective. Cambridge 2012; MA: MIT Press.
11. Beam AL and Kohane IS: Big data and machine learning in health care. The Journal of the American Medical Association 2018; 319(13): 1317-1318.
12. Executive summary of the 2019 ASH commission on goals: impact of artificial intelligence on healthcare and pharmacy practice. American Journal of Health-system Pharmacy 2019; 76(24): 2087-2092.
13. Topol EJ: High-performance medicine: the convergence of humanand artificial intelligence. Nature Medicine 2019; 25(1): 44-56.
14. Pennington J, Socher R and Manning CD: GloVe: Global Vectors for Word Representation 2014.
15. Zhou P, Shi W and Tian J: Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. Association for Computational Linguistics 2016; 207-212.
16. Ministry of Manpower, Singapore. Occupational wages by industry. (June 2021). http://stats.mom.gov.sg/Pages/Occupational-Wages-Tables2020.aspx (Accessed 2021 Oct 19)
17. Ministry of Manpower, Singapore. Employment practices. (November 2020).http://www.mom.gov.sg/employment-practices/hours-of-work-overtime-and-rest-days (Accessed 2021 Oct 19)
18. Hawes EM, Maxwell WD and White SF: Impact of an outpatient pharmacist intervention on medication discrepancies and health care resource utilization in post hospitalization care transitions. Journal of Primary Care & Community Health 2014; 5: 14-8.
19. Chew C, Chiang J and Yeoh TT: Impact of outpatient interventions made at an ambulatory cancer centre oncology pharmacy in Singapore. Journal of Oncology Pharmacy Practice 2015; 21(2): 93-101.
20. FDA Centre for devices and radiological health. Executive summary for the patient engagement advisory committee meeting. Artificial intelligence (AI) and machine learning (ML) in medical devices (October 2020).https://www.fda.gov/media/151482/download (Accessed 2021 Oct 17).
21. Rio‑Bermudez CD, Medrano IH, Yebes L and Poveda JL: Towards a symbiotic relationship between big data, artificial intelligence and hospital pharmacy. Journal of Pharmaceutical Policy and Practice 2020; 13: 75.
22. Davenport TH and Glaser J: Just-in-time delivery comes to knowledge management. Harvard Business Review 2002; 80: 107-126.
23. Dalal AK, Fuller T and Garabedian P: Systems engineering and human factors support ofa system of novel EHR-integrated tools to prevent harm in the hospital. J of the American Med Infor Associan 2019; 26(6): 553-560.
24. Bates DW, Saria S, Ohno-Machado L, Shah A and Escobar G: Big data in healthcare: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014; 33(7): 1123–31.
25. Ng TM, Hing WC and Koh TY: Merits of a harmonised system to classify drug-related problems in Singapore. Annals of the Academy of Med Singapore 2021; 50: 572-7
26. Ruhomauly Z, Betts K and Jayne-Coupe K: Improving the quality of handover: implementing SBAR. Future Healthcare Journal 2019; 6(2): 54.
    

    ---

    How to cite this article:

    Sia BY, Yap CW, Chen YR, Tan LT, Ng KT, Ng TM, Lin JW, Yap SL and Lim HY: Utilization of machine learning to improve productivity of interventions classification for a university teaching Hospital in Singapore. Int J Pharm Sci & Res 2023; 14(10): 4965-70. doi: 10.13040/IJPSR.0975-8232.14(10).4965-70.

    ---

    All © 2023 are reserved by International Journal of Pharmaceutical Sciences and Research. This Journal licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.