TY - JOUR AU - Rumoro, Dino P. AU - Shah, Shital C. AU - Gibbs, Gillian S. AU - Hallock, Marilyn M. AU - Trenholme, Gordon M. AU - Waddell, Michael J. PY - 2017/05/02 Y2 - 2024/03/28 TI - Utility of Natural Language Processing for Clinical Quality Measures Reporting JF - Online Journal of Public Health Informatics JA - OJPHI VL - 9 IS - 1 SE - Novel algorithms, statistical or mathematical methods DO - 10.5210/ojphi.v9i1.7605 UR - https://ojphi.org/ojs/index.php/ojphi/article/view/7605 SP - AB - <div style="left: 90px; top: 340.758px; font-size: 14.1667px; font-family: sans-serif; transform: scaleX(1.08129);" data-canvas-width="63.778333333333336">Objective</div><div style="left: 105px; top: 355.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.979675);" data-canvas-width="378.00775000000004">To explain the utility of using an automated syndromic surveillance</div><div style="left: 90px; top: 372.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.972855);" data-canvas-width="392.8388333333332">program with advanced natural language processing (NLP) to improve</div><div style="left: 90px; top: 389.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.00292);" data-canvas-width="358.44499999999977">clinical quality measures reporting for influenza immunization.</div><div style="left: 90px; top: 420.758px; font-size: 14.1667px; font-family: sans-serif; transform: scaleX(1.11768);" data-canvas-width="82.63416666666666">Introduction</div><div style="left: 105px; top: 435.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.0032);" data-canvas-width="378.65941666666663">Clinical quality measures (CQMs) are tools that help measure and</div><div style="left: 90px; top: 452.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.968407);" data-canvas-width="394.9326666666665">track the quality of health care services. Measuring and reporting</div><div style="left: 90px; top: 469.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.994994);" data-canvas-width="396.05750000000023">CQMs helps to ensure that our health care system is delivering</div><div style="left: 90px; top: 485.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.00627);" data-canvas-width="393.68458333333325">effective, safe, efficient, patient-centered, equitable, and timely care.</div><div style="left: 90px; top: 502.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.967116);" data-canvas-width="394.8250000000002">The CQM for influenza immunization measures the percentage of</div><div style="left: 90px; top: 519.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.04842);" data-canvas-width="394.85333333333324">patients aged 6 months and older seen for a visit between October</div><div style="left: 90px; top: 535.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.966923);" data-canvas-width="395.50358333333315">1 and March 31 who received (or reports previous receipt of) an</div><div style="left: 90px; top: 552.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.00658);" data-canvas-width="393.7412499999999">influenza immunization. Centers for Disease Control and Prevention</div><div style="left: 90px; top: 569.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.970707);" data-canvas-width="395.4497499999997">recommends that everyone 6 months of age and older receive an</div><div style="left: 90px; top: 585.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.03514);" data-canvas-width="389.7646666666667">influenza immunization every season, which can reduce influenza-</div><div style="left: 90px; top: 602.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.00107);" data-canvas-width="297.4858333333333">related morbidity and mortality and hospitalizations.</div><div style="left: 90px; top: 634.091px; font-size: 14.1667px; font-family: sans-serif; transform: scaleX(1.07287);" data-canvas-width="58.23916666666666">Methods</div><div style="left: 105px; top: 649.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.965998);" data-canvas-width="380.42599999999993">Patients at a large academic medical center who had a visit to</div><div style="left: 90px; top: 665.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.03952);" data-canvas-width="394.4963333333329">an affiliated outpatient clinic during June 1 - 8, 2016 were initially</div><div style="left: 90px; top: 682.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.967695);" data-canvas-width="394.9808333333334">identified using their electronic medical record (EMR). The 2,543</div><div style="left: 90px; top: 699.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.0166);" data-canvas-width="394.08408333333335">patients who were selected did not have documentation of influenza</div><div style="left: 90px; top: 715.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.03181);" data-canvas-width="394.4269166666668">immunization in a discrete field of the EMR. All free text notes for</div><div style="left: 90px; top: 732.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.966474);" data-canvas-width="395.3038333333334">these patients between August 1, 2015 and March 31, 2016 were</div><div style="left: 90px; top: 749.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.00131);" data-canvas-width="396.10000000000014">retrieved and analyzed using the sophisticated NLP built within</div><div style="left: 90px; top: 765.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.05482);" data-canvas-width="397.71924999999976">Geographic Utilization of Artificial Intelligence in Real-Time</div><div style="left: 90px; top: 782.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.02227);" data-canvas-width="396.298333333333">for Disease Identification and Alert Notification (GUARDIAN)</div><div style="left: 90px; top: 799.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.968073);" data-canvas-width="395.2641666666666">– a syndromic surveillance program – to identify any mention of</div><div style="left: 90px; top: 815.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.981075);" data-canvas-width="393.0839166666668">influenza immunization. The goal was to identify additional cases that</div><div style="left: 90px; top: 832.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.998442);" data-canvas-width="393.4026666666665">met the CQM measure for influenza immunization and to distinguish</div><div style="left: 90px; top: 849.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.975909);" data-canvas-width="394.8689166666667">documented exceptions. The patients with influenza immunization</div><div style="left: 90px; top: 865.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.02813);" data-canvas-width="397.06333333333333">mentioned were further categorized by GUARDIAN NLP into</div><div style="left: 90px; top: 882.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.04918);" data-canvas-width="397.7433333333332">Received, Recommended, Refused, Allergic, and Unavailable.</div><div style="left: 90px; top: 899.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.954023);" data-canvas-width="395.09274999999974">If more than one category was applicable for a patient, they were</div><div style="left: 90px; top: 915.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.03652);" data-canvas-width="394.8434166666664">independently counted in their respective categories. A descriptive</div><div style="left: 90px; top: 932.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.954017);" data-canvas-width="394.99500000000006">analysis was conducted, along with manual review of a sample of</div><div style="left: 90px; top: 949.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.00769);" data-canvas-width="136.87833333333333">cases per each category.</div><div style="left: 90px; top: 980.758px; font-size: 14.1667px; font-family: sans-serif; transform: scaleX(1.08488);" data-canvas-width="51.17">Results</div><div style="left: 105px; top: 995.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.03678);" data-canvas-width="379.53208333333316">For the 2,543 patients who did not have influenza immunization</div><div style="left: 90px; top: 1012.38px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.02622);" data-canvas-width="394.2569166666666">documentation in a discrete field of the EMR, a total of 78,642 free</div><div style="left: 90px; top: 1029.04px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.97704);" data-canvas-width="395.39166666666654">text notes were processed using GUARDIAN. Four hundred fifty</div><div style="left: 90px; top: 1045.71px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.01202);" data-canvas-width="393.9792499999999">three (17.8%) patients had some mention of influenza immunization</div><div style="left: 90px; top: 1062.38px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.982525);" data-canvas-width="392.9280833333334">within the notes, which could potentially be utilized to meet the CQM</div><div style="left: 90px; top: 1079.04px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.980147);" data-canvas-width="394.96666666666687">influenza immunization requirement. Twenty two percent (n=101)</div><div style="left: 90px; top: 1095.71px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.998688);" data-canvas-width="395.92999999999995">of patients mentioned already having received the immunization</div><div style="left: 90px; top: 1112.38px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.00177);" data-canvas-width="393.6123333333333">while 34.7% (n=157) patients refused it during the study time frame.</div><div style="left: 90px; top: 1129.04px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.01313);" data-canvas-width="393.98774999999995">There were 27 patients with the mention of influenza immunization,</div><div style="left: 90px; top: 1145.71px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.00827);" data-canvas-width="393.66333333333324">who could not be differentiated into a specific category. The number</div><div style="left: 90px; top: 1162.38px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.0336);" data-canvas-width="394.23283333333336">of patients placed into a single category of influenza immunization</div><div style="left: 90px; top: 1179.04px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.04065);" data-canvas-width="394.4920833333332">was 351 (77.5%), while 75 (16.6%) were classified into more than</div><div style="left: 90px; top: 1195.71px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.01369);" data-canvas-width="150.28000000000003">one category. See Table 1.</div><div style="left: 510px; top: 340.758px; font-size: 14.1667px; font-family: sans-serif; transform: scaleX(1.10336);" data-canvas-width="85.01416666666667">Conclusions</div><div style="left: 525px; top: 355.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.971597);" data-canvas-width="379.74883333333315">Using GUARDIAN’s NLP can identify additional patients who</div><div style="left: 510px; top: 372.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.98215);" data-canvas-width="395.57725">may meet the CQM measure for influenza immunization or who</div><div style="left: 510px; top: 389.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.948257);" data-canvas-width="394.9808333333332">may be exempt. This tool can be used to improve CQM reporting</div><div style="left: 510px; top: 405.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.0165);" data-canvas-width="394.0968333333333">and improve overall influenza immunization coverage by using it to</div><div style="left: 510px; top: 422.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(0.970253);" data-canvas-width="394.8930000000001">alert providers. Next steps involve further refinement of influenza</div><div style="left: 510px; top: 439.043px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.03435);" data-canvas-width="394.502">immunization categories, automating the process of using the NLP</div><div style="left: 510px; top: 455.709px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.02549);" data-canvas-width="394.31924999999995">to identify and report additional cases, as well as using the NLP for</div><div style="left: 510px; top: 472.376px; font-size: 14.1667px; font-family: serif; transform: scaleX(1.00106);" data-canvas-width="73.97833333333332">other CQMs.</div><div style="left: 510px; top: 500.528px; font-size: 12.5px; font-family: serif; transform: scaleX(1.00954);" data-canvas-width="393.33250000000004">Table 1. Categorization of influenza immunization documentation within free</div><div style="left: 510px; top: 513.861px; font-size: 12.5px; font-family: serif; transform: scaleX(1.00075);" data-canvas-width="182.6375">text notes of 453 patients using NLP</div> ER -