Real World Testing Results – 2022
Background & Instructions
Under the ONC Health IT Certification Program (Certification Program), health IT developers are required to conduct Real World Testing of their certified health IT (45 CFR 170.405). The Office of the National Coordinator for Health Information Technology (ONC) issues Real World Testing resources to clarify health IT developers’ responsibilities for conducting Real World Testing, to identify topics and specific elements of Real World Testing that ONC considers a priority, and to assist health IT developers in developing their Real World Testing plans and results reports.
A Real World Testing plan template was created to assist health IT developers in organizing the required information that must be submitted for each element in their Real World Testing plan. To accompany the plan template, ONC has also provided this results report template.
While the use of this template is voluntary, health IT developers may find it useful in preparing their Real World Testing results report(s). Health IT developers must submit one year of results to address the Real World Testing of eligible products as outlined in their previous year’s Real World Testing plan(s). If adjustments to approaches are made throughout Real World Testing, the health IT developer should reflect these adjustments in their Real World Testing results report. ONC expects that the results report will include a list of these changes, the reasons for them, and how intended outcomes were more efficiently met as a result.
While every effort has been made to ensure the accuracy of restatements of 45 CFR Part 170, this template is not a legal document. The official program requirements are contained in the relevant laws and regulations. This resource should be read and understood in conjunction with the following companion resources, which describe in detail many of the Certification Program requirements referenced in this resource.
- Real World Testing–What It Means for Health IT Developers – Fact Sheet
- Real World Testing Resource Guide
- Real World Testing Certification Companion Guide
Health IT developers should also review the following regulatory materials, which establish the core requirements and responsibilities for Real World Testing under the Certification Program.
- 21st Century Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification Program final rule, 85 FR 25642 (May 1, 2020) (ONC Cures Act Final Rule)
- Section VII.B.5 — “Real World Testing”
Introduction
This document contains a list of the steps taken to conduct the annual Real World Testing requirements for ONC certification. The Results within this document were reviewed as Screenshots and spreadsheets for their compliance with the criteria defined in the test plan. These artifacts will be maintained by the health IT developer for audit purposes or further requests.
General Information
Developer Name: Elation Health, Inc.
Product Name(s): Elation EMR
Version Number(s): Version 3
Certified Health IT Product List (CHPL) Product Number(s): 15.04.04.2717.Elat.03.00.1.181231
Developer Real World Testing Plan Page URL: https://www.elationhealth.com/real-world-test-plan
Changes to Original Plan
We have not made any changes to our approach for Real World Testing that differs from what was outlined in our test plan.
| Summary of Change | Reason | Impact |
| N/A | N/A | N/A |
Withdrawn Products
We have not withdrawn any products within the past year that we previously included in the Real World Testing plan.
Summary Of Testing Methods And Key Findings
Consistent with the ONC’s recommendation that “Real World Testing verify that deployed Certified Health IT continues to perform as intended by conducting and measuring observations of interoperability and data exchange”, our original test plan focused on capturing and documenting the number of instances that certified capability was successfully utilized in the real world. In instances where no evidence exists due to low or zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we tested and demonstrated the required certified capability in a semi-controlled setting as close to a “real world” implementation as possible.
As per the test plan, we leveraged a 3-fold approach to demonstrate successful real-world implementations.
- Adoption Rate
- Summative Testing
- Interactive Testing
Adoption rate was used to determine if/when certified capability is being used in the real world and to help identify differences in care settings. Evidence of high rates of implementation and usage indicate (but don’t by themselves prove) a certified capability’s usefulness and practical value. Evidence of low rates of implementation and usage might be accounted for by patient volume, location or provider preference among other reasons. Note, it is not the goal of this exercise to identify the individual causes of why a given certified capability may have a high or low adoption rate, but rather to identify the users and care settings for which a given test is relevant.
Summative assessments were used to measure which certified actions were performed at the conclusion of a given time period where the minimum time period was 90 days and longer where possible. These results are typically obtained by generating reports and examining audit logs from within the certified health IT module to help demonstrate the frequency of actions within the given time frame, and where possible, whether those actions were successful or unsuccessful. High success rates should be an indicator of a successful implementation of a given certified capability in a real-world setting.
Interactive testing was used to demonstrate conformance to requirements where the adoption rate of a given certified capability is zero and to demonstrate ongoing compliance with updated standards and code sets (SVAP). Interactive tests were live tested as opposed to examining historical usage statistics. The goal being to demonstrate the certified Health IT module being used in a way consistent within a practice or care setting.
This approach allowed for the successful testing and obtaining results for each criterion. Detailed below in the Metrics and Outcomes section the reader will find evidential data in the form of a Summative result(s) or Interactive test outcome for each certified criteria for Elation EMR.
Standards Updates (Including Standards Version Advancement Process (Svap) And United States Core Data For Interoperability (USCDI))
Indicates whether optional standards, via SVAP and/or USCDI, are leveraged as part of the certification of the health IT product(s).
Care Setting(s)
| Care Setting | Justification |
| Ambulatory | Elation EMR is marketed to primary care providers and certified accordingly for the ambulatory care setting. |
Metrics and Outcomes
Within this section is a list of the results collected from the Elation Health solution Real World Testing measures as defined in their Real World Test plan. Outcomes are listed as Pass, Pass with Exception, or Fail determined by the success of obtaining testing results. This determination was based on a thorough review by the Elation Health team. A link is included within the Outcomes column in the table below to a subsequent Outcomes Details table. This second table matches each outcome with additional detailed information such as supporting resources and descriptions of the tests that were performed.
Key components include:
- Customer created a comprehensive Test Results Report which details customer environment, patient data utilized for tests, locations of testing
- Customer attempted Summative and/or Interactive Testing
- Customer collected audit logs to support spreadsheets and as necessary, screen shots that demonstrate proof of Interactive Testing for each criteria with “0” values in Summative Testing. These files are referenced and remain on file with Elation Health.
The following metrics were measured by viewing audit logs in the client’s live production system from July 2022 – December 2022 of the current year. For each test, a screen shot was taken of the audit report criteria screen showing the auditing information being reported. The resultant report was then saved to show the usage (or lack thereof) of the criterion.
| Associated Criterion(a) | Measurement/Metric | Relied Upon Software | Outcomes | Challenges Encountered |
| 170.315(b)(1) Transitions of care | Over a 90-day period: 1) Number of CCDAs created 2) Number of CCDAs sent via edge protocols 3)Number of CCDAs received via edge protocols | Surescripts Clinical Direct Messaging | Pass 1) 11,924 2) 55,010 3) 188,777 | N/A |
| 170.315(b)(2) Clinical information reconciliation and incorporation | Over a 90-day period: 1) Number of times a user reconciled medication list data from a received CCDA 2) Number of times a user reconciled allergies and intolerance list data from a received CCDA 3) Number of times a user reconciled problem list data from a received CCDA | N/A | Pass 1) 1,215 2) 1,238 3) 1,470 | N/A |
170.315(b)(3) Electronic prescribing | Over a 90-day period: 1) Number of prescriptions created 2) Number of prescriptions changed 3) Number of prescriptions canceled 4) Number of prescriptions renewed | Surescripts ePrescribing | Pass 1)1,572,338 2) 28,773 3) 37,391 4) 569,583 | N/A |
| 170.315(b)(6) Data export | Over a 90-day period: 1) Number of times a data export was performed | N/A | Pass 1) 11,924 | N/A |
| 170.315(b)(9) Care Plan | Over a 90-day period: 1) Number of care plans recorded 2) Number of care plans changed 3) Number of care plans accessed 4) Number of care plans created 5) Number of care plans received | N/A | Pass 1) 0 2) 0 3) 0 4) 0 5) 0 | N/A |
| 170.315(c)(1-3) Clinical quality measures (CQMs) | Over a 90-day period: 1) Number of measures recorded during the period 2) Number of QRDA Category 1 files exported 3) Number of QRDA Category 1 files imported (if applicable) 4) Number of QRDA Category 3 aggregate report(s) created over the period | N/A | Pass 1)35,000,000 2) 7 3) 92,569 4) 18 | N/A |
| 170.315(e)(1) View, download, and transmit to 3rd party | Over a 90-day period: 1) Number of views of health information by a patient or authorized representative 2) Number of downloads of health information by a patient or authorized representative 3) Number of transmissions of health information by a patient or authorized representative | N/A | Pass 1) 7,441 2) 2,929 3) 542 | N/A |
| 170.315(f)(1) Transmission to immunization registries | Over 3 separate unique 10-day periods within a 90-day window: 1) Number (or percentage) of immunization records submitted to the immunization record | N/A | Pass 1) 0 | N/A |
| 170.315(f)(2) Transmission to public health agencies — syndromic surveillance | Over 3 separate unique 10-day periods within a 90-day window: 1) Total number of syndromic surveillance events created and submitted | N/A | Pass 1) 0 | N/A |
| 170.315(g)(7) Application access — patient selection | 1) Number of requests for a patient ID or token 2) Number of requests that provided sufficient information to provide a valid response 3) Number of follow-up requests made using the provided patient ID or token | N/A | Pass 1)19,231,577 2)19,018,976 3)75,930,872 | N/A |
| 170.315(g)(8) Application access — data category request | 1) Number of requests for a patient’s data made by an application via a data category request using a valid patient ID or token 2) Number of requests for a patient’s data made by an application via a data category request using a valid patient ID or token for a specific date range | N/A | Pass 1)71,593,414 2) 3,716,437 | N/A |
| 170.315(g)(9) Application access — all data request | 1) Number of requests for a patient’s Summary Record made by an application via an all data category request using a valid patient ID or token 2) Number of requests for a patient’s Summary Record made by an application via an all data category request using a valid patient ID or token for a specific date range | N/A | Pass 1) 1,490,509 2) 883 | N/A |
| 170.315(h)(1) Direct Project | 1) Number of Direct Messages sent 2) Number of Delivery Notifications received 3) Number of Direct Messages received 4) Number of Delivery Notifications sent | Surescripts Clinical Direct Messaging | Pass 1) 75,215 2) 75,215 3) 175,146 4) 175,146 | N/A |
Outcome Details
The following sections contain additional descriptions and test results supporting documentation to provide more context for the testing outcomes defined in the Metrics and Outcomes table above.
170.315(b)(1) Transitions of care
| Summary Description |
| Method: Summative Testing – Pass The purpose of this test was to show that CDA documents are able to be created and exported. A query on historical audit logs for 90-day periods was performed for the 170.315(b)(1) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result. |
| Justification |
| This criterion requires the ability of a certified Health IT module to create CCDAs according to specified standards and vocabulary code sets, as well as send and receive CCDAs via edge protocols. However, it is not possible to consistently and reliably demonstrate that all required standards and code sets were used because not all CCDAs created in a real-world setting contain all the necessary data elements. Furthermore, it is not feasible to obtain copies of CCDA documents from “outside” developers or providers who have no incentive to participate in this exercise. Therefore, we intend to demonstrate the required certified capabilities by demonstrating how often CCDAs are created and exchanged with other systems to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be high utilization by providers with a high success rate. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets if needed. |
170.315(b)(3) Electronic Prescribing
| Summary Description |
| Method: Summative Testing – Pass The purpose of this test was to show that an active connection from EHR customer sites to an ePrescribing solution was deployed. A query on historical audit logs for 90-day periods was performed for the 170.315(b)(3) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result. |
| Justification |
| This criterion requires the ability of a certified Health IT module to perform prescription-related electronic transactions (eRx) using required standards. However, it is not possible to demonstrate the correct standards were used because it is not feasible to obtain copies of eRx documents from “outside” companies or pharmacies who have no incentive to participate. Therefore, we intend to demonstrate the required certified capabilities are effective by demonstrating how often eRx transactions are performed by examining reports from our eRx partner. This will demonstrate that not only are the eRx transactions sent from the certified Health IT module, but that the transactions are successfully received by the eRx clearinghouse. Our expectation is there will be high utilization by providers with a high success rate. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets if needed. |
170.315(b)(6) Data Export
| Summary Description |
| Method: Summative Testing – Pass The purpose of this test was to show that our customer can export patient data from our EHR without any assistance from Elation Health. A query on historical audit logs for 90-day periods was performed for the 170.315(b)(6) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result. |
| Justification |
| This criterion requires the ability of a certified Health IT module to export a summary of a patient’s record in CCDA format according to specified standards and vocabulary code sets. However, it is not possible to consistently and reliably demonstrate that all required standards and code sets were used because not all CCDAs created in a real-world setting contain all the necessary data elements. Therefore, we intend to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be very low utilization by providers with a high success rate. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets and screenshots if needed. |
170.315(b)(9) Care Plan
| Summary Description |
| Method: Interactive Testing – Pass The purpose of this test was to show that our customer can record, change, access, create and receive a care plan from our EHR without any assistance from Elation Health. A query on historical audit logs for 90-day periods was performed for the 170.315(b)(9) criterion. Due to low or zero adoption of this criteria, health IT developer demonstrated the module function in their system as an interactive test demonstrating a compliant result. |
| Justification |
| This criterion requires the ability of a certified Health IT module to record, change, access, create, and receive care plan information according to the specified format. While Elation EMR provides this capability to our providers, we’ve seen low adoption of the capability and so we will demonstrate real world performance through interactive testing. We will select a provider of Elation EMR providers to run a real life test with real patient data. Reported results will remove all PHI but demonstrate the following expected outcomes:Expected outcomes:– Provider is able to record the care plan– Provider is able to change the care plan– Provider is able to access the care plan– Provider is able to create the care plan– Provider is able to receive the care plan |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets and screenshots if needed. |
170.315(c)(1-3) Clinical Quality Measures (CQMs)
| Summary Description |
| Method: Summative Testing – Pass The purpose of this test was to show that the EHR meets the QRDA reporting requirement for the designated care settings. A query on historical audit logs for 90-day periods was performed for the 170.315(c)(1-3) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result. |
| Justification |
| These criteria will be tested together. C1 requires a certified Health IT module to record required data, calculate CQMs from the recorded data, and export the data in QRDA Category 1 format. C2 requires a certified Health IT module must be able to import data from a QRDA Category 1 formatted file and calculate the CQMs based on that data. C3 requires a certified Health IT module must be able to create a QRDA Category 1 formatted file and a QRDA Category 3 aggregate report to be used for transmitting CQM data to CMS. We intend to record the frequency that CQM files are imported and/or exported by providers to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be moderate utilization by providers with a high success rate. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets and screenshots if needed. |
170.315(e)(1) View, Download, and Transmit to 3rd Party
| Summary Description |
| Method: Summative Testing – Pass The purpose of this test was to show that the EHR provides patients access to a patient portal with the ability to view, download, and send their health care records for the designated care settings. A query on historical audit logs for 90-day periods was performed for the 170.315(e)(1) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result. |
| Justification |
| This criterion requires the ability of a certified Health IT module to provide patients access to a patient portal with the ability to view, download, and send their health care records to other providers via encrypted or unencrypted transmission methods in CCDA format. We intend to record the frequency that patients are viewing, downloading, and transmitting their records from the portal using the certified capabilities to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be low utilization by patients for view and download and lower utilization for transmit with a high success rate for all certified capabilities. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets and screenshots if needed. |
170.315(f)(1) Transmission to Immunization Registries
| Summary Description |
| Method: Interactive Testing – Pass The purpose of this test was to show that the EHR is able to transmit immunization data to a registry and meets the reporting requirement for the designated care settings. A query on historical audit logs for 90-day periods was performed for the 170.315(f)(1) criterion. Due to low or zero adoption of this criteria, health IT developer demonstrated the module function in their system as an interactive test demonstrating a compliant result. |
| Justification |
| This criterion requires the ability of a certified Health IT module to transmit immunization data to a registry using a specified format. We intend to record the frequency that immunization data is submitted to registries by providers to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be zero adoption of this certified capability by our users, so we have added interactive testing methodology for these capabilities to the test plan below to demonstrate the feature is available and functions as expected should any users elect to begin using this feature. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets and screenshots if needed. |
170.315(f)(2) Transmission to Public Health Agencies — Syndromic Surveillance
| Summary Description |
| Method: Interactive Testing – Pass The purpose of this test was to show that the EHR is able to transmit syndrome-based public health surveillance data to a registry and meets the reporting requirement for the designated care settings. A query on historical audit logs for 90-day periods was performed for the 170.315(f)(2) criterion. Due to low or zero adoption of this criteria, health IT developer demonstrated the module function in their system as an interactive test demonstrating a compliant result. |
| Justification |
| This criterion requires the ability of a certified Health IT module to transmit syndrome-based public health surveillance data to a registry using a specified format. We intend to record the frequency that syndromic surveillance data is submitted to registries by providers to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be zero adoption of this certified capability by our users, so we have added interactive testing methodology for these capabilities to the test plan below to demonstrate the feature is available and functions as expected should any users elect to begin using this feature. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets and screenshots if needed. |
170.315(g)(7) Application Access — Patient Selection
| Summary Description |
| Method: Summative Testing – Pass The purpose of this test was to show that the EHR is able to fulfill an API request that enables external applications to request a unique patient identifier from the certified Health IT module that can be used to request additional patient data. A query on historical audit logs for 90-day periods was performed for the 170.315(g)(7) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result. |
| Justification |
| This criterion requires the certified Health IT module to provide an API and supporting documentation that enable external applications to request a unique patient identifier from the certified Health IT module that can be used to request additional patient data. We intend to record the frequency that patient ID requests are received by providers via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be low utilization by providers with a high success rate. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets and screenshots if needed. |
170.315(g)(8) Application Access — Data Category Request
| Summary Description |
| Method: Summative Testing – Pass The purpose of this test was to show that the EHR is able to fulfill an API request that enables external applications to request patient data categories from the certified Health IT module. A query on historical audit logs for 90-day periods was performed for the 170.315(g)(8) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result. |
| Justification |
| This criterion requires the certified Health IT module to provide an API and supporting documentation that enable external applications to request patient data by category from the certified Health IT module. We intend to record the frequency that patient data requests by category are received by providers and fulfilled via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be low utilization by providers with a high success rate. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets and screenshots if needed. |
170.315(g)(9) Application Access — All Data Request
| Summary Description |
Method: Summative Testing – Pass |
| Justification |
| This criterion requires the certified Health IT module to provide an API and supporting documentation that enable external applications to request all categories of patient data defined in the CCDS from the certified Health IT module. We intend to record the frequency that patient data requests for all categories are received by providers and fulfilled via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be low utilization by providers with a high success rate. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets and screenshots if needed. |
170.315(h)(1) Direct Project
| Summary Description |
Method: Summative Testing – Pass |
| Justification |
| This criterion requires the ability of a certified Health IT module to record the frequency that direct messages are sent and received by providers, along with how often MDNs are sent and received. Since not all systems respond with MDNs, we cannot reliably use that metric to define success. Furthermore, it is not feasible to obtain copies of Direct Messages from “outside” developers or providers who have no incentive to participate in this exercise. Therefore, we intend to demonstrate the required certified capabilities by demonstrating how often Direct Messages are exchanged with other systems to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be moderate utilization by providers with a high success rate. |
| Results Supporting Documents |
| Please Contact Elation Health for any Results spreadsheets and screenshots if needed. |
Key Milestones
Includes a list of key milestones that were met during the Real World Testing process. Includes details on how and when Elation Health implemented measures and collected data.
| Key Milestone | Care Setting | Date/Timeframe |
| Scheduling and logistics | Ambulatory | 7/1/2022-9/25/2022 |
| Data collection | Ambulatory | 9/26/2022-12/25/2022 |
| Review and Collect Data | Ambulatory | 12/27/2022-1/9/2023 |
| Writing Report | Ambulatory | 1/9/2023-submission date |
| Elation Health executed interactive testing to show that the criterion are functional. The following metrics were tested interactively as detailed in the outcomes section above: • 170.315 (b)(9) Care Plan • 170.315 (f)(1) Transmission to immunization registries • 170.315 (f)(2) Transmission to public health agencies — syndromic surveillance | Ambulatory | 10/31/2022 |
| Elation Health executed summative testing to show that the criteria are functional. The following metrics were pulled from transaction logs as detailed in the outcomes section above: • 170.315 (b)(1) Transitions of care • 170.315 (b)(2) Clinical Information Reconciliation and Incorporation • 170.315 (b)(3) Electronic Prescribing • 170.315 (b)(6) Data Export • 170.315 (c)(1-3) Clinical Quality Measures (CQMs) • 170.315 (e)(1) View, Download, and Transmit to 3rd Party • 170.315(g)(7) Application access—patient selection • 170.315(g)(8) Application access—data category request • 170.315(g)(9) Application access—all data request • 170.315 (h)(1) Direct Project | Ambulatory | Sep – Dec 2022 |