Enhance Transplant Program Performance Monitoring
Background
In December 2021, the OPTN Board of Directors approved bylaw changes that will allow for a more holistic evaluation of transplant programs. OPTN Bylaws (PDF) outline membership requirements for organizations and individuals to participate in the U.S. transplant network.
The OPTN Membership and Professional Standards Committee (MPSC) has historically used 1-year post-transplant graft and patient survival to identify underperforming transplant programs. This metric only evaluates one phase of patient care.
The new monitoring system involves four risk-adjusted measures related to the patient journey:
- 90-day graft survival hazard ratio: Took effect July 14, 2022
- 1-year conditional graft survival hazard ratio: Took effect July 14, 2022
- Pre-transplant- offer acceptance rate ratio: Took effect July 27, 2023
- Pre-transplant mortality rate ratio (previously waitlist mortality rate): Took effect July 25, 2024
In developing this new system, the MPSC focused on areas within the OPTN’s authority under the OPTN Final Rule. The MPSC strives to support and collaborate with transplant programs to provide performance improvement opportunities, while also meeting its responsibility to monitor member performance to identify potential patient safety issues and ensure public trust in the transplantation system. The MPSC aimed to develop a performance monitoring system that provides maximum support for the OPTN strategic plan by increasing the number of transplants, promoting equitable access to transplantation, and fostering innovation. The committee’s proposal received public comments from Aug. 3 - Sept. 30, 2021.
- Download the PDF: Enhanced Transplant Program Performance Metrics (PDF) provides a quick review about how risk adjustment works for the four metrics used to evaluate performance
Bylaws changes
- As outlined in the policy notice (PDF), these changes revise the OPTN Bylaws, Appendix D, D.12.A. Transplant Program Performance (PDF), and establish new criteria to identify transplant programs that will enter into MPSC performance review. Note: the changes will apply to Appendix D, D.12.A, until all approved changes to the bylaws have been implemented.
- The MPSC will review a transplant program’s performance if the transplant program meets the criteria (for either adult transplants or pediatric transplants) for:
- 90-day graft survival hazard ratio
- 1-year conditional graft survival hazard ratio
- Pre-transplant mortality rate ratio
- Offer acceptance rate ratio
- OPTN Bylaw, Appendix D, D.12.A. Transplant Program Performance also clarifies, but does not substantively change, transplant program obligations to participate in the performance review process.
- In addition, Appendix L: Review and Actions now includes a section that codifies the current peer review process.
- Appendix M: Definitions includes new definitions for the performance metrics and an updated definition of the Scientific Registry of Transplant Recipients (SRTR). Appendix M: Definitions no longer includes definitions of two defunct MPSC subcommittees.
Enhance Transplant Program Performance Monitoring System
- Read the policy proposal(PDF) (8/2021)
- Read the Board briefing paper (PDF) (11/2021)
- Read the full policy notice (PDF) (12/2021)
- Read the phase 1 policy notice (PDF) for inquiries related to post-transplant outcomes (12/2021)
- Read the phase 2 policy notice (PDF) for inquires related to offer acceptance outcomes (7/2023)
- Read the phase 3 policy notice (PDF) for inquires related to pre-transplant mortality rate ratio (7/2024)
- Read public comment on Enhance Transplant Program Performance Monitoring System
Implementation timeline
The OPTN is implementing the new performance monitoring system in phases. The post-transplant metrics took effect on July 14, 2022.
The MPSC delayed implementation the pre-transplant metrics to allow programs time to better understand the metrics, review their applicable data and practices, and make practice changes as desired.
Specifically, the MPSC postponed the implementation of the pre-transplant metrics until at least one complete evaluation interval had passed following the December 2021 Board approval. This ensures that the MPSC will only evaluate programs based on transplant program performance on these two metrics using data collected after Board approval.
- The offer acceptance model reports a program’s offer acceptance practices during a 1-year evaluation interval.
- The pre-transplant mortality model reports the mortality rate for candidates on the waiting list at any time over a 2-year evaluation interval.
The data collection period for the pre-transplant metrics began Jan. 1, 2022. The offer acceptance rate ratio criteria took effect on July 27, 2023 and the pre-transplant mortality rate ratio implemented on July 25, 2024.
The SRTR provides statistical and other analytic support to the OPTN. The SRTR produces the models to calculate each of the four performance metrics and will provide the MPSC with reports using these models. These reports allow the MPSC to identify transplant programs that meet the OPTN Bylaw requirements for review. The SRTR has prepared FAQs with more information.
The MPSC will begin evaluating transplant programs based on the new metrics and criteria on the following timeline:
90-day graft survival hazard ratio: A program’s rate of graft failure from date of transplant to 90 days post-transplant, relative to the expected rate based on transplants with similar recipient and donor organ characteristics across the nation.
1-year conditional on 90-day graft survival hazard ratio: A program’s rate of graft failure from day 90 post-transplant to day 365 post-transplant, conditional on the graft surviving for the first 90-days post-transplant, relative to the expected rate based on transplants with similar recipient and donor organ characteristics across the nation.
- Beginning July 14, 2022, the MPSC began to consider program performance on the above metrics for transplants that occurred from January 2019 through June 2021 (excluding the March 13– June 12, 2020, SRTR COVID carve-out).
Offer acceptance: A program’s actual rate of accepting organ offers relative to the expected rate, based on similar offers across the nation.
- Beginning July 27, 2023, the MPSC will begin to consider program performance on the above metric for offers received from January 2022 through December 2022.
- The MPSC sent notice to transplant programs six months prior to implementation.
- Download the PDF: Effective Practices to Improve Offer Acceptance (PDF)
-
Recorded presentations from the OPTN Offer Acceptance Collaborative's 2023 hybrid kickoff event are available in the OPTN Learning Management System (known as UNOS Connect) and offer access to a range of thought leadership on system performance.
Search for the OPTN Offer Acceptance Collaborative Playlist – CEPTCs available
The modules have information about:
- MPSC transplant program performance monitoring, the offer acceptance rate ratio, and OPTN tools to drive improvement.
- Effective practices related to offer acceptance and lessons learned from previous collaborative participants.
Pre-transplant mortality: A program’s pre-transplant mortality rate relative to the mortality rate experienced by similar patients across the nation.
- Beginning in July 2024, the MPSC will consider program performance on mortality for candidates appearing on the waiting list at any time during the period of January 2022 through December 2023.
- The MPSC will send notice to transplant programs six months prior to implementation.
- Download the PDF: Effective Practices to Improve Pre-transplant Outcomes
- Watch an educational webinar recording: National webinar for transplant professionals to provide information about July 2024 implementation of pre-transplant performance metric
Prior to implementation, the OPTN will provided educational offerings to help programs understand how various scenarios may impact each metric, and how to find and interpret data available on the SRTR secure site. Members will received more informational emails about how to register for events such as webinars and access pre-recorded materials.
The OPTN will support programs in this phased approach. The OPTN will notify programs at least one cycle prior to implementation if, based on current pre-implementation data, they would have been identified if the respective criteria had been in effect. This will allow programs to evaluate their performance and make adjustments ahead of implementation. The OPTN will also collaborate with high-performing programs to identify effective practices and develop resources for members.
Risk-adjustment and performance evaluation
The MPSC strongly believes that it is important to use risk-adjusted metrics to have a fair and accurate performance monitoring system. Risk adjustment allows the MPSC to focus on a program’s performance on those elements of patient care that are within the control of the program that the program can affect through performance improvement initiatives. In addition, the MPSC believes the use of risk-adjusted metrics is the best way to avoid risk-averse behavior that could result in the unintended negative consequence of greater organ non-use (discard) and a decrease in organ transplants.
Some specific questions regarding the importance of risk adjustment are addressed below. For more information on risk adjustment and the SRTR models developed for the MPSC performance evaluation metrics, a presentation titled “MPSC: Enhance Transplant Program Performance Monitoring System: Proposed Metrics” may be viewed here. This video addresses frequently asked questions. Additional information is being prepared for members.
All four metrics in the new performance monitoring system are risk-adjusted. Risk-adjusted models compare a program’s observed performance to its expected performance. The SRTR attempts to adjust for each program’s respective risk level by taking into account many factors. These include program-specific characteristics of the transplant candidates, recipients and donors. A program’s risk level is reflected in its expected performance number. A program’s performance will be compared to the national performance for patients or offers with similar characteristics.
- Download the PDF: Enhanced Transplant Program Performance Metrics (PDF) provides a quick review about how risk adjustment works for the four metrics used to evaluate performance
Without risk adjustment, all programs would be expected to achieve the same observed survival rate, regardless of the acuity of their patients or the quality of the donor organs they transplant. The goal of using risk adjustment is to be able to differentiate between program performance, given the known patient and donor factors.
The MPSC wants to understand if a program’s outcomes differ significantly from what would be expected from that program based on the outcomes of similar patients or organ offers nationally; or, in other words, what is the individual program’s effect on the outcome of interest (graft survival, pre-transplant mortality, offer acceptance).
Some transplants involve greater risk than others. The risk adjustment models do not expect higher-risk transplants to succeed at the same rate as lower-risk transplants. If the higher-risk transplants a program performs are associated with a bad outcome, and the model expects those transplants to have a poorer outcome, then the program can still be considered to be performing as expected.
In this example, assume two liver transplant programs perform the same number of transplants and that most of the donor and recipient characteristics are the same between the programs. One program utilizes a high volume of donor after cardiac death (DCD) livers, while the other program does not use any.
- The model expects that liver transplants using a DCD liver have an approximately 60 percent higher risk of graft failure based on national data.
- Because of risk-adjustment, the model would expect the program that utilizes DCD livers to have more graft failures (a lower “expected” rate) than the program that uses none.
- However, a program that rarely performs higher-risk transplants would be expected to have fewer poor outcomes, and would be identified as performing worse than expected if the program had poor outcomes in average-risk transplants.
If the SRTR models did not use risk adjustment, the MPSC would not be well-equipped to understand if a program’s outcomes differ from what would be expected based on patient and donor risk factors that are beyond the program’s control, such as the age of the recipient or whether a donor is a donor after cardiac death or donor after brain death. Instead, the MPSC would only have the raw, absolute number of events at a program. The MPSC would not be able to determine easily if those events were similar to events experienced by other programs, or were associated with more complex transplants.
The SRTR risk adjustment models are limited to the data currently collected by the OPTN. The MPSC acknowledges that current risk adjustment models cannot adjust for all risks; however, the MPSC does not believe that the lack of perfect risk adjustment supports the complete rejection of risk adjustment. If the MPSC and the transplant community acknowledge that transplant programs have different risk profiles – different patient acuity, different case mix, and different donor availability depending on where they are located and the donor supply - the way to reflect that difference is through risk adjustment.
No. Risk-adjusted models cannot be built for some types of transplants, because there are not enough events or available factors that predict the outcome. For example, there were no predictors identified for pediatric kidney deceased donor 1-year graft survival, so a risk-adjusted model was not built.
In cases such as these, the national failure rate is used to determine an expected rate for a program.
The SRTR is continuously improving its risk adjustment models and encourages any feedback from the community and the MPSC. Every six months, the SRTR analyzes national data to determine if the risk associated with a variable has changed and refits the models to capture the current risk profile.
On its website , the SRTR provides information on each of the performance monitoring metric models under the “Tools” tab. This information includes a list of the model elements, the model coefficients, and the element plots that provide a visualization of the relationship between the element and the predicted risk. These tools also include a tab for “Other Elements” that provides a list of elements that the SRTR investigated and found did not add predictive value to the model, or in other words, the element did not improve their ability to predict the outcomes.
- SRTR risk adjustment model documentation: Post-transplant outcomes
- SRTR risk adjustment model documentation: Offer acceptance models
- SRTR risk adjustment model documentation: Waiting list models: pre-transplant mortality rate
In order to include a data element in a risk adjustment model, the variable must be included in OPTN data collection. For any variable with data collected and available, the SRTR can evaluate whether the variable improves the risk adjustment. The SRTR Review Committee determines if a variable should be included in a model.
- Any potential adjuster must affect a different proportion of patients at different programs
For example, if every program in the country used 10 percent donor DCD organs for transplants, DCD would not be a good candidate for adjustment because adjusting for it would not make a difference, since it is the same at all programs. In reality, DCD is a good adjuster because the willingness to use DCD organs varies between programs.
- Any potential adjuster must predict, or be associated with, the outcome
We know from experience that DCD donor organs tend to have poorer outcomes on average than brain dead donor organs. For example, in liver transplants, DCD donors have approximately a 60 percent higher risk of failure nationally than brain dead donors. Yet, the increased risk of using DCD donor organs is acceptable as compared against the alternative outcome of not being transplanted. We want to adjust for this risk.
- Any potential adjuster cannot be a treatment by the program that would affect the outcome
For example, the program’s choice of immunosuppression regimen, which is a treatment decision by the program, may affect the outcome. This is precisely what the MPSC is trying to determine – if there is a treatment practice that is causing or contributing to poorer outcomes. We do not want to adjust that out of the metric, as it may be something the program needs to evaluate and adjust to achieve better outcomes.
MPSC members will evaluate whether factors that are not included in the risk adjustment model appear to be a factor at a program under review. Any factors identified will be referred either to the OPTN organ-specific committee or the OPTN Data Advisory Committee if the data point is not currently collected, or to the SRTR for investigation for inclusion in the model if the data point is currently collected.
Education and resources
The proposal's public comment page has information and educational resources about the metrics, including:
- A video introduction of the changes outlining what's involved
- A recording of an Aug. 4, 2021, webinar with Rich Formica, M.D., providing a more detailed analysis
- A September 2021 recording of MPSC chair, Ian Jamieson, and SRTR director, Jon Snyder, discussing the four evaluation metrics involved, including risk-adjustment and frequently asked questions
Community webinar
The OPTN hosted a webinar for transplant professionals June 28, 2024, with information on the the pre-transplant mortality rate ratio metric. Scientific Registry of Transplant Recipients (SRTR) Director Jon Snyder, Ph.D. and SRTR Biostatistician Grace Lyden, Ph.D., joined Zoe Stewart Lewis MD, chair of the OPTN Membership and Professional Standards Committee (MPSC) to discuss the new metric and answered participants' questions. Watch the webinar recording.
Instruction
- Download the PDF: Enhanced Transplant Program Performance Metrics provides a quick review about how risk adjustment works for the four metrics used to evaluate performance
- Download the PDF: Effective Practices to Improve Post-transplant Outcomes
- Download the PDF: Effective Practices to Improve Offer Acceptance
- Download the PDF: Effective Practices to Improve Pre-transplant Outcomes
Use this link to access these professional education resources and learning modules:
-
QLT150: MPSC transplant program performance monitoring
-
QLT151: MPSC post-transplant outcomes and risk adjustment (confers CEPTC credits)
- QLT152: MPSC Offer Acceptance Rate Ratio and Risk Adjustment (confers CEPTC credits)
-
QLT153: MPSC pre-transplant mortality rate ratio and risk adjustment (confers CEPTC credits)
- The modules have information about:
- MPSC transplant program performance monitoring, the offer acceptance rate ratio, and OPTN tools to drive improvement.
- Effective practices related to offer acceptance and hear lessons learned from previous collaborative participants.OPTN Offer Acceptance Collaborative Playlist (confers CEPTC credits)
- The modules have information about:
Access the Offer Acceptance Collaborative Improvement guide here.
Learn more about monitoring and performance reviews: What to Expect: Performance Reviews (PDF)
SRTR reporting
The SRTR has also added information to transplant programs’ SRTR secure sites to help you understand whether your program meets the new MPSC performance monitoring criteria. This information is intended to support you as you plan and evaluate your program’s performance.
Contact us
Email MPSCReports@unos.org with questions about the transplant program performance monitoring system.