Apr. 7, 2017

CAATE Pass Rate Algorithm Update and Accreditation Actions

The CAATE is providing you with this summary report and recommendation for your information and reference.

The Pass Rate Standard/COVID-19 Task Force was charged with reviewing and evaluating the 2020-2021 exam data and investigating the accreditation action algorithm for the upcoming February 2022 Commission meeting.

The task force proposed, and the Board of Commissioners approved, the following recommendation for applying the accreditation action algorithm for Standard 11 of the 2012 Professional Standards and Standard 6 of the 2020 Professional Standards:

Standard 11 (2012 Professional Standards) Data Analysis: Programs must meet or exceed a three-year aggregate of 70 percent first-time pass rate on the BOC (Board of Certification) examination.

Standard 6 (2020 Professional Standards): The program meets or exceeds a three-year aggregate of 70% first-time pass rate on the BOC (Board of Certification) examination.

Recommendation

A two-year recommendation (February 2022 and February 2023 accreditation actions): The Commission will return to the historical accreditation action algorithm with the exception of programs that have been on probation for one (1) year with a pass rate below 50%. (Historically, these programs would be subject to a Withdraw Accreditation action.) They now will be subject to a Show Cause action.

The accreditation action algorithm has been provided below for reference and outlines the historical action (gray), the 2021 action (red), and the recommended 2022 and 2023 action (blue).

The recommendation is based on the evaluation of 2020-2021 exam data and previous evaluation of 2019-2020 exam data, in addition to concerns expressed by stakeholders surrounding the unpreventable cancellation and subsequent rescheduling of the March/April 2020 BOC exam due to COVID-19 during the 2019-2020 exam cycle.

It is evident that the 2019-2020 exam data affected some programs, and these exam results will be included in a program’s 3-year, first-time pass rate aggregate through the February 2023 accreditation actions.

By amending the accreditation action to Show Cause for programs on probation for one (1) year with a pass rate below 50%, it allows a mechanism for the Commission to make an accreditation action and allows a program to justify why an action of impending Withdrawal of Accreditation should not be taken.

In addition, this recommendation does not require a change in the pass rate Standards and allows for consistent accreditation actions across programs. Examples of potential scenarios are provided below for illustration purposes only. Specific questions about your program should be directed to the CAATE staff team.

Individual Exam Cycle Report

*2020-2021 exam results (excluding the October exam results) show a 77% first-time pass rate as compared to a 65% first-time pass rate in 2019-2020.

Examples

(1) Program A was placed on Probation for Standard 6/11 (67% 3-year aggregate first-time pass rate) in February 2020. In February 2021, the program’s 3-year aggregate was 60% Based on the modified algorithm, the program remained on Probation for Standard 6/11. For February 2022, the program has a 3-year aggregate of 55%. Based on the new algorithm, the program will have to Show Cause because the program has been on Probation for one year for Standard 6/11.

(2) Program B was compliant with Standard 6/11 (75% 3-year aggregate first-time pass rate) in February 2020. In February 2021, the program’s 3-year aggregate was 60%. Based on the modified algorithm, the program was non-compliant with Standard 6/11 and was not placed on Probation. For February 2022, the program has a 3-year aggregate of 62%. Based on the new algorithm, the program will be placed on Probation for non-compliance with Standard 6/11.

(3) Program C was placed on Probation for Standard 6/11 (52% 3-year aggregate first-time pass rate) in February 2020. In February 2021, the program’s 3-year aggregate was 40%. Based on the modified algorithm, the program had to Show Cause. The Commission accepted the program’s Show Cause. For February 2022, the program has a 3-year aggregate of 42%. The program will have to Show Cause based on the new algorithm because the program has been on Probation for one year for Standard 6/11 with a pass rate below 50%.

(4) Program D was compliant with Standard 6/11 (86% 3-year aggregate first-time pass rate) in February 2020. In February 2021, the program’s 3-year aggregate was 66%. Based on the modified algorithm, the program was non-compliant with Standard 6/11 and was not placed on Probation. For February 2022, the program has a 3-year aggregate of 70%. No action will need to be taken for the program as it is compliant with Standard 6/11.

(5) Program E was placed on Probation for Standard 6/11 (69% 3-year aggregate first-time pass rate) in February 2020. In February 2021, the program’s 3-year aggregate was 57%. Based on the modified algorithm, the program remained on Probation for Standard 6/11. For February 2022, the program has a 3-year aggregate of 72%. Probation will be removed for the program based on the new algorithm.

--------------------------------------------------------------------------------------------------------------------

Standard 11 (the 70% first-time pass rate standard), in effect since 2013, is a method to measure program quality. Three years ago, the CAATE announced that starting in 2016, programs would be placed on Probation if they were not compliant with Standard 11.  In early 2016 the CAATE released its implementation algorithm for taking its first accreditation actions related Standard 11. The full release can be accessed HERE. That algorithm provided information on how a Standard 11 non-compliance would be reviewed and acted upon through the February 2018 Commission meeting.  The algorithm reflected a transitional enforcement strategy for Standard 11 as the Standard could have significant impact on some programs. The transitional approach provided the majority of non-compliant programs with an additional year’s extension of Probation in order to rectify their pass rate.  As programs have now had several years to adjust to the Standard 11 requirements, the transitional approach will come to an end and the Commission will apply the Standard in the same way that other Standards are applied. Beginning in 2019 (for 2016-18 BOC exam results), the Commission will no longer provide a second year of Probation prior to a program receiving a Show Cause action for Standard 11.  The Commission has provided the following review and extended timeline guidance, in the table below, to inform programs and stakeholders of how Standard 11 actions will be administered.

Temporary February 2021 BOC Pass Rate Algorithm

Updated BOC Pass Rate Algorithm (2019)

Analytic Progress Report and Action Plan 2022

A Show Cause action requires that a program provide evidence as to why the CAATE should not Withdraw accreditation.  Questions about the Show Cause action can be found on page 20 of the CAATE P & P manual available on the CAATE website.  Specific questions may be directed to the CAATE Office at 512-733-9700 or support@caate.net.

*Please note this article was included in the 2017 Spring edition of the CAATE Insight Newsletter that was eblasted to all program administrators.