Additional Outcome Information

Currently the CAATE accreditation standards for professional programs state the following standards regarding outcomes:

4. Develop a Plan: There must be a comprehensive assessment plan to evaluate all aspects of the educational program. Assessments used for this purpose must include those defined in Standards 6 and 7. Additional assessments may include, but are not limited to, clinical site evaluations, preceptor evaluations, completed clinical proficiency evaluations, academic course performance, retention and graduation rates, graduating student exit evaluations, and alumni placement rates one year post graduation.

5. Develop a Plan: The plan must be ongoing and document regular assessment of the educational program.

6. Assessment Measures: The program’s assessment measures must include those stated in this Standard (6) and Standard 7 in addition to any unique metrics that reflect the specific program, department, or college. The specific volume and nature of this information is influenced by the individual character of the institution and should be in keeping with other similar academic programs within the institution. The assessment tools must relate the program’s stated educational mission, goals and objectives to the quality of instruction, student learning, and overall program effectiveness.

7. Assessment Measures: The program’s BOC examination aggregate data for the most recent three test cycle years must be provided and include the following metrics: Number of students graduating from the program who took the examination, number and percentage of students who passed the examination CAATE Standards for Professional Programs on the first attempt, and overall number and percentage of students who passed the examination regardless of the number of attempts.

8. Assessment Measures: Programs must post the data from Standard 7 on the program’s home page or a direct link to the data must be on the program’s home webpage.

9. Collect the Data: Programs must obtain data to determine program outcomes as indicated in Standards 6-8 (above).

10. Data Analysis: Programs must analyze the outcomes data to determine the extent to which the program is meeting its stated mission, goals, and objectives.

11. Data Analysis: Programs must meet or exceed a three year aggregate of 70 percent first-time pass rate on the BOC examination.

12. Action Plan: The results of the data analysis are used to develop a plan for continual program improvement. This plan must:

a. Develop targeted goals and action plans if the program and student learning outcomes are not met; and
b. State the specific timelines for reaching those outcomes; and
c. Identify the person(s) responsible for those action steps; and
d. Provide evidence of periodic updating of action steps as they are met or circumstances change.

13. Action Plan: Programs that have a three-year aggregate BOC first-time pass rate below 70% must provide an analysis of the deficiencies and develop an action plan for correction.

Communications Regarding Outcomes

The Commission has received correspondence from some programs regarding several Outcomes standards, specifically those that address the pass rate on the Board of Certification (BOC) exam and those that require transparency and accountability to the public. The standards that directly relate to the pass rate on the BOC exam are standards 7, 8, 11, 12 and 13. The Commission has received positive feedback, concerns, and an unprecedented amount of inquiry from the public regarding specific programs’ BOC exam pass rates. At our winter meeting in Austin Texas, February 21 & 22, 2014, the Commission discussed the feedback that we have received regarding the Outcomes standards.

Congress, the U.S. Department of Education, the Council for Higher Education Accreditation (CHEA) and the public have all voiced recent concerns with consumer protection, assurance of quality outcomes in higher education and the transparent reporting of those outcomes. As such, the CAATE is accountable to the public and the federal government to ensure that the programs accredited by CAATE are achieving suitable minimum outcomes. For these reasons and after hearing public comment regarding the BOC pass rate, the Commission set a standard requiring a minimum three-year aggregate pass rate of 70%. Notification of this standard was released in the summer of 2012 and programs were notified that the standard was to be fully implemented by July 1, 2013 (the start of the 2013-14 accreditation cycle).

The intent of these standards was twofold: to meet accountability needs by driving program quality and to meet transparency needs by informing the public. There is evidence that both of these things have already begun occurring. The percentage of programs meeting a 70% three-year aggregate pass rate was 60% for the 2010-2012 graduation cohorts and has increased to 74% for the 2011-2013 cohorts. The BOC pass rate data page on CAATE’s website averages more than 100 visits a day and is the second most visited page other than the homepage.

The 2011-2013 cohort data include 365 professional programs that had students challenge the BOC examination during this timeframe. The following table shows the results by degree level across all programs, including the number (95) and percentage (26%) of programs failing to meet the 70% first-time pass rate.   The first-time pass rate across all programs was 78% while the overall (any attempt) pass rate across all programs was 90%.

Outcomes-Standards-Update

At the Winter 2014 meeting, the Commission discussed the comments we received and after careful consideration took the following actions:

STANDARD 8: “Programs must post the data from Standard 7 on the program’s home page or a direct link to the data must be on the program’s home webpage”.

The Commission affirmed its commitment to enforcement of Standard 8 (referenced above):

The Standard 7 data that must be posted are – “BOC examination aggregate data for the most recent three test cycle years must be provided and include the following metrics: Number of students graduating from the program who took the examination, number and percentage of students who passed the examination on the first attempt, and overall number and percentage of students who passed the examination regardless of the number of attempts”.

Public notification of this particular student outcome is something that the Commission highly values. Consistent with the nature of accreditation and the expectations of the public, the government, and the Commission, programs must be transparent in informing the public regarding their BOC pass rate as it has a direct impact on student cost and indebtedness, success and employability. These data are required to be reported accurately and be provided as of July 1, 2013 to include the 2011 -13 graduation cohorts. It is important to note that the CAATE obtains examination results from the BOC, but calculates pass rate according to graduation cohorts and not by the BOC exam year. Programs can view their pass rates in their e-accreditation account once they link their students to the students’ BOC ID number. The Commission understands and appreciates concerns regarding the use of historical BOC pass rates when calculating the initial three-year aggregate pass rate. However, the Commission feels that these data are a valid measure of program outcomes and the public posting outlined in standard 8 is necessary in order to provide transparency.

STANDARD 11: “Programs must meet or exceed a three year aggregate of 70 percent first-time pass rate on the BOC examination”.

To be fully compliant by July 1, 2013, data from 2011-2013 graduation cohorts were needed to obtain the initial 3-year aggregate. Programs were initially notified in December 2013 if they were non-compliant with Standard 11. However, the Commission determined and announced that programs would NOT be placed on probation this year for non-compliance with Standard 11 based upon their 2011-2013 cohort data. Programs that were newly accredited in 2011, 2012, and 2013 have not yet produced their 3rd graduation cohort. Therefore, they will not be eligible to be cited as non-compliant on standard 11 until their 3rd year of accreditation.

The Commission has taken the following action in regard to Standard 11:

Programs below the 70% three-year aggregate pass rate in the fall 2014 (using 2012-14 graduation cohort data) will NOT be placed on probation, but will continue to be listed as non-compliant with Standard 11. Furthermore, while the Commission remains committed to assuring minimum quality program outcomes, we will not consider probationary actions against programs that are non-compliant on Standard 11 until the Commission’s February 2016 meeting. At that time, the Commission will be reviewing information from the 2015 annual reports and it will have been 4 ½ years since Standard 11 was officially released to the public. During that review, only BOC exam data from cohorts graduating after the 2013 implementation of standard 11 (graduation cohorts 2013-15) will be used in the three-year aggregate pass rate calculation.

STANDARD 13: “Programs that have a three-year aggregate BOC first-time pass rate below 70% must provide an analysis of the deficiencies and develop an action plan for correction.”

Several programs have asked for further directions in developing their required action plan (Standard 13) for non-compliance with Standard 11.

The Commission provides the following clarification in regard to Standard 13:

Programs currently found in non-compliance with Standard 11 based on 2011-13 cohorts will be required to submit a progress report through e-Accreditation by June 1, 2014 providing an analysis of the deficiencies and an action plan for correction. This report must be approved and signed by the program’s Dean, Chair, and Program Director. The Commission is requiring these reports as a mechanism to ensure that institutions are engaging in a process of self-examination and improvement. The individual characteristics and nature of institutions will naturally mean that these reports will be highly variable from institution to institution. The Commission will receive these reports and will review them to determine appropriate approvals and the inclusion of the plan elements as required in Standard 12. Programs that do not come into compliance with Standard 11 based upon next year’s annual report data (2012- 2014 graduation cohorts) will again be cited as non-compliant and will be required to provide an update of their plan and an analysis of its effectiveness. The specific requirements of an appropriate action plan are detailed in Standard 12, which states:

Standard 12: The results of the data analysis are used to develop a plan for continual program improvement. This plan must:

a. Develop targeted goals and action plans if the program and student learning outcomes are not met; and
b. State the specific timelines for reaching those outcomes; and
c. Identify the person(s) responsible for those action steps; and
d. Provide evidence of periodic updating of action steps as they are met or circumstances change.

COHORT SIZE AND INSTITUTION DEMOGRAPHICS: The BOC pass rate threshold is the same for all professional programs wishing to maintain voluntary accreditation through the CAATE regardless of program demographics (e.g., cohort size, institution classification). The CAATE determined that 70% was as an initial absolute minimum to ensure a base level of program quality. A three-year average was adopted to adequately address any concerns regarding data variability based upon program demographics, to reduce concerns from a single aberrant year of data, to better reflect the overall quality of program outcomes and is consistent with other health profession accreditors. Certain variables that may significantly influence pass rate percentages in a large positive or negative magnitude, such as a small student cohort, are at the discretion of the program and are not regulated by the CAATE. Hence, such factors are not weighted when examining program compliance with the standard. Therefore, while the Commission appreciates concerns regarding the impact of certain program demographics (e.g., small cohort size) on BOC pass rates; we remain committed to assuring minimum quality program outcomes.

The Commission appreciates this opportunity to communicate with all accredited professional programs regarding actions taken at our most recent meeting. The Commission values public and stakeholder feedback and we are confident that the accreditation standards and our recent actions are promoting program quality, student success, and public transparency and accountability. Should you have any question or concerns regarding this communication, please do not hesitate to contact the Commission.