Student Outcome Studies

Since its inception in 2002, the Academy for College Excellence (ACE)—formerly called Digital Bridge Academy (DBA)—has been rigorously evaluated qualitatively and quantitatively.  Ten independent external evaluations have been conducted, all demonstrating significant impacts on at risk student populations and community college faculty. 

1. Long Term Academic and Wage Outcomes: Summary of a Qualitative and Quantitative Analysis (2018)

• Read a summary of the final report (see page 6 of document for six-year longitudinal salary outcomes)

• Read the Interim RP Group summary report

The findings from this 2018 study conducted by the RP Group and funded by the Joyce Foundation show ACE builds students’ ability to achieve educational awards, increase earnings over time, and complete transfer level courses. The long term outcomes for high risk students have strong implications for equity and student success.

Key findings:

1. The addition of an ACE Foundation Professional Leadership and Communication course to a career
   technical program (CTE) such as nursing appears to enhance academic and
   wage outcomes for nursing students six-years later.

2. Participation in ACE for high-risk student populations helps them attain
   academic and wage outcomes comparable to the general student
   population.

3. Participation in ACE boosts potential in earnings over time for all students,
   but especially for males of color and underrepresented minorities.

2. Evidence of Success

This document summarizes the results of the recent findings of three longitudinal evaluation studies of the ACE Program: the Columbia University’s Community College Research Center (CCRC) study published in 2009, the RTI International (RTI) study released in January 2014, and the Research and Planning Group for the California Community Colleges (RP Group) salary and academic outcome study released in July 2018. The two key findings: 

1. The ACE academic outcomes are replicable at colleges adopting the ACE model; i.e., the academic outcomes produced in the CCRC study are replicated when the ACE Program is implemented at multiple colleges where some of the colleges are scaling to serve over 350 students per year.

2. The ACE accelerated math approach shows strong results when integrated with the ACE social justice primary research course, and the accelerated math approach is successful when combined with accelerated English in the same semester.

3. The ACE 21st Century Professional Competencies approach has significant salary increases six-years after completing the one-week Professional Leadership and Communications course (2 Credit Hours).

3. Evaluation of the Academy for College Excellence MPR Final Report – January 2014

Our recent longitudinal study conducted by RTI International (formerly MPR Associates), shows that ACE students experience a high velocity of advancement toward transfer-level English, as well as shifts in affective factors that are important for academic success. The final report from RTI analyzes the achievement of multiple cohorts of ACE students at 4 colleges—Berkeley City College, Cabrillo College, Hartnell College, and Los Medanos College—over 3 semesters, at the end of each semester and again two semesters subsequent to the ACE Semester, using comparison groups constructed from institutional and program data. Data collected from the colleges includes transcript information (e.g., courses and grades), assessment/placement test results for English, math, reading, and/or ESL, and demographic information (i.e., date of birth, gender, ethnicity).  RTI measured credit accrual, retention, persistence, full-time enrollment, successful completion of math at one college, and successful completion of gatekeeper English courses for all ACE students as well as from a larger group of students used to construct comparison groups.

Velocity

Vulnerable students enrolled in ACE also make significant gains after they leave the ACE program, compared to non-ACE students. The ACE program is one semester long and focuses on completing college-level English as discussed above. The figure below shows the trajectory of transfer-level English completion for ACE students after they leave the ACE program at 4 colleges—Berkeley City College, Cabrillo College, Hartnell College, and Los Medanos College.  Notably, ACE students achieve more gains in completing transfer-level English by the end of the first semester as well as by the end of the second semester after completion of the ACE program than those for the control group.  

Even as the gap shrinks over time, ACE students have the advantage of completing transfer-level English, sooner than the control group and this advantage enables them to take other more advanced courses that require transfer-level English as a prerequisite sooner than would otherwise be possible. 

At Los Medanos College (LMC), the ACE program has led to significant improvements in student performance in both English and math. One semester after attending ACE at LMC, students were 7.8 times more likely to pass transfer-level math and English courses than the comparison group. Two semesters out, they were 4 times more likely to pass transfer-level math and English courses than the comparison group. 

Effects of ACE Program on Students' Affective Capacity

The RTI report also measured psychological constructs associated with the affective dimension, which are instrumental to student success. ACE students exhibited significant gains in the 8 factors measured: academic self-efficacy, personal responsibility, college identity, four factors of mindfulness, and leadership and teamwork efficacy. The total n for this analysis is 769 students from 6 colleges participating in the ACE program measured at three different times:

• Tan = Before Foundation Course

• Dark Blue = End of Foundation Course

• Light Blue = End of Semester

Results have shown that students who complete the ACE Foundation Course report a statistically significant improvement at a p<.001 level in six of the eight affective dimensions measured. 

ACE's Non-Cognitive Effect on Students - 6 College Study of 769 Students
ACE Participants' Mean Scale Scores for CSSAS Factors

• Notes: *p < .001; statistical significance is based on comparison with Time 1 scores.

• The Y-axis represents the mean (average) score for each factor.

• Survey responses were based on a five-point scale, from “strongly disagree to strongly agree” for the non-mindfulness items and from “never or very rarely true” to “always or almost always” true for the mindfulness items.

4. Educational Outcomes of Cabrillo College’s Digital Bridge Academy: Findings from a Multivariate Analysis (2009)

• See slides summarizing the study

• Read the CCRC report summary

• Read the full CCRC report

Community College Research Center (CCRC) at Columbia University released a report in 2009 of a longitudinal study comparing ACE students at Cabrillo College with a comparison group of 11,578 non-ACE Cabrillo College students. The outcomes of ACE students were separated into two groups: students in cohorts 1-3 who took an accelerated English class that was one-level-below transfer English as part of the ACE semester, and students in cohorts 4-9. Cohorts 4-9 did not take an accelerated English class like cohort 1-3 because it was determined that their enrollment in a course more advanced than that indicated by the assessment/placement process was prohibited by the California Educational Code. The reading lab course did not count as an English course, therefore the ACE students lost one semester of English progression, yet as the study indicated, they still progressed in English just as well as non-ACE students. The results were the following:

The graph below shows results of the accelerated model for Cohorts 1-3.

The results in the table below show a statistically significant positive difference between ACE students—both accelerated and non-accelerated—and the comparison group. The exceptions to this pattern are there are no significant differences between non-accelerated ACE students and the comparison group in passing English courses one level below transfer and transfer level, and in transfer credits earned. In these areas, performance matches the comparison group.

“In summary, this study found significant positive effects for participation in both the accelerated and non-accelerated versions of the DBA…

“To the extent that colleges are seeking strategies for increasing the rate at which academically underprepared students complete 'gatekeeper' courses such as college-level English and earn college credits, the accelerated version of the DBA program seems to hold particular promise.

“While the results of this analysis show that participation in the DBA program is correlated with better outcomes on most measures for students…DBA students are very likely more at-risk than are Cabrillo students with similar levels of academic preparation…

“To the extent that the DBA students in the sample were substantially more disadvantaged than other students, [see demographics] it may be that the estimates produced through this analysis understate the effect of participating in DBA.”

Note on demographics: Close to two-thirds of ACE students have at-risk characteristics such as being English Language Learners, being the first in their family to go to college, or placing two levels below transfer-level in English, mathematics, or reading. Approximately one-quarter of the students have high-risk factors such as not being a high school graduate or GED holder or being a parent of dependent children. Fifteen percent of the students have received public assistance.

5. Watsonville Digital Bridge Academy Report I: Student Outcomes Evaluation Cohorts 1 & 2 (2005) & 4. Watsonville Digital Bridge Academy Report 2: Persistence And Achievement (2007)

• Watsonville DBA Report I: Student Outcomes Evaluation Cohorts 1 & 2 (2005)

• Watsonville DBA Report 2: Persistence And Achievement (2007)

In these evaluations funded by the National Science Foundation, Dr. Norena Badway of theHigher Education Evaluation and Research Group conducted two sequential evaluations on the characteristics of Digital Bridge Academy students and their personal and academic growth related to their participation.

These studies used the metrics of academic progress: GPA, retention through the semester, progress toward an award, and persistence to the next term. The evaluators viewed these outcomes not only in terms of academic progress, but as markers of self-efficacy indicating confidence that the student can succeed in college. Their findings were that participation in DBA had substantial impact on those four measures.

For the initial two cohorts, which experienced the initial accelerated design, retention in the Bridge Semester was 83% and 79% respectively. Because of the tightly linked curriculum, students who completed the Bridge Semester earned 19.5 college credits.

The second report summarized the longitudinal effects for the five DBA cohorts from Fall 2003 to Fall 2005 and found similar results.

The retention rate for first semester DBA students varied from 74-100%, compared toretention rates of 60% for all Cabrillo basic skills students and 67% for students in credit-level coursework.  The majority of DBA students completed 16 credits with greater success than did their non-DBA peers at Cabrillo.

These findings alone commend the DBA program: under-prepared, at-risk students make marked progress towards academic, career and personal goals. The DBA evidence suggests a need to re-examine beliefs about sheltering at-risk underprepared students.