Student Outcome Studies

Since its inception in 2002, the Academy for College Excellence (ACE)—formerly called Digital Bridge Academy (DBA)—has been rigorously evaluated qualitatively and quantitatively.  Ten independent external evaluations have been conducted, all demonstrating significant impacts on vulnerable students and community college faculty. 

1. Evaluation of the Academy for College Excellence
MPR Interim Report–December 2012

Our current longitudinal study being conducted by MPR Associates, is showing that ACE students experience a high velocity of advancement toward transfer-level English, as well as shifts in affective factors that are important for academic success. The latest interim report from the MPR study analyzes the achievement of multiple cohorts of ACE students at 4 colleges—Berkeley City College, Cabrillo College, Hartnell College, and Los Medanos College—over 3 semesters, at the end of each semester and again two semesters subsequent to the ACE Semester, using comparison groups constructed from institutional and program data. Data collected from the colleges includes transcript information (e.g., courses and grades), assessment/placement test results for English, math, reading, and/or ESL, and demographic information (i.e., date of birth, gender, ethnicity).  MPR measured credit accrual, retention, persistence, full-time enrollment, successful completion of math at one college, and successful completion of gatekeeper English courses for all ACE students as well as from a larger group of students used to construct comparison groups.

Velocity

Vulnerable students enrolled in ACE also make significant gains after they leave the ACE program, compared to non-ACE students. The ACE program is one semester long and focuses on completing college-level English as discussed above. The figure below shows the trajectory of transfer-level English completion for ACE students after they leave the ACE program at 4 colleges—Berkeley City College, Cabrillo College, Hartnell College, and Los Medanos College.  Notably, ACE students achieve more gains in completing transfer-level English by the end of the first semester as well as by the end of the second semester after completion of the ACE program than those for the control group.  

Even as the gap shrinks over time, ACE students have the advantage of completing transfer-level English, sooner than the control group and this advantage enables them to take other more advanced courses that require transfer-level English as a prerequisite sooner than would otherwise be possible. 

4 College Longitudinal Study of 658 Students Transfer-Level English Completion

4 College Longitudinal Study of 658 Students
Transfer-Level English Completion

Effects of ACE Program on Students' Affective Capacity

This MPR update also reports measured psychological constructs associated with the affective dimension, which are instrumental to student success. ACE students exhibited significant gains in the 8 factors measured: academic self-efficacy, personal responsibility, college identity, four factors of mindfulness, and leadership and teamwork efficacy. The total n for this analysis is 535 students from 7 colleges participating in the ACE program measured at three different times:

  • Gold = Before Foundation Course
  • Brown = End of Foundation Course
  • Tan = End of Semester

Results have shown that students who complete the ACE Foundation Course report a statistically significant improvement at a p<.001 or p<.01 level in six of the eight affective dimensions measured, and at a p<.05 in an additional dimension.  

ACE's Non-Cognitive Effect on Students - 7 College Study of 535 Students
ACE Participants' Mean Scale Scores for CSSAS Factors

  • Notes: ***p < .001; **p < .01; *p < .05; statistical significance is based on comparison with Time 1 scores.
  • The Y-axis represents the mean (average) score for each factor.
  • Survey responses were based on a five-point scale, from “strongly disagree to strongly agree” for the non-mindfulness items and from “never or very rarely true” to “always or almost always” true for the mindfulness items. 

2. Evaluation of the Academy for College Excellence
Year 1 Interim Report (2011)

MPR Associates Inc. is conducting a comprehensive evaluation of ACE programs at six colleges.

The ACE semester begins with a two-week full-time Foundation Course that focuses on communication and work styles. This initial educational experience is designed to ‘light the fire of learning.” ACE believes that affective changes are the preparation and precursor for full-time academics. Working with ACE personnel and University of California Santa Cruz Psychology Professor (emeritus) Martin Chemers, MPR created the College Student Self-Assessment Survey (CSSAS) an instrument that measures students’ perspectives on themselves. This work is also reported in Change Magazine, in an article called “Supporting the Students of the Future” by Diego Navarro, the Director of the Academy for College Excellence.

Change Over Time in ACE Students' Scores on CSSAS Scales, Fall 2010 and Spring 2011; 
* = statistically significant

The Foundation Course focuses on building students’ capacity in each of the affective areas. Results show that this training is particularly effective in the dimensions of Self-Efficacy, Teamwork, Identity, Interaction, Mindfulness, Describing, Accepting, and Observing. The last four variables—mindfulness, describing, accepting and observing—are aspects of mindfulness as measured in the Kentucky Inventory of Mindfulness Skills used by researchers in medicine.

Based on CSSAS results over time, interventions early in the semester—between Time 1 (before starting ACE) and Time 2 (after the Foundation Course)—provide the base for change and remain constant or grow slightly over the course of the Bridge Semester, as measured at Time 3(end of semester). The change is statistically significant.

One more finding to date is that ACE students report ambitious long-term academic goals. Over half of the students hope to earn an Associate’s degree, ver one-third hope to earn a Bachelor’s degree, and nearly half hope to attain a Master’s or higher post-graduate degree.  

MPR is currently analyzing longitudinal college transcript data on student performance and will report findings in the first half of 2012.

3. Educational Outcomes of Cabrillo College’s Digital Bridge Academy: Findings from a Multivariate Analysis (2009)

Community College Research Center (CCRC) at Columbia University released a report in 2009 of a longitudinal study comparing ACE students at Cabrillo College with a comparison group of 11,578 non-ACE Cabrillo College students. The outcomes of ACE students were separated into two groups: students in cohorts 1-3 who took an accelerated English class that was one-level-below transfer English as part of the ACE semester, and students in cohorts 4-9. Cohorts 4-9 did not take an accelerated English class like cohort 1-3 because it was determined that their enrollment in a course more advanced than that indicated by the assessment/placement process was prohibited by the California Educational Code. The reading lab course did not count as an English course, therefore the ACE students lost one semester of English progression, yet as the study indicated, they still progressed in English just as well as non-ACE students. The results were the following:

The graph below shows results of the accelerated model for Cohorts 1-3.

The results in the table below show a statistically significant positive difference between ACE students—both accelerated and non-accelerated—and the comparison group. The exceptions to this pattern are there are no significant differences between non-accelerated ACE students and the comparison group in passing English courses one level below transfer and transfer level, and in transfer credits earned. In these areas, performance matches the comparison group.

“In summary, this study found significant positive effects for participation in both the accelerated and non-accelerated versions of the DBA…

“To the extent that colleges are seeking strategies for increasing the rate at which academically underprepared students complete 'gatekeeper' courses such as college-level English and earn college credits, the accelerated version of the DBA program seems to hold particular promise.

“While the results of this analysis show that participation in the DBA program is correlated with better outcomes on most measures for students…DBA students are very likely more at-risk than are Cabrillo students with similar levels of academic preparation…

“To the extent that the DBA students in the sample were substantially more disadvantaged than other students, [see demographics] it may be that the estimates produced through this analysis understate the effect of participating in DBA.”

Note on demographics: Close to two-thirds of ACE students have at-risk characteristics such as being English Language Learners, being the first in their family to go to college, or placing two levels below transfer-level in English, mathematics, or reading. Approximately one-quarter of the students have high-risk factors such as not being a high school graduate or GED holder or being a parent of dependent children. Fifteen percent of the students have received public assistance.

4. Watsonville Digital Bridge Academy Report I: Student Outcomes Evaluation Cohorts 1 & 2 (2005) & 4. Watsonville Digital Bridge Academy Report 2: Persistence And Achievement (2007)

In these evaluations funded by the National Science Foundation, Dr. Norena Badway of theHigher Education Evaluation and Research Group conducted two sequential evaluations on the characteristics of Digital Bridge Academy students and their personal and academic growth related to their participation.

These studies used the metrics of academic progress: GPA, retention through the semester, progress toward an award, and persistence to the next term. The evaluators viewed these outcomes not only in terms of academic progress, but as markers of self-efficacy indicating confidence that the student can succeed in college. Their findings were that participation in DBA had substantial impact on those four measures.

For the initial two cohorts, which experienced the initial accelerated design, retention in the Bridge Semester was 83% and 79% respectively. Because of the tightly linked curriculum, students who completed the Bridge Semester earned 19.5 college credits.

The second report summarized the longitudinal effects for the five DBA cohorts from Fall 2003 to Fall 2005 and found similar results.

The retention rate for first semester DBA students varied from 74-100%, compared toretention rates of 60% for all Cabrillo basic skills students and 67% for students in credit-level coursework.  The majority of DBA students completed 16 credits with greater success than did their non-DBA peers at Cabrillo.

These findings alone commend the DBA program: under-prepared, at-risk students make marked progress towards academic, career and personal goals. The DBA evidence suggests a need to re-examine beliefs about sheltering at-risk underprepared students.