Standard 4 and A.4

The provider demonstrates the impact of its completers on P-12 student learning and development, classroom instruction, and schools, and the satisfaction of its completers with the relevance and effectiveness of their preparation.

SUMMARY

Trend data indicate VCU completers successfully gain employment and retain in the profession in the formative first three years, with nearly half of EPP completers teaching in Title I schools. Standard 4 evidence, across measures, converges to illustrate that beginning VCU EPP teachers teach effectively and demonstrate positive impacts on P-12 students’ learning, and both employers and completers are satisfied with the preparation and performance of our initial completers. Evidence suggests there are opportunities to enhance preparation for teaching in hard-to-staff schools where many VCU EPP completers teach. However, VCU EPP completers are described as both “eager to teach” and “adaptable to feedback” that seems to serve them well in their initial entry to the profession across school settings.

EXAMPLES OF EXCELLENCE IN PRACTICE

• Candidates demonstrate positive impact on student learning
1. Nearly all recent completers in their first three years after graduation earned a rating of “Exemplary” or “Proficient” on the principal-evaluated Virginia Performance Standard 7, assessing Teacher Impact on Student Learning (92.8%-100%). Supervising principals assess “Overall positive impact on student growth and/or learning”, on average for three data cycles, at Proficient.
2. Candidate self-report examples of impact on student learning include instances where difficult content was finally understood, where students improved behavioral issues related to trauma and anxiety, where relationships were formed that enhanced social development, among many other powerful moments.
• Completers and Employers indicate satisfaction with VCU educator preparation
  1. Employer feedback indicates VCU EPP completers perform well, respond positively to feedback and are adaptable.
  2. In 2019, initial licensure completers indicated they are “Somewhat satisfied”, approaching “Extremely satisfied” with their VCU EPP preparation with a mean score of 4.48 on a 5-point scale.

IMPROVEMENTS EMERGING FROM THE SELF-STUDY

• The VCU EPP launched several efforts observing completers in their classrooms so as to provide structured feedback. These efforts support our beginning teachers, as well as our understanding of completer effectiveness and impact on learning and classrooms.
• The VCU EPP is engaged in the Virginia Education Assessment Collaborative, partnering with nearly half of Virginia colleges/schools of education in the development and validation of state-wide completer and employer surveys for Standard 4. This creates opportunities for state-wide benchmarks to better understand VCU EPP completer outcomes.
• Employer and completer feedback indicate opportunities to enhance preparation for hard-to-staff schools. The EPP is engaged with P-12 partners through several committees to co-construct clinical experiences, and understand school needs and experiences to best inform teacher preparation.
• Completer feedback emphasized the opportunity to diversify both program faculty and program candidates. The SOE strategic plan details strategies and efforts for the recruitment and retention of under-represented minority faculty and students.

VCU EPP has developed and implemented a multidimensional, systematic approach for collecting, analyzing, and using data to understand the post-preparation employment experiences of our completers. VCU EPP collects qualitative and quantitative evidence to create opportunities for triangulation of data across multiple measures, gathering information from completers, their employers, and their employing school divisions to seek saturation to support appropriate inferences about completer effectiveness and impact.

Standard 4 data sources and quality

The baseline data source for tracking VCU EPP completers derives from the Virginia Department of Education (VDOE) Teacher Licensure Application and Annual Instructional Personnel (IPAL) Data Collection. Beginning in 2017, the VDOE started providing each EPP employment information for recent completers (3-5 years out of the program) employed in a Virginia public school as of October 1st of the school year prior. For example, in June 2019, VCU received information on the employment of completers earning a license in 2014-2018 and employed in a Virginia public school as of 10/1/2018. The VDOE started providing these data in 2016; therefore, longitudinally, VCU can explore both employment and retention for recent completers year over year. Further, using employment information from VDOE, VCU EPP identifies the completer cohorts for the completer/employer surveys and impact data. Additional measures collected to corroborate the employment placement and satisfaction data include focus group qualitative data, other employment milestones, and pilot evidence of teacher effectiveness through case studies of recent completers. All measures align to CAEP and InTASC Standards, and surveys meet CAEP criteria for assessments at the "Sufficient" or higher level (85.2).

4.1 Completer impact on P-12 student learning

VCU EPP is committed to preparing teachers who positively impact the development and learning of their students. Positioned in a state that does not provide EPPs with data on completer impact, VCU developed and monitors multiples measures to understand the degree to which completers are contributing to expected learning growth in schools. This includes annual principal evaluation of impact, an employer survey item assessing impact, completer self-reported examples of impact, as well as a pilot case study of impact. The pilot case study provides evidence of both components 4.1 and 4.2. In addition, a Phase-in Plan is presented for component 4.1 as the EPP develops representative evidence for three cycles (76).

Virginia public school teachers receive an annual principal evaluation on Virginia Teaching Performance Standards. Virginia Performance Standard 7 (i.e., The work of the teacher results in acceptable, measurable, and appropriate student academic progress.) provides a measure of impact using a four-point scale of Exemplary, Proficient, Developing/Needs Improvement, Unacceptable, where a rating of "Proficient" is the expected level of performance. Following a pilot data collection cycle in 2017 to collect data on one year of completers, annually, since 2018, four school divisions in the greater Richmond Metropolitan area provided Virginia Performance Standard 7 data to VCU on recent completers employed in their division. Teachers in all licensure areas and grade levels establish student learning goals with their principal each year and monitor learning as measured using division, state, or national measures. As such, these data (75) capture evidence for all licensure programs and are not limited to core subject areas. In 2017, 100% (n=20) of 2016 completers earned a rating of "Exemplary" or "Proficient" on Standard 7: Impact on Student Learning. The rubric semantic rating for Proficient states "The work of the teacher results in acceptable, measurable, and appropriate student academic progress". "Exemplary" on the rubric means "In addition to meeting the standard, the work of the teacher results in a high level of student achievement with all populations of learners." Expanded data collection in 2018 yielded data for 2015-2017 completers (n=290), with 98.6% of completers demonstrating "Exemplary" or "Proficient" impact on learning. Results in 2019 (n=303) indicate 92.8% of evaluated completers demonstrated "Exemplary" or "Proficient" impact. These data are supported by Employer Survey ratings of "Acceptable" (2017-18: mean score of 3.96 out of 6; item: Overall positive impact on student growth and/or learning") and "Proficient" (2018-19: mean score of 3.16 out of 4; item: The work of the teacher results in acceptable, measurable, and appropriate student academic progress) on related items.

Our program completers in their first three years are invited, as part of the annual completer survey, to share examples of impact on a student(s) learning or development. Qualitative examples (79.1) further illustrate the impact of recent completers in a range of subject matter areas and in a variety of ways, such as academic, personal, and social growth for students. Examples include instances where difficult content was finally understood, where students improved behavioral issues related to trauma and anxiety, where relationships were developed that enhanced social development, among many other powerful moments.

The EPP also selected a case study to engage in an in-depth analysis of recent VCU completer impact and effectiveness (74). VCU implemented four cycles of data collection from spring 2019 through spring 2020, using three pilot strategies examining multiple data sources as detailed in the case study summary (74). Case study findings (74) related to CAEP component 4.1, completer impact on P-12 student learning, confirm that VCU completers demonstrate a positive impact on their P-12 students' learning. Half (4 of 8) of the 2019-2020 case study participating completers were observed for impact on student learning, due to either unforeseen school-specific issues (e.g., health issue of supervisor) or due to the COVID-19 pandemic that interrupted data collection. All assessed completers scored at Target or Exemplary levels indicating a positive impact on learning.

4.2 Impact on classroom instruction and schools

Across the case study data collection cycles (n=8 teachers), findings (74) exceed the case study propositions with six of seven completers rated at Target level on the Clinical Evaluation Continuum (CEC) and at Exemplary on the Virginia Performance Standards on all standards assessed. For student teachers during the teacher preparation program, the benchmark performance level is "Acceptable" with candidates demonstrating some "Target" behaviors reflecting performance of an accomplished teacher. As such, completer data at "Target" on the CEC suggest effectiveness meets the behavioral performance benchmarks of an accomplished teacher. Further, data for the completer assessed using the school division PGPP on Virginia Performance Standards indicates effectiveness that exceeds the benchmark (Proficient for the Virginia Performance Standards). Across all seven cases, data indicate a consistent strength in the use of instructional strategies to maintain and monitor student behavior and engagement, including both classroom setting strategies (e.g., seat arrangements) but also strategies for reinforcing behavioral expectations and intervening to maintain the learning environment.

The EPP piloted three different data collection approaches to gather evidence of completer effectiveness and impact (74). In the absence of a state-wide data system on teacher evaluation and impact data, VCU EPP employed each of these three strategies consecutively, each as a pilot, to both provide preliminary evidence of recent completer impact and effectiveness, aligned with CAEP Standard 4.1 and 4.2. Although the overall sample size is moderate, the pilot cases help inform subsequent data collection with a larger sample and data collection design across programs to support continuous improvement.

Use of 4.1 and 4.2 data for continuous improvement

Each of the case study pilot approaches involved various resources from assessment and accreditation staff, faculty, and completers (74). The faculty observations employed in data cycle three, did have some limitations; however, overall, as an approach, this strategy seems most promising for providing data that can guide program improvement. Piloted in three program areas, faculty found completers accepted the invitation to the case study openly for Math and Science. However, for Early Childhood Special Education, completers indicated a reluctance to introduce an observer to the classroom as it might disrupt an already challenging classroom and may provoke behavioral concerns. Further, the school-level approval for the observation in special education presented additional demands on the special education faculty to gain access. These important procedural considerations need to be addressed in future data collection to most effectively communicate with completers and school leaders. The goal is to increase participation while partnering in mutually beneficial ways.

Despite some logistical challenges to the faculty observations, the faculty observers shared that they directly benefited from observing their recent completers. First, directly engaging with recent completers provided a touchpoint to offer non-threatening feedback to completers. Second, faculty observed direct evidence of completer performance and observed a modern classroom supporting continued awareness of how classrooms and students are changing. Observations help ensure relevant instruction in the EPP programs. Next steps include sharing results of the case study with program faculty and P-12 partners to discuss inferences and limitations both to the data and the methods. These discussions will occur in fall 2020.

4.3 Satisfaction of employers

To address component 4.3 and to support EPP understanding of employer satisfaction with recent VCU EPP completers in practice, VCU EPP used surveys and focus groups for employers to provide feedback. Evidence 78.1 describes the processes used in the Employer Survey and Focus Group, and includes summary themes. Evidence item 78.2 showcases Employer Survey data by year and by program. Exhibit 82 provides archival evidence of employment, retention, promotion, and National Board Certification as supporting employment milestones.

Employment data

Employment data (82) suggest the majority of VCU EPP completers seek and gain employment in a Virginia public school and are retained at high rates year over year, indicating a degree of employer satisfaction with the performance of VCU EPP completers. Specifically, 66.8% of 2014-2016 completers recommended for licensure by VCU worked in a Virginia public school as of 10/1/2016; 71.3% of 2015-2017 completers as of 10/1/2017; and 67.6% of 2016-2018 completers as of 10/1/2018. With three years of employment data provided to VCU by the VDOE, the EPP explored year over year retention in both 2018 and 2019. Findings indicate the overwhelming majority of 2015 (84%) and 2016 (95.7%) completers retained from 10/1/2016 to 10/1/2017 and 2016 (98.7%) and 2017 (93.8%) retained from 10/1/2017 to 10/1/2018.

Employer Survey

The Employer Survey provides employers of initial licensure program completers the opportunity to indicate, based on their observations in the field, how well prepared VCU program completers are for their chosen profession. In 2018, VCU EPP facilitated their own Employer Survey of local school divisions. In 2019, VCU EPP partnered with the University of Richmond and University of Virginia to pilot a common form, creating the opportunity for benchmark data and a means to better partner with school divisions on a shared timeline across EPPs in contacting school leaders. Finally, in 2020, the three-EPP collaborative expanded, forming the state-wide Virginia Education Assessment Collaborative. Together, 18 participating EPPs across the state partnered to implement the common Employer Survey form; data collection is currently delayed due to COVID-19.

The pilot Employer Survey study in 2018 invited principals to provide survey feedback specifically for the recent graduate(s) they supervise. For the pilot cycle, the EPP sent the survey only to completers of the most recent calendar year to provide one cycle of pilot data to inform process and content. Using employment information from VDOE, VCU EPP identified the school-level employer of each initial licensure completer teaching in a Virginia public school in the four surrounding divisions of the greater Richmond metropolitan area for selection to the annual 2017-18 and 2018-19 VCU EPP Employer Survey. VDOE data indicate the majority of completers (as of 10/1/2016=70.2%; 10/1/2017=64.3%) practice in these four school divisions, and the distribution of teachers by licensure area in these four divisions is representative of the completer population (4.d, representativeness). Employers of 2016 completers in the pilot spring 2018 survey (response rate=39%; n =22) rated 86.3% of VCU completers at Good or Excellent on preparation and performance, and 100% of respondents indicated they are Likely or Very likely to hire future VCU graduates based on the preparation and performance of recent completers in their school. Of note, 31.8% of respondents were principals in a Title I school, with 27.3% described as Urban, 68.2% as Suburban, and 4.5% as Rural. Analysis of 2018 pilot data indicated no significant difference in employer ratings by licensure program or by school level. Data indicated completers in non-Title I schools were rated more highly than those in Title I schools on creating and maintaining a positive and safe learning environment, engaging and supporting students in learning, and overall impact on student learning (78.1; 78.2).

In fall 2018, the EPPs at VCU and the University of Richmond partnered to develop a common employer survey form, guided by the CAEP Evaluation Framework for EPP-Created Surveys, aligned to InTASC, CAEP, and VDOE teaching standards. Further, the two institutions established and shared an administration calendar and invitation language to facilitate a more streamlined effort in 2019 to reach out to partnering school divisions to support response rates and efficiency for both the participating employers and the two institutions. The University of Virginia partnered to administer the common form to their employers also. The three institutions create opportunities for shared benchmark data to inform each EPPs findings. Initial benchmark results are available (77).

Findings for the 2018-19 survey of employers (response rate=58%; n=144) satisfaction were stable with the previous cycle with 86.9% of 2015-2017 VCU completers rated at Good or Excellent on preparation and performance, and 95.9% of employers Likely or Very likely to hire future VCU graduates. However, for this data cycle, the profile of responding employers changed, with 62.1% leading at a Title 1 school (39.7% Urban, 53.4% Suburban, 6.8% Rural), and principals identified each of the following as characteristics of their school: High population of ELL (34.5%); High SES (29.7%); Low SES (63.4%). Analysis of 2018-19 data indicated no significant difference in employer ratings by licensure program, by school level, or by school region (e.g., urban, rural, suburban). Data indicated differences on select items for teachers in Title I compared to teachers in non-Title I schools and for teachers in schools characterized as high SES compared to teachers of other schools. Specifically, employers rated completers in non-Title 1 schools higher than completers in Title-1 schools on four items (i.e., understanding of curriculum/content/student needs; impact on student learning; reflective practice; and overall likelihood to hire). Employers indicated they were more likely to hire VCU completers based on the performance of completers in schools characterized as high-SES versus non-high-SES (78.1; 78.2).

Benchmark data (77) for the three participating Virginia EPPs indicate VCU EPP completers evaluated in the 2018-19 survey score slightly lower than the mean on each of the competency items on the common Employer Survey, but VCU EPP mean scores are all within one standard deviation of the state mean by item. Disaggregated program data indicates completers by program received ratings within one standard deviation above or below the benchmark mean with no notable outlier programs. Given the variation in VCU EPP ratings by Title I and SES status of the school, understanding the profile of employers' schools for the state-wide benchmark schools will be helpful moving forward to understand if the high proportion of VCU EPP employers responding from Title I and low SES schools skews the overall average rating of VCU completers in practice.

Building on the efforts of these three state EPPs, VCU EPP staff are engaged in VEAC (Virginia Education Assessment Collaborative), a collaborative created to develop and test common Standard 4 measures for Virginia EPPs. Therefore, for the 2019-20 data collection cycle, VCU EPP intends to utilize the common measure, shared with 17 institutions, leading to both shared processes for our P-12 partners and also more robust benchmark data to support reliable and valid inferences about employer satisfaction (77). Due to COVID-19 and the closure of Virginia public schools on March 13, 2020, VEAC delayed the spring 2020 Employer Survey and is partnering with school divisions to discern an appropriate time to collect this data in summer or fall 2020; findings will be forthcoming.

Employer focus group

Designed to supplement Employer Survey data (78.2), VCU EPP implemented an initial licensure employer focus group in 2019 (78.1). Faculty program chairs identified school- or division-level employers well-positioned to provide VCU with feedback on our initial licensure program completers. Content for the focus groups and interviews align to core concepts in the InTASC and CAEP Standards. Further, the content reflects key programmatic outcomes on the CEC completed during student teaching and mirrors the Employer Survey questions to support triangulation of data. Five employers participated. Themes emerged from the qualitative data from the focus group to confirm survey data suggesting VCU EPP completers generally perform well in their beginning years of teaching; however, those in Title-I or hard-to-staff schools do have opportunities for improvement in classroom management and cultural competency to create a student-centered learning environment. Focus group participants consistently agreed that VCU completers maintain a commitment to professional ethics and respond positively to feedback and are adaptable. Employers discussed the teacher shortage extensively, particularly in Title-I schools. Employers shared that teacher vacancies are often in Title 1 schools and recommended VCU match training experiences to job opportunities. Based on VDOE data as of 10/1/2018, 45% of 2016-2018 completers employed in a Virginia public school taught at a Title-I school. Thus, the EPP does directly serve this P-12 school need. Based on variability in the interview skills of VCU completers, employers did recommend VCU host an employer panel to help completers understand expectations for new teachers and the needs of schools.

Overall, triangulation of evidence across employment outcomes and milestones and employer feedback suggests employers are satisfied with the preparation and performance of beginning VCU EPP teachers, supporting that the EPP meets 4.3. The majority of completers gain employment in the first three years out of the program and then persist to teach in Virginia through those initial years. Feedback data indicate opportunities exist to improve preparation for hard-to-staff schools where nearly half of completers work, particularly in classroom management and understanding of trauma and needs of students in those settings. These challenges of beginning teaching are not unique to VCU completers but do suggest dialogue is needed on how to address this need through curriculum and clinical experiences, as well as induction support.

4.4 Completer satisfaction

Quantitative and qualitative data from a Completer/Alumni Survey and Focus Group (79.1; 79.2) converge to demonstrate that recent VCU EPP completers (1-3 years out from the program) are satisfied with their initial preparation.

Completer survey

The Completer Survey provides program completers from initial licensure programs the opportunity to indicate how well they were prepared to enter their chosen profession. Completers are identified from VDOE employment data. In the pilot data year (2017-18), only the most recent calendar year of completers participated in the survey, yielding 14 responses (7%). After pilot implementation, beginning 2018-19 data collection cycle, initial licensure completers from the three most recent calendar years as provided by the VDOE comprise the annual completer cohort.

VCU EPP employed an EPP-specific Completer Survey in 2017-18 and 2018-19, transitioning to a common form facilitated by VEAC in 2019-20. For the original form, faculty and accreditation staff developed the survey items to ensure item content reflected the InTASC and CAEP standards. The respondent profile (response rate=8.6%; n=51) indicated completers teach in schools with the following characteristics: 59.6% (2019) Title I school; 80.9% (2019) Low SES; 25.5% (2019) High SES; 40.4% (2019) High population of ELL, and 40.8% (2019) Urban; 42.9% (2019) Suburban, and 16.3% (2019) Rural. These data approximately mirror the Employer Survey school characteristics data, supporting representativeness of the sample. Overall, completers rated their preparation as "Good", approaching "Excellent" on a four-point scale (2018=3.71; 2019=3.45). Further, all items received a rating of at least "Good" or better. Review of qualitative survey comments for 2019 revealed 10% of survey respondents indicated a need for support with classroom management, children with ongoing behavioral needs, or trauma and social-emotional impacts on behavior. The small response set for the pilot year 2018 did not support disaggregation of data by program; however, disaggregated data for 2019, suggest most programs are also rated "Good" or better. Statistical MANOVA analysis on 2019 data indicated no significant differences in completer ratings by program or by graduation year.

In 2019-20, VCU EPP partnered with the pilot institutions in VEAC to administer a common form (77), facilitated by a common process. This revised form for the EPP mirrors the content of the VEAC Employer Survey, derives from the Virginia Teaching Performance Standards, and aligns with both InTASC and CAEP Standards. The VEAC survey did not include items to capture school characteristics. The respondent profile (response rate=17%; n=73) included feedback at reportable levels (n>10) for each completer year (2015=23%, 2016=32%, 2017=27%, 2018=18%) and for the following programs (Elementary=34%, History and social science=7%, Visual Arts P-12=15%, with other programs having fewer than 10 responses). Overall, completers indicated they are "Somewhat satisfied", approaching "Extremely satisfied" with their VCU EPP preparation with a mean score of 4.48 on a 5 point scale. These findings are consistent across programs (79.2).

Completer focus group

Completer Survey data are shared annually with program chairs and the Assessment Committee (92; 93). Faculty indicated limited confidence in using the data to improve their programs due to low response. Therefore, to supplement survey data, VCU EPP engaged in two focus groups with recent completers (n=14). Focus group data highlighted several key themes that echoed qualitative feedback from the survey data. Specifically, completers find, as strengths, that their coursework is relevant to practice and many completers complimented the high quality of teaching faculty. Focus group discussions emphasized three main opportunities to improve VCU EPP preparation programs: 1) Increase preparation for classroom management, 2) increase diversity in teaching faculty; more faculty with teaching experience in Title I schools; 3) offer faculty training on racial and gender bias and sensitivity to LGBTQIA student needs.

Although response rates are low for the three cycles of completer feedback data, the findings suggest consistency in response between cycles of the survey and with the focus group, suggesting completers are satisfied with their VCU EPP preparation. The focus group provided supplemental data that confirmed that overall theme. Evidence to date suggests VCU EPP meets component 4.4 and efforts continue to engage completers in providing a more robust response, including a plan for more strategic partnerships with the communications team to employ social media to promote response.

Use of 4.3 and 4.4 data for continuous improvement

Multiple stakeholder groups reviewed employer feedback data, including the Assessment Committee (AC), the Continuous Improvement Task Force (CITF), Clinical Experience Advisory Board, and the Professional Education Coordinating Council (PECC). P-12 partners serving on PECC reviewed employer feedback at the April 2019 meeting and confirmed themes as consistent with their experiences through hiring and supervising recent completers. AC reviews employer feedback annually and reviewed the summary analysis across survey, focus group, and milestone data at the October 2019 meeting (25). Faculty find the multiple measures to be a strength with data converging across sources, providing support for actionability of the data. Faculty gleaned that completers are doing well in the first three years. It was noted, as a point of context, that nearly half of recent completers work in Title I schools and both focus group feedback and Employer Survey t-test analysis indicated new teachers perform lower in those settings. VCU EPP engages P-12 partners through the PECC and CEAB committees in reviewing candidate and completer data, providing feedback on clinical performance and experiences, and contributing to the development of assignments (25, 23). Further, partners provide the EPP with school division needs and updates. Through these partnerships, the EPP seeks to understand the context of partner schools/divisions and engage partners in co-constructing learning experiences that are mutually beneficial by enhancing VCU EPP candidates' preparation for Title I schools.

Measures of completer satisfaction are reviewed to inform continuous improvement. For example, AC review of completer survey data highlighted that completers felt less prepared to integrate technology into instruction relative to other skill items on the survey. Thus, the elementary and secondary programs felt additional course content was needed on this topic. A new technology course was developed and approved by the Curriculum Committee as a formal requirement of those programs (79.1; 79.2). In addition, completer feedback emphasizes the value of diversity in faculty and candidates/peers. As described in Standard 3, VCU EPP engages in ongoing strategic recruitment efforts, aligned to the SOE strategic plan, to diversify the licensure applicant pool. In addition, the strategic plan focuses on strategies to increase diversity among faculty, as well as ensuring a culture that promotes retention of diverse candidates and faculty. Focus group findings confirm feedback on the Completer Survey and the PECC (25) found the findings consistent with their experiences in the field, providing supportive triangulation of data; however, additional focus groups and strategic efforts to increase survey response are needed to ensure the EPP has a broad representation of alumni experiences and perspectives. In October 2019, the AC recommended several strategies to support understanding completer satisfaction, including: repeating the focus group, exploring opportunities to connect with completers through Alumni Council programming or program-level events, faculty encouraging participation from candidates prior to graduation, offering an incentive (95).

Closing Statement

Trend data indicate VCU completers successfully gain employment and retain in the profession in the formative first three years, with nearly half of EPP completers teaching in Title I schools. Standard 4 evidence, across measures, converges to illustrate that beginning VCU EPP teachers teach effectively and demonstrate positive impacts on the learning of their P-12 students' learning, and both employers and completers are satisfied with the preparation and performance of our initial completers. Evidence suggests there are opportunities to enhance preparation for teaching in hard-to-staff schools where many VCU EPP completers teach. However, VCU EPP completers are described as both "eager to teach" and "adaptable to feedback" that seems to serve them well in their initial entry to the profession across school settings.

EPP faculty and leadership continue to unpack data to inform improvements to the QAS to generate robust data on completer effectiveness and impact. Multiple data-driven changes derive from a review of Standard 4 data, including the addition of an initial licensure technology course for elementary and secondary students (10) and ongoing pilot efforts to observe completers in the classroom to provide structured feedback to support beginning teachers' effectiveness and impact on learning and classrooms (74; 75). Additionally, future plans include continued engagement in the development and validation of state-wide measures for Standard 4 to enhance understanding across programs, in varied school settings, and over time.

Summary of Evidence and Supporting Documentation

• 74: 4.1 & 4.2 - Case Study of Impact and Effectiveness
• 75: 4.1 Summary of initial completer impact
• 76: 4.1 Initial Phase in Plan Completer Impact
• 77: VEAC Initial Licensure Standard 4 Survey Guide, Forms and Pilot Benchmarks
• 78.1: Initial employer survey and focus group summary feedback
• 78.2: Employer Survey Initial Licensure Summary Data
• 79.1: Initial licensure completer survey and focus group summary feedback
• 79.2: Initial Licensure Completer Survey Summary Data
• 82: Initial & advanced employment outcomes

The provider documents the satisfaction of its completers from advanced preparation programs and their employers with the relevance and effectiveness of their preparation.

SUMMARY

The VCU EPP values the feedback of both employers and completers in guiding programmatic improvements. The EPP engages in a systematic, multi-faceted approach to gather and analyze high-quality evidence on the satisfaction of employers and completers via surveys and employment milestone data. Taken as a whole, evidence suggests completer and employer satisfaction with both the relevance and quality of VCU advanced program preparation; however, small completer counts in select programs yields small sample sizes and limited data on select measures. As such, in addition to reporting evidence to date, the EPP presents plans to strengthen our evidence over time, increasing our confidence in our interpretations regarding the post-preparation experiences of our advanced completers.

EXAMPLES OF EXCELLENCE IN PRACTICE

• Advanced candidates gain employment and retain in education
1. Employment data made available to VCU by the VDOE indicate the majority of VCU EPP completers seek and gain employment in a Virginia public school and are retained at high rates year by year, across advanced programs.
2. Most advanced completers are moving into positions related to their advanced license; educational leadership graduates, however, do not always make that move immediately after program completion. Reading Specialists do, at times, stay in the classroom as teachers.
• Completers and Employers indicate satisfaction with VCU educator preparation
1. Employers and completers consistently rated the preparation and performance of advanced completers, on average, as “Good” or “Excellent”.
2. Completers in focus groups spoke very positively about their experiences with teaching faculty and advisors.

IMPROVEMENTS EMERGING FROM THE SELF-STUDY

• Like initial licensure programs, advanced completer feedback emphasized the need to diversify both program faculty and program candidates. The SOE strategic plan emphasizes strategies and increased efforts for the recruitment and retention of underrepresented minority faculty and students.
• The EPP developed a Phase-in Plan due to small response rates for both completer and employer feedback for the self-study cycle. This plan details opportunities to enhance our data quality to support confidence in our statements about completer and employer satisfaction with advanced programs.

VCU EPP engages in a systematic, multi-pronged approach to gather and analyze high-quality evidence on the satisfaction of employers and completers of our advanced programs. The quality assurance system supports annual, cyclical procedures to collect completer and employer feedback via surveys and employment milestones data derived from employment information shared with EPPs by the Virginia Department of Education. Taken as a whole, evidence suggests completer and employer satisfaction with both the relevance and quality of VCU EPP advanced program preparation; however, small completer counts in select programs yield small sample sizes and limited data on select measures. As such, in addition to reporting evidence to date, VCU presents plans to strengthen our evidence over time to be more confident in our interpretations regarding the post-preparation experiences of our advanced completers.

Employment data made available to the EPP by the VDOE indicate the majority of VCU EPP completers seek and gain employment in a Virginia public school and are retained at high rates year over year, across advanced programs (82). Specifically, 77.1% of 2015-2018 completers were working in education as of 10/1/2018, with 81.3% of those working in the Richmond area school divisions confirmed working in the advanced area for which they were trained. Review of data suggests most advanced completers are moving into positions related to their advanced license; however, educational leadership graduates do not always make that move immediately after program completion, and reading specialists do, at times, stay in the classroom as teachers. Data indicate 25% (31/125) 2015-2018 completers worked in a Title I school in the school year 2018-19: Math Specialist (22%); School Counselor (23%); Reading Specialist (28%) and Educational Leadership (29%) (82).

Standard A.4 data sources

Detailed Phase-in Plans for A.4.1 and A.4.2 (80) provide the timeline, measures, resources, and planned engagement of stakeholders in data use as the EPP continues efforts to ensure CAEP Standard A.4 is met and that completer and employer satisfaction data is an integral component of the program improvement quality assurance system. Much like the initial program Standard 4 procedures, the baseline data source for identifying and tracking VCU EPP advanced completers derives from the Virginia Department of Education (VDOE) Teacher Licensure Application and Annual Instructional Personnel (IPAL) Data Collection. VCU uses these data to determine the extent of employment in education and the field in which advanced completers were trained, and VCU EPP can explore both employment and retention for recent completers year over year. Further, VCU EPP uses the employment information from VDOE to identify who to survey for the Completer and Employer Surveys. All measures (81) align to CAEP Standards. Surveys meet CAEP criteria for assessments at the "Sufficient" or higher level and are under ongoing review and revision to ensure content validity (85.2).

A.4.1 Satisfaction of employers

VCU EPP administers annual Employer Surveys for each advanced program to understand their satisfaction with the preparation and performance of recent program completers. The Employer Survey provides employers of advanced program completers the opportunity to indicate, based on their observations in the field, how well prepared Virginia Commonwealth University program completers are for their chosen profession. They are also solicited to offer feedback on strengths and weaknesses of the program in light of current needs in the employers' classrooms and schools. Employers are identified using employment placement data provided by the Virginia Department of Education.

For the three data feedback cycles (83.1/83.2), employers consistently rated the preparation and performance of advanced completers, on average, as "Good" or "Excellent". Further, descriptive comments provided by employers highlight that advanced completers are very knowledgeable, collaborate well with colleagues, are positive in their work, and passionate about their field. However, with the exception of School Counseling that has high response rates (2017-18 = 75%; 2018-19 = 42%; 2019-20 = 53%), other advanced programs receive inconsistent response rates each year, given small program sizes. The EPP compared the number and percent of degree earning graduating VCU EPP completers for each data cycle by licensure program and compared those proportions to the proportion of employers responding to the Employer Survey for completers of each licensure area. Representativeness data indicate the proportion of employer survey respondents closely mirrors the proportion of completers for mathematics and reading specialists, suggesting a good representativeness of the feedback. However, educational leadership is not sufficiently represented in the results, and school counseling is over-represented relative to the number of completers. Efforts to ensure representation of educational leadership in response are detailed in the Phase-in Plan.

Use of employer satisfaction data to inform continuous improvement

The Assessment Committee remains actively engaged in the development of completer measures and data analysis (95). As such, they contributed to the Phase-in Plan that includes steps in the coming semesters to improve total response on the Employer Survey, as well as alignment of content to CAEP Standards, to enhance actionability on the evidence. Specifically, efforts will include: reviewing item content to ensure professional skills are fully represented and stated in terms that are measurable; partnering with program and department chairs to send follow up emails encouraging participation; potential incentives; discussion with P-12 partners for their input on the process.

Upon review of data by both program faculty and the Assessment Committee, the Assessment Committee (95, October 2018; January 2019) recommended focus groups and phone interviews as supplemental approaches to triangulate evidence collected via Employer Surveys. The Assessment Committee and program coordinators partnered with Assessment Office staff to develop scripts and protocols for focus groups/interviews in spring 2019; however, employer schedules created scheduling challenges that delayed data collection. The Phase-in Plan for A.4.1 (80) details a revised timeline for collecting qualitative feedback. The EPP will dialogue with P-12 partners in 2020 to discern a process, schedule, and appropriate incentive that might encourage and support greater participation in the focus groups/interviews proposed for implementation in spring 2021. The Phase-in Plan for A.4.1 also includes planned efforts to engage with advisory groups and partners, by way of program faculty, to collect additional feedback to supplement survey data.

A.4.2 Satisfaction of completers

VCU EPP combines the use of both completer surveys and focus groups to explore completer satisfaction with the relevance and effectiveness of their preparation program (84.1; 84.2). The two approaches allow the EPP to triangulate both quantitative and qualitative data while canvassing a broad sample via survey and also going in-depth with a representative sample in the focus group setting. On the Completer Survey, completers of the program in their first three years out of the EPP are invited to indicate how well they were prepared to enter their chosen profession and to offer feedback on their program to program faculty.

Findings for both 2017-18 and 2018-19 indicate completer satisfaction in the preparation for all programs, with ratings, on average, between "Good" and "Excellent", with only a few exceptions. Pilot data collected in 2017-18 (response rate for each program: Reading=8% (n=1); Ed Leadership=7.4% (n=2); School Counseling=17% (n=6)) sampled only completers in their first year of practice; whereas, in 2018-19, three years of completers participated and item mean scores rose relative to first-year completers only (response rate for each program: Reading=9.7% (n=3); EdLeadership=10% (n=8); School Counseling=14% (n=14)). Across programs, and within programs across years, there are no particular patterns to items that fell below the "Good" rating. Further, with limited response in all programs except School Counseling, one low rating can skew the averages significantly.

Prior to 2019-20, the Completer Survey acted more like an Alumni Survey, canvassing feedback from all licensure program alumni, three years post-program completion. However, guided by CAEP's definition of program completer (a completer of the approved licensure program that has been teaching/working in the professional role for at least six months), the sample for the 2019-20 Completer Survey changed. For this cycle, only completers who the EPP confirmed as working in the role for which they were trained received an invitation to provide feedback on satisfaction with the program. Under this model, only School Counseling received actionable completer satisfaction data with a 47% return rate (response rate for each program: Reading=0% (n=0); EdLeadership=0% (n=0); School Counseling=47% (n=14)). Data for this program indicate the same "Good", approaching "Excellent" pattern as years prior.

Based on ongoing review of Completer Survey data (84.2; 84.1) with potential limitations to inferences due to response rate, the Assessment Committee (95) recommended the EPP engage in focus groups, as detailed in the Phase-in Plan for A.4.2, to allow for in-depth dialogue with completers to understand their experiences in their profession and the extent to which VCU EPP prepared them for that role. In spring 2019, six alumni of advanced programs participated in a focus group; the script is available (84.1). Three alumni worked in elementary school settings: Reading (two completers) and Educational Leadership (one completer). Three worked in middle school settings: Reading (one completer), School Counseling (one completer), and Mathematics Specialist (one completer). Each participant completed their VCU EPP advanced program in the last one to three years and currently works in a Greater Metropolitan Richmond area school division. All participants were women, including three minority women.

Across advanced programs, key themes consistently emerged from the focus group. Across all programs, alumni spoke very positively of their experiences with teaching faculty and advisors. Alumni consistently reported on availability and support of advisors in the program. Further, participants mentioned the faculty as a strength of VCU's education program, citing passion for subject-matter; great professional experiences; knowledge of subjects taught; understanding of the working adult student & flexibility with assignments and projects. This finding is corroborated by qualitative comment feedback provided annually by completers on the Completer Survey who consistently note faculty expertise as a program strength. Focus group participants felt their program experience would be enhanced by a more diverse faculty, as well as a more diverse set of peers/classmates. Select alumni in advanced programs note the lack of diversity among classmates in the program, limiting the diversity of perspectives and experiences that could be shared and learned from among peers. Minority alumni lamented the experience of very often being the only minority in their VCU EPP classroom, though they acknowledged the limited diversity of the current profession. Reflection on this feedback with faculty highlighted the cohort model used to engage many division partners for course delivery has limitations of a homogenous grouping effect that leads to this experience of occasional limited diversity.

Use of completer satisfaction data for continuous improvement

Although some data are available from the survey and focus group, the EPP feels there are opportunities to enhance our data quality to support confidence in our statements about completer satisfaction with advanced programs. As such, the Assessment Committee established a Phase-in Plan for A.4.2 that details strategies to build on the preliminary findings collected to date to establish high-quality evidence consistent with CAEP criteria. Two significant improvements to the process started in spring 2020 with the pilot implementation and analysis of a revised form for math and reading specialist advanced programs and the targeted sample of CAEP-defined completers. Further, the Phase-in Plan (80) details the multi-pronged approaches to encourage response, including: partnering with the program and department chairs to send follow up emails encouraging participation, exploring opportunities to connect with completers through Alumni Council programming or program-level events, encouraging participation from candidates prior to graduation, and the possibility of incentives, in addition to other strategies to roll out in time.

In response to completer concerns about diversity in program peers, the EPP recruitment plan, aligned with CAEP and the VCU / SOE Strategic plans, emphasizes recruitment of diverse candidates into licensure programs as a key goal. Ongoing efforts of the graduate team of program coordinators and recruitment, admissions, and student success staff are focused on developing and implementing strategies to help the EPP increase candidate diversity in the number of applications and those who choose to enroll in VCU EPP programs. These data and their use are summarized in Standard 3, specifically.

Closing Statement

VCU EPP values the feedback of both employers and completers in guiding programmatic improvements and remains committed to implementing the Phase-in Plans as detailed. This will support a robust and representative understanding of their satisfaction with the relevance and effectiveness of VCU EPP advanced preparation, leading to data-driven ways the EPP can improve programs to ensure high-quality preparation curriculum and experiences leading to practice in P-12 schools.

Summary of Evidence and Supporting Documentation

• 80: A.4 Advanced Phase in Plan for Satisfaction with Preparation
• 81: Advanced Employer and Completer Survey forms
• 82: Initial & advanced employment outcomes
• 83.1: Advanced employer perceptions of completer preparation summary
• 83.2: Advanced Employer Survey Advanced Summary Data
• 84.1: Advanced Completer Survey Summary
• 84.2: Advanced Programs Completer Summary Feedback Data