Program Evaluation and Development
The institution operates a comprehensive, ongoing system of program evaluation and development that involves program participants and local practitioners, and that leads to substantive improvements in each program. The institution provides opportunities for meaningful involvement by diverse community members in program evaluation and development decisions.
Commitment to the increased use of data-informed decision making and an increased emphasis on performance assessments have been implemented and refined since the 2006 visit. Some key components include:
- Creation of an Assessment Committee for Initial Teacher Preparation programs
- Increased use of computer technology to make information more timely and accessible
- Implementation and refinement of performance assessment tasks and rubrics to measure candidate knowledge, skills, and dispositions
- Triangulation of date from multiple sources to inform program improvement efforts
- Inclusion of specific measures of candidate professional dispositions
Consistent with the emphasis on collaboration articulated in the conceptual framework, members of the educational community continue significant, long-established roles in the implementation and refinement of the system. Some of these activities include:
- Meetings of program advisory boards, the Kremen School of Education and Human Development (KSOEHD) Dean’s Advisory Council, and the President’s Commission on Teacher Education regarding results of graduate and employer surveys and their implications.
- Meetings of program advisory committees, program faculty, and K-12 master teachers and clinical supervisors regarding program assessments of candidate proficiencies and recommendations for improvement.
Regular meetings of coordinators representing all unit programs to facilitate communication and consistency of the system throughout the unit. There is collaboration among faculty of each advanced program to develop and implement a Student Outcomes Assessment Program (SOAP) containing performance objectives, assessment activities, assessment timelines, and an implementation plan.
Use of Technology for Data Management
Additional reliance on technology to support the collection and analysis of data has been incorporated into the assessment system.
- For all programs within the KSOEHD, the Data Technician in the Student Services office continues to maintain the database of candidate information. This database contains information on program candidates and is used in conjunction with the university’s Common Management System.
In order to monitor the progress of candidates, to evaluate and improve candidate
performance, and to evaluate and enhance program quality, the unit assessment system
includes multiple types of information on applicant qualifications; on candidate performance
data related to the knowledge, skills, and dispositions expected of an education professional
in each program; and from surveys of exiting candidates, graduates, and employers.
Use of data for program improvement
Schedules for collection, analysis, and use of data, results of assessments, surveys,
and other data collection are shared with candidates, program faculty, and advisory
boards in order to improve candidate and faculty performance, and program quality.
Coordinators are responsible for sharing results of program data with faculty and
The ECE program assessments are consistent with the unit and university policies. Beginning in 2001, all university graduate programs were required to develop a “Standards-Based Outcomes Assessment Plan,” (SOAP) consisting of program standards, assessments, course matrices and a timeline for administration. An initial assessment for admission is made by the graduate program coordinator for the unit. As part of the program, candidates must meet a graduate writing requirement. In the ECE program, this critical assessment is made in LEE 235: Concept Development in ECE. As part of SOAP, the Alumni and Employer Survey are required by the unit as a post-program assessment.
The ECE Program has also developed a unique set of program assessments that monitor
the progress of candidates through a spiraling series of steps toward competence as
an ECE Leader. During the program, five assessments are made of students that respond
directly to NAEYC Core Standards for Advanced Programs and the Essential Professional
Tools (see Assessments). A sixth assessment meets NCATE dispositional assessment criteria, but is currently
being replaced by a yet to be identified measure that not only meets the validy and
reliability needs of the program, but is also a practical instrument for implementation
within the advanced program. Assessment is an ongoing process in the ECE Graduate
Program and begins at the initial interview of a prospective program applicant. Formative
assessments are embedded in each class in the ECE program to provide candidate’s with
performance feedback. Five of the six summative assessments are embedded in course
and fieldwork. The Ethics Assessment has been a stand-alone pre- and post-test evaluation
administered in the first and last semesters of a candidate’s course of study. The
steps in the assessment system are as follows:
Step 1. The coordinator makes an assessment of student’s readiness for enrollment according
to the unit’s established criteria and the ECE program’s additional criteria (initial
certification in ECE). Using a ECE advisement form, a tentative program of studies
is planned. Based on the interview and an identification of interests, the ECE Specialization
isdecided and a faculty adviser assigned. Referrals to testing office, financial aid,
etc. are made based on student needs. The Ethics Assessment pretest is taken to provide a baseline for dispositional growth at the beginning of
Step II. During the first year in the program, students are assessed in the areas of professional knowledge, skills and dispositions mainly through the
DAP Charter School Assessment (Assessment 3). All students take ECE core electives in the first year having the DAP Charter School Component. Because the assessment requires a research paper, skills demonstration and commitment to exemplary practice, key aspects of ECE leadership, requires that students meet expectations for these areas early on.
Step III. During the first or early in the second year, the additional assessments, Action Research Project (Assessment 1)and graduate writing assessment (See LEE 235 Syllabus) are required of students for continuation in the program and advancement to candidacy.
Additional DAP Charter School Components require repeated demonstrations of competence.
Step IV. With their growing leadership and expertise by the end of the second year, assessments
are more rigorous and comprehensive but also self-evaluative. The cumulative Portfolio (Assessment 2), Leadership Activity (Assessment 4)and Ethics Assessment post-test are completed.
Step V. The most demanding assessment of the ECE Leader is the Summative Assessment (Assessment 5). The ECE Specialist Credential student completes a rigorous research paper while
the Master’s Degree candidate may choose between a thesis or project. Unit required
follow-up studies of alumni and employers provide follow-up evidence of program outcomes.
These six assessments evaluate students’ performances on the five Standards and nine
Essential Tools identified by the National Association of the Education of Young Children
as being representative of the quintessential Early Childhood educator (See Alignment of Standards andTools with Assessment). The five Standards and nine Essential Tools create the cornerstone of ECE Specialist
Credential and Master’s coursework.
The results of students’ performances on those six measures are reported to the Commission on Teacher Credentialing (CCTC) in the Biennial Report along with program improvement efforts that have been made based on those data. Current plans for program improvements include:
- Provide writing resources. The ECE Program will work with the University Dean of Graduate Studies and the Dean of the School of Education to provide expanded opportunities to develop graduate-level writing skills.
- Change in reporting methods. The ECE Program believes that if our data reporting method by assessment reflected discrete scores for the Standards and Essential Tools evaluated by that assessment, we would be in a much better position to evaluate individual candidate’s performance on each Standard and each Essential Tool; have more critical evidence with regard to specific coursework and fieldwork effectiveness; and finally, by aggregating Standard and Essential Tool data across assessments, be able to discern overall program strengths and weaknesses.
- Establishment of a permanent database. A database is being designed to coincide with the University’s BlackBoard system that allows faculty to enter student assessment data.
- Recruit more ECE Specialist Credential candidates. Continued effort needs to be made through school districts and from ECE credential program graduates to recruit students into the ECE Graduate Program.
- Identifying a more practical assessment of candidate’s dispositions. The current assessment tool, albeit valid and reliable, in not practical with regard to time required in its administration or analysis.