Educational Effectiveness Review
Opportunity

APPENDIX A4

Assessment of the General Education Program

Part 1. Report on the Review of the General Education
Program: AY 2003 – 2004

Section I. General Education Writing Rubrics

Course work of students currently in upper division General Education courses was analyzed as an assessment of writing skills gained in lower division General Education courses. This analysis was conducted in the 2002-03 academic year in the upper division integrated science and arts/humanities courses, and reported in the Fall 2003. Faculty were involved in developing rubrics, in scoring writing samples, and in providing feedback. General Education Scoring Guides were developed for four areas: Writing; Critical Thinking; Upper Division Integrative Science: Physical Universe and Its Life Forms (IB); and Upper Division Integrative Arts and Humanities (IC).

The Office for Institutional Research collected sets of student papers from each course offered in GE Area IB and IC, selected a random sample, photocopied them, and returned the sets of papers to the instructors. Student papers were coded so students could be identified by Institutional Research (to be able to distinguish between native and transfer students for example) but not by scorers; papers were scored anonymously. Copies of the respective writing assignments were also collected. In May 2003, twenty faculty participated in the scoring process.

The scoring results are summarized below. A score of 4 = accomplished, 3 = competent, 2 = developing, and 1 = beginning. In addition to providing a score on each dimension, scorers were asked to provide an overall score (not an average) that best represented the quality of the paper.

Writing			Knowledge of Conventions			Clarity & Coherence		Rhetorical Choices		Overall
IB (Science)			2.39			2.42		2.10		2.21
IC (A&H)			2.44			2.39		2.07		2.21
Critical Thinking		Interpretation			Analysis & Evaluation				Presentation	Overall
IB (Science)		2.37			2.31				2.26	2.38
IC (A&H)		2.48			2.50				2.55	2.56
IB Goals (Science)	Science & Society			Basic Concepts & Fundamental Principles			Scientific Approach		Nature of Science	Overall
	2.07			1.75			1.53		1.71	1.75
IC Goals (A&H)				Integration			Discipline			Overall
				2.56			2.40			2.50

The overall quality of student writing in the sciences (mean score, 1.75) was scored lower than the overall quality of writing in the arts and humanities (mean score, 2.50) or in writing overall (mean scores, 2.21). Student abilities to think critically, to capture and express their thinking generated some of the best scores (mean scores, 2.38 and 2.56). It is clear from the comments of faculty participating in the rubric evaluation process that the development of rubrics is important and that the alignment of the assignment to the rubric made a significant difference in the results. This could have been a factor in the lower than expected scores of student writing in the sciences compared to those in the arts and humanities. To the extent that the curriculum is aligned to outcomes and the assignments and rubrics are well designed, learning is documented, and embedded assessment can be used to measure educational effectiveness. One of the lessons is to encourage faculty to use rubrics and help them design effective writing assignments to apply them to. A second lesson is that assessment processes that are embedded in class work bridge potential problems with student motivation because they represent real work both to the students completing the assignments and to the faculty assigning them.

Section II. Syllabus Review of General Education Upper-Division Integrative Courses

Appointed in October of 2003, the GE Assessment Task Force (GE ATF) conducted a review of a total of 101 Fall 2003 course syllabi for all GE upper-division integrative courses. Integration courses are offered in four areas: Physical Universe and Its Life Forms (IB); Arts and Humanities (IC); Social, Political, and Economic Institutions and Behavior, Historical Background (ID); and Multicultural/International (MI). Reviews were done both in terms of the student learning outcomes outlined for the GE program and the requirements for campus syllabi set out in the Academic Policy Manual (APM 241). The Fall 2003 course syllabi were also compared to the model syllabi originally submitted to and approved by the General Education Committee. A number of recommendations came out of the task force’s review. Among them were the following.

1. Crisper statements of student learning outcomes should be developed to enhance future assessment activities in General Education Areas.

Discussion/Action: The task force reviewed the comparative educational benefits and cost effectiveness of the use of locally developed writing rubrics and national testing programs. The task force noted that embedding the use of writing rubrics in GE courses would likely enhance faculty development and lead to better alignment of the courses to desired learning outcomes. The use of writing rubrics reflects the cost of faculty time to develop instruments and apply them in the evaluation of student writing. On a per student basis, the costs to date for our use of the writing rubrics average just under $60 per student assessed.

Instruments associated with several national testing programs were reviewed in detail. The task force noted the potential to compare student abilities on a national basis. The task force also noted that many of the specific subject matter tests rely heavily on a student’s reading ability. Currently, components of our campus Upper Division Writing Examination utilize ACT’s CAAP Test of Writing Skills. On a per student basis, the costs of administering a national normed test are less than 25% of the costs of our development and use of writing rubrics.

After much reflection and discussion, the task force recommended:

• Every four years, students enrolled in our upper division writing-intensive “W” courses be given the same ACT CAAP Test of Writing Skills, which will give us a one-semester snapshot of general student writing skills.
• Alternating between the administrations of the ACT CAAP Test of Writing Skills beginning two years after its first use, students enrolled in our upper division writing-intensive “W” courses should be given the ACT CAAP Writing Essay Test, which will complement objective test results from the Test of Writing Skills with the evaluation of student writing samples.
• On an experimental one-semester basis, students enrolled in courses meeting the GE Area IC requirement be given the ACT CAAP Reading Test. At the same time, two large sections (50+ students each) of our freshman University 1 course should also be given the same reading test. Together we will have measures of student reading skills at entrance and at the upper division level. Once results are known, this recommendation will be revisited and reevaluated.
• Learning outcomes for each GE Area of study should be reviewed as the first set of test scores is made available, both to sharpen our own statements of student learning objectives and to evaluate the information contained in the set of test scores.

2. GE course syllabi should have a stronger connection to their model syllabi submitted by the department and approved by the General Education Committee.

Discussion/Action: The task force review found that, in many cases, there is a significant disconnect between the model syllabus submitted by the department and approved by the General Education committee and the syllabus distributed to students Fall 2003. This may be exacerbated by the use of new or part-time faculty who are simply unaware of the role the course plays in the campus’s General Education program. The Provost has communicated the request to all department chairs that they verify all faculty teaching General Education courses base their syllabi on the model developed by the department and submitted to the General Education Committee. Collection and review of GE course syllabi is scheduled again for Fall 2005. Future syllabi reviews will report on the degree of comparability between the model syllabi for the GE course and those distributed to students that semester.

3. Connections between expected student learning outcomes and specific course elements, which were included in the original course proposals, should be more explicitly mapped to course elements in GE course syllabi.

Discussion/Action: Toencourage a stronger linkbetween expected student learning outcomes and specific course elements, a Web-based application process has be designed for departments to use in submitting applications for consideration of new and renewed General Education courses. Whether the course being proposed is new to the General Education Program or is being reviewed for renewal after 5 years, the Web application requires course submissions to explicitly map student learning outcomes contained in the Policies for Inclusion and Evaluation of General Education Courses to specific course elements detailed in the model course syllabus.

4. Campus policy currently urging GE faculty to include an iterative approach in which students receive instructive feedback and an opportunity to revise their work should be amended to require that iterative approach.

Discussion/Action: The task force found that many of the writing assignments given to students were due within the last week of the semester and most did not include any provision for feedback to students. The task force has recommended that all student writing assignments made as part of the General Education Writing Requirement must include an “iterative approach” in which students receive instructive feedback and an opportunity to revise their work. Recommended changes to the General Education Writing Requirements that reflect this proposal have been submitted and are currently in the process of being reviewed in the Academic Senate.

Collection and review of GE course syllabi is scheduled again for Fall 2005. The call for syllabi will be supplemented with a request for all materials describing writing assignments, specifically as they incorporate the iterative design. Future syllabi reviews will report on the iterative design of writing assignments made in GE classes.

5. GE course syllabi should do a better job of detailing how they meet the 4000-word writing requirement, including a description of the assignment components, methodology, goals of the assignment, and criteria/standards against which they would be evaluated.

Discussion/Action: The task force found that nearly 20% of the course syllabi it reviewed do not meet the 4000-word writing requirement approved by the Academic Senate. An additional 30% of the syllabi were unclear about how that assignment is met. The current General Education Writing Requirement does specify at least half of the student writing consist of sustained writing. The task force review found a term paper of 5 to 9 pages in length is the most common writing assignment. The task force has recommended reducing the current number of words assigned in student writing by half, to reflect the fact that student writing will be iterative and will be evaluated twice. Recommended changes to the General Education Writing Requirements that reflect this proposal have been submitted and are currently in the process of being reviewed in the Academic Senate.

Collection and review of GE course syllabi is scheduled again for Fall 2005. The call for syllabi will be supplemented with a request for all materials describing writing assignments, including the assignment components, methodology, goals of the assignment and criteria/standards against which they are evaluated. Future syllabi reviews will report on the writing assignments made in GE classes.

6. GE course syllabi should require sustained reading of primary source or non-textbook assignments to enhance student command of language, rhetoric, and argumentation.

Discussion/Action: An important component of the requirements detailed for General Education is one that specifies students be exposed to primary source reading. In earlier reviews, findings pointed out a significant disconnect between the type of reading assigned in General Education courses and the type of reading assigned in courses that support the baccalaureate major. As a result, the Academic Senate required General Education courses include assignments of primary source reading where appropriate. Of the 101 course syllabi it reviewed, the task force found fewer than 60% of the syllabi to specify primary source or non-textbook reading assigned in the semester. The task force has recommended that students have opportunities for sustained reading that enhances their command of language, rhetoric, and argumentation. Recommended changes to the Policies for Inclusion and Evaluation of General Education Courses that reflect this proposal have been submitted and are currently in the process of being reviewed in the Academic Senate. Future syllabi reviews will report on the type and character of reading assignments made in GE classes.

7. Compliance with requirements for campus syllabi outlined in the Academic Policy Manual should be improved.

Discussion/Action: Because the campus has existing, well publicized policies regarding students with disabilities, cheating/plagiarism, disruptive classroom behavior, and computer usage, syllabi that did not reflect statements in those areas were not viewed as problematic even though anywhere from 13% to 41% of the syllabi had one or more of the statements missing. Still, improvement should be made in bringing all syllabi into compliance, and future syllabi reviews will report on progress made.

Of greater concern were syllabi that did not include a sufficiently detailed course calendar (14%) and that did not offer clear statements on attendance and makeup work policies (14%). Recent policies passed by the Academic Senate and signed by the President Spring 2004 place increased importance on these areas. The Provost has communicated the request to all department chairs that they verify all faculty teaching General Education courses include unambiguous statements of attendance and makeup work policies in GE course syllabi.

8. Errors of fact contained in some campus syllabi should be corrected.

Discussion/Action: Copies of syllabi containing errors of fact have been sent to appropriate department chairs with a cover letter from the Provost.

 

Section III. Student Surveys of General Education, Areas IB and IC

A student survey instrument was developed locally that gathers student feedback on the quality of the educational experience students have in the GE program. Student survey results for GE Areas IB and IC have been tabulated. Data from GE Areas IB and IC have been augmented by data elements pulled from the campus’s PeopleSoft student database; analyses will be available by Fall 2004 for all four GE areas of study. From analyses of the student surveys of GE Areas IB and IC thus far, some general comments are:

• As the student’s expected grade in the course went up, the average number of non-class hours they reported spending on the class tended to go up.
• Not surprisingly, students expecting Bs or As in their classes tended to respond that the number of class hours they devoted was adequate, and students expecting Cs or Ds did not.
• More students reported they retained 13-15 units of coursework through the semester than any other level of unit load. More students reported they worked 17-32 hours per week through the semester than any other number of employment hours. These patterns were consistent for students who expected As, Bs, and Cs. Students expecting D grades were too few to summarize. This evidence supports the conclusion that unit load and weekly employment hours are independent of student grade expectations.
• There were no differences in grade expectations by gender or by class standing.
• Student age categories and native language categories show some spotty differences in grade expectations.

Student reports of assignments done in GE Area IB and IC courses are consistent with faculty reports of frequency and character of the assignments they made in their classes.

 

Section IV. Faculty Surveys of General Education Areas IB, IC, ID, and MI

Faculty surveys were gathered at the same time the student instruments were distributed to GE courses. Responses came from 16 GE Area IB faculty in Spring 2002, 29 GE Area IC faculty in Spring 2003, 20 GE Area ID faculty in Spring 2004, and 41 GE Area MI faculty in Spring 2004 for a total of 106 faculty responses. Summary of the faculty responses fall into five general categories: 1. A summary of how frequently GE faculty make different classroom assignments, 2. A comparative analysis of how GE faculty make classroom assignments, 3. A report of how faculty spend their GE class time, 4. A summary of how faculty assess student preparedness, motivation, and abilities, and 5. A review of how faculty rank order the importance of generic course objectives. Some interesting patterns emerge from analysis.

1. Summary of Classroom Assignments

Faculty responded to a series of items whose common stem asked them to identify the number of times in their GE class they made specific assignments in their GE class. The assignments related to:

1. Textbooks or other book-length sets of course materials
2. Newspaper, magazine, or journal articles
3. Internet sources to develop assignments
4. Papers or reports of 10 pages or longer
5. Papers or reports of 5 to 9 pages in length
6. Papers or reports of 1 to 4 pages in length
7. Short answers of less than 1 page in length
8. Exams or quizzes completed in class
9. Exams or quizzes completed as take-home assignments
10. Exams or quizzes that included short answer, open-ended work, or essay questions
11. Web site development
12. In-class performances/presentations

Reading. Across the four areas of General Education, faculty tended to assign one or two textbooks or other book-length sets of course materials (average frequency assigned = 2.4 across all areas of GE). Other reading assignments made included newspapers/magazine articles (average frequency assigned = 2.7 across all areas of GE) and internet sources (average frequency assigned = 2.5 across all areas of GE).

Writing. Faculty tended to make writing assignments of 1 to 4 pages (average frequency assigned = 2.8 across all areas of GE), with short answers of less than a page (average frequency assigned = 2.6 across all areas of GE) a close second, followed at some distance by papers of 5 to 9 pages in length (average frequency assigned = 1.2 across all areas of GE) and papers of 10 pages or longer (average frequency assigned = 0.6 across all areas of GE). In addition, faculty reported providing written comments to student writing with an average frequency of 4.4 across all areas of GE and requesting revision of student writing with an average frequency of 1.3 across all areas of GE.

Testing. Faculty tended to give numerous in-class exams and/or quizzes (average frequency assigned = 3.8 across all areas of GE), often with essay questions included (average frequency assigned = 2.2 across all areas of GE). Infrequently did faculty assign take-home exams and/or quizzes (average frequency assigned = 0.7 across all areas of GE).

Other Activities. Web site development was rarely assigned (average frequency assigned = 0.2 across all areas of GE), but students were sometimes required to prepare presentations and/or give performances in class (average frequency assigned = 1.1 across all areas of GE).

2. Comparison of Classroom Assignments

Analysis of classroom assignments by GE Area showed significant differences across the four GE Areas at the 0.05 level for seven of the 12 areas queried, with two additional assignments different at the 0.10 level. In the table below, actual level of statistical significance is shown in parentheses. Assignments showing significance at the 0.05 level are bolded and marked with two asterisks (**) and areas showing significance at the 0.06 to 0.10 level are shown in italics and marked with a single asterisk (*).

1. Textbooks or other book-length sets of course materials (0.09) *
2. Newspaper, magazine, or journal articles (0.07) *
3. Internet sources to develop assignments (0.00007) **
4. Papers or reports of 10 pages or longer (0.00002) **
5. Papers or reports of 5 to 9 pages in length (0.22)
6. Papers or reports of 1 to 4 pages in length (0.57)
7. Short answers of less than 1 page in length (0.30)
8. Exams or quizzes completed in class (0.030) **
9. Exams or quizzes completed as take-home assignments (0.037) **
10. Exams or quizzes that included short answer, open-ended work, or essay questions (0.0002) **
11. Web site development (0.0004) **
12. In-class performances/presentations (0.0009) **

This is preliminary evidence of cultures differing by GE Area that faculty bring to structuring their General Education courses. In area results on internet sources, Area IB and MI faculty assigned more than the typical expected while Area IC faculty underutilized internet sources. In papers of 10 pages or longer, Area IB and MI faculty assigned more than the typical expected while Area IC faculty underutilized longer paper assignments. This is somewhat unexpected since Area IC courses are taught by faculty in the College of Arts and Humanities. In testing, courses in Areas IB and ID tend to give more in-class exams and quizzes and courses in Areas IC and MI tend to give more take-home exams and quizzes. Area IB, IC and ID faculty assigned more essay questions on their exams and quizzes while Area MI faculty assigned fewer essay questions on their exams and quizzes than expected. Area IC and ID faculty assigned fewer Web site developments and presentations/ performances than expected but Area MI faculty assigned more in both categories than expected.

Again with some irony in the results, although faculty tended to handle writing assignments with differing frequency, they reported providing students comments on their writing and requiring students to revise their written work with about the same frequency across the four GE Areas.

From a later section of the faculty survey, questions probing the use and character of a final examination showed that virtually all faculty give a final examination in GE courses. But faculty teaching Area IB courses give more final exams that are comprehensive in nature (69%) than faculty in Area IC (44%), Area ID (10%), and Area MI (46%).

Faculty reported how they allocated their class time in General Education classes. Across all 106 faculty responding to the Faculty Survey, their weighted averages and range among the four GE Area averages are shown below. The range among the four GE Area averages is displayed in the third column. In some activities, the range is broad, such as lecturing. Where the range is a spread of several percents, the fourth and fifth columns report the GE Areas with the low value and high value.

	Weighted	Range in GE	GE Area	GE Area
Mode	Average	Area Averages	Low	High
Lecturing	53%	39% to 63%	IC	MI
Answering student questions	10%	7% to 12%	MI	IC
Asking student questions	13%	8%to 23%	MI	IC
Small group interactions	6%	4% to 9%	IB	IC
Testing	7%	7% to 8%
Role playing	1%	0.3% to 1.5%
Student presentations	4%	2% to 9%	IC	ID
Student writing	2%	0.2% to 2.6%	IB	IC
Other (e.g. videos, guest lectures)	3%	2% to 6%	ID	IB

Faculty responded to a set of queries asking them to assess their GE students on measures of student preparedness, motivation, and abilities. After reviewing results of the GE Area IB Faculty Survey in 2002, the GE Statisticians decided to augment the faculty survey with the same set of queries in faculty assessments of students, but with the new direction asking faculty what percent of their students should be on each student measure. Below is the summary of four GE Areas on measures of where faculty assess their GE students are and a set of three GE Areas on measures of where faculty assess their GE students should be. Reported below are the weighted averages for each measure. GE Area faculty were fairly consistent across the GE Areas, so, while they are available upon request for the interested reader, ranges on each measure are not reported here.

	Weighted Average	Weighted Average
Student Attribute	% Students Are	% Students Should Be
Prepared for class	48%	90%
Independent learners	35%	78%
Intellectually curious	37%	86%
Motivated by getting degree	69%	71%
Motivated by gradess only	48%	22%
Competent user of technology	63%	85%
Competent analytically	44%	88%
Competent readers	53%	94%
Competent writers	48%	93%

Discussion of these important results will begin in the Fall 2004.

Faculty were asked to rank order six generic course objectives. Descriptive statistics of the results are shown below, where 1 = most important and 6 = least important.

Average Importance

Generic Course Objectives	Area IB	Area IC	Area ID	Area MI	TOTAL 4 Areas
Content mastery	2.5	3.1	2.0	2.2	2.45
Independent learning	4.1	2.8	3.8	3.9	3.65
Interest in further study	3.8	3.4	4.9	3.6	3.93
Creative thinking	3.6	2.6	2.9	3.3	3.10
Understanding applications	3.0	4.4	3.4	3.4	3.55
Problem solving	3.9	4.7	4.1	4.6	4.33

 Ranked Importance

Generic Course Objectives	Area IB	Area IC	Area ID	Area MI	TOTAL 4 Areas
Content mastery	1	3	1	1	1
Independent learning	6	2	4	5	4
Interest in further study	4	4	6	4	5
Creative thinking	3	1	2	2	2
Understanding applications	2	5	3	3	3
Problem solving	5	6	5	6	6

When analyzed as a two-way analysis of variance without replication, the means by GE Areas show no statistical significance (F = 0.0003, p-Value = 0.9997), but the means by course objective are highly significant (F = 5.01, p-Value = 0.0067). Ironically, while faculty showed differing patterns by GE Area in the frequency with which they assigned classroom tasks, they do not show any differing patterns in the goals they hope to achieve in the semester using those tasks.

Close Window