In spring 2000, the University Standing Committee on Evaluation of Teaching (EOTC) recommended to the Provost a campus-wide process for student evaluation of teaching (end-of-term course evaluation). The proposed process includes

1. anonymous evaluation - i.e., students are not identified;
2. a small "common core" of questions asked in all courses;
3. opportunity for colleges, departments, and individual faculty members to add their own questions;
4. standard reports summarizing the results for a course section and comparing them to results for similar courses and to other courses department-, college-, and campus-wide;
5. opportunity for custom analyses across core and college- or department-wide items;
6. faculty ownership of data, as in the current evaluation process; and
7. a central office coordinating administration and handling form preparation and reporting.

UEI (University-wide Evaluation of Instruction) is the team's working name for the proposed course evaluation process. Compared to the current college- and/or department-specific end-of-term course evaluations, the campus-wide process can make it easier to interpret students' ratings, with a common context against which to view uniquenesses across departments in a college or across colleges and more powerful comparisons of how ratings in one course compare to ratings of other courses similar in size, level, type, etc. Combining campus-wide and college-, department-, and faculty-specific questions in an integrated process eliminates the need for departments/faculty to participate in both a campus-wide processes and their own separate one, and an integrated process offers efficiencies of scale. With a coordinated process, the money and much of the time that departments and colleges currently use for course evaluation will be available for other purposes.

The planning team recommends that UPA be asked to administer the process, on the basis of their experience with survey administration, and the confidentiality and security procedures needed for data that contains information about individuals. It is clear, however, that UPA cannot handle a process of this size with its current staff, equipment, and space. NC State's academic departments offer approximately 11,000 course sections each year, with nearly 290,000 student/section combinations in 1999-2000. End-of-term evaluations typically have high return rates, and we expect approximately 235,000 evaluations to be processed over the course of a year, with summary reports to each section's faculty member(s) plus department- and college-level summaries. Web-based administration is faster and cheaper that a paper-based process, but there is not yet enough evidence from other universities' experience to determine how moving to web-based course evaluation affects response rates and data quality. For now, we recommend a paper-based process, but will continue to monitor experiments with web-based course evaluation at NC State and elsewhere.

The suggested procedures for reporting course evaluation results are consistent with current university policies. Summaries of results from individual course sections will go to the faculty member(s) involved, along with the section's original opscan sheets. A separate report for department heads will list results for each question by faculty member and course section. Aggregate reports that do not identify individual faculty members or sections, such as tables of means and distributions across an entire department or across all courses in a college with certain characteristics, will be available to individual faculty members, department heads, deans, and the Provost's office, and can probably be made available via a web site that is not linked to any of the university's public pages. As now, departments, colleges, or the provost or chancellor may choose to make aggregate data publically available as performance indicators or as part of program assessments.

To be an effective test of the proposed process, the fall 2000 pilot study must involve a range of section characteristics such as section size, level (introductory, lower-division, upper division, graduate), type (lecture, lab, seminar, etc.), etc., and include full departments, with college- or department-specific questions in addition to the "core" items. This will test the interaction with departments for information and additional questions, and faculty and department reaction to how the reports compare a section's results with the department's. Plans for the pilot study also include statistical analyses of the results' reliability. At the time this report was written, three departments were considering participation in the pilot study: Economics, Mechanical & Aerospace Engineering, and Adult & Community College Education. Between them, these departments offer enough sections and enrollments for a reasonable test but still manageable by a small temporary staff. They have a wide range of course types and levels including laboratory sections, graduate courses, and, particularly in ACCE, off-campus sections with a variety of time-enhanced teaching methods including Web-based courses and courses taught via interactive video.

Current budget estimates for the full campus-wide process are between $150,000 and $165,000 per year. This is consistent with costs at other large campuses when all expenses are taken into account. Costs include the optically-scannable forms themselves, space, office supplies, and the staff needed to supervise and manage the process, to process the forms and get forms and reports to and from the faculty, and to develop and maintain computer programs to gather information and customize each section's forms. There are also start-up costs for scanning and printing equipment and staff computers. The much smaller Spring 2000 pilot project, with temporary staff and existing equipment, is estimated to cost approximately $25,000. The full report includes detailed budget estimates and descriptions of the procedures that would be involved and the resources needed to implement them, as well as mockups of the proposed evaluation instrument and standard reports.

EOTC described the proposal to Deans' Council and Faculty Senate during spring 2000. The Faculty Senate and its Academic Policy Committee reacted positively to the general concept and the proposed pilot study. In Spring 2001, the Academic Policy Committee will review the results of the pilot and advise the Senate on the proposal to change university policy and implement a campus-wide process. If the decision is to proceed with the full campus-wide process, timing will be important. A decision to implement the campus-wide process in Spring 2001 would have to be made by mid-February to allow time for all necessary steps. If the campus-wide process did not begin until Fall 2001, the pilot departments would be able to continue using the new evaluation in Spring and Summer 2001. It might be possible for a few additional departments to "beta test" the process that spring and/or summer as well. We recommend active publicity by EOTC and the UEI planning team in Fall 2000 while the pilot is under way, to continue the campus discussion that has already begun in the Faculty Senate. Possibilities include (but are not limited to): discussing the proposal and pilot with the associate deans, some individual departments, and the student government; having one or more articles about it in the Bulletin; and adding a comment form to the UEI web page and periodically summarizing the comments.

The complete UEI Planning Team report is available as a *.pdf file. If your browser cannot read this file, download the free Adobe Acrobat Reader.

Executive Summary

Proposal, rationale, & history

Issues

1. General issues
1. Scope of the campus-wide process
2. Paper vs web-based administration
3. Integrating campus-wide and college/department-specific parts of the process
4. Offices involved in the centralized process
5. Reports and data ownership
6. Data storage and security
2. For the fall 2000 pilot study

 
3. Faculty acceptance

1. Campus-wide process
1. The "core" questions
2. Preprinted opscan forms
3. Preparing the customized opscan forms
4. Getting the forms to and from the students
5. Scanning
6. Completed section packets
7. Standard reports
8. Custom analyses of results from forced-choice questions
9. Resources needed
2. Fall 2000 pilot study
1. Participating departments
2. Modifications to the campus-wide procedure
3. Evaluating the pilot study
4. Resources needed
3. Spring and Summer 2001
1. Deciding whether to implement the full process
2. If the full process is not implemented in Spring 2001
3. If the full process is implemented in Spring 2001

Appendix 2: Mockup of the evaluation form

Appendix 3: Mockups of standard reports

• Section report: Core and optional question results for a single section
• Faculty results list: Each faculty and course section in a department
• Department summary: Summary across faculty and course sections
• College summary: Summary across faculty and course sections

University Planning & Analysis -- k:\survey\uei\plancmte\report\uei_execsum.htm -- 8-11-00