Industry-University Cooperative Research Centers (IUCRC),
Program Evaluation Project
|
2.1 BACKGROUND
The Evaluator's Report provides a narrative description of the circumstances, events, and actions, that contribute to the center’s development and growth. The Evaluator’s Report is prepared based on a structured outline. In new centers, the focus of the Evaluator’s Report will be on the ideas, plans, and efforts that led to the creation of the center. In centers that have been in existence for several years the focus will be on the changes that occur as the center evolves and matures and various center outcomes and achievements.
The intent of this instrument is to provide center evaluators with a vehicle for recording the unique history of each center while allowing the data to be aggregated across centers for purposes of overall program evaluation. The report provides the director with a qualitative evaluation from an involved yet objective observer. Some centers share this report with new members as a way of briefing them about the center’s history. The report is also intended to provide NSF with a record of the center’s progress and accomplishments.
2.2 GENERAL INSTRUCTIONS
All evaluators are expected to complete and submit an Evaluator’s Report annually. The structure of the report is defined by the outline provided in Section 2.4. Major topics to be addressed by the report include: overview, environmental/ institutional, organizational, research, accomplishments and analysis. Brief instructions for completing these sections are included on the report outline. The report submitted to your director for inclusion in the center’s renewal report should not exceed five pages.
It should be noted that each of sections requires the evaluator to directly address issues of stability and/or change for that year and note the impact of these changes. Much of the report can be completed based on information obtained from participant observation. However, evaluators may also need to examine center documents, refer to national databases and interview relevant center participants (e.g., director, business manager, and/or other key center faculty). Section 2.5 provides an outline of the kinds of issues, which can be addressed in each section. Section 2.6 provides a sample of a completed report.
Schedule for Submitting Report
NSF requires that a copy of the Evaluator’s Report be included with all center renewal proposals. Directors are required to submit their renewal proposals to NSF at least 90 days prior to the anniversary of their award date. Evaluators must consult with their directors on their due date and should plan to submit a copy of their report to their director at least a month before this deadline in order to allow for review and integration. Evaluators must also forward copies of the report to the NCSU Evaluation Team. Since most renewal proposals are due in the spring, evaluators are asked to submit their report by March 15.
Access to the Evaluator’s Report Database
The Evaluation Team at NCSU will continue to monitor and archive the Evaluator’s Reports submitted each year. It is anticipated that individual evaluators will propose studies, which would include access to the Evaluator’s Report database. Use of the database for this purpose is encouraged. Any evaluator interested in using the database should submit a written request to Alex Schwarzkopf of NSF for permission to receive any portion of the database.
2.3 HELPFUL HINTS FOR PREPARING AND COMPLETING YOUR EVALUATOR’S REPORT
Below are some "helpful hints" you may want to consider as you embark on preparing your Evaluator’s Report.
Preparation:
1) Gather all relevant files, notes, center reports, etc. organize them and become familiar with the information they contain.
2) Keep a copy of the Evaluator’s Report handy (blank or the previous year's) and periodically (especially after board meetings, etc.) and periodically jot down notes which will form the basis for that year’s answers
3) Prepare a timeline of major events for the past year.
First Draft:
4) Make brief cryptic notes on each question outlining what should be included in your answer and noting where you will have to refer to documents or other sources to get adequate details (including results from Process/Outcomes assessment).
5) Begin by answering questions for which you have the information at your finger tips; move on to other questions later, perhaps after you have had the chance to do some research.
Revisions and Final Draft:
6) Schedule interviews with center director and other relevant parties to get information you lack and/or clarify points of uncertainty. Do not waste people's time going over issues which are obvious.
7) Refer to outside documents as necessary (e.g., U.S. Industrial Outlook)
8) Make final revisions incorporating this information.
9) Solicit feedback from center director on accuracy and completeness.
2.4. EVALUATOR’S REPORT
Evaluator’s Report
(Revised September 22, 2008)
General Instructions: Summarize the center's recent history (last twelve months or since the last history was prepared). Please address each domain. Emphasis should be on significant changes, which might influence the Center's ability to achieve its goals and objectives.
Length: 5 pages.
1. Overview
Provide a general overview of the center’s status.
2. Goals and objectives
Please describe the center's primary technical and organizational goals and objectives.
Instructions: This question can be answered by listing a center's "official" (written) and/or unofficial (but acknowledged) goals and objectives. Goals and objectives should be updated or elaborated upon (e.g., the basis for inferring informal goals) as needed. You should indicate if your answer reflects a change from previous years.
3. Environmental/Institutional
Please describe any environmental (e.g., decline in industry's competitive position) or institutional/university (e.g., partnerships with other universities, shift in university priorities) changes which have occurred during the last year.
Instructions: Evaluator’s are encouraged to include industry data based from outside sources.
4. Organizational
Please describe any changes in the center’s personnel, structure, policies, financial status, and operations, which have occurred during the last year.
Instructions: The section can be answered based on participant observation, reference to program documents and data collected via the Process/Outcome questionnaire.
5. Research Program
Please describe any changes in the center’s research program (e.g., new or modified thrust areas), which have occurred during the last year.
Instructions: The section can be answered based on participant observation, reference to program documents and data collected via the Process/Outcome questionnaire.
6. Center accomplishments
Please describe any accomplishments or impact the Center has had in the following areas:
knowledge/technical advances; technology transfer; educational impacts. In addition, comment on accomplishments, which may reflect unique center objectives
(e.g., forging international linkages, etc.).
Instructions: Evaluators are encouraged to obtain the information needed to answer this question from a variety of sources, including: program documents (e.g., Center's annual report), responses or comments provided in the Process/Outcome questionnaire, discussions with CD, IAB members and faculty.
6.1. Center Success Case Study
Effective 2008-9, evaluators are required to produce a mini-case (one to two paragraphs) description of a recent center “success story”. A center success story might be scientific, technological, technology transfer, or educational in nature. The purpose of this part of the report is to try to document some center activity, output or outcome. This mini-case might be used by the center, NSF or others as evidence the center or program is achieving its objectives. While not exhaustive, a center success might include: a significant scientific breakthrough or award; a firm development or commercialization activity that was related to center research (perhaps listed in the P/O survey); a new significant patent filing; a start-up or spinout company linked to center research; a scientific, technological or commercial development achieved by a center alumni, etc. The evaluator is expected to obtain background information from relevant sources (e.g., faculty, members, licensees) about the success story to prepare a mini-case (e.g., think in terms of who, what, where, how and when). Eventually, some of these mini-cases will be examined for inclusion into the “technology breakthroughs” report NSF IUCRC produce. The evaluator is encouraged to select a success story based on input from the center director and/or IAB. The NSF has provide a template for documenting Research Highlights. You may want to use this template to document Center Success Case Studies.
7. Analysis
Based on the information provided above and other relevant information, comment on the health and vitality of the Center. What are the implications of the various environmental, institutional, organizational and research changes? Is the center making adequate progress in achieving its objectives? Is the research program still vital and current?
Instructions: Comments should reflect the evaluator’s view of the major obstacles and/or opportunities, which may affect the Center's success during the next one to three years?
8. Timeline
Attach an updated timeline of significant events and milestones, which have occurred over the center’s lifetime.
2.5 EVALUATOR’S REPORT -- OUTLINE OF RELEVANT ISSUES
Instructions: Evaluators can address a variety of issues within the different sections of the report. The bulleted items listed below are a sampling of issues addressed by evaluators in previous reports.
1. Overview
Provide a general overview of the center’s status.
2. Goals and objectives
Technical Goals
Organizational Goals
3. Environmental/Institutional
Environment
Institutional
4. Organizational
Personnel
Structure
Policies
Financial stability
Operational
5. Research Program
6. Center accomplishments
Knowledge/technical advances
Technology transfer
Educational impacts
6.1 Center Success Case Study.
- A one to two paragraph description of some significant scientific, technological or other output or outcome of center activities (see description above).
- You have the option of using the Research Highlights template provided by NSF
7. Analysis
8. Timeline
2.6 For a sample of an evaluator's report, please click here.