3.1 INTRODUCTION

The Process/Outcome (P/O) instruments are designed to assess Center processes and outcomes based on the perceptions and reports of industrial and university participants. Process items tend to focus on participant characteristics and perceptions of the center's operations. Outcome items include the assessment of immediate, intermediate, and ultimate outcomes of Center activities. There are currently three instruments used in this activity: Industrial Sponsor Questionnaire; Short Industrial Sponsor Questionnaire, and Faculty Questionnaire. Data may be collected through a mail survey, or via a web survey format. These P/O instruments are designed to provide directors with critical feedback from industry and faculty participants about how the center is operating. This should allow directors the opportunity to correct problems and reinforce positive aspects of center operations. As a consequence, it’s critical that results from the questionnaires be shared with your director as quickly as possible. Section 3.5 provides some suggestion for providing concise and clear feedback to your director. Also provided is a sample report(PDF) that incorporates the process/outcome questionnaire feedback.

3.2 PROCESS/OUTCOME ASSESSMENT/ADMINISTRATION

3.2.1 FREQUENCY OF ADMINISTRATION

The instruments are administered yearly during late summer or early fall. Begin administration after your center has been in operation for one full year (do not count the planning grant period). Thus, the first administration of these instruments would begin in the fall after the first anniversary of the center's birth (usually the starting date for the operating grant), the second administration the following fall, etc. Evaluators of Centers that have been in the program for at least 5 years may use the Short Industry Sponsor Questionnaire in place of the full version. If an evaluator believes they cannot collect questionnaires from either industry or faculty for some reason (e.g., response set too small), they should contact the IUCRC Evaluation Team for advice on alternative data collection options.

 

3.2.2 RESPONDENTS

In general, potential respondents for the Process/Outcome instruments are all faculty and all IAB members involved in the center during the preceding year.

Generation of Respondent List. Experience has shown that the best way to obtain a respondent list is to contact the Center’s administrative assistant for an up-to-date contact list. However, if there is any doubt about a respondent, INCLUDE the individual in the sample. This includes individuals who participated in Center activities for most of the year but are no longer active at the anniversary date.

Extensive experience has shown that a different data collection strategy works best with the two target populations: industry and faculty.

 

3.2.3 INDUSTRY SURVEY IMPLEMENTATION

Industry respondents are incredibly busy and are typically only on-site during the center’s twice-a-year IAB meetings. Experience with mailing questionnaires, email and web surveys have typically produced very disappointing response rates. As a consequence, we now strongly recommend (and will only provide support for) collecting data by administering paper questionnaires during the fall IAB meeting.

A. Collecting Surveys at IAB Meeting

Steps in Preparing and Collecting Data.

We recommend you prepare your questionnaires for administration in the following manner:

1. Get an up-to-date list of IAB members from your center’s administrative assistant.
2. Contact your center director and ask permission to get 15-20 minutes on the next IAB agenda so members can complete their questionnaires on-site. Time blocks right before breaks usually work the best.
3. Download a copy of either the long or short Industry Process/Outcome questionnaire
4. Complete the cover page of the questionnaire. Personalize your questionnaire by typing in the name of your Center. Facilitate questionnaire returns by typing your name and address and a "Return by" date. Make enough questionnaires for your respondent group.
5. Coding the ID# Each questionnaire requires a six-digit identification code. Assign this code in the following manner:
   a) Columns 1-2: 2-digit Center code. If you are a new center please call the NCSU evaluation team to be assigned a center identification number.
   
   b) Column 3: 1-digit Center age code. Assign a "1" the first time administered, a "2" the second year, etc. If your Center is older than a single digit allows, continue to use "9" to reflect age.
   
   c) Columns 4-6: 2-digit Respondent Code. Assign a unique code number to each individual receiving the questionnaire. We suggest you use the following convention for industry respondents: use column 4-5 as a firm (or university) code and column 6 for the individual representative who may change over time.
   
   d) Be sure to maintain a codebook for your respondents to assist in follow-up.
   
6. Attach a cover letter to each questionnaire (personalized if possible) that explains the purpose of the overall evaluation and questionaire. Sample letters are provided at the end of this section (see Section 3.6.1 - 3.7.8).
7. When the data collection part of the IAB meeting happens, briefly explain the purpose of the assessment, ask them to fill out the questionnaires right then, hand out the pre-coded questionnaires, and collect the completed questionnaires.

Plan to have a few self-addressed envelopes ready if a respondents asks to complete the questionnaire outside the meeting. Quickly mail or fax questionnaires to any IAB member who did not attend the meeting. Since respondents do not put their name on the questionnaire, you will need to consult the identification numbers recorded in your respondent codebook to identify non-respondents.

Coding Completed Questionnaires

With a few exceptions, the questionnaires contain all the information you will need for coding answers and entering them on a machine-readable medium. Specifically, the response code is part of the item for all forced choice questions; numeric answers are coded as stated; the columns for entering the response code are shown next to each question. You should be aware of the following miscellaneous coding conventions:

1. Coding missing data or "don't know" responses. Leave columns blank if using a database program such as Excel. Enter a space if using a word processing program such as Word.

2. Coding multi-column data. Always right-justify the data. For example to code a response of 50% in a 3-column space you would enter 0 5 0 or 5 0.

3. Open-ended questions. We have not established coding conventions for open-ended questions. For the purposes of the national evaluation data set please indicate whether or not a answer was given by:

Code "0" if the respondent offered no answer.
Code "1" if the respondent made a comment.

4. Coding Question 21, Industry: Question 21 (functional groups) allows more than one response. Codes for up to two choices can be coded in the specified column.

 

3.2.4 FACULTY SURVEY IMPLEMENTATION

Generally speaking, faculty respondents are a lot easier to track down than industry respondents. As a consequence, faculty questionnaires can be administered anytime during the fall semester but generally no later than the IAB meeting. We are currently supporting two methods of data collection: mail (or fax) questionnaire; web survey.

• Mail Survey

After preparing the questionnaires as described in steps 1, 3-6 above.

• Mail or fax questionnaires to your faculty list (See Section 3.7.6).
• Send a follow up letter or email after about 10 days reminding non-respondents to complete and return the questionnaire. Repeat this step until you obtain an adequate response rate.
  
• Web Survey

We now support a web version of the faculty questionnaire (click here to see a pdf version). This data collection method involves local evaluators using an email to contact and direct faculty to a web-based Process/Outcome questionnaire maintained by NCSU Evaluation Team. When data collection is completed, NCSU will send you a dataset. Here are some specific steps for using this option.

1. Notify NCSU Evaluation Team that you plan to collect your data by web questionnaire (iucrc@ncsu.edu).
2. Get an up-to-date list of faculty and research scientists from your center’s administrative assistant.
3. Send an email to these individuals requesting they complete the web-based questionnaire (see Section 3.7.7). Please “cc” us at (IUCRC@ncsu.edu).
4. NC State will send you bi-weekly aggregate updates on your response rate (number of questionnaires submitted). Updates will also be available upon request.
5. Depending on your response rate, you should send several reminders to increase your response rate. Since submitted questionnaires are anonymous, you will need to send this to all respondents (see Section 3.7.8).
6. At your request, but no later than December 31, 2005, NC State will send you a documented excel data set for you to analyze and summarize for your center.

Submitting Data to NCSU

When you have all or a good portion of the total response and have recorded your data, send the hard copies of all the surveys (both faculty and industry) to the evaluation team at NCSU, coded according to the guidelines in this book. Please retain copies of all questionnaires. If you collect faculty data by web-questionnaire, you can skip this step.

 

3.5 ANALYSIS OF RESULTS

Evaluators have the responsibility for providing the Center with feedback on questionnaire results. At a minimum, evaluators should prepare a report which summarizes both faculty and industry responses (frequency counts and percentages) to each item. Many evaluators are also asked to give a powerpoint presentation on the results of the survey at the next IAB meeting. After the first administration, the report should emphasize significant changes in outcome/satisfaction items over the reporting periods. Evaluators have also received positive feedback when they compare their Center to IUCRC normative scores (distributed every summer by NCSU team at the June NSF Evaluator’s Meeting). The sample report(PDF) provides such comparisons. Please remember you should promise confidentiality to your respondents. If you feel your director would benefit from knowing who gave a particular response (e.g., intention to quit the Center), you must check with the respondent before providing this information.

The NCSU evaluation team will be responsible for aggregating across Centers and for any comparative analyses of the Process/Outcome data.

3.5.1 Tools for summarizing and reporting your results

Under construction

 

3.6 PROCESS/OUTCOME ASSESSMENT REPORT

To see a sample of the Process/Outcome assessment report, please click here(PDF).

3.6.1 SAMPLE CENTER DIRECTOR LETTER

October 12, 20xx Dr. Vice President, Research Dear We are collaborating with the National Science Foundation in a study of the NSF Industry University Cooperative Research Centers program. The Center for MY RESEARCH (CMR) at MY University is one of the Centers funded by NSF. As you know, NSF support over five years have been and still is very crucial to the development and growth of the MU program. Comparative data for NSF-supported research Centers are being obtained. This information will be used to assess and develop NSF's internal policy and administration for establishing new cooperative research Centers. In addition, this data will provide CMR with useful feedback on communication patterns within CMR and your evaluation of our structure, procedures and achievements. The final report of last year's survey is available from the Center office on request. Since you were involved in CMR during the time frame covered in this study (199x-199x) we need feedback from you on the operations of our Center. Dr. EVALUATOR from the Department of RESEARCH at MU is responsible for conducting the CMR study. He will ensure the anonymity of individual responses. Dr. EVALUATOR and I would greatly appreciate it if you would please fill out the enclosed questionnaires and return them in the self-addressed envelope. If you have any questions, please feel free to call Dr. EVALUATOR at (XXX) XXX-XXXX. Sincerely, Director Enclosure

3.7.4 SAMPLE EVALUATOR LETTER

November 21, 20xx President, Multi Systems Dear The enclosed survey is part of the National Science Foundation- sponsored research program on industry-university cooperative research Centers. This is the final year in which the questionnaire will be administered to CMR participants. The purpose of the questionnaire is three-fold: to provide information that will be useful to CMR administration, to provide information that will contribute to the development and administration of the NSF Cooperative Research Centers Program, and to provide data that will enrich our understanding of industry-university cooperation in general. Since you were involved with CMR during the year ending June, 19XX, I am asking you to participate, regardless of your (or your company's) current involvement with CMR. The questionnaire should take about 15 minutes to complete. Your responses will be held in strict confidence; no individual will be identified by name in any report resulting from this survey. Your responses will not be shared with other Center participants except in aggregated form. Please return the questionnaire to me by December 5, 19XX. Thank you in advance for your cooperation. Please do not hesitate to call if you have any questions. Your response is greatly appreciated. Sincerely, Dr. Kermit Evaluator Center Evaluator (XXX) XXX-XXXX

3.7.5 SAMPLE FOLLOW-UP LETTER

November 12, 20xx Dear I have begun coding our CMR questionnaires and discovered that I have not received a completed questionnaire from you. I hope to proceed shortly with preliminary analysis but need a complete data set before I begin. I understand how busy you must be this time of year, but I hope you will take a few minutes to complete and return the survey by Monday, November 25, 19XX. If there are any questions, please call me at (XXX) XXX-XXXX. Thanks for your help. Sincerely, Kermit Evaluator Project Evaluator

 

3.7.6 SAMPLE FACUTLY PAPER BASED EMAIL

Dear Center Faculty Member and Research Scientists, As you likely recall, each year the National Science Foundation requires that we conduct a survey of Center faculty. This survey is a tool for making interim assessments of the progress of the center and is part of the overall database for the study of all the cooperative research centers. It is designed to yield information that may be helpful to the center and for helping to establish successful centers at other universities. Please complete the attached survey and e-mail it or fax it back to me [ATTACH SURVEY]. **Note: make sure to “save as” the survey on your hard drive. Saving changes after opening the attachment will not save your data unless you save it to your hard drive. Please notify me if you'd rather have me mail or fax this survey to you. Otherwise, please take a few minutes to fill out the attached survey and email or fax us back the feedback. PLEASE RETURN BY: [ENTER DEADLINE DATE], 2005 Thanks in advance, EVALUATOR NAME: ADDRESS: E-MAIL: PHONE: FAX:

3.7.7 SAMPLE FACULTY WEB-BASED EMAIL

Dear Center Faculty Member and Research Scientists, As you likely recall, each year the National Science Foundation requires that we conduct a survey of Center faculty and research scientists. This survey is a tool for making interim assessments of the progress of the center and is part of the overall database for the study of all the cooperative research centers. It is designed to yield information that may be helpful to the center and for helping to establish successful centers at other universities. This year you are able to complete this survey via the Internet; thus, you will not be receiving a separate evaluation survey. The link below will take you to the survey, which is a revised version to the one that has been used in previous years. The link to the survey is: [PASTE IN LINK TO QUESTIONNAIRE] At the beginning of the survey, please make sure to select: [ENTER CENTER NAME AS LISTED ON WEBSITE] (Please be careful not to scroll so that the appropriate center remains highlighted). Please remember that your responses will be held strictly CONFIDENTIAL. No individual will be identified by name in any reports resulting from this questionnaire. Your responses will NOT be shared with other Center participants except in AGGREGATED form. Please submit your survey no later than: [ENTER APPRORIATE DEADLINE], 2005 As always, if you have any questions, please do not hesitate to contact either [EVALUATOR’S CONTACT INFORMATION] or the IUCRC Evaluation Project at NC State University (Phone: 919-515-3237, email at iucrc@ncsu.edu). Thank you for your cooperation and we look forward to your participation!

 

3.7.8 SAMPLE FACULTY WEB-BASED EMAIL REMINDER

Dear <Center Name> Faculty Member or Research Scientist; A couple of weeks ago, I sent you an email requesting that you complete a web-based questionnaire based on your participation in <Center Name>. If you have already completed the questionnaire, thank you for your help and please ignore this reminder. If you have not completed the questionnaire, I would appreciate it if you could click on the following link and complete the questionnaire. It should only take about 5-10 minutes of your time. Since we do not ask for any identifying information on the web survey, your answers will be completely confidential. <LINK> Thanks in advance for your help. XXXXXXXXXX Center Evaluator