Process/Outcome Surveys

Introduction

The Process/Outcome (P/O) instruments are designed to assess Center processes and outcomes based on the perceptions and reports of industrial and university participants. Process items tend to focus on participant characteristics and perceptions of the Center’s operations. Outcome items include the assessment of immediate, intermediate, and ultimate outcomes of Center activities. There are currently five instruments used in this activity: Industry Pulse Survey, Industry Benefits Inventory, Faculty Year 1-5 P/O Questionnaire, Faculty Year 6-10 P/O Questionnaire, and the Student Survey. Data may be collected in person or through an email survey, or via a web survey format. These P/O instruments are designed to provide directors and NSF with critical feedback from industry, faculty, and student participants about how the Center is operating. This should allow directors the opportunity to correct problems and reinforce positive aspects of Center operations. As a consequence, it’s critical that results from the questionnaires be shared with your director as quickly as possible. The I/UCRC Evaluation Team has also developed a Checklist for Optimizing LIFE and PO Response at IAB Meeting to help Evaluators collect their data and enhance response rate as much as possible. Collecting Process/Outcome (and LIFE) data has become challenging. This document highlights steps the evaluator can take to optimize the compliance and response rate for both the LIFE forms and P/O questionnaire during the IAB meeting. Obviously, the effectiveness of these steps depends on satisfactory attendance by IAB members. We have also developed a Process/Outcome Workbook, which includes codebooks and data entry shells to help streamline the data entry and submission process.

Frequency of Administration

The instruments are administeredannually. Below is the recommended survey administration schedule. You man need to deviate from this depending on the particular circumstances of your Center. However, each survey should be administed annually.

Surveys and Instruments

Below you will find Word and PDF versions of the Industry Member Pulse Survey, the Industry Member Benefits Inventory, the Standard Faculty Process/Outcome Survey for use in Centers’ first five years, the Short Faculty Process/Outcome Survey for use in Centers’ 6 years and older, and the Student Survey. Please see the Process/Outcome Workbook for detailed codebooks and data entry shells. Please return survey data by entering it into the Process/Outcome Workbook and emailing it to the NCSU Evaluation Team. If you plan to use the web surveys, please send an email to ncsuiucrc@ncsu.edu notifying the NCSU Evaluation Team of your plans and requesting a copy of the data for your Center(s).

IUCRC Process/Outcome Survey Reporting Templates

Respondents

In general, potential respondents for the Process/Outcome instruments are all faculty (and research scientists), all IAB members, and all students participating in center research during the preceding year.

Generation of Respondent List

Experience has shown that the best way to obtain a respondent list is to contact the Center’s administrative assistant for an up-to-date contact list. However, if there is any doubt about a respondent, INCLUDE the individual in the sample. This includes individuals who participated in Center activities for most of the year but are no longer active at the anniversary date.

Extensive experience has shown that a different data collection strategy works best with the target populations: industry, faculty, and students.

Industry Survey Implementation

Industry respondents are incredibly busy and are typically only on-site during the Center’s twice-a-year IAB meetings. This is an excellent time for collecting data by administering paper questionnaires during the fall IAB meeting. However, response rates may suffer if attendance is low at IAB meetings. The interpretability of survey findings is greatly influenced by who is completing the survey. Therefore, we also support a web and email based survey administration. Evaluators are encouraged to employ all data collection methods necessary to achieve maximum response rate.

Faculty Survey Implementation

Generally speaking, faculty respondents are a lot easier to track down than industry respondents. As a consequence, faculty questionnaires can be administered anytime during the fall semester but generally no later than the IAB meeting. We currently support two methods of data collection: mail/email questionnaires and web surveys.

Email/Web Survey

We now support a web version (see above) of both the long and short faculty questionnaires. This data collection method involves local evaluators using an email to contact and direct faculty to a web-based Process/Outcome questionnaire maintained by NCSU Evaluation Team. When data collection is completed, NCSU will send you a dataset. Here are some specific steps for using this option.

1. 1. Notify the NCSU Evaluation Team that you plan to collect your data by web questionnaire.
   2. Get an up-to-date list of faculty and research scientists from your Center’s administrative assistant.
   3. Send an email to these individuals requesting they complete the web-based questionnaire.
   4. NC State will send you updates on your response rate (number of questionnaires submitted) upon request.
   5. Depending on your response rate, you should send several reminders to increase your response rate. Since submitted questionnaires are anonymous, you will need to send this to all respondents.
   6. At your request NC State will send you a documented excel data set for you to analyze and summarize for your Center.

Submitting Data to NCSU

When you have all of the total response and have recorded your data, send the completed Process/Outcome Workbook with data entered to the evaluation team at NCSU, coded according to the associated codebooks. Please retain copies of all questionnaires. If you collect data by web-questionnaire, you can skip this step.

Analysis of Results

Evaluators have the responsibility for providing the Center with feedback on questionnaire results. At a minimum, evaluators should prepare a report which summarizes both faculty and industry responses (frequency counts and percentages) to each item. Many evaluators are also asked to give a PowerPoint presentation on the results of the survey at the next IAB meeting. After the first administration, the report should emphasize significant changes in outcome/satisfaction items over the reporting periods. Evaluators have also received positive feedback when they compare their Center to I/UCRC normative scores (distributed every summer by NCSU team at the June NSF Evaluator’s Meeting). Please remember you should promise confidentiality to your respondents. If you feel your director would benefit from knowing who gave a particular response (e.g., intention to quit the Center), you must check with the respondent before providing this information.

The NCSU Evaluation Team will be responsible for aggregating across Centers and for any comparative analyses of the Process/Outcome data.

Process/Outcome Assessment Report

To see a sample of the Process/Outcome assessment report, please click here(PDF).

Archived and Optional Instruments

Over the years, the I/UCRC Evaluation Project and individual evaluators have developed a number of assessment devices.  Some were used as part of the I/UCRC protocol but then discontinued, others were developed by individual evaluators and used for specific supplemental evaluation projects.  Below is a list of these instruments.  Evaluators may wish to consider using these instruments to supplement their required assessments. Please seek permission prior to using them for research or as the basis for developing another instrument.

All of these instruments are available for downloading and printing using Adobe Acrobat Reader.