Center Evaluator Reports

Background

The Evaluator’s Report provides a narrative description of the circumstances, events, and actions that contribute to the Center’s development and growth. The Evaluator’s Report is prepared based on a structured outline. In new Centers, the focus of the Evaluator’s Report will be on the ideas, plans, and efforts that led to the creation of the Center. In Centers that have been in existence for several years the focus will be on the changes that occur as the Center evolves and matures and various Center outcomes and achievements with a particular emphasis on measurable economic impacts. In addition to the report itself, please also include the Evaluator Report Cover Sheet, this document is intended to capture critical information and make it available at a glance, and the Semi-annual Meeting Best Practice Checklist – one for each IAB meeting.

The intent of this report is to provide Center evaluators with a vehicle for recording the unique history of each Center while allowing the data to be aggregated across Centers for purposes of overall program evaluation. The report provides the director with a qualitative evaluation from an involved yet objective observer. Some Centers share this report with new members as a way of briefing them about the Center’s history. The report is also intended to provide NSF with a record of the Center’s progress and accomplishments.

General Instructions

All evaluators are expected to complete and submit an Evaluator’s Report annually. The structure of the report is defined by the outline provided in the Evaluator’s Report Section. Major topics to be addressed by the report include: overview, environmental/ institutional, organizational, research, accomplishments and analysis. Brief instructions for completing these sections are included on the report outline. A particular point of emphasis for the evaluator report beginning Fall, 2012 is a focused effort to document quantifiable economic impacts for members and other organizations. The report submitted to your director for inclusion in the Center’s renewal report should typically not exceed five pages plus the cover sheet.

It should be noted that each of the sections requires the evaluator to directly address issues of stability and/or change for that year and note the impact of these changes. Much of the report can be completed based on information obtained from participant observation. However, evaluators may also need to examine Center documents, refer to national databases and interview relevant Center participants (e.g., director, business manager, and/or other key Center faculty, beneficiary firms). The Outline of Relevant Issues section provides an outline of the kinds of issues, which can be addressed in each section. The Sample Evaluator’s Report section provides a sample of a completed report.

Schedule for Submitting Report

NSF requires that a copy of the Evaluator’s Report be included with all Center renewal proposals. Directors are required to submit their renewal proposals to NSF at least 90 days prior to the anniversary of their award date (although it is not uncommon for directors submit it after this deadline). Directors submitting proposals for Phase 2 or 3 funding are required to include their evaluator report with their proposal and may need your report earlier in these instances. Evaluators must consult with their directors on their due date and should plan to submit a copy of their report to their director at least a month before this deadline in order to allow for review and integration. You may consult the NSF website to identify the award date(s) for your Center and it’s sites. Evaluators must also forward copies of the report to the NCSU Evaluation Team.

Access to the Evaluator’s Report Database

The Evaluation Team at NCSU will continue to monitor and archive the Evaluator’s Reports submitted each year. It is anticipated that individual evaluators will propose studies, which would include access to the Evaluator’s Report database. Use of the database for this purpose is encouraged. Any evaluator interested in using the database should submit a written request to the NSF IUCRC Program Director for permission to receive any portion of the database.

Below are some “helpful hints” you may want to consider as you embark on preparing your Evaluator’s Report.
Preparation:

    1. Gather all relevant files, notes, Center reports, etc. organize them and become familiar with the information they contain.
    2. Keep a copy of the Evaluator’s Report handy (blank or the previous year’s) and periodically (especially after board meetings, etc.) jot down notes which will form the basis for that year’s answers.
    3. Prepare a timeline of major events for the past year.
    4. First Draft
    5. Make brief cryptic notes on each question outlining what should be included in your answer and noting where you will have to refer to documents or other sources to get adequate details (including results from Process/Outcomes assessment).
    6. Begin by answering questions for which you have the information at your fingertips; move on to other questions later, perhaps after you have had the chance to do some research.
    7. Revisions and Final Draft:
    8. Schedule interviews with Center director and other relevant parties to get information you lack and/or clarify points of uncertainty. Do not waste people’s time going over issues which are obvious.
    9. Refer to outside documents as necessary (e.g., U.S. Industrial Outlook)
    10. Make final revisions incorporating this information.
    11. Solicit feedback from Center director on accuracy and completeness.
    12. Fill in the Evaluator Report Cover Sheet.

General Instructions: Summarize the Center’s recent history (last twelve months or since the last history was prepared). Please address each domain. Emphasis should be on significant changes, which might influence the Center’s ability to achieve its goals and objectives.
Length:  about 5 pages.

  1. Evaluator Report Cover Sheet
    • Fill in the provided form to capture basic information and make it available at a glance.
  2. Overview
    • Provide a general overview of the Center’s status.
  3. Goals and Objectives
    • Please describe the Center’s primary technical and organizational goals and objectives.
    • Instructions: This question can be answered by listing a Center’s “official” (written) and/or unofficial (but acknowledged) goals and objectives. Goals and objectives should be updated or elaborated upon (e.g., the basis for inferring informal goals) as needed. You should indicate if your answer reflects a change from previous years.
  4. Environmental/Institutional
    • Please describe any environmental (e.g., decline in industry’s competitive position) or institutional/university (e.g., partnerships with other universities, shift in university priorities) changes which have occurred during the last year.
    • Instructions: Evaluator’s are encouraged to include industry data based from outside sources.
  5. Organizational
    • Please describe any changes in the Center’s personnel, structure, policies, financial status, and operations, which have occurred during the last year.
    • Instructions: The section can be answered based on participant observation, reference to program documents, data collected via the Process/Outcome questionnaire, and information reflected in the Semi-annual Meeting Best Practice Checklist.
  6. Research Program
    • Please describe any changes in the Center’s research program (e.g., new or modified thrust areas), which have occurred during the last year.
    • Instructions: The section can be answered based on participant observation, reference to program documents and data collected via the Process/Outcome questionnaire.
  7. Center Accomplishments
    • Please describe any accomplishments or impact the Center has had in the following areas: knowledge/technical advances; technology transfer; educational impacts. In addition, comment on accomplishments, which may reflect unique Center objectives (e.g., forging international linkages, etc.).
    • Instructions: Evaluators are encouraged to obtain the information needed to answer this question from a variety of sources, including: program documents (e.g., Center’s annual report), responses or comments provided in the Process/Outcome questionnaire, discussions with CD, IAB members and faculty.
      1. Center Success Case Study and Economic Impact Assessment
        1. Effective 2008-9, evaluators are required to produce a mini-case (one to two paragraphs) description of a recent Center “success story” (click here for an example). A Center success story might be scientific, technological, technology transfer, or educational in nature. The purpose of this part of the report is to try to document some Center activity, output or outcome. This mini-case might be used by the Center, NSF or others as evidence the Center or program is achieving its objectives. While not exhaustive, a Center success might include: a significant scientific breakthrough or award; a firm development or commercialization activity that was related to Center research (perhaps listed in the P/O survey); a new significant patent filing; a start-up or spinout company linked to Center research; a scientific, technological or commercial development achieved by a Center alumni, etc. The evaluator is expected to obtain background information from relevant sources (e.g., faculty, members, licensees) about the success story to prepare a mini-case (e.g., think in terms of who, what, where, how and when). Eventually, some of these mini-cases will be examined for inclusion into the “technology breakthroughs” report NSF I/UCRC produce. The evaluator is encouraged to select a success story based on input from the Center director and/or IAB.
        2. Effective fall 2012, evaluators are required to attempt to identify and document via an interview any Center outcome that might represent a significant economic impact. Please refer to the Economic Impact Assessment section for additional information on how to fulfill this responsibility. Please pay particular attention to whether or not the beneficiaries that you interview desire confidentiality or not for the information and estimates that they provide you.
  8. Analysis
    • Based on the information provided above and other relevant information, comment on the health and vitality of the Center. What are the implications of the various environmental, institutional, organizational and research changes? Is the Center making adequate progress in achieving its objectives? Is the research program still vital and current
    • Instructions: Comments should reflect the evaluator’s view of the major obstacles and/or opportunities, which may affect the Center’s success during the next one to three years?
  9. Timeline
    • Attach an updated timeline of significant events and milestones, which have occurred over the Center’s lifetime.
  10. Semi-annual Meeting Best Practice Checklist
    • Report on the extent to which the Center adheres to IAB meeting best practices.

Instructions: Evaluators can address a variety of issues within the different sections of the report. The bulleted items listed below are a sampling of issues addressed by evaluators in previous reports.

  1. Evaluator Report Cover Sheet
    • Provide as summary of Center status and activity available at a glance
  2. Overview
    • Provide a general overview of the Center’s status.
      • Purpose of the report
      • Age of Center
      • Funding snapshot
      • Membership snapshot including names of firms
      • Number of years remaining on NSF award
  3. Goals and Objectives
    • Technical Goals
      • Center’s mission statement
      • Technical goals and objectives
    • Organizational Goals
      • Increase in number of industry sponsors
      • Increase in technology transferred
      • Reduction in production costs
      • Number of patents produced
  4. Environmental/Institutional
    • Environment
      • Name and number of industries Center serves (e.g., energy and chemical)
      • Relevant economic indicators for industry
      • Percentage of companies in particular industry that have downsized or closed
      • Percentage of change of R&D dollars by firms or industry
      • # of membership terminated due to economic downturns and industry reorganizations
      • Percentage of change in employment in particular industry
      • Level of analysis may be local, regional, and/or international
      • Competition from other Centers
    • Institutional
      • Changes in university leadership positions
      • Changes in policies and procedures (e.g. overhead, conflict of interest)
      • Willingness and ability of the university, department to support the Center (e.g., facilities, equipment, personnel)
      • Interdepartmental issues
      • Status of linkages with other universities (e.g., multi-Center)
  5. Organizational
    • Personnel
      • Turnover of key faculty and staff (describe transition)
      • # of personnel (faculty) by department supported by Center research
      • # of faculty who have left the Center
      • # of graduate/undergraduates supported by Center
      • Visiting scholars involved in Center research
      • Cohesiveness of key Center staff and faculty
    • Structure
      • Changes in organizational chart
      • Changes in roles
      • Changes in reporting relations
    • Policies
      • New or changed policy or procedures
      • Center SOP – are they adequate? Used? Stifle creativity?
    • Financial stability
      • Status relative to NSF criteria
      • Amount of funding by source
      • Membership status (current, new, left)
      • Major grants awarded
      • In-kind contributions
      • Prospects for new funding or loss of funding
      • Major equipment donations
    • Operational
      • P/O results on Administration items
      • Recruitment efforts
      • Proposal development and selection efforts
      • Planning efforts
      • Center newsletter activity
      • Formal decision making process (L.I.F.E. form used?)
      • Basic description of semi-annual meetings (e.g., projects funded, voting procedures)
      • Assessment of the effectiveness of meetings (i.e. findings from Semi-annual Meeting Best Practice Checklist)
  6. Research Program
    • P/O data from research items
    • Major thrust areas
    • Number and changes in core, enhancement projects
    • Average dollar value of projects
    • # of total number of faculty/research assistants for each project; multidisciplinary teams
    • # of pre-proposals generated
    • Percentage of team-based research project initiated by the Center
    • L.I.F.E. form ratings
    • TIE projects
    • # of project that were voted on, accepted, rejected
    • # of collaborative research projects with industry
    • Typical length of projects
    • Basic or applied perspective
  7. Center Accomplishments
    • Knowledge/technical advances
      • P/O data from faculty and IAB
      • Technical advances
      • Impact of publications
      • # of articles published; prestige of journals
      • # of presentations made
      • # of posters presented
      • # of abstracts submitted
      • Awards for research or teaching
    • Technology transfer
      • P/O data from follow-on funding and benefits
      • # of invention disclosures
      • # of patents filed
      • # of visits to and from industry and with other research laboratories
      • # of presentations made during industry visits
      • # of web sites (hits, citations from other web sites)
      • # of workshops conducted
      • # of open conferences
      • # of short courses offered
      • Commercialization
      • Dollar benefits
    • Educational impacts
      • Curriculum changes and improvements
      • # of graduates hired by industry
      • # of undergraduates that work at the Center
      • # of Masters and doctorates completed with Center research
      • IAB offers on-site training for graduate students/undergraduate students
      • Post doctoral instructional components
      • Any kind of pre-college educational program
      • Impact on student training
  8. Center Success Case Study
    • A one to two paragraph description of some significant scientific, technological or other output or outcome of Center activities (see description above).
    • Please use the guidelines and supporting materials provided in Economic Impact Assessment Section.
    • Since it may take a number of years for Center to produce economically significant impacts for their members, evaluators may complete this reporting requirement by indicating that there is nothing significant to report at this time.
  9. Analysis
    • Commentary on major trends and developments noted in other sections of the report
  10. Timeline
    • Chronology of major events since Center began
  11. Appendix A:Semi-annual Meeting Best Practice Checklist (one for each IAB meeting).

For a sample of an evaluator’s report, please click here. You can click here for an example of a Center Success Case Study