Economic Impact Assessment

In a recent report, the NCSU IUCRC Evaluation Team was able to demonstrate that they could document significant economic impacts in mature IUCRCs by engaging in a proactive assessment strategy that involved either face-to-face or telephone interview methodology and provision of confidentiality, if requested, to the beneficiary who would be providing the economic estimates. Based on these findings, beginning Fall 2012, all evaluators are expected to attempt to obtain such information as part of their effort to document Center “success stories” within the Evaluators Report. There is no assumption every Center will produce such outcomes, particularly early in their history. However, evaluators should indicate in their report that they attempted to document such impacts. The NCSU Evaluation Team is prepared to provide consultation including some selected site visits to help evaluators fulfill this new data collection requirement.

Assessment Objective:

To provide early detection and documentation information on Center impacts that may represent significant economic or societal impacts.

The following are assessment guidelines, scheduling guidelines and supporting materials for fulfilling these responsibilities:

Assessment Guidelines:

  • Impact data collection should become a higher priority for evaluators at Centers as they become more mature with a greater emphasis placed on this activity during Phase 2 and 3 of NSF funding.
  • Special effort should be taken to document impacts that appear to have the potential to produce “significant” economic impacts.
  • Assessment should emphasize data collection via personal interviews of targeted high impact beneficiaries.
  • The evaluator is not expected to perform a precise economic impact assessment. Rather s/he is expected to serve as an informed source who can “prospect” for impacts that appear to be potentially significant and that could be subject to more rigorous assessment by NSF
  • Evaluators should confirm with interviewees whether their organization can be identified in their reports or whether they would like their identity to remain confidential. This may require the evaluator to prepare cases where the identity of the beneficiary is kept confidential and/or to restrict access to their evaluator report.
  • Evaluators should attempt to document forecasted impacts (impacts the respondent predicts will happen in the future) by conducting follow-up interviews with informants in order to validate these estimates.

Scheduling:

Evaluator Reports must be attached to a Center’s Annual Report which is due 90 days prior to the Center’s award date anniversary. As a consequence, evaluators should typically begin the process of exploring whether a significant Center impact has occurred about 60 days earlier or five months before the Center’s award anniversary. Beginning Fall 2012, the NCSU IUCRC Evaluation Project will begin reminding evaluators when they should begin this process.  Centers that are submitting Phase II or Phase III proposals will be required to submit their annual report at the same time as their Center’s proposal. In these instances, the evaluator may need to produce their report earlier.

The typical sequence of events for the assessment would include:

  • Contacting and interviewing the director, faculty or other informants and using archival information to identify a member organization or other beneficiary that may have realized a significant benefit from the Center (See EconImpact 1, 2, 3).
  • If such a potential beneficiary organization is identified, asking the director to send an email introducing you to the beneficiary (See EconImpact 4).
  • Sending an email to the beneficiary representative explaining the assessment and scheduling a screening and assessment interview (See EconImpact 5, 6, 7).
  • Preparing an impact assessment summary that can be included in your Evaluator Report (See EconImpact 8).

Supporting Materials:

The following materials and tools are intended to facilitate the collection of well documented success cases that include credible economic impacts.

Goals:

  • Identify firms that you believe have realized the greatest economic impact from IUCRC research and/or technology;
  • Attempt to obtain a more detailed, preferably quantitative estimate of impact

Criteria for selecting firms:

  • Any firm/organization that you suspect used previous or current IUCRC research and/or technology to improve and/or create products, processes or services.
  • Firm/organization could be an IUCRC member, an ex-IUCRC member, a start-up or spin-out based on IUCRC technology, or even a non-member that has acquired a license to use IUCRC technology.
  • You believe firm/organization’s efforts are far enough along that they have already realized some economic benefits by virtue of cost savings, sales, improved performance, etc.

Examples of Types of Impacts
Impact could be technology created within the Center or research that leads to the creation of IP or commercialized products and processes within the firm. Examples include:

  • The discovery of new knowledge (e.g., regarding the properties or characteristics of relevant materials) that led industry in new promising directions
  • A new research method or technique that helped accelerate technological progress
  • A new measurement tool, device, software, or algorithm that works with or impacted an existing commercial process (either whole or in part)
  • An entirely new process that replaced an existing process
  • A new subsystem or component that improved an existing product
  • A whole new system or product that replaced an existing system or product
  • A whole new system or product that opens a new market or industry
  • Other developments that might have commercial value like creation of an accepted industry standard

Sources of Information on Potential Beneficiary
A variety of sources may led you to identify a firm/organization as a potential beneficiary

  • Discussions you have had with firms/organizations
  • Center PIs who are familiar with member applications of Center knowledge and technology
  • Examples you have documented in previous Center’s annual reports
  • The Center Evaluator’s report, including member survey data and Center structure data
  • Items you have nominated for the Compendium of Technology Breakthroughs  (produced by Craig Scott)
  • University Technology Transfer Office (for patent licensees)

Background: The NSF is interested in identifying organizations that have benefited from various IUCRCs. I would like your help in identifying organizations that you believe have benefited significantly from the transfer of knowledge and/or technology from the Center. The Center research or technology they have benefitted from may have been performed at any time during the Center’s existence. The purpose of this handout is to give you some guidance on the type of firm that might be a good candidate for my assessment.

Types of Beneficiary firms.  Since we’re interested in quantifiable, already realized economic impacts, we are primarily investigating knowledge and/or technology that you suspect a firm used to improve or create products, processes or services.  The ‘firm’ could be an IUCRC member, an ex-IUCRC member, a start-up or spin-out based on IUCRC technology, or even a non-member that has acquired license to use IUCRC technology.

Criteria for Selecting Firms
The IUCRC could have economic impact on firms through various channels, including IP created within the Center or research that leads to the creation of IP or commercialized products and processes within the firm. Specific examples include:

  • The discovery of new knowledge (e.g., regarding the properties or characteristics of relevant materials) that points industry in new promising directions
  • A new research method or technique that could help accelerate technological progress
  • A new measurement tool, device, software, or algorithm that works with or impacts an existing commercial process (either whole or in part)
  • An entirely new commercial process that could replace an existing process
  • A new commercial subsystem or component that improves an existing product
  • A whole new commercial system or product that replaces an existing system or product
  • A whole new commercial system or product that opens a new market or industry

What I plan to do. I recognize that you may only have a general idea about how or if a firm may have benefited from your IUCRC work.  As a consequence, I would like to talk directly with firms that you believe are beneficiaries of IUCRC knowledge and technology. Their participation will be voluntary and we will not share information they provide me without their permission. Depending on the type of knowledge or technology, I will ask a firm representative about specific impacts, like follow-on spending, related employment, total sales, cost savings to the firm, cost savings transferred to customers, and similar economic impact questions. We realize that precise information is likely to be proprietary, too difficult to develop, or both, so rough estimates will be perfectly adequate for our purposes.

Reference sources or materials that could help identify possible impact cases include;

  • The director’s own knowledge and experience with the Center and its members
  • Items reported in the Center’s past annual report
  • Previous Center Evaluator’s reports and member survey data
  • Items included in past copies of the Compendium of Technology Breakthroughs  (produced by Craig Scott)
  • Center PIs who are familiar with member applications of Center knowledge and technology
  • University Technology Transfer Office (for patent licensees)

Note: Ask about each impact separately. Obtain as many impact cases as possible. At this point look for general understanding of the knowledge/technology before interviewing the firm contact. Remember to ask for any written documents or weblinks that may relate to successes.

  1. Taking them one at a time, can you identify any firms that you think have realized quantifiable economic impacts from Center research? Who is the beneficiary firm? How is the firm related to the Center—e.g., member, non-member, start-up?
  2. How do you think the firm has benefitted?
  3. Can you give a short, layperson’s description of the knowledge or technology that has had quantifiable, already realized economic impacts? Note: If other than basic knowledge, ask:
    • How does this relate to similar technologies on the market—e.g., incremental improvement, whole new technology, replacement technology, etc.
    • What are the benefits over existing technologies on the market?
  4. Can you explain the technology path or trajectory and how the Center fits in? Did the knowledge or technology originate within the Center?
    • When did the firm adopt the knowledge or technology?
    • At what stage of development did the firm adopt it?
    • In your opinion, would this development have happened without the Center’s research?
  5. What do you know about the impact of the knowledge or technology on the firm—for example, impact on processes, products, services, new hires, total sales, etc.?  At this point we’re only looking for a general sense of the impact before we talk with the firm representative.
    [Note: ask for  any written documents and/or weblinks that may be available.]
  6. Who can we contact within the firm for a short interview about the realized economic benefits related to the IUCRC knowledge or technology (e.g., profits, unit sales, cost savings to customers, etc)? [Note: Get name, email and telephone number].  We need an introduction to this individual.  If I send you a generic “heads up” email can you follow up with your contact?
    • Do you think this firm representative will be forthcoming in sharing impact-related information, even in general, rough estimate terms?

*The ‘technology readiness level’ (TRL) offers excellent definitions of different stages of development and can be found here: http://en.wikipedia.org/wiki/Technology_readiness_level
TRL specific to manufacturing can be found here: http://www.dodmrl.com/
Other information can be found in the DoD Deskbook, including TRLs specific to software, hardware and biomedical: http://www.acqnotes.com/Attachments/Technology Readiness Assessment Deskbook.pdf

Dear [Beneficiary]

I am writing you to give you a heads-up and an introduction to [evaluator] who will be contacting you soon concerning your involvement in [Center] and benefits you may have realized.

The [evaluator] is interested in talking with beneficiaries of [Center] research, and would like to talk to you about participating in this effort and potentially participating in an interview.

By copy of this email, [evaluator] will follow up with you. I encourage you to take part, as this study could potentially influence policy on industry-university collaboration.

Sincerely,

[Center Director]

Dear [Beneficiary]

I wanted to follow-up on [Center Director’s] email to your regarding our study of economic impacts of the National Science Foundation’s (NSF) Industry-University Cooperative Research Center (IUCRC) program.

The NSF is especially interested in the measurable economic impacts of IUCRCs, but we realize that many, perhaps most, of these impacts are very long-term and difficult or impossible to measure in strictly economic terms. So, we’re asking companies who may have benefited from working with IUCRCs to estimate (roughly) for us the impact that Center “outputs”—ideas and technology– have had on the company and its customers. We realize that precise information is likely to be proprietary, too difficult to develop, or both, so rough estimates are perfectly adequate for our purposes.

I would like to schedule a brief phone interview with you to discuss the nature of our assessment, what kinds of impacts we are looking for and see if you are willing to help us out. If you decide to participate in our assessment, we can conduct our interview right then or schedule it for a separate time. I will not share information specific to you or your company outside the research team without your permission.

Please let me know when would be a good time to talk or I will follow up with you by phone.

Regards,

[Evaluator]

General Information: Company Name; Center Name;  Participation Years; Company Contact; Title/Position; E-Mail Address; Date of Interview; Interviewed By

Introduction. Thank for agreeing to talk with me. I am assessing the economic impact of ideas and technologies emerging from industry-university cooperative research. [Director] at [Center] suggested that your organization may have benefitted from Center research and/or possibly taken research into commercialization. The purpose of this phone call is to explain the nature of our study,  to confirm that your organization has realized commercial benefits from ideas or technology that could be attributed in whole or in part to the [Center] and see if you’re willing to share this information with us. Also, if some of the information we ask about is contained in written documents or web-based sources, please direct us to those sources.

Nature of the assessment. The National Science Foundation (NSF) is interested in getting information from organizations that have benefitted from ideas and technologies from IUCRCs. My report will highlight documented quantifiable outcomes and impacts. We’d like to report specific successes to the NSF; however we will not identify you or your company without your explicit permission.

Part I: Screening Question

  • Has your firm/organization benefited technologically or economically from its involvement with [Center]? If so, how?
    • Can you describe briefly, in layperson terms? (If necessary, clarify as product, process, service, component, tool, software, method, etc.)
    • Have these ideas and/or technologies reached commercialization (or commercial application)?  If so, how long have they been commercialized?
    • Would you be able and willing to provide estimates of the economic impact (e.g., sales, personnel growth) for your firm/organization?  We realize that precise information is likely to be proprietary, too difficult to develop, or both, so rough estimates are perfectly adequate for our purposes.

Research note:

  • IF THE ANSWERS PROVIDED BY THE RESPONDENT APPEAR TO CONFIRM THAT THEY HAVE NOT BENEFITTED IN SOME SIGNIFICANT WAY AT LEAST CURRENTLY, THANK THEM FOR THEIR TIME AND ASK IF YOU COULD FOLLOW UP WITH THEM IN THE FUTURE TO ASSESS FUTURE IMPACTS.
  • IF THE ANSWERS PROVIDED BY THE RESPONDENT APPEAR TO CONFIRM THAT THEY HAVE BENEFITTED IN SOME SIGNIFICANT WAY THEN ASK
    • Thanks that is very interesting. I would be interested in including information about this development in our assessment.  I would need some detailed information about this.  Would you like us to go into our more detailed questions about this now or would you like to set up a separate time to talk?

Research Note: IF THEY WANT TO KEEP GOING, BEGIN WITH PART II OF THIS INTERVIEW

Research Note: IF THEY WANT TO SCHEDULE SEPARATE INTERVIEW ASK THEM TO CONFIRM A DATE AND TIME.

National Science Foundation (NSF) is interested in doing a better job of estimating the economic impact of Industry/University Cooperative Research Centers (IUCRCs) on organizations and their customers.  They have asked the Center evaluators to try to conduct follow up interviews with organizations that may have received significant benefits.

The NSF is especially interested in the measurable economic impacts of IUCRCs, but we realize that many, perhaps most, of these impacts are very long-term and difficult or impossible to measure in strictly economic terms.  So, we’re asking companies who may have benefited from working with IUCRCs to estimate (roughly) for us the impact that Center “outputs”–ideas, technology, and student hires–have had on (a) the company and (b) the industry.  We realize that precise information is likely to be proprietary, too difficult to develop, or both, so rough estimates are perfectly adequate for our purposes.

Unless you explicitly tell us otherwise, your name and that of your organization will remain confidential. The NSF may be interested in publicizing case summaries that highlight organizations that have received exceptional benefits from an IUCRC. If we think your organization would make a good case study, we will ask for your explicit permission before it is disseminated in this fashion. If you agree, we will share a draft of our written summary for your review and approval, before it is disseminated by NSF. 

Let’s talk about the [insert idea/technology from director or screening interview].
Research Note: Depending on firm/organization type, go to outline for:

  • I. Established Firms: Product or product-related technology
  • II. Established Firms: Process or process-related technology
  • III. Start-ups: (with no realized sales or revenues to-date)

I. Established Firms: Product or product-related technology

  1. Confirm understanding of the product/technology.
    • Is this a product or a component of a larger product?
    • What are the benefits over existing, similar products on the market?
  2. How does this technology relate to the Center?
    • Triggered by the Center and developed into technology/IP outside the Center
    • IP/technology developed by the Center
    • Would the technology have been developed without the Center?
      • If yes- time lag or delay without the Center?
    • How long did it take to reach commercialization, from when your organization first became involved with the technology? (in years / months)
  3. Since first commercialization what have been the impacts on your organization?
    • Sales and net profits (either total, annually, or per unit)
    • Percent of sales / profits attributable to the Center
    • Personnel associated with the product
  4. Can you provide future or prospective estimates on market growth (percent or dollars)
    • Conservative estimate
    • Best case scenario estimate
  5. What are the impacts on downstream customers of the technology?
    • Cost savings over prior technologies
    • Qualitative benefits over prior technologies
    • What percent of savings/benefits are attributable to the Center?
  6. Let me review some of the specifics of what you have told me. I would like to clarify what if anything I can attribute to your firm or if you would like all information to remain confidential.

II. Established Firms: Process or process-related technology

  1. Confirm understanding of the process/technology.
    • Is this a process or a component of a larger process?
    • What are the benefits over existing, similar technologies on the market?
  2. How does this technology relate to the Center?
    • Triggered by the Center and developed into technology/IP outside the Center
    • IP/technology developed by the Center
    • Would the technology have been developed without the Center?
      • If yes- time lag or delay without the Center?
    • How long did it take to reach commercialization, from when your organization first became involved with the technology? (in years / months)
  3. Since first commercialization what have been the impacts on your organization?
    • Cost savings (either total, annually, or per unit)
    • Percent of savings attributable to the Center
    • Growth or improvement in sales, net profits
    • Growth in personnel associated with market growth
  4. What are the impacts on downstream customers of the technology?
    • Cost savings over prior technologies
    • Qualitative benefits over prior technologies
    • What percent of savings/benefits are attributable to the Center?
  5. Let me review some of the specifics of what you have told me. I would like to clarify what if anything I can attribute to your firm or if you would like all information to remain confidential.

III. Start-ups (with no realized sales or revenues to-date)

  1. Confirm understanding of the technology and firm.
    • What are the benefits over existing similar technologies on the market?
    • How long since the start-up was established?
  2. How does this technology relate to the Center?
    • Triggered by the Center and developed into technology/IP outside the Center
    • IP/technology developed by the Center
    • Would the technology have been developed without the Center?
      • If yes- time lag or delay without the Center?
  3. How much did your organization invest to get this technology ready to commercialize?
    • Total spending to date (wages, supplies, facilities, equipment, etc)Total funding received from all sources: VC, private, government grants
    • Jobs created person-years through life of the firm
    • [If licensed technology] what would alternative technology have cost the organization?
  4. Can you provide future or prospective estimates on market size and growth
    • Conservative estimate (total market and firm-specific)
    • Best case scenario estimate (total market and firm-specific)
  5. What are the impacts on downstream customers of the technology?
    • Cost savings over prior technologies
    • Qualitative benefits over prior technologies
    • What percent of savings/benefits are attributable to the Center?
  6. Let me review some of the specifics of what you have told me. I would like to clarify what if anything I can attribute to your firm or if you would like all information to remain confidential.

Company: Confidential
Years as member: 5 (2004 – 2008)
Interviewee: Confidential
Interviewer: Drew Rivers
Interview Date: December 2010

Orientation to Center
The Company does not currently hold a membership in the Center; however, our informant indicated a renewed membership could be forthcoming. During the company’s five consecutive years as a member we identified no quantifiable benefits aside from the hiring of a [Center] student near or just after the final year of membership. Otherwise, all realized economic impacts came after the termination of membership.

Table 1: Technology Transfer Events

No. Event/Project Description Stage Impact
1 Air compressors The company’s air compressor technology was operating at below desired efficiencies due to unexpected surges and other factors.
This technology was most efficient at near-surge levels.
The Center’s technology helped the company to predict surges before they occurred.		 Testing of a new algorithm has been completed and the new technology is currently being rolled out across the company. Testing of the new technology indicated savings of $50,000/year, per air compressor.
The company maintains dozens of air compressors at each of 8 manufacturing plants.
2 Robot joints The company was searching for a method to predict impending failure of robot joints in a production process.
After testing various factors, a correlation was found between torque and joint health, resulting in the development of a predictive algorithm.		 The new predictive algorithm is in the testing phase. Based on initial testing results, the technology could save each manufacturing plant 400-500 hours of downtime each year,
at $7,500/hour in cost savings. Implementation would take 3 years for the first plant, and 5 years for all 8 plants.

Table 2: Other Impact Events

No. Event/Project Description Stage Impact
1 Student hire BA level student hired by company. Hired Estimated at $50k savings in mentoring costs.

Notable Qualitative Impacts
The company sees additional value from involvement in the Center through collaborative project work and general knowledge sharing among members, as well as an increase in the Company‘s capacity to search and absorb new technologies.

Quantified Impacts
For prospective benefits the informant estimated initial planned deployment of the technologies in years 1 through 3 at a single plant, then accelerating across all 8 plants in years 4 and 5. Applying a best case scenario, we assume the Company will realize full benefits across its operations within the next five years, with greater benefits realized in years 4 and 5. Table 3 below shows these prospective benefits.

Table 3: Retrospective and Prospective Economic Impacts (estimates in actual dollars)

Year Air compressors Robot joints Other Total
2004
2005
2006
2007
2008
2009 $50,000 $50,000
2010 $50,000 $50,000
2011 (est) $594,000 $1,113,750 $1,707,750
2012 (est) $1,188,000 $2,227,500 $3,415,500
2013 (est) $1,800,000 $3,375,000 $5,175,000
2014 (est) $7,200,000 $13,500,000 $20,700,000
2015 (est) $14,400,000 $27,000,000 $41,400,000

Total retrospective impacts: $100,000
Total estimated impacts through 2015: $72,498,250

Attribution to the Center
The company views the impact as a collaborative effort; the Center bringing advanced skills and the company bringing knowledge of their manufacturing system. As such, both parties were necessary to achieving the outcomes. Specifically for the robot joints project, the company initially attempted to perform the work internally but later opted to involve their prognostics suppliers and the Center. The Center proved a key player in the effort, with advanced skills in collecting and analyzing data. The Center played an instrumental role but was one of several actors in the development of the technology.

Summary
Despite clear linkages between the technology and the Center, the interviewee expressed uncertainty regarding attribution; a host of actors and resources outside the Center, including the Company itself, contributed to getting the technology from early stages of development and into commercial application. Further, we assumed a best case scenario with the roll-out of these technologies across the Company’s operations. Alternatively, a fully implemented evaluation might also include a conservative estimate or likelihood of complete deployment in the next five years, along with periodic follow-up by the evaluator with the Company representative to gauge progress on implementation and realized benefits from the technology.