Erfahrungen als Gutachter der EU Evaluator, Reviewer, Intern. Observer Prof. Dr. Jürgen Kreyssig ProDekan Fakultät Informatik Goslar, 19. Nov. 2013
Zur Person: Prof. Dr. Jürgen Kreyssig TU Braunschweig; Forschungsprojekt mit IBM 7 Jahre Tätigkeit in einem Technologie-Transfer- Zentrum Seit 1994 Professor in der Fakultät Informatik der Hochschule Braunschweig/Wolfenbüttel Embedded Systems, System On Chip, RFID RTD-Performer in mehren EU-Projekten Projektkoordinator in mehren EU-Projekten Prof. Dr. Jürgen Kreyssig j.kreyssig@ostfalia.de
Zur Person: Prof. Dr. Jürgen Kreyssig Seit 1992 als Gutachter der EU tätig Evaluation: Review: Observer: RTD, SME, Research Potentials International Cooperation, Marie Curie Research for SMEs/SME associations International Cooperation Research Potentials International Cooperation Prof. Dr. Jürgen Kreyssig j.kreyssig@ostfalia.de
Inhalt Beispiel: Research for the benefit of SMEs Typische Calls for Proposals: RTD for SMEs (1 stage evaluation) RTD for SME associations (2 stage evaluation) Ablauf der Evalution; Bewertungskriterien Review der EU: Begutachtung eines laufenden Projektes International Observer der EU Prof. Dr. Jürgen Kreyssig j.kreyssig@ostfalia.de
Das 7. Rahmenprogramm 50.000 Mio. Ideas (ERC) 7.46 Mio. (14.8 %) Research for SMEs Reserach for SME associations 1.300 Mio. People (Marie Curie) 4.728 Mio. (9.4 %) Capacities 4.217 Mio. (8.3 %) Joint Research Centre 1.751 Mio. (3.5 %) Cooperation 32.365 Mio. (64 %) Prof. Dr. Jürgen Kreyssig j.kreyssig@ostfalia.de (Stand: 2006)
Beispiel: Research for SME Regelmäßige Ausschreibung (ca. jedes Jahr): Research for SME associations Research for SME companies Beispiele Call Identifier: FP7-SME-20XX-1 (one step) Call Identifier: FP7-SME-20XX-2 (two steps) Prof. Dr. Jürgen Kreyssig j.kreyssig@ostfalia.de
Research for SMEs Principle: Outsourcing scheme customer-seller relationship SMEs Investing in Research Results & IPR RTD- Performers Other enterprises, End users. The grant covers a big part, but not all! Participants, SMEs as well, need to co-finance!
Indications Consortium: 5 10 partners Total budget (costs): 0.5 1.5 Million Duration: 1 2 years Deviations from these recommendations are possible, but should be well justified.
Research for SMEs Call Identifier: Call title: FP7-SME-2008-1 Research for SMEs Published on: 30 November 2007 Budget: 94 million Call Deadline: 11 April 2008
Number of proposals submitted: 673 Number of ineligible proposals: 10 Research for SMEs Evaluation Budget: about 100 million Requested funding: ~ 1.0 million About 100 proposals can be funded!! Result: Evaluated above thresholds: 267 Requested budget: 277 million
Research for SMEs Objectives Support innovative SMEs to solve common or complementary technological problems in order to strengthen their competitiveness by enhancing their investment in RTD-activities and acquisition of intellectual property rights and knowledge. Projects must be centred on the innovation needs of the SMEs which outsource research to RTD performers and must demonstrate a clear exploitation potential for the SMEs concerned. Projects are centred around the economic interest of the SME participants who will take an active role by defining their specific technological needs and will ensure that the research performed complies with their requirements.
Principle: SMEs outsource part of their research needs Customer-seller relationship between SMEs and RTDperformers. SMEs invest in the RTD project and outsource part of the research activities to "RTD performers. RTD performers invoice their services to SMEparticipants. EC contribution: Grant to the project (and not to individual participants) which covers only part of the total investment. SMEs and "Other participants and end users" contribute with own resources to the project. Collective leadership of the SMEs in the project.
Evaluation: Place Rogier, Brussels
Evaluation Process Basic Principles EXCELLENCE FAIRNESS & TRANSPARENCY CONFIDENTIALITY Evaluation Rules Guide for applicants (annex 2) EFFICIENCY & SPEED IMPARTIALITY ETHICAL & SECURITY CONSIDERATIONS
Overview of the evaluation Example Reading and Individual Assessment Consensus Meetings & Reports Panel Review a) Monday to Tuesday Tuesday to Thursday Friday b) Remote: two weeks Monday to Thursday Friday About six panels representing the different research areas
The Evaluation Process Thematic Panels Individual assessment by at least 3 experts Consensus Reports Quality Check of CRs Panel Review Final Ranking List
Part B A maximum number of pages has been indicated for the proposal as a whole: Stage 2 Part B proposal: Section 1: Max. 15 pages +tables Section 2.1.1: Max. 4 pages Section 2.1.2: Max. 3 pages + ½ page per partner Section 2.2: Max. 4 pages Section 3: Max. 10 pages + tables Exceeding pages might be ignored Each proposal will first be assessed independently by three experts. One of them is assigned as rapporteur.
Evaluation criteria & sub-criteria 1. S/T QUALITY Sound concept, and quality of objectives Innovative character in relation to the state-of-the art Contribution to advancement of knowledge / technological progress Quality and effectiveness of S/T methodology and associated work. 2. IMPLEMENTATION Appropriateness of the management structures and procedures Quality and relevant experience of the individual participants Quality of the consortium as a whole (including complementarity and balance) Appropriate allocation and justification of the resources to be committed (budget, staff, equipment) 3. IMPACT Contribution, at the European [and/or international level], to the expected impacts listed in the work programme under the relevant topic/activity Appropriateness of measures envisaged for the dissemination and/or exploitation of project results, and management of intellectual property.
Interpretation of scores Each criterion will be scored out of 5. The scores indicate the following with respect to the criterion under examination: 0 The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information 1 Very poor. The criterion is addressed in a cursory and unsatisfactory manner. 2 Poor. There are serious inherent weaknesses in relation to the criterion in question. 3 Fair. While the proposal broadly addresses the criterion, there are significant weaknesses that would need correcting. 4 Good. The proposal addresses the criterion well, although certain improvements are possible. 5 Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.
Scoring Each criterion is scored 0-5 Half-scores allowed Whole range (0 to 5) should be considered No weightings will be applied Thresholds will be applied to the scores. The thresholds for criteria: 3 (1 and 2); 4 (3); The overall threshold, applying to the sum of the three individual scores, will be 11. When scoring proposals, experts must only apply the above evaluation criteria.
Evaluation Jeder Gutachter geht anders vor: I. a) Summary; (Partner; Budget Overview) b) S/T Quality c) Implementation d) Impact II. a) Summary III. a) Summary b) Impact b) Implementation c) Rest c) Rest Jeder muss nach fünf Minuten begeistert sein!!
Guiding questions to evaluate Implementation Description of the Consortium Is the consortium as a whole well balanced and reasonably sized with respect to the proposed work plan? Does the proposal clearly present the profile and role of each participant in terms of competencies, skills, key personnel, business / industrial activity, research field? Are the SME participants the real driving forces for the project with each having an active role in the consortium? Do the RTD performers demonstrate a high level of scientific excellence and complement each other? Does the proposal show the interest of all participating companies in the project with no or little overlap?
Guiding questions to evaluate Impact To what extend demonstrates the proposal a clear economic impact for the SME participants in terms of economic growth, employment, market strategy, distribution channels, etc? How strongly will the project improve the competitiveness of the involved SMEs. How sound are the economic justification of the proposed research? (potential direct economic benefits for the individual SME participants) Does the project address adequately Community societal objectives. Is the transnational approach well justified and geared towards increased transnational technological cooperation amongst SMEs and between SMEs and Research organisations or other organisations at the European level?
Consensus meeting: Who s who Evaluator Independent Observer Rapporteur Moderator EC Staff
Consensus meetings 3 Evaluators + Moderator as chairman No longer than 30 min Focus on discrepancies In case of no agreement the Panel Coordinator can decide for a 4th reading (to be avoided!) Only half points can be given
IAR IAR IAR Consensus meeting FINALISED in RIVET Ready for Consensus Meeting WORKFLOW CR v.1 Quality check by the EC staff COMMENTS from EC By Rapporteur, FINALISED on Rivet APPROVED & FINALISED in RIVET by the 2 other experts ONLY IF NO COMMENTS: final CR ready for printing by EC To be signed
The Evaluation Process Individual assessment by at least 3 experts (REMOTE) Consensus (CENTRAL EVAL.) Panel Review Establishing Priority Order of Proposals (CENTRAL EVAL.)
Ranking Proposals from all areas Section I: Above cut-off-score; above all thresholds Section II: Equal cut-off-score; above all thresholds Section III: Below cut-off-score; above all thresholds Section IV: At least one threshold was not met Will be ranked
Wie wird man Gutachter Eintrag in die Datenbank der EU Lebenslauf Veröffentlichungen Erfahrung im Bereich nationaler und internationaler Forschungsprogramme Interessensgebiete Einladung durch die EU Daten aktuell halten! Prof. Dr. Jürgen Kreyssig j.kreyssig@ostfalia.de
. mehrmals Gutachter von Anträgen Begutachtung der geförderten Projekte (Reviewer) ca. alle 12 Monate Begutachtung: Antrag (DoW) / Projekstand Entscheidung über Fortführung und/oder Kosten Begutachtung des Evaluationsprozesses (International Observer) Ein bis drei sehr erfahrene Gutachter pro Call Weitgehende Befugnisse Vorschläge zur Verbesserung des Prozesses Prof. Dr. Jürgen Kreyssig j.kreyssig@ostfalia.de
Begutachtung der geförderten Projekte (Review) Ca. ein Review pro Jahr Ein bis fünf Gutachter und der zuständige Projekt- Officer der EU Begutachtung von u.a.: Progress towards project objectives Milestones and deliverables Assessment of the use of resources Technical, administrative and financial management of the project Impact; Use of results; Dissemination Recommendations (period; future work) Approvement of costs. Prof. Dr. Jürgen Kreyssig j.kreyssig@ostfalia.de
Vielen Dank für die Aufmerksamkeit und viel Erfolg! Email: j.kreyssig@ostfalila.de Prof. Dr. Jürgen Kreyssig j.kreyssig@ostfalia.de