Erfahrungsbericht einer Evaluatorin im 7.EU Rahmenprogramm

Transkript

1 FFG Informationsveranstaltung Umwelt im 7. EU Rahmenprogramm Wien, 25.September 2008 Erfahrungsbericht einer Evaluatorin im 7.EU Rahmenprogramm DI Monika Schönerklee-Grasser, MSc Geschäftsfeld Wasser Austrian Research Centers GmbH ARC ( ) 2444 Seibersdorf, Austria Monika.schoenerklee-grasser@arcs.ac.at

2 INHALT DER PRESENTATION Allgemeines, Evaluation principles Evaluation main actors Evaluation Process (central remote) Individual reading individual assessment report Consensus finding consensus report Panel review Evaluation Criteria Scoring Einige persönliche Erfahrungen Summary und Benefits Source: EU Commission Community Research Guidance for Evaluators 2

3 Zu meiner Person: ALLGEMEINES Projektleiterin und stv. Leiterin des Geschäftsfeldes Wasser der Austrian Research Centers (seit 1995) Zuständig für integriertes Wassermanagement und internationale Projekte Projektentwicklung, Projekteinreichungen und Partner in diversen EU Projekten (5., 6. Rahmenprogramm sowie AsiaUrbs, AsiaProEco, MEDA etc.) Eintragung in EU FP ExpertInnendatenbank (auf Anregung des Wissenschaftsministeriums) Mehrmalige Teilnahme als Independent Expert an EU Projektevaluierungen (6. und 7. Rahmenprogramm) seit 2004 Cooperative Research ENVIRONMENT CRAFT Programme/Research for SMEs, Panel: Environment Zum generellen Ablauf der Evaluierung: Kontaktaufnahme durch Scientific Officer per Inquiry about availability Zusendung und Unterzeichnung eines Appointment Letters für jede Evaluierung (Declaration of No Conflict of Interest, Confidentiality) Central bzw. teilweise remote evaluation (Electronic Evaluation Tool - RIVET) Briefing Session am Beginn bzw. während jeder Evaluierung 3

4 Evaluation Process - Basic Principles EXCELLENCE TRANSPARENCY CONFIDENTIALITY Evaluation Rules Guide for applicants (annex 2) EFFICIENCY & SPEED FAIRNESS & IMPARTIALITY ETHICAL & SECURITY CONSIDERATIONS Source: EU Commission Community Research Guidance for Evaluators 4

5 Evaluation basic rules and principles Excellence Projects selected for funding must demonstrate a high quality in the context of the topics and criteria set out in the calls. Transparency Funding decisions must be based on clearly described rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals. Fairness and impartiality All proposals submitted to a call are treated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants. Confidentiality All proposals and related data, knowledge and documents communicated to the Commission are treated in confidence. Efficiency and speed Evaluation, award and grant preparation should be as rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework. Ethical and security considerations Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award. Source: EU Commission Community Research Guidance for Evaluators 5

6 Evaluation main actors Independent experts Independent, impartial and objective advice to the Commission Represent neither a employer, nor a country Commission staff Check the eligibility of the proposals Oversee work of experts Conduct briefings Moderate discussions Organise the panel and its work Ensure coherence and consistency Observers From the point of view of evaluation process working and execution Provides assurance that the process is fair and transparent Their reports are made available to the Programme Committee Source: EU Commission Community Research Guidance for Evaluators 6

7 Overview of the Evaluation Process May be remote May be remote Submission Individual reading Consensus Panel Finalisation Full Proposal Evaluators Evaluators Evaluators Final ranking list Proposal forms Eligibility Criteria Criteria Criteria Proposals in suggested priority order Rejection list COMMISSION COMMISSION Role of experts Source: EU Commission Community Research Guidance for Evaluators 7

8 The evaluation process remote Proposal X copy 1 Proposal X copy 2 IER expert 1 IER expert 2 Consensus meeting Initial draft CR prepared remotely, then finalised centrally CR 3 experts Proposal X copy 3 IER expert 3 Note: There may be more than 3 evaluators IER=Individual assessment report CR=Consensus Report Individual reading: Peer-review by 3 or more evaluators (expert panel) Evaluate the proposal individually without discussing with the other evaluators Check whether the proposal is in scope Complete an Individual Evaluation Report (IER) form giving scores and comments on all criteria Consensus: Built on the basis of the individual evaluations The aim is agreement on scores and comments Usually involves a discussion Outlying opinions need to be explored Not just a simple averaging exercise It is quite normal for individual views to change Moderated by a Commission staff-member Helps the group reach a conclusion Provides information if necessary Does not contribute opinions Source: EU Commission Community Research Guidance for Evaluators 8

9 The evaluation criteria Criteria adapted to each funding scheme and each thematic area (specified in the work programme) Three main criteria - applicable to ALL funding schemes (guiding questions for each criteria): 1. S/T quality (in relation to the topics addressed by the call) Sound concept, and quality of objectives Innovative character in relation to the state-of-the art Contribution to advancement of knowledge / technological progress Quality and effectiveness of S/T methodology and associated work. 2. Implementation Appropriateness of the management structures and procedures Quality and relevant experience of the individual participants Quality of the consortium as a whole (including complementarity and balance) Appropriate allocation and justification of the resources to be committed (budget, staff, equipment) 3. Impact Contribution, at the European [and/or international level], to the expected impacts listed in the work programme under the relevant topic/activity Appropriateness of measures envisaged for the dissemination and/or exploitation of project results, and management of intellectual property. 9

10 Specific evaluation criteria Research for SMEs Programme S/T QUALITY Offering technological solutions to SMEs Not a requirement to develop cutting edge technology at world class level Adoption of existing technologies to new applications in a concrete SME business case is also worth considering Are the objectives SMART (quantified)? IMPLEMENTATION Are the SME Associations participants the real driving forces for the project with each having an active role in the consortium? Does the proposal demonstrate that the SME Associations participants posses their own resources necessary for the total investment in the project? IMPACT Proposals must demonstrate: Clear exploitation potential and economic benefits for the SMEs members of the associations involved Improvement of industrial competitiveness IPR issues must be considered at the proposal stage clear description of how the participants will organise IPR ownership and user rights among themselves (licenses, royalties, ownership of the foreground, exploitation strategy, benefits etc.) 10

11 The evaluation process Scoring Each criterion is scored 0-5 Marks can go from 0 5 in steps of 0.5, i.e half-marks are allowed Experts are encouraged to use the whole range Scores must pass thresholds if a proposal is to be considered for funding 0: The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information 1: Very poor. The criterion is addressed in a cursory and unsatisfactory manner. 2: Poor. There are serious inherent weaknesses in relation to the criterion in question. 3: Fair. While the proposal broadly addresses the criterion, there are significant weaknesses that would need correcting. 4: Good. The proposal addresses the criterion well, although certain improvements are possible. 5: Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor. Thresholds apply to individual criteria Default threshold is 3 (for SME projects threshold for Impact is 4) and to the total score Higher than the sum of the individual thresholds Default threshold is 10 (for SME projects overall threshold is 11) 11

12 The evaluation process Individual evaluation report 12

13 The Evaluation process Individual assessment by at least 3 experts Consensus Quality Check of CRs Final Ranking List Source: EU Commission Community Research Guidance for Evaluators 13

14 Consensus finding and Consensus report Once the 3 (or more) evaluators assigned to each proposal have completed and FINALISED their IERs, the evaluation progresses to a consensus report (CR), representing their common views. CR is built on the basis of the individual evaluations reports (IER) The aim is: Agreement on scores and comments A clear assessment of the proposal, with justifications Clear feedback on weaknesses & strengths Comments in line with the consensus scores Rapporteur Evaluator A rapporteur is appointed, who is responsible for drafting the consensus report (CR) - mostly remote The final version of Consensus Reports are finalised in Brussels during the Central Evaluation. Independent Observer Moderator EC Staff Source: EU Commission Community Research Guidance for Evaluators 14

15 The Panel Review Per sub-activity a limited number of experts (Panel) are invited to examine and compare the CRs of proposals passing all thresholds Key function is to ensure consistency The Panel will recommend a priority order including final marks and comments for each proposal Evaluation Summary Reports (ESR) Any new scores (if necessary) should be carefully justified Split proposals with identical consensus scores Prioritise certain criteria? Consider overall balance? Budget? Clear guidance for contract negotiation 15

16 Meine persönlichen Erfahrungen Evaluierung von 5-8 Proposals zu einem Topic des Work Programms Vergleich unterschiedlicher methodischer Ansätze für das gleiche Topic Innovationsgrad Projektstruktur und Zieldefinition Lesbarkeit des Proposals Projektumsetzung Vorteile durch remote evaluation (reading): Mehr Zeit zum Lesen der Proposals, freie Zeiteinteilung (kein Zeitdruck) Möglichkeiten zum Recherchieren (Literatur, Internet etc.) Consensus Finding (tw. remote): meist: gute Übereinstimmung der Bewertung (Scores) mit den anderen EvaluatorInnen einfacher Konsens manchmal: eine outlying Bewertung eines Evaluators Consensus Findung relativ problematisch, lange Diskussionen, kann Gesamt-Score des Proposals deutlich senken 16

17 Meine persönlichen Erfahrungen Was macht einen guten Projektantrag aus? Keine Romane, kurze und klare (überzeugende) Formulierung der wesentlichen Elemente Klare und gut definierte Projektziele Präzise Darstellung der state-of-the-art und Innovation Übersichtliche Beschreibung der Methodik und des Work Plans Kohärenz des Projektantrages Workplan Aktivitäten Ressourcen Konsortium - Partnership Expertise der einzelnen Partner Komplementarität Klare Definition der Roles and Responsiblities (inkl. Schnittstellen) der einzelnen Partner Relevanz für das Workprogramme (Call Topic) IMPACT!!!!!!!!!! 17

18 Summary und benefits Fachlich interessante Tätigkeit Unterschiedliche wissenschaftliche Ansätze und Methoden Diskussion und Konsensfindung mit den anderen EvaluatorInnen Nutzen für eigene Projekteinreichungen Kenntnis der Bewertungskriterien und des prozederes Außensicht wie gestalte ich eine Einreichung, die fundiert und gut lesbar ist? Vergleich zwischen erfolgreichen und nicht erfolgreichen Anträgen Erweiterung des persönlichen Netzwerkes Potentielle Partner für zukünftige Forschungsprojekte Informations- und Erfahrungsaustausch VIEL ERFOLG BEI IHRER PROJEKTEINREICHUNG! 18

19 DANKE FÜR IHRE AUFMERKSAMKEIT! FRAGEN???????????? 19