Erfahrungsbericht einer Evaluatorin im 7.EU Rahmenprogramm

Ähnliche Dokumente
Redress Procedure in Horizont 2020

Analyse typischer Fehler bei der Antragstellung Bonn, Brigitte Ulamec, DLR, NKS Raumfahrt

The DFG Review Process

Darstellung und Anwendung der Assessmentergebnisse

Scenario Building Workshop - Interplay of problem framings

Antragsverfahren. Katerina Kotzia, SusAn Call Secretariat

How to develop and improve the functioning of the audit committee The Auditor s View

Rollen im Participant Portal

HIR Method & Tools for Fit Gap analysis

Mock Exam Behavioral Finance

Evaluierung in der Praxis: Analyse der Evaluation Summary Reports und Services der FFG

Making quality visible. National Quality Certificate for Old Age and Nursing Homes in Austria (NQC)

prorm Budget Planning promx GmbH Nordring Nuremberg

Level 2 German, 2015

Corporate Digital Learning, How to Get It Right. Learning Café

Seminar: Advanced Internet Technology. Lehrveranstaltungsnummer: 12ws Lehrveranstaltungstyp: Seminar engl. Erfasste Fragebögen: 9

Introduction to the diploma and master seminar in FSS Prof. Dr. Armin Heinzl. Sven Scheibmayr

Level 1 German, 2014

Level 1 German, 2016

Level 2 German, 2013

ISO Reference Model

New Call for IMI Proposals: Topics, Rules and Tips for Submission

Registration of residence at Citizens Office (Bürgerbüro)

Big Data Analytics. Fifth Munich Data Protection Day, March 23, Dr. Stefan Krätschmer, Data Privacy Officer, Europe, IBM

VGM. VGM information. HAMBURG SÜD VGM WEB PORTAL USER GUIDE June 2016

Lehrstuhl für Allgemeine BWL Strategisches und Internationales Management Prof. Dr. Mike Geppert Carl-Zeiß-Str Jena

Cooperation Project Sao Paulo - Bavaria. Licensing of Waste to Energy Plants (WEP/URE)

Hinweise aus der Sicht eines Gutachters Arne Jungstand Dornier Consulting GmbH

Level 1 German, 2012

EVANGELISCHES GESANGBUCH: AUSGABE FUR DIE EVANGELISCH-LUTHERISCHE LANDESKIRCHE SACHSEN. BLAU (GERMAN EDITION) FROM EVANGELISCHE VERLAGSAN

Word-CRM-Upload-Button. User manual

Measuring and improving user satisfaction Insights from a global corporation

How does the Institute for quality and efficiency in health care work?

Erfolgsfaktoren für Ihren Projektantrag

Support Technologies based on Bi-Modal Network Analysis. H. Ulrich Hoppe. Virtuelles Arbeiten und Lernen in projektartigen Netzwerken

Call Centers and Low Wage Employment in International Comparison

Security of Pensions

SAMPLE EXAMINATION BOOKLET

Customer-specific software for autonomous driving and driver assistance (ADAS)

VGM. VGM information. HAMBURG SÜD VGM WEB PORTAL - USER GUIDE June 2016

Wer bin ich - und wenn ja wie viele?: Eine philosophische Reise. Click here if your download doesn"t start automatically

Women Entrepreneurship in Germany and Access to Capital

Possible Solutions for Development of Multilevel Pension System in the Republic of Azerbaijan

GIPS 2010 Gesamtüberblick. Dr. Stefan J. Illmer Credit Suisse. Seminar der SBVg "GIPS Aperitif" 15. April 2010 Referat von Stefan Illmer

ISO Reference Model

INTERREG IIIa Project R&D - Ready for Research and Development Project results and ongoing activities

Less is more? Checks and balances in sport organisations

Exercise (Part XI) Anastasia Mochalova, Lehrstuhl für ABWL und Wirtschaftsinformatik, Kath. Universität Eichstätt-Ingolstadt 1

TU WIEN RESEARCH & TRANSFER SUPPORT

Der wissenschaftliche Teil des Projektantrags

STARCOS 3.4 Health AHC

Technology Transfer in Steep Terrain Timber Harvesting (TechnoSteep)

Repositioning University Collections as Scientific Infrastructures.

POST MARKET CLINICAL FOLLOW UP

Level 2 German, 2016

DIBELS TM. German Translations of Administration Directions

Einsatz einer Dokumentenverwaltungslösung zur Optimierung der unternehmensübergreifenden Kommunikation

Level 1 German, 2013

U N D E R S T A N D I N G P E N S I O N A N D E M P L O Y E E B E N E F I T S I N T R A N S A C T I O N S

GAUSS towards a common certification process for GNSS applications using the European Satellite System Galileo

Labour law and Consumer protection principles usage in non-state pension system

WP2. Communication and Dissemination. Wirtschafts- und Wissenschaftsförderung im Freistaat Thüringen

Non users after Cochlear Implantation in Single Sided Deafness

LOC Pharma. Anlage. Lieferantenfragebogen Supplier Questionnaire. 9. Is the warehouse temperature controlled or air-conditioned?

Prof. Dr. Bryan T. Adey

From a Qualification Project to the Foundation of a Logistics Network Thuringia. Logistik Netzwerk Thüringen e.v.

TMF projects on IT infrastructure for clinical research

Notice: All mentioned inventors have to sign the Report of Invention (see page 3)!!!

Social Sciences and Humanities role in Social Innovation Some key observations from the experience of TU Dortmund University sfs

MASTER THESIS GRADING PROTOCOL FH BFI WIEN

ELBA2 ILIAS TOOLS AS SINGLE APPLICATIONS

Exercise (Part II) Anastasia Mochalova, Lehrstuhl für ABWL und Wirtschaftsinformatik, Kath. Universität Eichstätt-Ingolstadt 1

Algorithms for graph visualization

Wie man heute die Liebe fürs Leben findet

Ein Stern in dunkler Nacht Die schoensten Weihnachtsgeschichten. Click here if your download doesn"t start automatically

1. General information Login Home Current applications... 3

Beschwerdemanagement / Complaint Management

INFORMATIONEN ZUR PARTNEREINRICHTUNG UND MANDAT

Cloud Architektur Workshop

Themen für Seminararbeiten WS 15/16

WP 4 Developing S E C Strategy

COMET K2-Zentren 3. Call Förderungsansuchen/ Inhaltliche Projektbeschreibung. Informationsveranstaltung zur 3. K2-Ausschreibung 18.

Criteria to Assess the Quality of Virtual Patients (VP)

Challenges for the future between extern and intern evaluation

STRATEGISCHES BETEILIGUNGSCONTROLLING BEI KOMMUNALEN UNTERNEHMEN DER FFENTLICHE ZWECK ALS RICHTSCHNUR FR EIN ZIELGERICHTETE

Temporary Backup Schemes SNSF Consolidator Grants

Ich habe eine Nachricht für Sie

Developing the business case for investing in corporate health and workplace partnership indicators and instruments Input

Elektronische Identifikation und Vertrauensdienste für Europa

Workshop on Copernicus and the CAP. A technology vision for IACS

Wirtschaftsstandort. Augsburg. Dr. Marcus Cieleback, Group Head of Research, PATRIZIA Immobilien AG. 27. Juni PATRIZIA Immobilien AG

Auswertungsbericht Lehrveranstaltungsevaluation an die Lehrenden

Im Fluss der Zeit: Gedanken beim Älterwerden (HERDER spektrum) (German Edition)

Level 2 German, 2011

Proseminar - Organisation und Personal Seminar Organisational Theory and Human Resource Management

IF: Einreichung und Evaluierung

Emotion Recognition of Call Center Conversations Robert Bosch Engineering and Business Solutions Private Limited

Prof. Dr. Margit Scholl, Mr. RD Guldner Mr. Coskun, Mr. Yigitbas. Mr. Niemczik, Mr. Koppatz (SuDiLe GbR)

Inequality Utilitarian and Capabilities Perspectives (and what they may imply for public health)

Level 1 German, 2011

Transkript:

FFG Informationsveranstaltung Umwelt im 7. EU Rahmenprogramm Wien, 25.September 2008 Erfahrungsbericht einer Evaluatorin im 7.EU Rahmenprogramm DI Monika Schönerklee-Grasser, MSc Geschäftsfeld Wasser Austrian Research Centers GmbH ARC (www.arcs.ac.at ) 2444 Seibersdorf, Austria Monika.schoenerklee-grasser@arcs.ac.at

INHALT DER PRESENTATION Allgemeines, Evaluation principles Evaluation main actors Evaluation Process (central remote) Individual reading individual assessment report Consensus finding consensus report Panel review Evaluation Criteria Scoring Einige persönliche Erfahrungen Summary und Benefits Source: EU Commission Community Research Guidance for Evaluators 2

Zu meiner Person: ALLGEMEINES Projektleiterin und stv. Leiterin des Geschäftsfeldes Wasser der Austrian Research Centers (seit 1995) Zuständig für integriertes Wassermanagement und internationale Projekte Projektentwicklung, Projekteinreichungen und Partner in diversen EU Projekten (5., 6. Rahmenprogramm sowie AsiaUrbs, AsiaProEco, MEDA etc.) Eintragung in EU FP ExpertInnendatenbank (auf Anregung des Wissenschaftsministeriums) - 2003 Mehrmalige Teilnahme als Independent Expert an EU Projektevaluierungen (6. und 7. Rahmenprogramm) seit 2004 Cooperative Research ENVIRONMENT CRAFT Programme/Research for SMEs, Panel: Environment Zum generellen Ablauf der Evaluierung: Kontaktaufnahme durch Scientific Officer per email Inquiry about availability Zusendung und Unterzeichnung eines Appointment Letters für jede Evaluierung (Declaration of No Conflict of Interest, Confidentiality) Central bzw. teilweise remote evaluation (Electronic Evaluation Tool - RIVET) Briefing Session am Beginn bzw. während jeder Evaluierung 3

Evaluation Process - Basic Principles EXCELLENCE TRANSPARENCY CONFIDENTIALITY Evaluation Rules Guide for applicants (annex 2) EFFICIENCY & SPEED FAIRNESS & IMPARTIALITY ETHICAL & SECURITY CONSIDERATIONS Source: EU Commission Community Research Guidance for Evaluators 4

Evaluation basic rules and principles Excellence Projects selected for funding must demonstrate a high quality in the context of the topics and criteria set out in the calls. Transparency Funding decisions must be based on clearly described rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals. Fairness and impartiality All proposals submitted to a call are treated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants. Confidentiality All proposals and related data, knowledge and documents communicated to the Commission are treated in confidence. Efficiency and speed Evaluation, award and grant preparation should be as rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework. Ethical and security considerations Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award. Source: EU Commission Community Research Guidance for Evaluators 5

Evaluation main actors Independent experts Independent, impartial and objective advice to the Commission Represent neither a employer, nor a country Commission staff Check the eligibility of the proposals Oversee work of experts Conduct briefings Moderate discussions Organise the panel and its work Ensure coherence and consistency Observers From the point of view of evaluation process working and execution Provides assurance that the process is fair and transparent Their reports are made available to the Programme Committee Source: EU Commission Community Research Guidance for Evaluators 6

Overview of the Evaluation Process May be remote May be remote Submission Individual reading Consensus Panel Finalisation Full Proposal Evaluators Evaluators Evaluators Final ranking list Proposal forms Eligibility Criteria Criteria Criteria Proposals in suggested priority order Rejection list COMMISSION COMMISSION Role of experts Source: EU Commission Community Research Guidance for Evaluators 7

The evaluation process remote Proposal X copy 1 Proposal X copy 2 IER expert 1 IER expert 2 Consensus meeting Initial draft CR prepared remotely, then finalised centrally CR 3 experts Proposal X copy 3 IER expert 3 Note: There may be more than 3 evaluators IER=Individual assessment report CR=Consensus Report Individual reading: Peer-review by 3 or more evaluators (expert panel) Evaluate the proposal individually without discussing with the other evaluators Check whether the proposal is in scope Complete an Individual Evaluation Report (IER) form giving scores and comments on all criteria Consensus: Built on the basis of the individual evaluations The aim is agreement on scores and comments Usually involves a discussion Outlying opinions need to be explored Not just a simple averaging exercise It is quite normal for individual views to change Moderated by a Commission staff-member Helps the group reach a conclusion Provides information if necessary Does not contribute opinions Source: EU Commission Community Research Guidance for Evaluators 8

The evaluation criteria Criteria adapted to each funding scheme and each thematic area (specified in the work programme) Three main criteria - applicable to ALL funding schemes (guiding questions for each criteria): 1. S/T quality (in relation to the topics addressed by the call) Sound concept, and quality of objectives Innovative character in relation to the state-of-the art Contribution to advancement of knowledge / technological progress Quality and effectiveness of S/T methodology and associated work. 2. Implementation Appropriateness of the management structures and procedures Quality and relevant experience of the individual participants Quality of the consortium as a whole (including complementarity and balance) Appropriate allocation and justification of the resources to be committed (budget, staff, equipment) 3. Impact Contribution, at the European [and/or international level], to the expected impacts listed in the work programme under the relevant topic/activity Appropriateness of measures envisaged for the dissemination and/or exploitation of project results, and management of intellectual property. 9

Specific evaluation criteria Research for SMEs Programme S/T QUALITY Offering technological solutions to SMEs Not a requirement to develop cutting edge technology at world class level Adoption of existing technologies to new applications in a concrete SME business case is also worth considering Are the objectives SMART (quantified)? IMPLEMENTATION Are the SME Associations participants the real driving forces for the project with each having an active role in the consortium? Does the proposal demonstrate that the SME Associations participants posses their own resources necessary for the total investment in the project? IMPACT Proposals must demonstrate: Clear exploitation potential and economic benefits for the SMEs members of the associations involved Improvement of industrial competitiveness IPR issues must be considered at the proposal stage clear description of how the participants will organise IPR ownership and user rights among themselves (licenses, royalties, ownership of the foreground, exploitation strategy, benefits etc.) 10

The evaluation process Scoring Each criterion is scored 0-5 Marks can go from 0 5 in steps of 0.5, i.e half-marks are allowed Experts are encouraged to use the whole range Scores must pass thresholds if a proposal is to be considered for funding 0: The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information 1: Very poor. The criterion is addressed in a cursory and unsatisfactory manner. 2: Poor. There are serious inherent weaknesses in relation to the criterion in question. 3: Fair. While the proposal broadly addresses the criterion, there are significant weaknesses that would need correcting. 4: Good. The proposal addresses the criterion well, although certain improvements are possible. 5: Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor. Thresholds apply to individual criteria Default threshold is 3 (for SME projects threshold for Impact is 4) and to the total score Higher than the sum of the individual thresholds Default threshold is 10 (for SME projects overall threshold is 11) 11

The evaluation process Individual evaluation report 12

The Evaluation process Individual assessment by at least 3 experts Consensus Quality Check of CRs Final Ranking List Source: EU Commission Community Research Guidance for Evaluators 13

Consensus finding and Consensus report Once the 3 (or more) evaluators assigned to each proposal have completed and FINALISED their IERs, the evaluation progresses to a consensus report (CR), representing their common views. CR is built on the basis of the individual evaluations reports (IER) The aim is: Agreement on scores and comments A clear assessment of the proposal, with justifications Clear feedback on weaknesses & strengths Comments in line with the consensus scores Rapporteur Evaluator A rapporteur is appointed, who is responsible for drafting the consensus report (CR) - mostly remote The final version of Consensus Reports are finalised in Brussels during the Central Evaluation. Independent Observer Moderator EC Staff Source: EU Commission Community Research Guidance for Evaluators 14

The Panel Review Per sub-activity a limited number of experts (Panel) are invited to examine and compare the CRs of proposals passing all thresholds Key function is to ensure consistency The Panel will recommend a priority order including final marks and comments for each proposal Evaluation Summary Reports (ESR) Any new scores (if necessary) should be carefully justified Split proposals with identical consensus scores Prioritise certain criteria? Consider overall balance? Budget? Clear guidance for contract negotiation 15

Meine persönlichen Erfahrungen Evaluierung von 5-8 Proposals zu einem Topic des Work Programms Vergleich unterschiedlicher methodischer Ansätze für das gleiche Topic Innovationsgrad Projektstruktur und Zieldefinition Lesbarkeit des Proposals Projektumsetzung Vorteile durch remote evaluation (reading): Mehr Zeit zum Lesen der Proposals, freie Zeiteinteilung (kein Zeitdruck) Möglichkeiten zum Recherchieren (Literatur, Internet etc.) Consensus Finding (tw. remote): meist: gute Übereinstimmung der Bewertung (Scores) mit den anderen EvaluatorInnen einfacher Konsens manchmal: eine outlying Bewertung eines Evaluators Consensus Findung relativ problematisch, lange Diskussionen, kann Gesamt-Score des Proposals deutlich senken 16

Meine persönlichen Erfahrungen Was macht einen guten Projektantrag aus? Keine Romane, kurze und klare (überzeugende) Formulierung der wesentlichen Elemente Klare und gut definierte Projektziele Präzise Darstellung der state-of-the-art und Innovation Übersichtliche Beschreibung der Methodik und des Work Plans Kohärenz des Projektantrages Workplan Aktivitäten Ressourcen Konsortium - Partnership Expertise der einzelnen Partner Komplementarität Klare Definition der Roles and Responsiblities (inkl. Schnittstellen) der einzelnen Partner Relevanz für das Workprogramme (Call Topic) IMPACT!!!!!!!!!! 17

Summary und benefits Fachlich interessante Tätigkeit Unterschiedliche wissenschaftliche Ansätze und Methoden Diskussion und Konsensfindung mit den anderen EvaluatorInnen Nutzen für eigene Projekteinreichungen Kenntnis der Bewertungskriterien und des prozederes Außensicht wie gestalte ich eine Einreichung, die fundiert und gut lesbar ist? Vergleich zwischen erfolgreichen und nicht erfolgreichen Anträgen Erweiterung des persönlichen Netzwerkes Potentielle Partner für zukünftige Forschungsprojekte Informations- und Erfahrungsaustausch VIEL ERFOLG BEI IHRER PROJEKTEINREICHUNG! 18

DANKE FÜR IHRE AUFMERKSAMKEIT! FRAGEN???????????? 19