FP7 evaluation & selection: the point of view of an evaluator
- 2. About myself • Associate professor in ULB, Faculty of Sciences, Computer Science Department. • Head of the Computer Science Department. • Head of the Machine Learning Group • Research interests: machine learning, data mining, adap*ve systems, modelling and simula*on, bioinforma*cs
- 3. Outline • My exper*se as EU evaluator. • The mission of an evaluator. • The FP7 evalua*on process in short. • Personal considera*ons. • Some advices.
- 4. About my exper*se as EU reviewer • 2002: reviewer of IST project on Electronic commerce • 2003: evaluator of FP6 projects proposals (Specific Targeted Research Projects, STREP) for the strategic objec*ve « Mul*modal interfaces » (1st call) • 2005: evaluator of FP6 projects proposals (STREP) for the strategic objec*ve « Mul*modal interfaces » (5th call) • 2005: evaluator at Integrated Projects IP Hearings • 2007: evaluator of FP7 projects proposals (STREP & CA) for the strategic objec*ve «Cogni*ve Systems, Interac*on and Robo*cs in Framework »
- 5. How to become an EU expert? • The Commission appoints independent experts to assist in the evalua*on of IST proposals. • They are individuals from the field of science, industry and/or with experience in the field of innova*on with the highest level of knowledge and who are interna*onally recognized authori*es in the relevant specialist area. • In appoin*ng independent experts the Commission also takes into account their abili*es to appreciate the challenges and industrial and/or societal dimension of the proposed work. • Experts must also have the appropriate language skills and come from countries other than the Member/Associated States. • List of EU experts are open for applica*on.
- 6. Profiles involved in the evalua*on process • Project officers. • Interna*onal experts. • External observers.
- 7. Conflict of interests and confiden*ality • The Commission takes all the reasonable steps to ensure that the expert is not faced with a conflict of interest in rela*on to the proposals on which he/she is required to give an opinion. • To this end, experts sign a declara*on that no such conflict exists. • Experts are obliged to maintain the confiden*ality of the informa*on contained within the proposals they evaluate and to act with strict impar*ality. • The evaluator works as an independent person. He/she is deemed to work in a personal capacity and, in performing the work, does not represent any organisa*on.
- 8. Code of conduct of evaluators • Evaluators may not discuss any proposal with others, including other evaluators or Commission officials not directly involved in the evalua*on of the proposal, except during the formal discussions at the scheduled mee*ngs. • Evaluators may not communicate with proposers, except in case of panel hearings. • Evaluators are not allowed to take outside the evalua*on building any parts of proposals copies or notes, either in paper or electronic forms. • Evaluator may be given the possibility of seeking further informa*on (e.g. through internet)
- 10. Guidelines on proposal evaluation and selection procedures Eligibility Individual Evaluation Consensus Thresholds Hearings Panel Ranking by Commission Ethical Issues Negotiation Negotiation Result (OPTIONAL) (OPTIONAL) Proposal Commission Funding Decision and/or RejectionDecision Consultation of Programme Committee, if required Commission Rejection Decision
- 11. Before the evalua*on • Only proposals that meet all the eligibility criteria (deadline, minimum number of par*cipants, presence of all the requested administra*ve forms) are retained for evalua*on. • The decision to exclude a proposal for failing one or more eligibility criteria is taken by the Commission. • Sanity check, no assessment of the S&T contribu*on.
- 12. The evalua*on session 1. Briefing of the independent experts (presenta*on of the process, expected schedule, WP objec*ves, recall of code of conduct). 2. Individual evalua*on of adributed proposals. 3. Consensus and prepara*on of a consensus report. 4. Panel evalua*on. The whole process takes approximately one week and takes place in EU premises.
- 13. Individual evalua*on • Each expert is assigned a set of proposals (about 4‐5) and each proposal is assigned a set of experts. • Each expert works independently and gives marks and comments for each block of criteria • Experts are required to provide comments to accompany each of their marks in a form suitable for providing feedback to the proposers. The comments must be consistent with the marks • An individual evalua*on form is filled, signed by the expert and delivered to the Commission officer. This form may not subsequently be charged and will be archived.
- 15. IP (Integrated projects) criteria 1. Relevance (thr 3 out of 5): the extent to which the proposal addresses the objec*ves of the work programme. 2. Poten*al impact (thr 3 out of 5): the extent to which – The proposal is suitable ambi*ous in terms of its strategic impact on reinforcing compe**veness or on solving societal problems – Exploita*on and dissemina*on plans are adequate to ensure op*mal use of the results – The proposal is a clear added value to society.
- 16. 3. Scien*fic and technology excellence (thr 4 out of 5): – The project has clearly defined objec*ves. – The objec*ves represent clear progress beyond the current S&T state‐of‐the‐art – The proposed approach is likely to enable the project to achieve the objec*ves 4. Quality of the consor*um (thr 3 out of 5) – The par*cipants cons*tute a consor*um of high quality. – The par*cipants are well‐suited and commided to the tasks. – There is a good complementarity. – The involvement of SME has been adequately addressed.
- 17. 5. Quality of the management (thr 3 out of 5) – Organisa*onal structure is well matched to the complexity of the project. – Project management is of high quality. – Plan for the management of knowledge, intellectual property and innova*on ac*vi*es. 6. Mobilisa*on of resources (thr 3 out of 5) – Cri*cal mass of resources (personnel, equipment, finance) – Resources are convincingly integrated – Overall financial plan is adequate
- 18. Proposal marking • 0: the proposal fails to address the issue under examina2on • 1: poor • 2: fair • 3: good • 4: very good • 5: excellent When appropriate half marks may be given
- 19. Consensus mee*ng • Once all the evaluators to whom a proposal has been assigned have completed their individual assessment, a consensus discussion is convened. • The experts adempt to agree on a consensus mark for each of the blocks of criteria. They jus*fy their marks with comments suitable for feedback. • A Commission official acts as moderator and the group will design a « rapporteur » for the proposal. The proposal rapporteur is responsible for presen*ng the proposal at the panel mee*ng. • If during the consensus discussion it is found to be impossible to bring all the experts to a common point of view the Commission officials may ask up to 3 addi*onal independent experts (it happened only once). • The outcome is a consensus report (with scores) signed by all the independent experts (containing, if needed, any dissen*ng views).
- 20. Panel evalua*on • This is the final mee*ng where all the proposals are finally ranked. • Each proposal arrives at this evalua*on with a global mark and a rapporteur. • The panel is expect to examine and compare the proposal consensus reports in order to check on the consistency of the marks applied during the previous process. • Proposals receiving the same marks are re‐examined with a view to recommending a priority order (if possible and relevant) • Recommenda*on to any possible grouping or combina*on of proposals into larger projects may be issued.
- 21. Outcome of the panel mee*ng • The outcome of the panel mee*ng is – An evalua*on summary report (ESR) for each proposal. It will be sent back to the coordinator of each proposal as feedback. It contains comments and marks and a final overall score. The comments must give sufficient and clear reasons for the scores and in case of proposal with high scores, any recommenda*on for modifica*ons. – A list of proposals passing thresholds along with a final mark for each proposal passing the thresholds and the panel recommenda*ons for priority order. – The iden*fica*on of proposals requiring a special aden*on due to either the importance of any ethical issues raised or the inadequacy of the way ethical issues are addressed. – Recommenda*ons for hearings (if needed)
- 24. Recommenda*ons • Work‐programme is the “bible”: ask yourself whether your proposal fits really in that vision. • Ask yourself: “Is my project a clearly recognizable advance with respect to the state‐of‐the‐art?” • Try to have the largest number of proof‐readers (colleagues, external experts) before submijng • Plenty of very good proposals did not pass the selec*on (ogen due to weaknesses in a single criterion), then… – ordinary (“let’s try”) proposals have prac*cally no chance of being funded • Submit only proposals of very high quality. Do not waste your *me in just trying.. • Invest your *me in wri*ng/submijng proposals only if the added value goes beyond the expected funding • Ask yourself the ques*on: “Would I do that anyway, even if I knew that I will be rejected?” • If the answer is yes, it is worthy to setup a project.
- 25. Conclusion • The selec*on procedure is fair yet very difficult to pass. • What you could learn in brainstorming, conceiving, wri*ng, taking part to project setup, and receiving feedbacks, should be worthier than the expected EU funding • “Do it for love of science/technology and not for money…” • Have fun! • Good luck !!