Evaluators
The CoARA Steering Board is responsible for evaluating the Working Group Proposals. Each submission will be reviewed by three reviewers from. the Steering Board members, then discussed collegially by the Steering Board.
Members of the Steering Board are elected by the CoARA General Assembly. The group has been elected and exhibits diversity in terms of geographical and disciplinary background as well as career stages.
Potential Conflicts of Interest
Steering Board members are allowed to participate in Working Group submission. In cases where Steering Board members are directly involved in an application, either in their individual or institutional capacities, the Steering Board member in question will abstain from the evaluation of the affected proposals.
In addition to flagging Steering Board members’ individual or institutional involvement as cases of Conflict of Interest, each reviewer is encouraged to thoroughly and conscientiously reflect upon their own biases regarding a submission and, when in doubt, decline to review.
Evaluation and selection process
The evaluation procedure consists of two main stages: scoring and selection via panel evaluation. After the CoARA Secretariat checked the eligibility of all proposals received through the submission portal, the eligible proposals will be forwarded to the Steering Board for scoring. Assignment of proposals to reviewers is done by the Secretariat based on ruling out Conflicts of Interest, random assignment or in accordance with areas of member expertise. topical relevance, random allocation and most importantly, on ruling out Conflicts of Interest. Each submission will be reviewed by one Lead Reviewer and two Non-Lead Reviewers, in total three reviewers per submission. During the scoring phase, Steering Board members fill in their scoring sheets (see attached below, Appendix 2) and send them back to the CoARA Secretariat who complies the scoring results. These results will form the base of the deliberation discussion among the Steering Board by which the Steering Board will select the CoARA Working Group portfolio. Decision will be made in the light of the evaluation criteria (see below). In the course of the selection, the Steering Board ensures that the CoARA Working Group portfolio is fully aligned with the core mission of the Coalition. The final outcome of the selection process will reflect a collective judgement of the Steering Board. One Steering Board member will not participate in the evaluation process to act as an independent observer to ensure maximum transparency and integrity of the process.
Evaluation criteria
In line with the Rules of Procedure, the CoARA Steering Board will evaluate the proposals considering nine criteria. Criteria will bear an equal weight in the assessment and each criterion will be qualitatively assessed following the scales provided in the table below.
| Criterion | Assessment scale | Scoring guidelines |
| Alignment with the vision of CoARA; |
|
Fully aligned: the mission, impact, workplan and planned outputs of the Working Group are carefully and specifically aligned with CoARA’s vision as specified in the Agreement. Addressing key topics specified in the call are an advantage.
Partially aligned: the mission, impact, work plan and planned outputs are not specifically or not fully consistently aligned with CoARA’s vision as specified in the Agreement.
Poorly aligned: the mission, impact, workplan and planned outputs of the WG are only superficially aligned with the vision of CoARA
|
| Support the implementation of at least one of the four Agreement’s core commitments |
|
Beyond listing commitments to be supported in the relevant fields in the applications, it is to be assessed how consistently and thoroughly the listed commitments are addressed in mission, impact and work plan sections.
|
| Added value of the proposed Working Group over and above what is currently being done within the community; |
|
High score: the proposal shows awareness of the state of the art (e.g. the latest policy developments, bottlenecks, currently active initiatives and players) and clearly indicates the proposed added value of the WG on top of them (within the character limit of the section of course). As such, it has high potential to foster dialogue, build consensus, or help identify bottlenecks on the implementation level.
Medium score: the proposed work has some overlap with existing initiatives or does not present robust novelties, only moderately contributes to building consensus.
Low score: the Working Group’s contribution is likely to only minimally exceed or duplicate the work that the proposers or others have already done or are doing in other work contexts of related projects, initiatives and organisations. |
| Diverse types and sizes of organisations, and wide range of experiences level, represented. For institution communities, proposals from one unique type of organisation are acceptable; |
|
The assessment of this criterion essentially depends on the type, mission and scope of the proposed Working Group (in the case of WGs with a very specific, and focussed outputs e.g. developing infrastructural underpinnings of the reform, we see the diversity of contributing organisation types less crucial as in more horizontal topics such as developing responsible metrics). In generic terms:
Highly diverse representation: the proposal brings together different stakeholders from a great variety of organisations of different sizes and missions. However, and again depending on the scope of the WG, the engagement of RPOs and individual researchers from all career stages is an added value. . Diverse representation: one or few angles of diversity measures (stakeholder types, organisation types, sizes, mission) are limited in the proposed membership structure.
Low diversity: the Working Group exhibits a homogeneous membership structure or only very few organisation types are represented that show similarity in terms of size and mission. |
| Broad and balanced geographical participation from the CoARA member organisations; |
|
Here again, depending on the scope and mission of the Working Group:
High geographical balance: the proposal represents a broad balance of organisations within and outside of Europe and demonstrates a clear and well-integrated mechanism to ensure geographical diversity through its work plan and membership structure.
Moderate geographical balance: the proposal represents a balance of organisations at a European regional level and demonstrates a mechanism to ensure (and expand upon) geographical diversity through its work plan or membership structure.
Low geographical balance: the proposal is limited in its balance of organisations and does not provide a clear indication of mechanisms to ensure geographical inclusiveness in its work plan or membership structure.
|
| Direct involvement of researchers at all career stages (including early career researchers) |
|
Highly diverse representation: the proposal put established mechanisms in place to ensure that researchers at all career stages are welcomed and have equal chances to contribute to the Working Group.
Diverse representation: the Working Group welcomes researchers at all career stages and ensures equal chances for participation.
Low diversity: no indication of inclusiveness of researchers from all career stages. |
| Feasibility of the proposed work plan and outcomes/deliverables within the indicated timeframe; |
|
Highly feasible: the mission and proposed impact of the Working Group translates well into the work plan and the work plan seems to be feasible within the proposed time frame even if the work will be carried out largely by volunteer contributors. The work plan comes with clear and achievable milestones and deliverables.
Feasible: the mission and proposed impact of the Working Group translates moderately well intro the work plan (only slight mismatches present) and the work plan seems to be feasible within the proposed time frame even if the work will be carried out largely by volunteer contributors. Milestones in the work plan are indicated.
Feasibility uncertain: the mission and proposed impact of the Working Group is not specified enough or does not translate well into the work plan. The work plan is not specific enough, no milestones outlines and/or the proposed work is not likely to be safely carried out within the proposed time frame. |
| Expected impact, notably expected adoption and implementation scenarios; |
|
High expected impact: the proposal comes with a clear and specific vision that, regardless of the granularity, scope and type of the work, is very likely to provide solid contribution to a systemic change of research assessment. There are safeguards in place for that in the Working Group’s implementation plan.
Medium expected impact: proposal comes with a clear and specific vision that, regardless of the granularity, scope and type of the work, is very likely to meaningfully contribute to the reform of research assessment but not necessarily on a systemic level.
Low expected impact: the mission and proposed impact if the Working Group is to be refined and specified. No guarantees in place for the solid implementation of the expected impact. |
| Robust plan to engage with other CoARA member organisations interested in the work of the Working Group, and with other organisations including outside Europe. |
|
|
Scoring sheet and guide
The scoring sheet can be found in Appendix 2 of the current document. The sheet guides reviewers to assess the quality of each Working Group proposal against the evaluation criteria specified above and express their evaluation in terms of scores (1-3) as well as in short written comments. Reviewers are encouraged to use the large spectrum of ranking scale (1-3) in order to make a clear distinction of high quality, average and low-quality proposals. Reviewers are also encouraged to suggest possible mergers across Working Groups.
The role of reviewers is vital in keeping the standards of the selection high. It includes:
- Assessing the proposals based on their soundness, integrity, feasibility, possible impact and relevance to CoARA’s mission of facilitating a systemic change in research assessment practices across Europe and beyond.
- Contributing to the development of the proposals by sharing their remarks and maintaining a constructive tone.
Keeping an awareness of one’s possible cultural, disciplinary etc. biases, implicit or explicit, can help avoid having them negatively affect reviewers’ judgements.
Guidelines for the panel evaluation and selection phase
These guidelines apply specifically to the panel evaluation and selection phase of the Working Groups and National Chapters. The Review Panel will be composed by the Steering Board and it will be chaired by members from the Executive Committee. The Secretariat will assist the Review Panel and will be responsible for the smooth operation and progress of the review process, and for the management of the Conflict of Interest during the entire evaluation at the Review Panel meeting. One Executive Committee member will act as independent Observer and as such will remain abstain from the scoring and selection discussion. They will observe and monitor the panel evaluation and selection phase and to ensure a consistent, fair and impartial panel evaluation process.
Management of Conflicts of Interest
Declared Conflicts of Interest of Steering Board members will be carefully addressed with the assistance of the Secretariat. Members will be invited to leave the discussion room before a Working Group proposal with which they are in conflict of interest is subject to discussion and until the end of this discussion. They will under no circumstances take part at the discussion and at decision regarding proposals for which they are in Conflict of Interest.
Review process of the panel evaluation
Working Groups and National Chapters proposals will be discussed separately. The discussion will be based on the scoring sheets filled in by Lead and Non-Lead Reviewers. All scoring sheets’ content will be available to the Review Panel before the Review Panel meeting. Scores of each proposal will be compiled and summed up and each proposal will have a total score in a scale between 0-81 (9x3x3) in order to facilitate the discussion of the Review Panel. This compilation will enable to distinguish lowest scores (0-54) from average scores (55-72) and highest scores (73-81) and to give a starting point for discussion”
Panel discussion will be focused and always related to the evaluation criteria and to the quality of proposals. The lead reviewer will present the proposal and his/her evaluation and the non-lead reviewer will be asked to complete with his/her comments; discussion will then open to the entire Panel. Based on discussion, scores from the lead and non-lead reviewer will either move up or move down to express that the proposal is fully aligned (3), partially aligned (2), not aligned (1) with the vision of CoARA. The total score will either confirm a proposal in one of the three groups distinguished or move a proposal up to the higher scores proposals (4-5 or 6) or down to the group of proposals with lower scores (0-3 or 4-5). Comments made along this line during the Review Panel will be recorded by the Lead reviewers for the feedback to applicants in line with the final score given. Scores are an internal tool to facilitate the Review Panel discussion and will not be disclosed to applicants.
In the end of panel discussion, the Steering Board identifies the high/medium/low scoring proposals, have the presentation of each proposal, and then collectively discuss the identification of a minimum of 5 proposals with the aim of producing a balanced WG portfolio based on existing assessment criteria (e.g. balance in topics, geographical balance, institutional balance). The selection will be carried out based on how the portfolio of groups support achieving CoARA’s mission and make the biggest possible impact towards a systemic reform of research assessment from the resources available.
The cut-off line will not apply to National Chapters since there is no limit as to the number of National Chapters that can be approved.
The selection of Working Groups and National Chapters
The final discussion for selection of the Working Groups will be based on the group of highest quality proposals determined by the Review Panel. This highest quality Working Group proposals will be discussed and ranked according to criteria of evaluation and portfolio management considerations (e.g. balance in topics, geographical balance, institutional balance).
Following this final discussion for selection, the Review Panel determine the minimum of 5 selected Working Groups based on discussion and ranking. Decision will be enacted by an unanimity vote of the Review Panel/Steering Board.
National Chapters will not be discussed nor ranked at the selection phase; they will be subject to formal eligibility check and approval by unanimity vote of the Review Panel/Steering Board.
Feedback to applicants
Comments made on the evaluation sheets will be disclosed to the applicants alongside the results of the selection process along the lines of the timeline specified in Appendix 1. In addition to the evaluation results, applicants will also receive a one-page document to present to applicants how the evaluation was carried out – based on elements from the finalised Evaluation Guide.
Contact us at info@coara.eu