The Commitments
The Agreement, based on 10 commitments, establishes a common direction for research assessment reform, while respecting organisations’ autonomy.
The Commitments
- valuable contributions that researchers make to science and for the benefit of society, including diverse outputs beyond journal publications and irrespective of the language in which they are communicated;
- practices that contribute to robustness, openness, transparency, and the inclusiveness of research and the research process including: peer review, teamwork and collaboration;
- activities including teaching, leadership, supervision, training and mentoring.
- relying exclusively on author-based metrics (e.g. counting papers, patents, citations, grants, etc.) to assess quality and/or impact;
- assessing outputs based on metrics relating to publication venue, format or language;
- relying on any other metrics that do not properly capture quality and/or impact.
- implement changes in research assessment, including planning and progress monitoring;
- raise awareness among all actors;
- educate, train and support researchers and any other staff involved in assessment, including peer-reviewers and assessors; and
- support the necessary infrastructure such as tools and services for the transparent collection and processing of data on research assessment practices.
Purpose: This commitment will broaden recognition of the diverse practices, activities and careers in research, considering the specific nature of research disciplines and other research endeavours.
Scope: Changes in assessment practices should enable recognition of the broad diversity of:
It is also important that assessment facilitates the recognition and valorisation of diverse roles and careers in research, including: data steward, software engineer and data scientist roles, technical roles, public outreach, science diplomacy, science advice and science communicator roles to name a few. It is recognised that current practice is often too narrow and limiting, so the goal cannot be to replace the narrow criteria we wish to move away from with different but equally narrow criteria. Instead, the aim is to allow organisations to broaden the spectrum
of what they value in research, while acknowledging that this may vary across disciplines and that each individual researcher should not be expected to contribute to all activities at once.
Purpose: This commitment will enable the move towards research assessment criteria that focus primarily on quality, while recognising that responsible use of quantitative indicators can support assessment where meaningful and relevant, which is context dependent.
Scope: Research assessment should rely primarily on qualitative assessment for which peer review is central, supported by responsibly used quantitative indicators where appropriate. Peer review is the most robust method known for assessing quality and has the advantage that it is in the hands of the research community. It is important that peer review processes are designed to meet the fundamental principles of rigor and transparency: expert assessment, transparency, impartiality, appropriateness, confidentiality, integrity and ethical considerations, gender, equality and diversity. To address the biases and imperfections to which any method is prone, the research community re-assesses and improves peer review
practices regularly. Revised, or potentially new, criteria, tools and processes appropriate for assessing quality could be explored alongside peer review. Moving towards assessment practices that rely more heavily on qualitative methods may require additional efforts from researchers. Researchers should be recognised for these efforts and their contributions to reviewing peers’ work should be valued as part of their career progression.
Purpose: This commitment will reduce the dominance of a narrow set of quantitative journal- and publication-based metrics.
Scope: Inappropriate uses of journal- and publication-based metrics in research assessment should be abandoned. In particular, this means moving away from using metrics like the Journal Impact Factor (JIF), Article Influence Score (AIS) and h-index as proxies for quality and impact. ‘Inappropriate uses’ include:
Purpose: This commitment will help avoid that metrics used by international rankings, which are inappropriate for assessing researchers, trickle down to research and researcher assessment. It will help the research community and research organisations regain the autonomy to shape assessment practices, rather than having to abide by criteria and methodologies set by external commercial companies. This could include retaining control over ranking methodologies and data.
Scope: Recognising that the international rankings most often referred to by research organisations are currently not ‘fair and responsible’, the criteria these rankings use should not trickle down to the evaluation of individual researchers, research teams and research units. Research organisations should also be mindful that public communication (e.g. the active advertising of an institution’s rank) can contribute to the perception that research quality conflates with ranking positions.
Where ranking approaches are deemed unavoidable, as may be the case in forms of evaluation beyond the scope of this Agreement such as benchmarking and performance reviews of countries or institutions, the methodological limitations of such approaches should be acknowledged, and institutions should avoid trickle-down effects on research and researcher assessment.
Purpose: This commitment will ensure that organisations allocate the necessary resources, whether in the form of budget or staff capacity, to improve research assessment practices within their agreed timeframe.
Scope: Resource allocation by assessment authorities and research funding and performing organisations is a necessary condition for reforming assessment practices. Resources should be allocated as is needed for each organisation to achieve the changes that will enable adherence to the Principles and to implement the Commitments. This includes resources to:
Particular attention should be paid to making resources available to enable the engagement of researchers at all career stages in reforming research assessment.
6.1 CRITERIA FOR UNITS AND INSTITUTIONS
With the direct involvement of research organisations and researchers at all career stages, review and develop criteria for assessing research units and research performing organisations, while promoting interoperability
Purpose: This commitment will ensure that national / regional / organisational authorities and evaluation agencies review and, where needed, develop criteria for the assessment of research performing units and organisations, in accordance with the Principles. It will foster the responsible use of metrics in assessing research performing units and organisations, and help to prevent contradictions or incompatibilities between the assessment of research, researchers and research performing organisations. It will also safeguard the interoperability of adapted or newly developed assessment processes.
Scope: Criteria for the assessment of research performing units and organisations, including universities, research centres, and research infrastructures, should be reviewed and adapted, and new criteria developed where needed, based on evidence. This should be done in close collaboration with assessors and those that will be assessed, including research organisations and researchers. The changes should increase the ability to assess quality by enabling recognition of all contributions to quality research by research units and institutions. Such recognition includes that of early sharing of data and results, open collaboration, teamwork; and consideration of contributions to the research ecosystem, knowledge generation and scientific, technological, economic, cultural and societal impact. National / regional / organisational authorities and evaluation agencies should coordinate to ensure their methodologies and processes are interoperable, while simultaneously respecting the necessary adaptation to each context.
6.2 CRITERIA FOR PROJECTS AND RESEARCHERS
With the direct involvement of researchers at all career stages, review and develop criteria, tools and processes for the assessment of research projects, research teams and researchers that are adapted to their context of application
Purpose: This commitment will enable recognition of the diverse research activities and practices through the revision and development of assessment criteria, tools, and processes. It will ensure that organisations review their processes and make tangible changes by developing existing or new assessment approaches, individually or in collaboration with others, in accordance with the Principles.
Scope: Criteria, tools and processes should be reviewed and developed together with researchers in different disciplines and at different career stages; and should enable recognition of the diversity of research activities and practices that contribute to research quality, including diverse outputs in different languages. This should increase the ability to assess quality by enabling recognition of all contributions to quality research from research projects and by researchers and research teams. This includes recognition of early sharing of data and results, open collaboration, and teamwork. Reformed practices for assessing individual researchers should consider future potential alongside track record and take into account researchers’ individual contexts and careers. They should also recognise that researchers cannot excel in all types of tasks and provide for a framework that allows researchers to contribute to the definition of their research goals and aspirations. Research assessment by research funders should consider disciplinary, multi-, inter-, and trans-disciplinary research as well as contributions to knowledge generation and scientific, technological, economic, cultural and societal impact.
Purpose: This commitment will ensure that organisations raise awareness of the reform among all actors. It will ensure that organisations transparently communicate the criteria, tools and processes used for research assessment and train researchers and assessors in their use.
Scope: Without widespread awareness of the reform and training of those assessed and, crucially, assessors, progress will be slow – if not impossible. Organisations should be clear and transparent about assessment processes and the tools and criteria they use. They should make guidance on their assessment approaches openly available and train those involved in the assessment process. They should allow those assessed to have access to the criteria, data and reviews or deliberation outcomes used in their assessment within the limits of confidentiality. Particular attention should be paid to raising awareness among researchers at all career stages.
Purpose: This commitment will ensure organisations exchange and make use of information for mutual learning. It will help avoid fragmentation, contribute to the coherence of assessment practices between organisations, and enable researcher mobility. It also will allow those further ahead to share approaches and lessons learned, to benefit those who have further to go on their reform journey.
Scope: While respecting each other’s autonomy, organisations should share practices and experiences to facilitate mutual learning. This exchange should include contributing to the development of guidance and common approaches in order to minimise contradictions or incompatibilities between the assessment practices used by different organisations. It should also include sharing of lessons learned to ensure continuous mutual improvements.
Purpose: This commitment will ensure organisations update one another on the progress made. It will foster careful self-reflection and monitoring of their own adherence to the Principles and progress towards meeting the Commitments.
Scope: Demonstrating progress made towards implementing the Commitments and adherence to the Principles is an important part of this initiative. Organisations should commit to regularly update each other and their communities on their adherence and progress. This process involves being open to scrutiny from their own communities, sharing successes as well as challenges, and communicating their experiences to facilitate collective progress.
Purpose: This commitment will ensure that assessment approach decisions are evidence informed. It will help organisations reflect on their own processes, gain understanding about whether assessment practices achieve the desired goals, and engage in evolutive assessment based on new evidence as it becomes available. It will also help to ensure control and ownership of research assessment data by the research community.
Scope: Growing evidence shows that current assessment processes that rely on publication- and journal-based metrics are prone to multiple biases. As approaches using more qualitative research assessment are piloted by several organisations (e.g. narrative and evidence-based CVs, new assessment frameworks and indicators), it is important to evaluate and monitor their impact based on evidence and rigorous methods. Organisations should contribute to the evidence base on research assessment in order to make this possible. For example, it could be achieved by making data that can be used for research on research available, by participating in research on research, or by funding research on research. Data sharing should be the minimum commitment and data should be shared through open infrastructure, while respecting personal data protection.
Contact us at info@coara.eu