Administrative procedure for Collaborative and Knowledge-building Project applications

As soon as the Research Council has received the applications, it will conduct a preliminary administrative review. Applications that do not meet the formal requirements described in the call may be rejected.

The following formal requirements are checked:

• Is the applicant an approved research organisation?
• Are the application and all attachments written in English?
• Has the project description been attached, and has the correct template been used?
• Has the project manager’s CV been attached?
• Does the project involve at least two Norwegian partners?
• Does the project manager meet the competence requirements?
• Does the application meet the requirements relating to partners’ contribution or cash financing?
• Has a Letter of Intent from the partners been enclosed?

Well before the application deadline, we start the work of finding international referees to consider the applications. However, this work is only complete once the applications have been received and sorted under their respective topics.

The referees we use must meet a number of general requirements:

• Several referees in the panel must be active researchers who have produced a significant body of work in terms of both quantity and quality. In a Collaborative and Knowledge-building Project, panels can include members from outside academia with good industry know-how about the research challenges in question.
• Researchers should have professor qualifications. The minimum requirement is associate professor qualifications or similar. 
• The referees’ place of work should preferably be located abroad.

We also aim to avoid using the same panels several years in a row. Ideally, a panel should have some referees who have participated before while others have been replaced.

To find new experts, we use various databases and tools, such as Expert Lookup. We perform automated searches on the basis of project titles and summaries as well as manual searches. We supplement these with searches on other websites such as the Web of Science and Google Scholar, as well as foreign universities with high profiles in the various fields. We also consider the applicants’ own suggestion for suitable referees. The list of sources varies from field to field.

Case officers thoroughly assess each of the potential referees. Our priority is finding good referees, including someone with broad expertise who can be active in discussions and do a good job on giving feedback to applicants. The referees must be able to participate in the panel meeting at the scheduled time and be motivated to do good work, both prior to and during the panel meeting.

We publish the names of all referees after the application assessment process has concluded.

The panels normally comprises four to five referees, and never fewer than three. We endeavour to ensure a good gender balance in the panels. The panel’s overall expertise must cover the breadth of the applications as regards topics, disciplines and industry and sector knowledge, and the axis from basic to applied research. The panels are also put together with a view to the need to consider interdisciplinary applications.

Before the referees are given access to the titles and project summaries of the applications assigned to the panel, they must consent to a confidentiality statement. They then declare their level of expertise on the basis of this information.

The referees indicate one of the following levels of expertise for each application:

• Specialist: The proposal is within your primary area(s) of expertise. You are well qualified to evaluate the proposal.
• Generalist: You have a general knowledge of at least one of the main subjects of the proposal. You are qualified to evaluate the proposal.
• Minor: You have only minor relevant expertise on the main subjects of the proposal.

The Research Council's expertise requirement states that at least two of the referees must have generalist or specialist expertise in the application’s field of research. If the mapping of the panel’s expertise shows that some applications are not adequately covered, we will bring in additional external referees with special expertise. The referees we bring in to assess individual applications will do this on par with the ordinary referee panel members.

In addition to the mapping of expertise, the referees will be sent a list of all project participants mentioned in the applications the panel will assess. This also includes project participants affiliated to partners in the project. To be permitted to assess the application, the referees must declare their impartiality in relation to everyone participating in the project. The referees are asked to pay special attention to the points in the impartiality provisions that concern collaboration, friendships and conflicts.

See our regulations on impartiality and confidence.

All panel members read and assess all applications to be considered by the panel. The goal of the panel assessment is to arrive at a consensus-based assessment of the three main criteria Excellence, Impact and Implementation. The assessment consists of numeric marks, in addition to a comment explaining why the panel chose this mark.

An application consists of the application form, project description, the CVs of the project manager and key project participants, as well as Letters of Intent. If the applicant has appended any other information, the Research Council’s administration ensures that these documents are removed from the material sent to the referees to ensure that the applications are assessed on the same grounds.

Before the panel meeting, there is a preliminary meeting, the main purpose of which is to ensure good technical implementation and to give the panel members an opportunity to ask questions. The questions can concern the call itself, the topic or the understanding of or emphasis on the marks.

The panel members submit their individual assessment of each application prior to the panel meeting. The individual assessments will not be forwarded to the applicant, but will be shared between the panel members when everyone has submitted their assessments and form the basis for the discussions during the panel plenum meeting.

For each application, one of the panel members will be assigned special responsibility as principal assessor. The principal assessor usually has specialist expertise in the application’s thematic area, and has a special responsibility for the discussion in the panel meeting and for compiling the panel’s assessments.

The members attend the panel meeting with their different professional views and interpretations of the scale of marks and will also emphasise the various elements of the application differently. The members discuss each application and consider them from various perspectives, and the panel reach a consensus-based assessment and a joint understanding of the scale of marks.

Research Council employees with responsibility for the panels will not participate in the scientific discussion. Their role is to provide guidance and ensure progress, contribute to a joint understanding of the assessment criteria and scale of marks, and ensure that everyone gets to speak. They also handle impartiality according to the rules and stop discussions about matters that fall outside the panel's remit.

When referees are formally appointed to serve as a panel member, they receive information about the administrative procedure through the document Guidelines for referee panels Collaborative and Knowledge-building project (pdf).

When the panel have submitted their assessments, the Research Council’s administration calculate the average of the three main marks awarded. The panel’s marks are not weighted.

If the average is lower than 5, the application will be rejected immediately. If the application was awarded ≤3 for one of the main criteria, it will also be rejected. That means that an application that has been awarded the mark 7 for Excellence cannot make up for a mark 2 for Implementation.

The remaining applications will be assessed for relevance.

One important objective for a Collaborative and Knowledge-building Project is to develop R&D competence in Norwegian R&D groups in relation to topics of importance to Norwegian societal actors and Norwegian industry. We also want to stimulate collaboration between the research groups and those representing the societal challenge for which funding is sought. The call and the topics therefore include requirements and guidelines to that end.

The Relevance criterion ensures that the requirements and guidelines described in the call are followed up. The Research Council’s administration carries out the assessment of relevance, which is intended to supplement the referees’ assessments.

In the assessment of relevance, we use a standardised assessment table with the same scale of marks as for the other criteria. A brief written comment will also be included in the feedback to applicants once the application assessment process has concluded.

The assessment table for the Relevance criterion consists of the following points:

• The extent to which the project satisfies the thematic approach the topic requires
  • The extent to which the project satisfies the thematic guidelines and delimitations

• The extent to which the project satisfies the requirements and characteristics set out in the call and that otherwise apply to the topic
  • The extent to which the project satisfies the requirements relating to project partners
  • The extent to which the project satisfies the purpose of knowledge-building in the research groups

if the applicant has selected more than one topic under the call, the application's relevance will be assessed accordingly, and can become relevant for funding in several portfolios.

The overall mark is the average of the marks awarded for the four main criteria, i.e. the referees’ three criteria and the administration’s assessment of relevance. The four main criteria are equally weighted.

For each topic, our goal is to have a comprehensive project portfolio based on the information given to the applicants through the call and the relevant plans referred to therein. The administration therefore carries out an overall assessment of the portfolio. This assessment does not affect the mark awarded to the applications, but is a tool used to prioritise between applications whose ranking is relatively similar after the assessment of individual applications.

The portfolio assessment takes the following factors into account:

• The applications’ assigned marks based on the assessments
• A good distribution of projects in relation to priorities set out for the specific topic
• The relative volume and quality of grant applications under other calls within the same thematic area in the same application year
• Any changes in the financial or scientific framework set by the ministries
• Priority will be given to projects led by women project managers when the applications are otherwise considered to be on a par

Based on the referee panel’s assessment, the assessment of relevance and the portfolio assessment, the administration writes a recommendation for the portfolio board on whether to grant or reject the application. Applications targeting more than one topic can be included on several lists of recommendations given to different portfolio boards. The Research Council’s administration ensures that the information provided enables the portfolio board to quality assure that the processing of applications has complied with the stipulated procedures, and that all applications have been properly assessed.

The portfolio board is presented with the recommendation documents for all applications, including those defined as being below the threshold value and are therefore in principle not relevant for funding. They are also sent overviews showing the marks awarded to the applications for the individual criteria.

The recommendations (one for each application) are intended to give the portfolio board good insight into the assessments made. The recommendations are also intended to make the portfolio board aware of applications where the assessment of relevance and/or portfolio assessment has had a major bearing on the application’s overall ranking. In this way, the portfolio board can quality assure the administration’s assessments. The portfolio board’s discussion may result in decisions that do not correspond with the recommendation the Research Council has submitted.

The Research Council has 15 portfolio boards with variation in the budgets at their disposal, the number of application types announced, the number of applications for processing, thematic breadth and the expected number of applications granted.