Abstract
Practical tips and guidance on peer review are provided by three scientists in the respiratory field, to help early career researchers who may be invited to review papers for respiratory journals https://bit.ly/3EuWpoH
Introduction
Peer review is an important part of scientists’ activities and the major system for evaluation of scientific reports [1, 2]. Besides making an important contribution to science in terms of ensuring high standards in research reporting, conducting peer review has many values at an individual level: it can be seen as a way to keep updated with recent literature, including new findings and novel approaches; it is also an opportunity to read manuscripts in a particularly structured way and reflect on the hypotheses as well as how the data were obtained and how this might impact the conclusions of the paper; and it may even contribute to generating novel questions and ideas in your own research. Finally, evaluating papers is one of the best ways to improve your own paper writing.
There is no single way to conduct peer review, and the best practices for peer review are a topic of active research [1]. Guides to peer review, as well as editorials summarising thoughts on the peer review process, have been published in various biomedical fields, including in the respiratory field [2–4]. General views on peer review have been previously published in Breathe, in 2014, from both the editor's and reviewer's perspective [5]. However, peer review has changed rapidly over the past few years due to several factors: an increasing number of interdisciplinary papers covering a wider range of disparate scientific fields within one paper, the emergence of preprint repositories, and more rapid review processes which emerged during the coronavirus disease 2019 (COVID-19) pandemic in an effort to disseminate research findings to the broader scientific community with greater speed [6]. The current article includes practical tips and guidance from three scientists in the respiratory field, with different backgrounds (clinical research, epidemiology, and basic/translational research). It is particularly intended to help early career researchers who may be invited to review papers for European Respiratory Society (ERS) journals (European Respiratory Journal (ERJ), European Respiratory Review (ERR), ERJ Open Research and Breathe), and other respiratory journals. Although we focus on the review of original research papers, the advice given here is applicable to different contexts (e.g. review articles and abstracts).
To write this article, we first established a list of questions that early career researchers may have when invited to review a manuscript, especially if it is their first time reviewing an article. Each of the co-authors provided his/her own answers to these questions. Because there was a large overlap in the responses, we combined and summarised them, but kept specific tips that may apply to different fields where applicable.
I'm invited to review a manuscript: should I accept?
If you are invited to review a manuscript, it is probably because you have recently published papers in a closely related field, and your records indicate that you have a certain expertise. Early career researchers might be invited to review a manuscript even if they only have a few publications as a first author. From the editors’ point of view, the best reviewers may be those who recently conducted a very similar study on a similar research question, and are aware of any potential limitations, but can also value novel/additional studies in the field.
However, such reviewers are not always available, and editors may extend their search to authors of closely related publications, even though they do not address the exact same research question. In addition, the editors may not know your exact competencies and field of expertise. For these reasons, it is important that you carefully read the abstract, which is generally attached to the “invitation to review” email, to check that you are confident enough in your capability to assess the value (e.g. novelty, interest of the research question, added value to the field) and evaluate the adequacy of the methods and the potential limitations. The journal editors will expect that you are familiar with the relevant literature and research methods. If you only feel partially confident, you can consult the editor about the expertise that she/he is looking for, and clarify yours. This is particularly important in interdisciplinary studies or studies which use diverse methodological and analytical approaches. The editor may be looking for a reviewer with particular expertise concerning some aspect of the study and not necessarily a reviewer who can cover all areas of the paper.
Before accepting to peer review a paper, you should also ensure that you will have the time to complete it on time [7]. Delays are unpleasant for the editors and the authors, who want their work published. For the same reason, if you choose not to accept a peer review assignment, you should still respond to the journal as soon as possible, to avoid any delays.
Finally, you should evaluate your potential conflicts of interest in evaluating the manuscript. For example, you should refuse to review a manuscript if you have personal relationships or close collaborations with some of the authors (even though you have not yet been co-authors on published papers) or with the research group; if you receive fees or shares from the pharmaceutical industry with relevant vested interests; or if you are working on a competing study or group. For various reasons, you may also feel that you may not be able to provide an objective assessment of the research work (both positive and negative bias may exist). Therefore, to maintain scientific integrity in the peer review process, you should decline the invite.
Questions to consider when invited to peer review are summarised in figure 1.
Questions to consider when invited to peer review.
What is the aim of the peer review process?
The simplest answer to this question is that peer review aims to ensure that only papers of a high methodological rigour, which support the conclusions stated by the authors, are accepted for publication in a peer-reviewed journal. Peer review also helps the authors to improve these manuscripts and better address their research questions. You should remember that the peer review process is a crucial component of the scientific discussion, where colleagues are invited to offer their expertise and their honest views. It can help authors to reflect on how strong their conclusions are given the data presented in their manuscript, as well as point out alternative conclusions which could be drawn from their data. As a result, peer review comments should not aim to diminish a manuscript (or its authors) but to improve it. A good peer review report could also serve as great educational material and a source of inspiration for future/follow-up studies. Finally, under no circumstances should the peer review be condemning or rough.
It's the first time I've been asked to peer review: where do I start?
An overview of the main steps of the peer review workflow is presented in figure 2. Once you have accepted the invitation to review, you will receive a link to access all submitted files, and a deadline to send your review. First, you should plan the review work in your calendar. It will take several hours at a minimum, even if you have core expertise in the area of the paper, and it is best to avoid fragmented work for this task. Ask yourself when you will be able to devote significant time to your review with minimal interruptions. When starting the review, make sure to download all the files, not limited to the main manuscript, but also the figures and tables (sometimes provided separately) and if applicable, the supplementary material. It is always a good practice to read the journal's guidelines to authors and/or reviewers before reading the manuscript to have a general idea about the aims and scope of the journal as well as specific formatting requirements.
Main steps of the peer review workflow.
Then, although each reviewer may have their own way to proceed, most experienced reviewers advise reading the full manuscript a first time to understand the content and make a first general impression. Some reviewers initially skim through the paper to look for any major flaw that may lead to a quick rejection (e.g. inappropriate methodology or significant, unaddressed bias). If you identify such a flaw, you can describe it in your comments and reject the manuscript without reading in more detail. A “major flaw” depends on the state of knowledge around a specific topic area as well as practical limitations when it comes to available methods to address basic research questions. For example, the first studies assessing risk factors for adverse COVID-19 outcomes were not adjusted for potential confounders and at the time this was acceptable, provided that it was highlighted as a limitation in the discussion [6]. However, a few months later it became very clear that factors like age, gender or specific comorbidities are strongly correlated with the adverse outcomes of COVID-19 and lack of adjustments in newer prognostic studies would be considered a major flaw leading to a quick rejection.
The first read through the manuscript can already be an active reading, highlighting some parts which seem important (i.e. which will later help in evaluating the strengths and limitations of the manuscript), and taking quick notes. Although it remains a first impression, this first reading is already a time to ask yourself the following questions:
Is it novel?
Is the research question important and well introduced?
Are the main findings clearly described?
Do the methods seem adequate?
Do you agree with the interpretation of the results?
On many occasions, the reviewer knows at the end of this first read through what his/her recommendation to the editors will probably be (acceptance, major/minor revision or rejection).
After this first overall impression of the manuscript, you should go back to each section for a very careful reading, taking detailed notes referring to specific sentences or paragraphs. This more thorough reading will be the basis for writing your reviewer report.
What should I check in each section of the manuscript?
We list below the main points of attention that are common to most research manuscripts. Research reporting guidelines, such as the CONSORT guidelines [8] for randomised controlled trials or the STROBE guidelines [9] for observational studies, provide useful checklists of items that need to be addressed in research papers. A large group of editors of respiratory journals have also provided specific guidance regarding the reporting of observational causal inference studies [10]. In basic and translational science, other guidelines, such as ARRIVE 2.0 [11] for animal work or FAIR Data Principles to allow for reuse of scholarly data [12], have become standard for most journals. Such material can be useful to reviewers who may refer to this as a list of good practices in their field. However, it is important to note that no exhaustive review checklist can be provided. The reviewer, as an expert in his/her field is expected to be able to evaluate the research beyond pre-established checklists and considering the various specificities of the research question.
Title
The title should reflect the content of the article and its main findings. Generally, it should be short but informative. Some journals have specific requirements for the title in terms of length and style (e.g. an “informative” rather than a “descriptive” title). It is important to read the title carefully before, but also after, reading the full manuscript, to check the consistency between what is stated, in terms of content and conclusion, and what was actually done and well supported by the findings of the paper. In particular, you should check that the title does not over-emphasise selected findings (e.g. only positive fndings) or use overly strong statements as compared with findings reported in the abstract/manuscript.
Abstract
The abstract should contain brief information about the rationale or hypothesis of the study, the main research methods used to address this, and the major findings. Ensure the key components of the methods and results are presented. For example, in epidemiological research, the abstract should include the main information about the study design and methods (e.g. cross-sectional versus longitudinal study, number of subjects included in the analyses), and provide the main results with important estimates (e.g. relative risks and confidence intervals rather than only statements such as “was significantly associated with”). Importantly, the abstract should reflect the main findings and conclusion from the manuscript. Often authors cannot summarise all their results in the abstract due to the word limit; however, it is important to ensure that the summary is balanced and does not only include the positive findings, but also pertinent negative findings. Similarly, you should carefully check that findings are not overinterpreted in the abstract compared with the manuscript, as the short format sometimes pushes authors to over-emphasise some findings. A recent systematic review found that the abstracts of systematic reviews are often inconsistent with their actual findings [13]. This suggests poor writing as well as a poor peer review process.
Introduction
The introduction should include a balanced account of the available data/evidence around the authors’ research question. While the introduction should not be too long (an exhaustive description of all related data is not required), it should include all pertinent data and previous major research findings that will allow the reader to understand the current state of the research field, the remaining knowledge gaps, and the rationale of the research question. Moreover, the research question should be very clearly described, usually in the last paragraph of the introduction. You should also pay particular attention to the references provided in the introduction. Are they up to date? Do they appropriately reflect the field? Any omissions should be highlighted.
Methods
The methods may be the most scrutinised section in peer review. A first crucial question is whether the described methods can adequately address the question posed. It is also important to check that the methods are described with enough clarity. One way to do this is to ask yourself “If I had to replicate this research/experiment, do I have all the information I need to do so?”. Another good question is “If I wanted to address this research question with the same means (e.g. the same data/study population/experimental material), how would I proceed?”. This generally implies a series of methodological choices at different steps of the study. Some of the authors’ choices may differ from yours. Do you find justification for their choices? How did they address uncertainties (e.g. sensitivity analyses)? Are there any pertinent experiments or analyses missing?
What needs to be checked specifically differs in each research field. In epidemiological and clinical research studies, ensure adequate description is provided for the sample size selection, study population (including the main inclusion and exclusion criteria), as well as the outcomes of interest. Moreover, studies evaluating treatments should describe the interventions, comparators and co-interventions; prognostic studies should list the prognostic factors; and diagnostic studies the diagnostic tests and gold standards they are compared to. Especially, but not only, observational studies need to describe how they accounted for potential confounding factors. In epidemiology, the study design, exposure assessment and statistical methods should be additional points of attention. Any potential source of bias should be considered and appropriately addressed and/or discussed. Clinical trials and systematic reviews should be based on a prospectively registered protocol. Nowadays, high-quality observational studies may also be registered prospectively. This should be described in the methods. Also, any differences from the prospectively registered protocol should be described and justified. It is advisable to have a look at the study protocol while peer reviewing.
For basic and translational studies, it is crucial that the methods are described such that the research can be replicated. Therefore, information such as the source of all chemicals and biological reagents should be listed, including lot numbers or batch numbers when appropriate. If tools have been designed in the laboratory or given as a gift by another laboratory, such as antibodies, plasmids or primer sequences, then details or references to original papers describing these reagents should be listed. With the rapidly increasing use of bioinformatic analyses in biomedical research, papers which analyse or re-analyse large datasets should provide information regarding access to the raw data, all the software used, databases (including version numbers if applicable), as well as access to all the code used to analyse data or generate plots. External resources such as deposition of data in data repositories, such as the European life-sciences Infrastructure for biological Information (ELIXIR), as well as custom code should be provided during peer review and/or posted on resources such as GitHub.
Results (including figures/tables)
In the results section, the authors should describe in a balanced fashion all their results (positive and negative). Importantly, the presented results should correspond to the paper's main objectives, and to what is announced in the methods. While parsimony may be welcome (e.g. authors should avoid presenting “side” results not related to their research question), any selective presentation or emphasis of the most significant results should be pointed out. For example, it is the responsibility of the peer reviewer to ensure that all outcome measures that are described in the study protocol and methods, or that are required to address the research question, are included in the results.
In terms of content and style, it is important to check that the authors report the absolute or relative effect estimates, along with their confidence intervals, and not only the p-values, which are less informative. The results should be described in an objective manner, without trying to predispose the reader (e.g. “treatment X decreased the frequency of exacerbations by 2%” and not “treatment X decreased the frequency of exacerbations by ONLY 2%”). As a reviewer, you may also comment on the choice of tables and figures: are they clear and adequate? Are the figure legends and all labelling on the figures adequate for interpreting the findings presented? For basic and translational science: could the data be presented in a more transparent way to see the effects of batches or patient/animal/primary cell culture differences? Feel free to suggest alternative ways to present graphs or figures which will help to display the data more transparently or in a way that makes it easier for the reader to understand the main conclusions which can be drawn.
Discussion
In all research papers, the discussion should contain the following components: a short description of the main study findings and their interpretation; a comparison of the study findings with other relevant studies, along with a potential explanation of the important similarities and differences; a description of the main strengths and limitations of the study; clinical and research implications of the study findings; followed by a conclusion. Besides checking that all these components are present, the main questions you should ask yourself to evaluate the paper at this stage are: “Is the conclusion supported by the findings?”, “Is the interpretation of the results balanced?”, and “Are the study limitations and alternative potential interpretations clearly stated?”. Comments on specific points in the discussion, e.g. if you disagree or do not fully agree with some statements made by the authors, can also be noted in your review.
References
All statements that the authors make should be justified either by their findings or by appropriate references. The references should include some of the most relevant, high-quality studies in the field. Generally, the most recent literature reviews or meta-analyses on a given topic, and papers published afterwards, should be cited. If you are aware of potential controversies in a field, you should check that papers reflecting different points of views are cited. You may also look at authors’ self-citations, which should appear justified and not be excessive. Moreover, you should not ask the authors to add references to your studies, unless they are directly linked to their work. It is not appropriate, and it does not look nice to the editor, or the authors (who may be able to identify you or your supervisor if you inappropriately ask them to cite your articles).
Funding/conflicts of interest
This section should not be neglected during peer review. If there are potential conflicts of interest, the authors should describe how these were managed during the study process. For example, if a study was funded by the pharmaceutical industry, the authors should explain what the role of the funder in the design, conduct and reporting of the study has been and how the impartiality of the study was maintained. If you think that that authors may have competing interests (explicitly declared or not), you should inform the editors.
Miscellaneous sections, e.g. “key points”
Several journals request additional summary sections (e.g. key points, graphical abstracts or a “research in context” summary). They should also be reviewed carefully, as this may be the only part that some readers will read. The peer reviewer should ensure these are a balanced representation of the study and its findings. As in the title and abstract, authors may tend to present only the positive findings, or describe conclusions that do not directly result from their study.
How should I organise my report?
Once you are done with reading and taking notes on different parts of the manuscript, you should organise and format your report. The report is intended first for the editors to make a decision on the manuscript, and then for the authors either to appropriately revise their manuscript or if rejected, to understand the reasons for this decision and how the manuscript can be improved if submitted elsewhere. Keep in mind that the editor's final decision may be different from your recommendation, as she/he generally takes into account comments from several reviewers. Therefore, it is preferable to provide rather detailed comments regardless of the recommendation. Depending on the journal, some journals will have a section for comments to the editor and comments to the authors. You may provide more direct feedback about your thoughts to the editor in the “comments to the editor” box (see later). Comments to the authors should be written constructively, even if there are aspects which you perceived to be major flaws in the original submission.
A good report starts with a general assessment of the quality and interest of the manuscript, with 2–3 sentences describing the research question and main findings, focusing on the main strengths and limitations, for the benefit of the editor (see the example in Box 1). Comments on novelty and interest of the study can also be included.
BOX 1 Example of 2–3 sentences summary in the reviewer report
“This observational study explored predictors of the future risk of COPD exacerbations in two population cohorts totalling over 1 million eligible participants. All analyses were adjusted for multiple demographic parameters, important comorbidities and respiratory treatments. However, exacerbations were defined based on the prescription of systemic corticosteroids only. The investigators could not differentiate the use of steroids for an acute exacerbation, for another disease, or as a rescue pack that was perhaps never used. Moreover, hospitalisation data were lacking and it is not clear whether severe exacerbations were captured in this study.”
Then, most reports continue with a list of “major comments” and then “minor comments”. The former refers to major issues in the paper that require additional work to be resolved (e.g. additional experiments, or additional dimensions to be described). The latter refers to small changes to improve the flow of the manuscript or add clarifications. The ordering of the comments depends on the reviewers and manuscripts. Reviewers may start with general points which apply to the manuscript as a whole and reflect general concerns that need to be addressed, and then provide detailed comments referring to specific parts of the manuscript. A common approach is to group the comments depending on the section of the manuscript they refer to (e.g. abstract, introduction, methods). When making very specific comments (e.g. when referring to a paragraph, sentence or word), it is useful to provide page and line numbering and to try to be as explicit as possible. Further, it may be helpful or appropriate to provide citations in your report if you would like to refute or question statements made by the authors in their original submission.
How long should my report be?
There is no requirement regarding the length of the reviewer report. Sometimes, you only need a few lines to state why you believe the manuscript has critical flaws that should lead to a rejection. Longer reports are usually required for papers that have some value, but could be improved significantly upon major revision.
In such cases, you should take the space that you need to make all your comments and to explain them clearly. This will help the authors to address them, and shorten the review process by avoiding going back and forth between the authors and reviewers (i.e. having a second and sometimes third round of review). However, excessively long reports may also be counterproductive. The reviewer's role is not to redo the study or rewrite the manuscript. If very extensive changes are needed to make the manuscript suitable for publication, you should inform the editor of this assessment and summarise the major concerns in the comments to the authors.
What if I don't feel skilled to evaluate specific parts of the manuscript?
First, you should only accept to peer review manuscripts that are relevant to your expertise. Therefore, you should feel confident to evaluate most parts of the manuscripts that you peer review. If after accepting to review the manuscript, you realise that you are not skilled to have an expert opinion on some parts (e.g. a specific statistical method), you can still write a report on the parts of the manuscript you are comfortable with, but you should inform the editor about the parts that you did not comment on because of lack of expertise (e.g. in the “comments to the editor” section). However, depending on the journal and type of paper, your input as a “non-expert” can also be relevant. Highly interdisciplinary work will be read by diverse audiences and, therefore, it can be helpful to point out field-specific terminology to authors in the minor comments or some suggestions of areas to improve. It is important that you do not list these as major comments but specify that these may help non-specialists understand the paper, the methods used, as well as the conclusions which can be drawn.
How will the editor make his/her decision? What should I recommend?
Editors usually assess the comments of at least two, and sometimes more, peer reviewers. The full reports from the reviewers, and not only the actual recommendations (e.g. “revision” or “rejection”) are important in the decision process. Editors usually review the manuscript as well, in order to make an informed decision. In cases with contradictory comments, they may invite additional peer reviewers.
Editors will make a decision considering: 1) the methodological rigour and all strengths and limitations of the manuscript, as described previously, 2) the pertinence and novelty of the findings, and 3) the relevance of the study to the scope and readership of the journal. If major issues are raised by the reviewer, the editor will also evaluate the chances that the authors can adequately revise the manuscript (e.g. based on the material/data that appear available and the feasibility of the requested revisions).
Your recommendation should be supported by your comments. After carefully appraising a manuscript and considering the strengths and limitations in the methods, content and message, you should consider whether it is appropriate for the journal. Keep in mind that negative results may also be pertinent (but ideally a study needs to have the power to demonstrate lack of effect). Examples of types of studies which are generally considered suitable for different kinds of journals are presented in Box 2.
BOX 2 Which kind of study would fit in which journal? Some examples
If a study has methodological errors that significantly limit the confidence of the findings and cannot be addressed, then it should be rejected by any journal.
A very preliminary report of pilot data around an innovative idea, a study confirming well-known and validated evidence, or an innovative study that is based on a shaky methodology or limited power are unlikely to fit the scope of a high-impact specialty journal, such as the European Respiratory Journal.
Findings that are only relevant for researchers with a very narrow expertise are more likely to be published in more specialised or more inclusive journals (e.g. ERJ Open Research).
Only studies of the highest rigour that are likely to imminently lead to changes in clinical practice would be accepted for publication at the New England Journal of Medicine or the Lancet.
What are the “confidential comments to the editor” for?
You can use the “confidential comments to the editor” to express any sensitive concern that you, or perhaps the editor, may not want to convey to the authors. Sometimes, it is also a space to justify your recommendation, or to moderate it. Keep in mind that you should not share your recommendation with the authors. You can also use this space to describe any relevant conflicts of interest that you may have or any concerns regarding ethical aspects of the study and manuscript, including data fabrication or plagiarism.
After revision, I'm not happy with the authors’ responses to my comments: what should I do?
Disagreements between the authors and peer reviewers are not unusual. First, you should ensure that you have reviewed the authors responses carefully and impartially. If you think that the methodology/results or any part of the manuscript is still significantly flawed and might be misinterpreted by the reader, then it may be appropriate to explain that to the authors and editor through an additional round of peer review. For this, you will need to write a new report, stating point by point the remaining issues and, if applicable, inadequate responses. Sometimes, the authors may not follow the advice of the reviewer but provide adequate justifications for this. In this case, it might be more appropriate to accept their responses.
I'm willing to peer review: how do I attract relevant peer review invitations?
To facilitate editors’ work in identifying suitable reviewers, and to avoid inadequate solicitations, it is recommended to have a personal or institutional webpage with a short description of your field of expertise, your current work, a list of publications and contact details. If you would like to maximise the likelihood that you will receive papers that fall within your expertise, you could register a profile at a few journals relevant to your field and add specific keywords (e.g. “COPD”, or “COPD exacerbations”, rather than “Respiratory medicine”). Also, make sure to add keywords reflecting both your disease expertise and methods expertise (e.g. clinical trials, epidemiology, or mouse models). There are also early career reviewer programmes at several of the respiratory journals, including the ERS and the American Thoracic Society family of journals. You can also typically attend sessions at all of the major respiratory meetings with editors where you can find out more details about which programmes they have for early career reviewers.
How do I get recognition for my work as a reviewer?
After submitting a peer review report, journals usually send you a confirmation email that you could include in your CV. There are online services, such as Web of Science Reviewer Recognition Service (http://webofscience.help.clarivate.com/en-us/Content/wos-researcher-profile.html), that can collect and validate all your peer review records. At any time, you can download a summary of your peer review activity for your CV. Some journals are directly linked with the service and your peer review records are updated by the journals. In other cases, you only have to forward the confirmation email that you receive from the journal. Additionally, some journals have an option for your name to be published with the final published article. This is a personal choice and optional in most cases. There are obviously pros and cons that go with having your name disclosed as a reviewer and so this should be carefully considered.
Conclusion
We strongly recommend all scientists, including early career researchers, dedicate time to peer review activities, which are valuable both at the community level and at an individual level. When engaging in peer review work, keep in mind what is valued in this process from the perspective of an author: receiving clear, fair and scientifically relevant reviews, and when limitations are pointed out, comments that help progress the work. When the peer review work is well conducted, editors often see respectful and positive exchanges between the authors and reviewers, in which authors honestly and warmly thank the anonymous reviewers who helped them improve their paper, and reviewers thank and congratulate the authors for their work. In general, peer review should aim to improve the overall quality of the paper for the team of authors and for the field. This speaks about how as a peer reviewer you can also make great contribution to science.
Footnotes
Conflict of interest: None declared.
- Received June 20, 2022.
- Accepted October 5, 2022.
- Copyright ©ERS 2022
Breathe articles are open access and distributed under the terms of the Creative Commons Attribution Non-Commercial Licence 4.0.
Jump To
- Article
- Abstract
- Introduction
- I'm invited to review a manuscript: should I accept?
- What is the aim of the peer review process?
- It's the first time I've been asked to peer review: where do I start?
- What should I check in each section of the manuscript?
- How should I organise my report?
- How long should my report be?
- What if I don't feel skilled to evaluate specific parts of the manuscript?
- How will the editor make his/her decision? What should I recommend?
- What are the “confidential comments to the editor” for?
- After revision, I'm not happy with the authors’ responses to my comments: what should I do?
- I'm willing to peer review: how do I attract relevant peer review invitations?
- How do I get recognition for my work as a reviewer?
- Conclusion
- Footnotes
- References
- Figures & Data
- Info & Metrics