Hello, and welcome to the Physics World Weekly Podcast. I'm Hamish Johnston. Physics World is brought to you by Institute of Physics Publishing, which produces more than 90 Scholarly Journals. I OPP is one of more than 35 organizations around the world that's celebrating Peer Review Week on September 23rd 27th. In this podcast, we explore how the Publisher is raising awareness of the importance of constructive and respectful peer review feedback, and how innovations can help to create a positive peer review culture. That's coming up after this message. Progress in Physics 2024 is a proud supporter of this week's podcast. The 2 day event takes place on the 8th 9th October at the Institute of Physics in London. It will take a deep dive into the exciting world of condensed matter physics. And you can secure your place now to avoid disappointment. Just visit i0p.events air.comforward/pip2024. Peer review is an essential part of the modern scientific process. In a nutshell, a piece of research is described in a paper, which is sent to several independent experts for review, As well as providing an important check on what gets published in scientific journals, peer review can also be a constructive process. Reviewers will often ask for revisions before publication, with the authors and the wider scientific community benefiting from their expertise. But like any human endeavor, the process doesn't always go smoothly. Sometimes authors receive very unhelpful or outright rude feedback about their work. Such reviewer misconduct is never acceptable, and it could dampen the enthusiasm of early career physicists, turning talented people away from careers in science. IOP publishing, which brings you physics world, publishes over 100 scholarly journals and engages with thousands of authors and reviewers. I'm joined by Laura Fethom Walker, who is IOPP's reviewer engagement manager, to talk about unhelpful reviews and how authors and reviewers can better engage with each other. Hi, Laura. Welcome to the podcast. Hi, Hamish. Thank you so much for having me on the podcast to talk about this topic. So as as part of your work, Laura, you've asked physicists for examples of poor reviews. Can you share a few examples that, they got back to you with? Sure. I can definitely give you some, examples, some of which are quite shocking. But for context, the reason that we we did this and we went out and asked some senior members of the community about their experiences receiving unprofessional peer review comments is because I think it's something that isn't very widely talked about, and it is an issue. So where I've spent the last 4 years or so running workshops with members of the physical sciences community, early career researchers, all the way up to very senior professors. And often when we have discussions that they talk about these really unprofessional, rude, peer review comments that they received where they felt personally attacked. And it's quite clear that this is something that really sticks with people. And often there was quite a sense of grievance or confusion or upset. And what I noticed was when you talk to really senior people who've been in the field, for decades and who've had really successful careers, some of these comments they received early in their career, they still remember them, you know, like it's yesterday. So it obviously have a real impact. And so what we've done for peer review week this year, 2024, is we've created a short video where we asked for really senior, physicists to talk to us about the negative, and unprofessional and rude comments that they received early in their career. And then we, we contrast that with their amazing achievements, since then. And and the reason we wanted to do this is to kind of, let earlier career researchers know that when they receive, unprofessional comments or ad hominem attacks in the peer review reports on their papers, it's not necessarily a reflect a reflection on them as a researcher. And they shouldn't let, let it get them downhearted. And it shouldn't affect their confidence. So that's hopefully the goal. We spoke to 4 senior researchers. One of them is professor Aklesh Lactakia, who's an incredibly, senior and accomplished, physical scientist. And the comment that he, highlighted in the video, which he received early in his career is, this is rubbish. Obviously, the author or authors had no electromagnetic training and no physical intuition. So somebody said that that was written down on a on a piece of paper or or in a some sort of review document, and someone thought that that was appropriate and constructive. Absolutely. I I don't know whether they thought it was constructive, but it definitely wasn't constructive. And and I think, you know, there's there's subtleties here and there's nuance. Right? You you can have a negative review. You can even have a little bit a harsh review where the reviewer is strongly critiquing the science, right? They're looking at that paper and they're saying, I really don't think this is up to par. I'm I'm not sure about the methodology and things like that. When we talk about unprofessional and rude reviews, they are always ad hominem attacks. They are talking about the authors themselves and not the research. And so in this comment that, Aklesh highlighted, you know, to say the authors have no training or intuition, that's downright rude and unprofessional. Unfortunately, it sounds like a lot of researchers have experienced this at some point in their career. Editors, you know, our editors follow cope guidelines, which is to say that when we see unprofessional or rude comments in a report, we do send the review back to the reviewer. And we say, please, can you amend this? Please, can you tone it down? And if the reviewer says no, then we might make some changes ourselves because we really don't want these to get to the authors. However, as you said, this is a human process. Things slip through the net, and it's quite clear that people are receiving these kind of comments. Doctor Cathy Foley, who's the chief chief scientist of Australia, who's, again, like an incredibly senior physicist, is also in the video, holding up a card with the negative comments that she received early in her career. It's across the board, regardless of career level. And we just felt that this is something that really needs to be highlighted and talked about a bit more. I see. Okay. And I should point out that Kathy Foley has been on the Physics World Weekly podcast. So, yeah. Definitely a very esteemed researcher. So, Laura, not all papers are immediately acceptable for publication, and some might have significant flaws. What is the best way to identify problems in a paper in a polite and professional manner? I mean, obviously, avoiding the word rubbish would be a good start. But, what can you can you give some guidelines maybe for how how reviewers should respond to work that, okay, is substandard and and the author has to be told. Sure. I mean, peer reviewing is a is a skill. And and to some extent, it's more of an art than a science. I don't think that's widely appreciated. It's something that people get better at throughout their careers and with practice. And so it's understandable if people don't get it right all the time. We do offer really comprehensive peer review training here at IOP Publishing, which is tailored for the physical sciences. And one of the things that we talk about in that training is focus on the science, not the authors. Assess your own biases. So try and be aware of your own biases. That might be biases against, particular methodologies, for example. That's quite a common one. If you receive an invitation to review a paper and you know you are just biased against that particular methodology, it's probably verging on a conflict of interest. And maybe you should, you know, say no because you're not able to offer an objective review. The other tip that we give people is that it's good practice to say at least one positive thing in your review. So you might, really have problems with the paper. You might not think it should be published for various reasons, and you might think it's unsalvageable. You might think that the reviewers can't revise it in a way that would make it publishable. And that's okay. You know, that's part of the job of a reviewer. But try and find one positive thing to say. And I think, fundamentally, remember that there are humans, human beings on the other end of this review who are going to sit down and read it, and who have worked really hard to create a research paper, and probably really care about furthering, the field. So those are the main tips we'd give people. There are tips about how to structure reviews and things like that. But fundamentally, to not send a rude and unprofessional review, that's what people should be keeping in mind. And science is is by its very nature, and it's an international thing. You know, IOPP has authors from all over the world. So, you know, I suppose culture can come into this as well. I mean, even in the, you know, I suppose the microcosm of of Northwest Europe. A reviewer from the Netherlands, might be much more blunt than a British reviewer. And, a Brit might, you know, they might try to inject a bit of humor into their review, which, you know, could fall flat with, I don't know, an American. So, is it possible to have guidelines to mitigate these cultural differences, on what is and isn't polite discourse? I mean, you know, could you would you tell a Brit? Well, you know, maybe you shouldn't be you shouldn't try to make it funny, or, you know, no sarcasm, please, or no irony, please. Is it possible to deal with those, those cultural differences? And indeed, language. Because a lot of, you know, some reviewers, English will be their first language. Others will be very good, with English. But others might, you know, maybe struggle a bit if it's not their first language. So, you know, how, are are those issues important when it comes to to reviews? It's a really interesting point and I think it's something that hasn't been studied. It'd be really difficult to study these kind of cultural differences in in peer review. But it almost certainly does happen. It's the job of the editor really to pick these things out. And editors are certainly trained on exactly that. So I think the issue of, for example, British people, using allegories or certain kind of terminologies or indeed trying to make jokes, I've seen that in cases. And and the editor should really intervene and say, you might wanna be a little bit more clear in exactly what you mean here. It's it's not possible, I think, to give clear guidelines to reviewers because, you know, it's just not gonna be feasible to do that. I think, fundamentally, when it comes to unprofessional reviews, I think fundamentally, when it comes to unprofessional reviews, you can say to everyone around the world, regardless of their culture or their first language, if you are attacking the authors, or the country of the authors, the institution of the authors, anything like that, then it's unacceptable. Fundamentally, what we're really looking out for and what we're really against is ad hominem attacks in peer review reports. And that's regardless of the reviewers' geography, culture, or anything else. So if the reviewer is in any way attacking the authors, their their institution, their location, or anything like that, that's a problem. And if they're critiquing the science to an extent, the tone matters a little bit less. One thing I will say is it's really interesting looking at the reports of, what researchers say they have received because some of the con some of the reported comments are really shocking. So there was an article published in peer j back in 2019 by Nissa Silbiger and Amber Stubler, based in California. And to my mind, it's the best analysis of this, that's been done. And they basically went to, a geographically diverse group of STEM researchers, and they asked them to self report what kind of comments have you received on your peer review reports. And, they really are shocking, and I'd recommend anyone interested to look up that paper. Because it's things like, I didn't even bother reading the paper because the authors are from country x and there's no point because I know the quality is low. Or the author is a woman, so she should be in the kitchen. I mean, really, really shocking things where I think that any any right minded person would read that and say, that's got nothing to do with culture or or tone. You know, that really is just unacceptable. So so this does happen. And what was really interesting about that study is they didn't just look at the comments that people received, but they looked at the impact that it had on people. And what they found was, men, women, non binary people, people of different different ethnic minority backgrounds were equally likely to report receiving these unprofessional and rude rude comments. And the rate at which everyone reported this was quite high. I'd say quite shockingly high. However, the impact of those comments was different depending on people's backgrounds. And so women and non binary people who were already under represented in STEM were more likely to report that these comments had a negative effect on them, really knocked their confidence. And sometimes they said, you know, it affected my career trajectory. And the same with people ethnic minority backgrounds. And so this is why fundamentally, you know, we know that this is a problem that's not being talked about or tackled enough. Mhmm. And I suppose that's why it's really important as a publisher. We don't want to enable this. We don't want this to be part of our process. Mhmm. So yeah. You know, it's fantastic that you and your colleagues here at IOPP are tackling it. And, listeners, we I'll I'll put a link to that paper in the notes, of the podcast so you can you can take a look at that for yourself. I mean, do you have a feeling, Laura, about why these personal attacks enter, reviews? Is it is it well, maybe laziness isn't the right word. But, you know, are are people just not paying attention to what they're doing, or maybe they find it easier to attack the person rather than attack the science? Or are there are there ulterior motives such as personal animosity that come into play as well? Or are people just not thinking for the most part? I think it's probably all of the above. There's definitely cases where there's personal animosity. I've seen cases where that where that's clearly what's happening. We know that it's so competitive nowadays to be a researcher and a scientist. And, you know, those kinds of conflicts of interest do exist. What we say to people is, if you have a personal animosity towards some authors or if you are essentially competing with those authors directly to try and get some results out, you have a conflict of interest. And if you have a conflict of interest, you shouldn't accept a review invitation. So there's that. There's the online disinhibition effect. There's the online disinhibition effect, which has been studied really broadly on things like social media sites. However, peer review is quite an interesting case study because, reviewers aren't actually anonymous. The editors know who they are. And so there's this moderating factor in the relationship here. You know, the editors can see exactly who's made these comments. But still, it that doesn't seem to stop some people saying these things. Generally speaking, you know, editors will remove these before they get to the authors, but that causes real inefficiencies in the peer review process. You know, it really slows things down. So I think it's a lack of awareness. I think a lot of researchers really, really care about their field. And I genuinely think people get quite worked up about papers that they received to peer review. You know, they really care, and sometimes that comes across in quite a negative way. And again, you know, peer reviewing is a skill. People need to learn how to deliver constructive criticism in a way that is constructive, that has impact for the right reasons, and doesn't, you know, and and doesn't attack the authors. So it's a skill that people are constantly learning. Okay. And I I thought we could talk a bit more actually about anonymity and open processes. The there is a process called double anonymous peer review, whereby the the peer reviewer doesn't know who's written the article, where they're from, you know, anything about them. Maybe they can guess, but, you know, they don't. It's not explicitly told to them. It cannot help in terms of getting people to to, not to go for ad hominem arguments if they if they don't really know who the who the authors are. Does that help? Well, anecdotally, it does. But we don't have any official figures on this. Because again, it's something that's really difficult to track in a in a sort of objective way. Anecdotally speaking to the editorial teams here, it does help. We certainly know the statistics that we have on our double anonymous policy show that it reduces bias. So we do know that it reduces bias in the publishing process. For example, the last year were years worth of data that we have, show that women who choose to anonymize their manuscripts, those manuscripts are 10% more likely to be accepted. And that is due to, you know, reviewers being less biased. Similarly, authors outside of Western Europe, when they anonymize their manuscripts, and importantly, this includes authors in North America. When they anonymize their manuscripts, those manuscripts are also more likely to be accepted. So we know that it reduces bias in this sort of objective way. Does it reduce the number of rude peer review comments? Probably. Interestingly, you say maybe they can guess because this is, this is an issue that we've looked at. We actually ask people. We have a survey that we give out to our peer reviewers. And one of the questions we ask them is, did you review a double anonymous paper? If so, are you, confident that you know which research group or which person wrote this? And if so, person wrote this? And if so, do you want to guess? And so we've recently done an analysis of those guesses because we've got a few years worth of data now. And people are nowhere near as good at guessing as they think they are. So this was some of the, this was one of the things we thought about when we implemented our double anonymous policy is, is it possible to ever properly anonymize a paper? Because often, authors are building on historical work. They have to cite themselves. And also in small fields, you know, everyone knows each other. So everyone might say, oh, that's so and so because there's only 2 groups working on this, and it's not us. So that was one of our concerns, as well as preprint servers. Actually, we found that a lot of, reviewers don't try and look the paper up on preprint servers. We find people are generally quite honest. And secondarily, we've seen a lot of cases where someone is absolutely sure they know who the author of the paper is. And then when we go and check, they're off by an entire continent, you know. So, so that's quite heartening. So we know it's reducing bias, and we know that people can't necessarily figure out who wrote the paper. And Laura, what about transparent peer review where the reviewer report is published openly for everyone to see? I could imagine, you know, there could be some serious shaming going on of a rude inappropriate reviewer when when when that happens. I mean, does that force people to behave for lack of a better way of expressing it. It absolutely does. So spoken to people again, the editorial team internally, anecdotally, everyone agrees that they just haven't seen, sort of rude and unprofessional comments in transparent peer review reports. However, what I will say is the transparent peer review process sort of kicks in at the acceptance stage, really. So, the flip side of that is if a reviewer made very unprofessional or harsh comments, they will probably turn around and say, actually, I don't want you to publish my report. So reviewers choose whether or not to publish their reports. They can opt in or out. We don't, make it compulsory on any of our journals. So there is an element of self selection there. Generally speaking, the reviews of transparent peer review papers are polite and constructive and positive. And I imagine the reviewers are particularly proud of their work and and are happy for it to be published. It's also noteworthy that reviewers can choose whether or not to sign their reports. So they can have their reports published in the transparent peer review, but not have their name next to them. So it's all very complicated, but generally speaking, we think that transparent peer review does nudge people towards submitting more polite and and, constructive reports. Yeah. I see. And it was it I mean, I think it am I right in thinking that transparent peer review is sort of a new thing? And, you know, transparent peer review is sort of a new thing? And if so, was it difficult to get the community on board? Because, you know, I could imagine if you're doing a peer review and and it is negative and it's a perfectly acceptable negative review, you I don't know. You might not want to be associated with it or, you know, as as the sayer of bad things. I mean, what what is it difficult to get people on board with it? Or it is I'd say it can be controversial in certain ways. There are definitely pros and cons. We found that uptake is actually pretty good. Is it new in the grand scheme of peer review of 200 years of peer review? Yes. But other fields and and other publishers and other journals, have been doing this for maybe 10, 15 years on certain journals. When when we adopted transparent peer review, we found the opt in rates for authors are about 50%. So on the journals when transparent peer review is available, 50% of authors say, sure, you publish the reports. Let's go for transparency. Opt in rates are lower for reviewers. For the reports to be published, all 3 people, the author, and usually 2 reviewers need to say yes. So so you can see that there's in general, there's quite a small percentage of cases where all 3 of those parties say yes to publishing their reports. The concern with fully open and transparent peer review where the reviewers' names are published is people worry about, particularly if they're junior researchers, they worry about criticizing someone more senior in their field and the potential for retaliation there. And I suppose that is a legitimate concern. But again, it's completely up to the reviewers whether they publish their reports. And importantly, they don't have to publish the names next to their reports. So Laura, you you've spoken of of a few things that IOPP is doing to improve, the quality of reviews and to make them kinder. Is there anything else that that you and your colleagues are up to? Yeah. We've been trying to kind of innovate this peer review process for a few years now. The peer review process has essentially been pretty much the same for over 200 years, and the scientific landscape and research landscape has changed massively in that time. So we need to do things, that make it easier for for our reviewers and for our authors. One of the things we've done is we've implemented a cut, we've implemented a co review policy. So this is when 2 reviewers, usually a junior person, perhaps even as, you know, a senior PhD student or something like that, can review alongside, much more senior seasoned peer reviewers. So a professor professor or their supervisor. Anecdotally, again, with with co review reports where 2 people have created this report together and they're kind of answerable to each other in the writing of that report, we've noticed that the quality is higher and we see far fewer of these unprofessional and rude comments. So that has really, you know, helped to improve the quality of the peer reviews we're receiving. And it's also a training opportunity for those early career researchers to build this skill and this reviewing muscle. That's really interesting, Laura, because, you know, the reality is that science is a very collaborative thing. Isn't it? And it really, it just seems natural for for 2 people to to collaborate on a peer review. Why not? Mhmm. So yeah. I mean, that that sounds like a really a really interesting idea. And we know the reason we implemented this, is because we know that it's it happens. You know, it's always happened. When you talk to early career researchers, they'll often, express concern or upset that they've done so many peer review reports for their supervisor, and yet their supervisor is, getting the air quotes sort of credit, whatever that might be, the recognition. And so we know it's happening behind the scenes and it's frustrating people. We wanted to bring it out in the open. And the way that our coreview system works, everyone gets recognition. Everyone gets credit. If they get a trusted reviewer certification, they both get the trusted reviewer certification. In fact, we've had cases where 3 people have collaborated. And like you say, it's a natural thing. It makes complete sense. And what we get when junior researchers collaborate with senior researchers is the junior researchers tend to have more time. They're really keen. They're kind of, you know, excited to get involved in the peer review process. And the more senior researcher, you know, really, really knows that stuff around the science. So you get these excellent reports. And Laura, I I also understand that you're offering peer reviewers feedback on their reviews. How how's that going? Yeah. This is an interesting one. We actually rate all of our reviews that we receive on a scale of 1 to 5, with 5 representing kind of an outstanding review. This is something that we've been doing for years as an editorial team, as kind of a signifier of who the who the great reviewers are and to try and give them recognition. And last year, we decided to be much more open and transparent with our reviewing community about this. So now all of our reviewers, when they submit their report, there's a question on the report form which says, do you want to get feedback on your review? And essentially, if they opt in and only if they opt in, that's the important part, they will be told what their review was, how it was evaluated by the editor essentially. And that also includes feedback on how the rating system works and what we expect a 5 out of 5 review to look like. Generally speaking, the feedback we've got from the community about that has been very positive. I think particularly early career researchers appreciate the opportunity to hear back about their reviews. Because so often, they spend a lot of time and effort crafting this review. They hit send, and it just disappears into the ether. And they don't get to find out, you know, how much did I influence the editorial decision? What did the editor think of my comments and my thoughts? So we're now giving them that opportunity. And, again, it's hopefully, you know, improving the overall quality of peer review and also making our reviewers feel more valued. Well, that that's really interesting. I mean, would you would you incorporate, if appropriate, feedback from authors? I mean, if an author said, wow, you know, that review was really good. It, you know, it it improved our paper. It even gave us an idea for future research. You know, can you thank the reviewer? Would that would would could could you incorporate that as well, into that feedback? Yeah. That does happen. You know, it does occasionally happen where an author is really grateful to a reviewer. I think there have been a few isolated cases where the reviewers actually been added in the acknowledgments of a paper because they, you know, that their comments really, really had an impact on the manuscript. It's quite rare. I think if we were to create a system where, authors were giving feedback to reviewers, it would be much more common that the feedback would be negative, unfortunately. And and, you know, it's it's our job as editors and publishers to moderate that relationship. And we think that we are much more objective in the way that we evaluate review reports potentially than authors would be. I'm not sure how appreciative reviewers would be about receiving feedback directly from authors, particularly if it was negative. Mhmm. Yeah. Okay. Well, that's great. Thanks so much for coming on the podcast, Laura. And, and happy peer review week as well. Thank you. Thanks so much for having me. You can find out more about Peer Review Week at peerreviewweek.wordpress.com. And you can browse all of IOPP's journals at iopscience.iop.org. You may have noticed that this podcast has come out on a Tuesday, 2 days earlier than usual. That's because on Thursday, 26th September, we're presenting the Future of Particle Physics. This is a Physics World live event produced in partnership with the journal Reports on Progress in Physics. The live panel discussion will feature Tara Shears, of the University of Liverpool. Phil Burrows, at the University of Oxford. And Talika Bowes, at the University of Wisconsin, Madison. They'll explore what the future holds for high energy physics, and where the next particle collider should be built. You can register now for this free event on the Physics World website. Just click on the Physics World Live tab at the top right of the home page. I'm afraid that's all the time we have for this week's podcast. Thanks to Laura Fietham Walker for joining me today. And, a special thanks to our producer, Fred Iles. We'll be back again next week. Progress in Physics 2024 is a proud supporter of this week's podcast. The 2 day event takes place on the 8th 9th October at the Institute of Physics in London. It will take a deep dive into the exciting world of condensed matter physics, And you can secure your place now to avoid disappointment. Just visit i0p.eventsair.comforward/pip2024.