Ugh, this hits me hard! Working on a research organism that isn't one of the established ones, I have always had to do tool development myself, I actually enjoy the tinkering and wish we could have more freedom to do that. I have been trying to balance things for my trainees by getting them started on both risky and bread-and-butter projects. One way I proactively help them reframe the "fails" is that "success is measured by whether you are trying new things and troubleshooting, not by positive results". But, I also tell them, we need papers, and we should check in often about the progress to decide whether we should move onto something else. It's always an ongoing reassess and adjust.
It sounds like you are handling it as well as you can--having a back up project already in the hopper for my student was key to getting them graduated. But it is so frustrating when you know how much the focus on publishing in our current system takes away from discovery and training, isn't it?
I wonder if preprint servers like bioRXiv, now used so much more commonly, can help to bridge that gap. Writing up a pile of rigorously performed experiments but negative data in anticipation of peer reviewers inevitably asking for yet more experiments is enough to make even the most enthusiastic trainee (or PI) throw up their hands. Getting this info out as a a preprint at least gets it into the public record, although it doesn't quite as well address your main point of risk to the trainee. Originally it seemed like the idea of journals like PLoS One or some of the solid society journals was that people could publish stories that were technically rigorous but didn't necessarily come to a flashy conclusion. In practice, even for these journals, reviewers still sometimes ask for more experiments to expand the scope rather just assessing the rigor of the experiments presented.
The problem is, it still takes so much time and effort to even post a preprint, which means you had to write a whole manuscript with figures, intro etc. I don't think the current system still incentivizes that.
Definitely! I agree that preprints help reduce the barrier to getting negative results out into the world, as do sound science journals! We used that approach a few times with our society journal, Plant Direct, and had a good experience, but I’ve definitely heard the horror stories.d
Our obsession with “positive” results is harmful to the scientific enterprise. Most studies “fail” (see what I did there?) to reject the null hypothesis. Most journals favor studies with significant main effects, so meta-analyses will over-estimate the strength of an effect because most of the non-significant studies were rejected. Add to that the drug companies that don’t even submit studies that don’t “prove” their drug works, and you have a real distortion of biomedical science. Not to mention the effect on PI’s and their grad students about taking on a “risky” study and the fear of harm to their careers as you mention. We don’t have a “replicability” crisis in science. We have an “ego” problem!
Thanks for sharing this, Liz. I concur -- some of my biggest regrets when it comes to training and mentoring relate to cases where either I didn't fully think through or didn't properly communicate the risks for a trainee's career.
Ugh, this hits me hard! Working on a research organism that isn't one of the established ones, I have always had to do tool development myself, I actually enjoy the tinkering and wish we could have more freedom to do that. I have been trying to balance things for my trainees by getting them started on both risky and bread-and-butter projects. One way I proactively help them reframe the "fails" is that "success is measured by whether you are trying new things and troubleshooting, not by positive results". But, I also tell them, we need papers, and we should check in often about the progress to decide whether we should move onto something else. It's always an ongoing reassess and adjust.
It sounds like you are handling it as well as you can--having a back up project already in the hopper for my student was key to getting them graduated. But it is so frustrating when you know how much the focus on publishing in our current system takes away from discovery and training, isn't it?
I wonder if preprint servers like bioRXiv, now used so much more commonly, can help to bridge that gap. Writing up a pile of rigorously performed experiments but negative data in anticipation of peer reviewers inevitably asking for yet more experiments is enough to make even the most enthusiastic trainee (or PI) throw up their hands. Getting this info out as a a preprint at least gets it into the public record, although it doesn't quite as well address your main point of risk to the trainee. Originally it seemed like the idea of journals like PLoS One or some of the solid society journals was that people could publish stories that were technically rigorous but didn't necessarily come to a flashy conclusion. In practice, even for these journals, reviewers still sometimes ask for more experiments to expand the scope rather just assessing the rigor of the experiments presented.
The problem is, it still takes so much time and effort to even post a preprint, which means you had to write a whole manuscript with figures, intro etc. I don't think the current system still incentivizes that.
Exactly. And the value for a trainee is so small compared to what a published paper with an "exciting conclusion" would be.
Definitely! I agree that preprints help reduce the barrier to getting negative results out into the world, as do sound science journals! We used that approach a few times with our society journal, Plant Direct, and had a good experience, but I’ve definitely heard the horror stories.d
Well done! Clever - and though the science was foreign to my ears the lesson is a good one and the illustration apt!! Thanks!
Thank you, Beth!
Our obsession with “positive” results is harmful to the scientific enterprise. Most studies “fail” (see what I did there?) to reject the null hypothesis. Most journals favor studies with significant main effects, so meta-analyses will over-estimate the strength of an effect because most of the non-significant studies were rejected. Add to that the drug companies that don’t even submit studies that don’t “prove” their drug works, and you have a real distortion of biomedical science. Not to mention the effect on PI’s and their grad students about taking on a “risky” study and the fear of harm to their careers as you mention. We don’t have a “replicability” crisis in science. We have an “ego” problem!
If we could just shift the burden of risk to the scientific community instead of making individuals carry it . . .
Thanks for sharing this, Liz. I concur -- some of my biggest regrets when it comes to training and mentoring relate to cases where either I didn't fully think through or didn't properly communicate the risks for a trainee's career.
Good to know I'm not alone there!