Is Most Published Research Wrong?
- Thread starter Winston
- Start date
Help Support The Rocketry Forum:
- Joined
- Feb 3, 2012
- Messages
- 6,339
- Reaction score
- 5,537
Somebody needs to research this...
Most of the published literature I have tried to reproduce has worked, probably 60% success rate when applied to my particular task. Of the 40% that didn't, I suspect most of that is the reported reaction (I'm an organic chemist, so reactions is what I mainly use the literature for) wasn't close enough to my substrate to work (~30%), there is some tiny detail missing in the prep or my technique isn't good enough to pull it off (~9%), or the report was total flaming BS (<=1%). I do know that twice I suffered from wasted time working from a blatantly made up result. Both of those people were drug into the light by the scientific community, the papers were retracted, and they couldn't work as janitors in a research lab anywhere in the world now. I can also swear that every scientific paper, book and patent with my name on works EXACTLY like we reported. Of course, I'm a physical scientist...natural sciences and "social sciences" are entirely different animals.
Cabernut
Well-Known Member
- Joined
- Oct 15, 2015
- Messages
- 1,384
- Reaction score
- 9
Doesn't matter if it's wrong, as long as it what your Funding Source is looking for.
Scientists have bills to pay too.
The hostility these days towards skeptics that don't go with the mainstream scientific community is appalling. Agree with the collective and you get your funding.
Scientists have bills to pay too.
The hostility these days towards skeptics that don't go with the mainstream scientific community is appalling. Agree with the collective and you get your funding.
BDB
Absent Minded Professor
- Joined
- Aug 22, 2015
- Messages
- 2,556
- Reaction score
- 938
I agree completely with Cl(VII), but we are both organic chemists, so the sampling for this data set is definitely skewed.
When I was a postdoc a grad student in my lab falsified a bunch of data. We ended up retracting 7 papers that were published in top-tier peer-reviewed journals. It was a really big deal in the organic community. There were stories about it in Nature, Science and even the ​New York Times. But the good news is that she tried to cheat, but got caught. The system isn't perfect, but it works--at least in the physical sciences.
When I was a postdoc a grad student in my lab falsified a bunch of data. We ended up retracting 7 papers that were published in top-tier peer-reviewed journals. It was a really big deal in the organic community. There were stories about it in Nature, Science and even the ​New York Times. But the good news is that she tried to cheat, but got caught. The system isn't perfect, but it works--at least in the physical sciences.
- Joined
- Feb 3, 2012
- Messages
- 6,339
- Reaction score
- 5,537
If this thread goes the way I think it might, it may be a candidate for publishing in the Journal of Irreproducible Results...
Did you work for Sames? Because that was one of the two incidences where I chased false results. Fortunately, a former grad student in our lab was a post doc with Sames at the time, so when my PI called to ask him what we were doing wrong he said to stop trying, and we would know why shortly.
Zeus-cat
Well-Known Member
- Joined
- Mar 14, 2009
- Messages
- 4,987
- Reaction score
- 1,439
Publishing research results is not to state that someone has found proof of something. It is to say that we studied this thing and there appears to be a correlation. Other people should study it to and verify what we found. If enough people replicate our results and don't find a problem with our data or methods then we may have found something.
A study for weight loss with 5 people in each of three groups is not a large enough sample size for a valid study. How did this get published??? Where was it published? The Hershey Foundation for Weight Loss? The Oreo Center to Prevent Skinniness?
A study for weight loss with 5 people in each of three groups is not a large enough sample size for a valid study. How did this get published??? Where was it published? The Hershey Foundation for Weight Loss? The Oreo Center to Prevent Skinniness?
BDB
Absent Minded Professor
- Joined
- Aug 22, 2015
- Messages
- 2,556
- Reaction score
- 938
Yep. That's the scandal. Sorry you wasted time trying to reproduce her work. I'm sure I know the postdoc who you are referring to. Luckily I worked on a different project in the group, and I got a tenure-track job and left Columbia right as everything started to come out. I didn't have to spend months trying to reproduce her bogus results, but I did get to participate in a crazy-tense group meeting before I left.
Wow, small (chemistry) world. I didn't spend too much time on that one, I was the second set of hands when the first guy had trouble. When neither of us could get even a GC trace of product the flags went up. Silas did save us more head pounding on hood, or at least Ross beating on us, when he gave us the stop trying warning.
Bet that group meeting ended with people dispersed to varying bars, yikes!
Like I said earlier though. The scientific community raised concerns, the liar's own group publicly disproved her BS, the papers were retracted, and the scientific world moves on without her. The system worked.
Winston
Lorenzo von Matterhorn
- Joined
- Jan 31, 2009
- Messages
- 9,560
- Reaction score
- 1,748
Har! Lots of nuances, but the bottom line is too great of a confidence in statistical rules of thumb supposedly signifying statistically significant results both by authors as well as the reviewers at science journals. He didn't even mention journals that don't even QC check the studies they publish (typically on-line only) or those that even CHARGE to publish papers.
BDB
Absent Minded Professor
- Joined
- Aug 22, 2015
- Messages
- 2,556
- Reaction score
- 938
The more I think about this thread, the more I think the kind of work that Chris [Cl(VII)] and I do is pretty different from the kind of work that is described in this video, and I'm not trying to say that my stuff doesn't stink.
We make things, they just happen to be the size of molecules, but they are real things that people use. It's like one of us claiming to have a new, revolutionary design for a rocket and then posting about it here. Surely one of us would try to replicate the work, and if it was total BS, we would report that too. The product of our science is a physical object, so it is difficulty to misrepresent.
The problems arise when someone works in a field that relies on statistics. Our field is often derided for not publishing data with error bars, but maybe that's the point. A homebuilder doesn't need error bars when he builds a house because it is a physical thing. But when the final "product" of the study is a graph, beware.
We make things, they just happen to be the size of molecules, but they are real things that people use. It's like one of us claiming to have a new, revolutionary design for a rocket and then posting about it here. Surely one of us would try to replicate the work, and if it was total BS, we would report that too. The product of our science is a physical object, so it is difficulty to misrepresent.
The problems arise when someone works in a field that relies on statistics. Our field is often derided for not publishing data with error bars, but maybe that's the point. A homebuilder doesn't need error bars when he builds a house because it is a physical thing. But when the final "product" of the study is a graph, beware.
