Quantcast

Is Most Published Research Wrong?

The Rocketry Forum

Help Support The Rocketry Forum:

Winston

Lorenzo von Matterhorn
Joined
Jan 31, 2009
Messages
9,050
Reaction score
1,160
Wow...

[video=youtube;42QuXLucH3Q]https://www.youtube.com/watch?v=42QuXLucH3Q[/video]
 

Bat-mite

Rocketeer in MD
Joined
Dec 5, 2013
Messages
10,986
Reaction score
1,738
Location
Maryland
Can't watch videos at work and home PC is down for repairs. Care to summarize?
 

cerving

Owner, Eggtimer Rocketry
TRF Sponsor
TRF Supporter
Joined
Feb 3, 2012
Messages
4,162
Reaction score
1,436
Somebody needs to research this...
 

Cl(VII)

Chris Bender, Lab Rat
TRF Sponsor
Joined
Feb 22, 2013
Messages
4,595
Reaction score
834
Location
Garland, TX
Most of the published literature I have tried to reproduce has worked, probably 60% success rate when applied to my particular task. Of the 40% that didn't, I suspect most of that is the reported reaction (I'm an organic chemist, so reactions is what I mainly use the literature for) wasn't close enough to my substrate to work (~30%), there is some tiny detail missing in the prep or my technique isn't good enough to pull it off (~9%), or the report was total flaming BS (<=1%). I do know that twice I suffered from wasted time working from a blatantly made up result. Both of those people were drug into the light by the scientific community, the papers were retracted, and they couldn't work as janitors in a research lab anywhere in the world now. I can also swear that every scientific paper, book and patent with my name on works EXACTLY like we reported. Of course, I'm a physical scientist...natural sciences and "social sciences" are entirely different animals.
 

Cabernut

Well-Known Member
Joined
Oct 15, 2015
Messages
1,384
Reaction score
2
Doesn't matter if it's wrong, as long as it what your Funding Source is looking for.

Scientists have bills to pay too.

The hostility these days towards skeptics that don't go with the mainstream scientific community is appalling. Agree with the collective and you get your funding.
 

BDB

Absent Minded Professor
Joined
Aug 22, 2015
Messages
2,184
Reaction score
417
I agree completely with Cl(VII), but we are both organic chemists, so the sampling for this data set is definitely skewed.

When I was a postdoc a grad student in my lab falsified a bunch of data. We ended up retracting 7 papers that were published in top-tier peer-reviewed journals. It was a really big deal in the organic community. There were stories about it in Nature, Science and even the &#8203;New York Times. But the good news is that she tried to cheat, but got caught. The system isn't perfect, but it works--at least in the physical sciences.
 

cerving

Owner, Eggtimer Rocketry
TRF Sponsor
TRF Supporter
Joined
Feb 3, 2012
Messages
4,162
Reaction score
1,436
If this thread goes the way I think it might, it may be a candidate for publishing in the Journal of Irreproducible Results...
 

Cl(VII)

Chris Bender, Lab Rat
TRF Sponsor
Joined
Feb 22, 2013
Messages
4,595
Reaction score
834
Location
Garland, TX
I agree completely with Cl(VII), but we are both organic chemists, so the sampling for this data set is definitely skewed.

When I was a postdoc a grad student in my lab falsified a bunch of data. We ended up retracting 7 papers that were published in top-tier peer-reviewed journals. It was a really big deal in the organic community. There were stories about it in Nature, Science and even the &#8203;New York Times. But the good news is that she tried to cheat, but got caught. The system isn't perfect, but it works--at least in the physical sciences.
Did you work for Sames? Because that was one of the two incidences where I chased false results. Fortunately, a former grad student in our lab was a post doc with Sames at the time, so when my PI called to ask him what we were doing wrong he said to stop trying, and we would know why shortly.
 

Zeus-cat

Well-Known Member
Joined
Mar 14, 2009
Messages
4,592
Reaction score
796
Publishing research results is not to state that someone has found proof of something. It is to say that we studied this thing and there appears to be a correlation. Other people should study it to and verify what we found. If enough people replicate our results and don't find a problem with our data or methods then we may have found something.

A study for weight loss with 5 people in each of three groups is not a large enough sample size for a valid study. How did this get published??? Where was it published? The Hershey Foundation for Weight Loss? The Oreo Center to Prevent Skinniness?
 

BDB

Absent Minded Professor
Joined
Aug 22, 2015
Messages
2,184
Reaction score
417
Did you work for Sames? Because that was one of the two incidences where I chased false results. Fortunately, a former grad student in our lab was a post doc with Sames at the time, so when my PI called to ask him what we were doing wrong he said to stop trying, and we would know why shortly.
Yep. That's the scandal. Sorry you wasted time trying to reproduce her work. I'm sure I know the postdoc who you are referring to. Luckily I worked on a different project in the group, and I got a tenure-track job and left Columbia right as everything started to come out. I didn't have to spend months trying to reproduce her bogus results, but I did get to participate in a crazy-tense group meeting before I left.
 

Cl(VII)

Chris Bender, Lab Rat
TRF Sponsor
Joined
Feb 22, 2013
Messages
4,595
Reaction score
834
Location
Garland, TX
Yep. That's the scandal. Sorry you wasted time trying to reproduce her work. I'm sure I know the postdoc who you are referring to. Luckily I worked on a different project in the group, and I got a tenure-track job and left Columbia right as everything started to come out. I didn't have to spend months trying to reproduce her bogus results, but I did get to participate in a crazy-tense group meeting before I left.
Wow, small (chemistry) world. I didn't spend too much time on that one, I was the second set of hands when the first guy had trouble. When neither of us could get even a GC trace of product the flags went up. Silas did save us more head pounding on hood, or at least Ross beating on us, when he gave us the stop trying warning.

Bet that group meeting ended with people dispersed to varying bars, yikes!

Like I said earlier though. The scientific community raised concerns, the liar's own group publicly disproved her BS, the papers were retracted, and the scientific world moves on without her. The system worked.
 

Winston

Lorenzo von Matterhorn
Joined
Jan 31, 2009
Messages
9,050
Reaction score
1,160
Can't watch videos at work and home PC is down for repairs. Care to summarize?
Har! Lots of nuances, but the bottom line is too great of a confidence in statistical rules of thumb supposedly signifying statistically significant results both by authors as well as the reviewers at science journals. He didn't even mention journals that don't even QC check the studies they publish (typically on-line only) or those that even CHARGE to publish papers.
 

BDB

Absent Minded Professor
Joined
Aug 22, 2015
Messages
2,184
Reaction score
417
The more I think about this thread, the more I think the kind of work that Chris [Cl(VII)] and I do is pretty different from the kind of work that is described in this video, and I'm not trying to say that my stuff doesn't stink.

We make things, they just happen to be the size of molecules, but they are real things that people use. It's like one of us claiming to have a new, revolutionary design for a rocket and then posting about it here. Surely one of us would try to replicate the work, and if it was total BS, we would report that too. The product of our science is a physical object, so it is difficulty to misrepresent.

The problems arise when someone works in a field that relies on statistics. Our field is often derided for not publishing data with error bars, but maybe that's the point. A homebuilder doesn't need error bars when he builds a house because it is a physical thing. But when the final "product" of the study is a graph, beware.
 

BDB

Absent Minded Professor
Joined
Aug 22, 2015
Messages
2,184
Reaction score
417
I know, I know. I'm probably beating a dead horse by now. That video has just been banging around in my head all week.

readImage.jpeg
 
Top