What struck me first about Okbay et al. 2016 was the sheer number of people who had worked on it–2 pages worth of authors and affiliations! Now, having lots of authors can be great because:
1) they can collect TONS of data
b) they bring together complementary expertise/resources
BUT science is expensive and there are limited resources ($$$, time, lab space, research positions, etc.) to go around so we need think about how we’re allocating those precious dollars.
So let’s decide whether the extraordinary amount of money that must have been spent on this study was worth it.
This article was called Genome-wide association study identifies 74 loci associated with educational attainment. What was this study about?
These biologists were looking for sequences in the genome that varied between those individuals that had little education and those who had finished many years of school. These sequence changes are called single nucleotide polymorphisms or SNPs (pronounced “snips”).
Say hypothetically that the scientists had found a gene in which most people who had a PhD had a T in one specific spot whereas most people who hadn’t finished high school had a G. This would be great evidence that that gene affects educational success and that changing that specific letter in the gene affects something in the body (probably the brain) that drives an individual along their educational path.
The authors identified 74 gene sequence changes that seemed to make a difference for education. On its face, that’s quite exciting! But how much can we actually predict by knowing what sequences someone has in these 74 genes?
What did these authors actually look at? They were testing a variable called educational attainment, which is number of years of schooling. They found that together the 74 genes sequences can predict about 3-9 weeks worth of educational attainment.
This is troubling for at least 2 reasons:
1) Number of years of schooling is only one measure of educational success. Did these students learn? Did they remember and apply the knowledge? Did they think critically and learn to communicate effectively?
2) 3-9 weeks is not a meaningful amount of time. If we care about education, it’s because people can achieve more with more education. And sure, people with a college diploma tend to do better than people who didn’t graduate high school. But is 9 weeks (which was their upper range!) going to make a substantial difference in one’s earnings/meaningfulness/ability to give back to society?
And sure, if there was a single gene that predicted 3-9 weeks worth of achievement, fine, perhaps that would be something interesting. But these are 74 genes that you have to combine to even see this measly effect.
Other potential problems (a couple extra are described in the video):
1) This study was done on white guys, all over 30 years old. Let’s just set aside for a minute the fact that this is in no way representative of the general population. Instead, as Bill Nye (who I once hugged!) would say:
When we think about education, we’re trying to help the next generation. So how relevant are these findings to a population that grew up in a totally different environment? (just one example: Today’s kids are born holding a smartphone and that access to technology is totally changing education!)
2) These findings are statistically significant but not meaningful. This is a complicated topic that you can learn more about here, but the takeaway is that the statistics here say that while these findings are likely reliable (=you’ll see them over and over again if you keep testing the same question), they don’t actually matter. That is, the independent variable only predicts a small amount (~3%) of the dependent variable. (And that’s not even getting into the fact that this is a correlative study not a causative one, so the words “independent” and “dependent” are pretty questionable.)
3) The authors found that the implicated genes are mostly involved with neural development, but there’s no indication of whether that was their hypothesis or whether they just tested all the possible pathways and only showed us the ones involved in neural development? Deciding ahead of time what hypotheses to test is important for avoid introducing common statistical biases, and we don’t know if the authors did that.
4) If it’s true that this collection of genes makes a difference for how the brain develops, how is it that this difference only changes educational attainment by a few months? How are cells changing their function in a way that has such a small effect?
Are you convinced that 9 weeks worth of extra education is something to get excited about? To pile this many resources into? To publish in Nature, which is a big deal?
And here’s the catch: it was already known going into this study that >80% of educational success is predicted by nurture, not nature. So why aren’t we pouring more resources into understanding and solving the environmental factors that affect education in a huge way? Especially since we’re going to be hard-pressed to change people’s genes large-scale (even if we wanted to), but we can do a hell of a lot to change their environment.
Please chime in with thoughts, corrections, criticisms, or thoughts about:
a) educational studies
b) allocation of resources
c) publishing in Nature
d) nature vs. nurture
f) anything else!