Thursday, January 3, 2013

The Link Between Leaded Gasoline and Crime


Kevin Drum from Mother Jones has a fascinating new article detailing the hypothesis that exposure to lead, particularly tetraethyl lead (TEL) explains the rise and fall of violent crime rates from the 1960s through the 1990s, after the compound was phased out of gasoline worldwide. It's a good bit of journalism on issues of public health compared to much of what you see, but I'd like to provide a little bit of epidemiology background to the article because there's so many studies listed that it's a really good intro to the types of study designs you'll see in public health. It also illustrates the concept of confirmation bias, and why regulatory agencies seem to drag their feet when we read such compelling stories as this one.

Drum correctly notes that simply looking at the correlation shown in the graph to the right is insufficient to draw any conclusions regarding causality. The investigator, Rick Nevin, was simply looking at associations, and saw that the curves were heavily correlated, as you can quite clearly see. When you look at data involving large populations, such as violent crime rates, and compare with an indirect measure of exposure to some environmental risk factor such as levels of TEL in gasoline during that same time, the best you can say is that your alternative hypothesis of there being an association (null hypothesis always being no association) deserves more investigation. This type of design is called a cross-sectional study, and it's been documented that values for a population do not always match those of individuals when looking at cross-sectional data. This is the ecological fallacy, and it's a serious limitation in these types of studies. Finding a causal link in a complex behavior like violent crime, as opposed to something like a specific disease, with an environmental risk factor is exceptionally difficult, and the burden of proof is very high. We need several additional tests of our hypothesis using different study designs to really turn this into a viable theory. As Drum notes:

During the '70s and '80s, the introduction of the catalytic converter, combined with increasingly stringent Environmental Protection Agency rules, steadily reduced the amount of leaded gasoline used in America, but Reyes discovered that this reduction wasn't uniform. In fact, use of leaded gasoline varied widely among states, and this gave Reyes the opening she needed. If childhood lead exposure really did produce criminal behavior in adults, you'd expect that in states where consumption of leaded gasoline declined slowly, crime would decline slowly too. Conversely, in states where it declined quickly, crime would decline quickly. And that's exactly what she found.

Well that's interesting, so I looked a bit further at Reyes's study. In the study, she estimates prenatal and early childhood exposure to TEL based on population-wide figures, and accounts for potential migration from state to state, as well as other potential causes of violent crime to get a stronger estimate of the effect of TEL alone. After all of this, she found that the fall in TEL levels by state account for a very significant 56% of the reduction in violent crime. Again, though, this is essentially a measure of association on population-level statistics, estimated on the individual-level. It's well-thought out and heavily controlled for other factors, but we still need more than this. Drum goes on to describe significant associations found at the city level in New Orleans. This is pretty good stuff too, but we really need a new type of study, specifically, a study measuring many individuals' exposure to lead, and to follow them over a long period of time to find out what happened to them. This type of design is called a prospective cohort study. Props again to Drum for directly addressing all of this.

The graph title pretty much says it all (Source)
The article continues by discussing a cohort study done by researchers at the University of Cincinnati where 376 children were recruited at birth between 1979 and 1984 to test lead levels in their blood over time and to measure their risk of being arrested in general, and also specifically for violent crime. Ultimately, some of these babies were dropped from the study by the end, and 250 were selected for the results. The researchers found that for each increase of 5 micrograms of lead per deciliter of blood, there was a higher risk for being arrested for a violent crime, but a further look at the numbers shows a more mixed picture than they let on. In prenatal blood lead, this effect was not significant. If these infants were to have no additional risk over the median exposure level among all prenatal infants, the ratio would be 1.0. They found that for their cohort, the risk ratio was 1.34. However, the sample size was small enough where the confidence interval for this rate was as low as 0.88 (paradoxically indicating that additional 5 µg/dl during this period of development would actually be protective), and as high as 2.03. This is not very helpful data for the hypothesis. For early childhood exposure, the risk is 1.30, but the sample size was higher, leading to a tighter confidence interval of 1.03-1.64. It's possible that the real effect is as little as a 3% increase in violent crime arrests, but this is still statistically significant. For 6-year-olds, it's a much more significant 1.48 (95% CI 1.15-1.89)It seems unusual to me that lead would have such a more profound effect the older the child gets, but I need to look into it further. For a quick review of the concept of CI, see my previous post on it. It really matters.  

Obviously, we can't take this a step further into experimental data to enhance the hypothesis. We can't expose some children to lead and not others on purpose to see the direct effects. This is the best we can do, and it's possibly quite meaningful, but perhaps not. There's no way to say with much authority one way or another at this point, not just because of the smallish sample size and the mixed results on significance. Despite an improved study design from cross-sectional studies, a cohort study is still measuring correlations, and we need more than one significant result. More cohort studies just like this, or perhaps done more quickly on previously collected blood samples and looking retrospectively at the connection, are absolutely necessary to draw any conclusion on causality. Right now, this all still amounts to a hypothesis without a clear mechanism for action, although it's a hypothesis that definitely deserves more investigation. There's a number of other studies mentioned in the article showing other negative cognitive and neurological effects that could certainly have an indirect impact on violent crime, such as ADHD, aggressiveness, and low IQ, but that's not going to cut it either. By all means, we should try to make a stronger case for government to actively minimize exposure to lead in children more than we currently do, but we really, really should avoid statements like this:


Needless to say, not every child exposed to lead is destined for a life of crime. Everyone over the age of 40 was probably exposed to too much lead during childhood, and most of us suffered nothing more than a few points of IQ loss. But there were plenty of kids already on the margin, and millions of those kids were pushed over the edge from being merely slow or disruptive to becoming part of a nationwide epidemic of violent crime. Once you understand that, it all becomes blindingly obvious (emphasis mine). Of course massive lead exposure among children of the postwar era led to larger numbers of violent criminals in the '60s and beyond. And of course when that lead was removed in the '70s and '80s, the children of that generation lost those artificially heightened violent tendencies.

Woah. That's, um, a bit overconfident. Still, it's beyond debate that lead can have terrible effects on people, and although there is no real scientific basis for calling this violent crime link closed with such strong language, it's a mostly benign case of confirmation bias, complete with putting blame of inaction on powerful interest groups. His motive is clearly to argue that we can safely add violent crime reduction to the cost-benefit analysis of lead abatement programs paid for by the government. I'd love to, but we just can't do that yet.



The $60B figure seems pretty contrived, but is a generally accepted way to quantify a benefit of removing of neurotoxins in wonk world. The $150B is almost completely contrived, and its very inclusion on the infographic is suspect. I certainly believe that spending $10B on cleaning up lead would be well worth it regardless, and even question the value of a cost-benefit analysis in situations like this, but that doesn't mean I'm willing to more or less pick numbers out of a hat. That's essentially what you're doing if you only have one study that aims to address the ecological fallacy.

The big criticism of appealing to evidence would obviously be that it moves at a snail's pace, and there's a possibility we could be hemming and hawing over and delaying what really is a dire public health threat. Even if that were the case, though, public policy often works at a snail's pace too. If you're going to go after it, you gotta have more than one cohort study and a bunch of cross-sectional associations. Hopefully this gives you a bit more insight onto how regulatory agencies like the EPA look at these issues. If this were to go up in front of them right now I can guarantee you they would not act on the solutions Drum presents based on this evidence, and instead of throwing your hands up, I figure it's better to have an understanding of why that would be the case. It's a bit more calming, at least.

Update: I reworded the discussion on the proper hypothesis of a cross-sectional study to make it more clear. Your initial hypothesis in any cross-sectional study should be that the exposure has no association to the outcome.

Update 2: An edited version of this blog now appears on Discover's Crux blog. I'm amazed to see the response this entry got today, and I can't say enough about how refreshing it is to see Kevin Drum respond and refine his position a little. In my mind, this went exactly how the relationship between journalism and science should work. Perhaps he should have some latitude to make overly strong conclusions if the goal is really to get scientists to seriously look at it.

Update 3: This just keeps going and going! Some good criticism from commenters at Tyler Cowen's blog, as well as Andrew Gelman regarding whether I'm fishing for insignificant values. You can find my responses to each in their comment sections. Perhaps I did focus on the lower bounds of CIs inappropriately, but I think the context makes it clear I'm not claiming there's no evidence, just that I'd like to see replication. In that case, I think it's arguably pretty fair.

Update 4!!! This thing is almost a year old now! I've thought a lot about everything since, and wanted to revisit. Read if ya wanna.

20 comments:

  1. Yes, but part of the problem with your analysis is that you're using frequentist statistics for each study, when what you really need to be doing is using a bayesian analysis. We already know lead is a poison, and we know with a fair degree of accuracy the damage it does to kids brains. That's why it was phased out in the 1970s and 1980s. That's important prior knowledge that you *must* take into account if you're investigating second and third order effects. Your frequentist statistics cannot do this, because they always start with "no relation" as the null hypothesis. But there's plenty of theory and evidence from first order effects that this should not be the null hypothesis.

    In any case, where there is demonstrable harm from an environmental pollutant, the burden of proof should be on those wishing to claim it's safe, not on those investigating the impacts of that harm. So you need to swap your type I and type II error thresholds, and pick more appropriate null hypotheses. Otherwise you're using using a flawed statistical argument to (potentially) slow down implementation of policy that prevents avoidable damage to public health.

    ReplyDelete
    Replies
    1. Steve, that's definitely a fair criticism. As is the case, I think, with a lot of frequentist approaches, it comes out of what we know best. I'm perfectly able to accept that a Bayesian analysis could be more appropriate here, and might totally undermine my whole point. There's obviously good use of the subjective, and there's not so good use of it, but I'm not fluent enough in the analytic tools to do it justice, nor to properly criticize someone who does. Your comment is much appreciated.

      Delete
    2. Even if a Bayesian approach is more appropriate for the research question, it wouldn't address the other weaknesses of the cited studies. Where is the evidence that lower IQ or other neurodevelopmental changes causes violent crime? It's just not there. Another issue that isn't mentioned here is that Nevin leans heavily on The Bell Curve to justify his lead-violent crime hypothesis and that books is pretty much a racist sack of crap. Some of Nevin's arguments in the original publications are also racist.

      Delete
    3. I'd just like to pass this Department of Justice data on:

      http://bjs.ojp.usdoj.gov/content/homicide/race.cfm
      http://bjs.ojp.usdoj.gov/content/homicide/tables/vracetab.cfm

      When you view violent crime data by race, you really get the sense that Nevin has done a bit of fudging to make his point. From the DOJ data, crime is seen on a general and slow decline since 1976, with 2 peaks in the African American data -- one of which begins with the introduction of crack cocaine in 1985.

      If Nevin was willing to conveniently ignore one very obvious stressor in the African American community, I wonder what other stressors he's ignored in favor of lead. And how does the fact that violent crime has been on decline in white communities (thanks to the fact that we have never had to deal with the terrible crap that black communities have had to), despite the rise and fall of airborne environmental lead levels, that should have completely ignored all social boundaries?

      Also, to address question of IQ and lead levels, why not look at SAT data? When you do, you can see pretty clearly that scores tend to ignore peaks in leaded gasoline use.

      Anywho, I'm all for removing any and all environmental toxins. But I'm completely against saddling an entire generation with the stigma of childhood lead poisoning, just to make a point.

      Delete
  2. Loved this. My initial knee-jerk reaction was, "Yeah, that sounds like an interesting hypothesis, and lead abatement is unquestionably a good thing anyway... but I'm not feeling particularly convinced." It's nice to have somebody who actually knows what they are talking about confirm that my gut feeling wasn't too far off.

    ReplyDelete
  3. Yes, Drum goes too far in his "of course" statements.

    I agree with Easterbrook's Bayesian comment above, except the problem is not "frequentist statistics". The problem is black/white thinking. One of your statements also goes too far: "Simply looking at the correlation shown in the graph to the right is insufficient to draw any conclusions regarding causality." ANY conclusions? A reasonable conclusion from the correlation is that it makes the lead-causes-crime hypothesis more plausible. There were plenty of plausible outcomes for these two sets of measurements (measuring lead over a certain set of years and measuring crime over a later set of years) that would have made the lead-causes-crime hypothesis less plausible.

    ReplyDelete
  4. That's really just poor wording on my part. Saying you have evidence that your inference is more plausible is a conclusion, of course, and the proper conclusion as well. I was referring to the conclusion that we've found the cause of ~50% of the variation in crime rates, which I didn't make clear.

    ReplyDelete
  5. "Another issue that isn't mentioned here is that Nevin leans heavily on The Bell Curve to justify his lead-violent crime hypothesis and that books is pretty much a racist sack of crap. Some of Nevin's arguments in the original publications are also racist."

    Oh boy ...

    Any frank discussion of crime statistics is going to make a lot of people hot under the collar like this, because crime in the U.S. is to a sizable extent a black problem. According the Obama Administration's Bureau of Justice Statistics, a slight majority of homicide offenders in the U.S. since the federal government began keeping uniform records in 1976 have been black, despite being about 1/8th of the population. I wouldn't be surprised if some of this high rate of black homicide is due to some urban blacks being more exposed to lead poisoning.

    In particular, in New Orleans, which has often led the country in homicide rate, both before and after Hurricane Katrina, poor blacks tend to live below sea level, so heavy metal toxins can't wash away down to the sea because the Lower 9th Ward is below the sea, so lead and other bad stuff just piles up in the soil.

    ReplyDelete
  6. I spent an evening looking into Rick Nevin and Jessica Wolpaw's lead-caused-crime-wave hypothesis in 2007, and found it more plausible than the celebrated abortion-cut-crime hypothesis in Freakonomics. However, I also identified a number of reality checks that the theory didn't do too well upon (e.g., why no crime wave ever in crowded, car-filled Japan?)

    http://isteve.blogspot.com/2007/07/lead-poisoning-and-great-1960s-freakout.html

    These anomalies shouldn't cause us to reject the theory, but instead to investigate it further.

    ReplyDelete
    Replies
    1. Steve, do you know of any studies into lead blood levels in Japan or the historical market share of leaded to unleaded fuel in Japan, anything along those lines? I did a quick google, and found

      a) this which says that Japan was one of the first countries to respond to the health implications of leaded fuel and phase it out.

      b) this which found the lead blood-levels of 72 Japanese women '93-'95 to be 5.67 μg/dL which is slightly higher than the current average today in the US, but not by much (@ 3μg/dL)

      So you need more than just 'there's a lot of cars in Japan' to call it an anomaly.

      Delete
    2. " this which found the lead blood-levels of 72 Japanese women '93-'95 to be 5.67 μg/dL which is slightly higher than the current average today in the US, but not by much (@ 3μg/dL)"

      Okay, and the crime rate is vastly higher in America today than in Japan at any point in the last 50 years.

      Delete
  7. Most of those who argue for the effect of lead make much of the supposed effect on intelligence, which in turn is supposed to affect criminality*. But they fail to cite population data which would indicate a corresponding rise and fall of intelligence. Rather than direct measures of intelligence they may cite classification of mental retardation, which is subject to various extraneous influences (availability of facilities or monetary support). It seems very odd that something like this would only affect criminality (and maybe even murder and not property crime) and not other psychological properties which would be easier to measure - especially intelligence, which is measured routinely. Surely the intelligence data could be analyzed more closely with respect to broad indicators of lead exposure.

    *Intelligence is supposed to be negatively correlated with intelligence, but do we really know that? We only know the intelligence of criminals who are caught - it is a reasonable hypothesis that the smart ones are not caught.

    ReplyDelete
    Replies
    1. Good point. But is increased violence linked more lower IQ or poor impulse control?

      Delete
  8. I think you are off base when saying that EPA would not go after this issue based on the case Drum makes. Actually if people get excited about this issue because of Drum's article, it makes it more likely that EPA will act, since they are politically influenced. Secondly, there are many many experimental studies on the in vitro and in vivo effects of lead that essentially support these associations. These studies would not be ignored if broad scale lead cleanup was considered. Many highly regulated contaminants have a much smaller body of evidence showing toxicity.

    ReplyDelete
    Replies
    1. That's certainly up for debate, and I'm glad you brought it up. It's really the context for why I posted this to begin with. In a more sane political climate, I'd probably agree with you, but after seeing what went down with the EPA's mercury rules, which have been over 20 years in the making (introduced by the George HW Bush administration), plus President Obama nixing tighter ozone regulations, I'm going to have to say that it's nearly impossible for me to see the necessary political pressure happening, especially when the uncertainties involved provide a justifiable skepticism.

      Right now this is story that appeals very strongly to progressives, and it should, but the dialogue between progressives and the President obviously is strained a bit, not just on the environment. Also, we're not talking about regulations to reduce exposure so much as a pretty unprecedented national $20B remediation project that goes beyond setting up Superfund sites.

      I'm quite optimistic over the long-term in these sorts of things, but not so much in the near-term. Kevin Drum is doing a great job advocating the issue, and I know he'll continue to do a great job. It's become clear as he's responded to my criticism that I may have misjudged his motivation for the article. He really wants this to be a higher priority for researchers, and so do I. If the political will to do something now happens out of it, fantastic, seriously, but I don't see it.

      Delete
    2. Good points; hopefully the Obama administration pays more attention to these issues this term. As you point out the funding issue is problematic though.

      Delete
  9. I'm not suggesting that lead should remain in our environment, but I do have lots of questions. For example, would someone care to explain why, if lead exposure peaked in 1970/71, SAT scores declined dramatically between 1966 and 1981, and increased from 1981 onward? What about the Flynn Effect, and the fact that IQ scores continued to rise up until the late '90s, when we started seeing signs of them leveling off? And why don't we see a dip in college entry rates in the early '90s (or do we)? And most interesting to me, what effect have the introduction of anti-depressants and anti-anxiety medications into our water supply starting in the late '80s, had on crime rates?

    I'm all for eliminating environmental lead, but suggesting that kids born in the '60s and '70s are essentially all lead poisoned, and therefore 'broken' is ludicrous, especially when you look at their testing/educational data.

    ReplyDelete
  10. Without looking it up, just spitballing some problems with using the SAT scores - I'm thinking SAT scores are not a good measure for this, because there's going to a be a huge overlap between children at highest risk for lead exposure (not just from gas, but from other sources) and those children who never take the SAT in the first place. Using SAT, you're losing a great number of kids who may have high levels of exposure but who either drop out of high school or graduate high school but have no prospects for college or $ for tests and applications, so never take the test. Relatedly, I grew up in a low income area of Appalachia not known for environmental protections, but the ACT was the usual test there for kids with any college prospects - everybody else took the ASVAB.

    ReplyDelete
  11. Thanks for this blog I will be certainly following you in the future as I do accept your proposal that informatics is becoming essential for anyone seriously following media coverage, particularly as it relates to health issues.
    As an aside concerning your questions surrounding violence and lead poisoning in children growing worse with age, you will want to particularly research neurological recovery in children and adults. While we have overall been surprised by the resilience of the human brain in studies over the past 20 years the resilience of children in adaptation and the brain's self repair should be of particular interest to this study.

    ReplyDelete
    Replies
    1. Thanks for your comment, Carl. I still haven't bothered to look too much into the neuroscience aspect. I have, however, had a couple of months to marinate on the overall argument I made. One larger theme of this blog is showing how the study designs we use to assess environmental risks don't really have the capacity to give the public the answers they need in isolation. This is especially the case here when we're talking about teasing out a potential effect of 30%-50%.

      This is very different than how researchers used the same tools to determine causation in smoking and lung cancer, for instance. The effect there was ~1000x greater, and even then, they still had to reproduce the findings. That's just how it works.

      We've firmly established some extremely harmful effects of lead, and that there's no safe level of exposure. My concern is that throwing in non-established effects as if they are known or "blindly obvious" could undermine the argument to remediate lead pollution. If you want to mention the link to violent crime, it would go a long way to be up front about the uncertainties, at least. Perhaps a shift to more Bayesian analysis can help get to these answers quicker, but that brings with it its own set of problems.

      Delete