On Getting Near The Center of the Bullseye On The First Try.
Quickly and With Some Degree of Confidence
[Editor’s note: This is a popular essay from my old blog, that I wanted to bring home to FAFDL.]
One of the things that keeps me busy is administering and moderating a Facebook group called Food and Farm Discussion Lab and helping to moderate GMO SkeptiForum. This has led to a steady stream of requests for help find or evaluate information and evidence, often on things I know little about (i.e. vaccines, Morgellons). If I can, I try to see if I can get a beginner’s handle on the subject on the fly. The hitch is that I want to do this in a responsible way.
Last night I got a message from someone wanting help getting a discussion thread going to find out more about the impact of pesticide exposure on cancer rates for farmers. This morning somebody wanted to know what I knew about pesticides and water contamination. On both questions, I had read things in passing, but they weren’t issues where I have a confident handle on the topic. The member who approached me about farmers and cancer didn’t feel comfortable initiating the conversation, so I took on the task of getting the discussion rolling on cancer rates and farmer/farmworker pesticide exposure. I expect to become better informed as the discussion matures, but I wanted to start with a firm footing. I didn’t want to spend more than 20 minutes getting oriented, but I wanted to start with solid sources.
Here is how I approached that problem. It’s how I approach beginning to learn about any technical issue with non-obvious answers.
When looking at the technical literature, I start with recent broad reads. This means literature reviews and meta-analyses from the last five years. If I can’t find lit reviews or meta-analyses I move on to the largest, most robust studies I can find. On this topic there are not going to be RCT’s (random controlled trials), we’ll be looking at epidemiology. That means we are looking for longitudinal cohort studies with the largest cohorts over the longest period of time.
A literature review is just what it sounds like. Researchers systematically find all the relevant literature, they read it, they weigh it all out and then they try to fairly summarize what it all means and what conclusions can be drawn from the best available evidence at the time. To over-simplify, a meta-analysis is done by gathering all the well conducted studies on a given topic and trying to combine the data into a single larger data set from which more robust conclusions can be drawn. A longitudinal cohort study follows a group of people over time, trying to accurately record data on a number of variables and then crunch the data to see if their are significant correlations that correspond to credible hypotheses.
If I’m new to the topic, then I will not have the chops to evaluate single studies. I don’t know what variables aren’t being taken into account. I will not have an eye for possible confounders. I may have prejudices that affect which single studies I give more weight to. The first studies I look at will cause have a first impression effect that can be hard to shake.  Cognitive psychologists call this anchor bias. If I start with an poorly conducted or unrepresentative study, that could bias me in unproductive ways. That’s why I try to stick to lit reviews and meta-analyses.
In this case, I have previously read that farmers have lower cancer rates than the rest of the population.  My union organizer loyalties tell me that farmworkers are more likely to be getting screwed when it comes to exposure and proper handling of pesticides. Those things may or may not be true, but I’m inclined to believe them and I don’t need Daniel Kahneman to tell me that if I dive into a stack of conflicting small studies, then my confirmation bias will almost certainly lead me to those two conclusions because of anchor bias and tribal loyalties.
Before I dive into the technical literature, I try to see if I can get the lay of the land first.
I’m going to let you in on a nifty google hack for separating the wheat from the chafe and find reliable answers to difficult questions. We are going to limit our search to university websites. That means when we do a search on “farmers cancer pesticides” we will be looking almost entirely at results from university ag school and ag extension pages. This could spare us hundreds of pages of irresponsible headlines and poor reporting on the subject. Instead we will find pages soberly summarizing what is known about the subject, put together by professors or more likely grad students with a background on the subject. A big step up from your average health reporter.
This how you do it. Enter your search term into the search box followed by site:edu. This is a variation on site search which allows you to search single website. In this case our search looks like this:
I clicked around and found a few pages that gave me an overview. Poke around, see what you find.
3. Now that I’ve read some summaries by academics, I’m ready to look at the research. I go to Google Scholar and enter “farmers cancer pesticides” I give a quick scan to the results and see that they are mostly from the 90’s. I know I don’t want to rely on those studies, because I know that pesticides have changed drastically in the last few decades. More importantly (and universally) I don’t know the body of literature, so I have no way of knowing if the results of those old studies have since been replicated, overturned, withdrawn. retracted, etc. The other advantage recent research offers me is that papers summarize the relavant literature, so one of my next steps, if I decide I really want to dig into the topic is to start ransacking the footnotes for further study. Recent papers will give me a better map of the literature.
So, in the interest of speed, instead of custom defining a five year window, I click on the box to the left and redefine my search to “Since 2010”. Scanning down through the first thirty results, three look promising.
A Review of Pesticide Exposure and Cancer Incidence in the Agricultural Health Study Cohort
We reviewed epidemiologic evidence related to occupational pesticide exposures and cancer incidence in the Agricultural Health Study (AHS) cohort.
Data sources:Studies were identified from the AHS publication list available at http://aghealth.nci.nih.gov as well as through a Medline/PubMed database search in March 2009. We also examined citation lists. Findings related to lifetime-days and/or intensity-weighted lifetime-days of pesticide use are the primary focus of this review, because these measures allow for the evaluation of potential exposure–response relationships.
Data synthesis: We reviewed 28 studies; most of the 32 pesticides examined were not strongly associated with cancer incidence in pesticide applicators. Increased rate ratios (or odds ratios) and positive exposure–response patterns were reported for 12 pesticides currently registered in Canada and/or the United States (alachlor, aldicarb, carbaryl, chlorpyrifos, diazinon, dicamba, S-ethyl-N,N-dipropylthiocarbamate, imazethapyr, metolachlor, pendimethalin, permethrin, trifluralin).
Mortality in the Agricultural Health Study, 1993–2007
Comparing agricultural cohorts with the general population is challenging because the general healthiness of farmers may mask potential adverse health effects of farming. Using data from the Agricultural Health Study, a cohort of 89,656 pesticide applicators and their spouses (N = 89, 656) in North Carolina and Iowa, the authors computed standardized mortality ratios (SMRs) comparing deaths from time of the enrollment (1993–1997) through 2007 to state-specific rates.
Objective: To systematically evaluate epidemiologic studies on pesticides and colon cancer and rectal cancer in agricultural pesticide applicator populations using a transparent “weight-of-evidence” (WOE) methodological approach.
Methods: Twenty-nine (29) publications from the Agricultural Health Study (AHS) and 13 additional epidemiologic studies were identified that reported data for pesticide applicators and/or specific pesticide compounds and colorectal, colon, or rectal cancer. The AHS evaluated pesticide applicators as well as dose–response associations for specific pesticide compounds, whereas the large majority of non-AHS evaluated applicators but did not analyze specific compounds or dose–response trends. This WOE assessment of 153 different pesticide–outcome pairs emphasized several key evidentiary features: existence of statistically significant relative risks, magnitude of observed associations, results from the most reliable exposure assessments, and evidence of convincing dose–response relationships (i.e., those monotonically increasing, with statistically significant trend tests).
There is one more that is about kids, not farmers but it catches my eye as a pretty robust study, so I flag that as well: Exposure to pesticides and risk of childhood cancer: a meta-analysis of recent epidemiological studies
Now, I feel like I have a firm footing to start from. Will it be the last word? No. Is it guaranteed to get me to the right answer? No. But by and large I think this approach points me in the right direction better than 9 out of 10 times. This is how I do a quick and dirty search. There are a two dozen limitations to this approach, but my goal isn’t to master the topic in an hour. I just want to be confident that I’m heading in the right direction and won’t embarrass myself in public.
I want to make Google my bitch. Not the other way around.
How I approach a deeper dive will have to wait for another day.
