Aug 18, 2009

News

Send

Meta-studies for biosafety reserach

"It’s not surprising that different studies arrive at contradictory results."

In biosafety studies that address genetically modified plants, a recurring problem is that individual studies deliver contradictory results. This makes risk assessment more difficult. In the field of medical research, which is confronted with the same problem, ‘meta-studies’ have gained importance in the past years. These combine the results of numerous studies and achieve thereby a level of validity that is significantly higher than is the case with individual studies. GMO Safety spoke with Michelle Marvier from the University of Santa Clara (USA), who has conducted meta-studies that address the ecological effects of Bt maize and Bt cotton. She recommends the establishment of central and publicly-accessible databases on biosafety research and that meta-studies should be routine in the safety assessment of genetically modified plants.

GMO Safety: Scientific studies seem often to be contradictory. One study reveals that a certain GM crop is harming a non-target organism, another study excludes this. How does this happen? Is this caused by flawed science and inappropriate statistics or by biased researchers?

Michelle Marvier: Actually, given the small sample sizes and high variation that characterise these types of risk assessment studies, it’s not surprising at all that different studies yield contradictory results. Many of the studies that have been conducted to look at risk for non-target organisms are “weak.” What I mean is this - in the lab, the number of animals exposed to the treatments is typically small and in the field, relatively few plots or fields are used. In contrast, think of a drug trial. When a pharmaceutical company is looking to see whether a drug has side effects, they conduct studies involving hundreds, sometimes thousands, of people. But for GM crops, the studies might involve just a handful of insects or field plots. When studies have low replication, the results are more likely to reflect chance outcomes.

GMO Safety: You conducted what are known as meta-analyses to improve the significance of single studies. How do they work?

Michelle Marvier: Meta-analysis has gained prominence in clinical trials and the medical arena, where one has to be very careful before deciding that a treatment is relatively risk-free. The crux of meta-analysis is the realisation that an absence of significant effects in a collection of individual studies is not necessarily as convincing as it might first seem. The problem is – as already mentioned - that the individual risk assessment or toxicity studies may be poorly replicated and thus have low statistical power.

Even a tally of the results - that means the number that found versus didn’t find significant effects - from a collection of weak studies is not much more convincing than the findings of each individual study on its own. Among statisticians, such tallies are called “vote counts” and if one thinks about it a bit, it is pretty obvious that even a dozen studies, all with poor replication, finding no effect would not constitute convincing evidence that no effect actually exists.

Fortunately, meta-analysis provides an alternative to vote counting. By statistically combining the observed differences between treatments and controls across a group of independent studies, and weighting the results of each experiment by the variance in the data, one comes up with an estimate of the general effect size across experiments. This resultant effect size is much richer than simply stating that 9 of 11 experiments or even 11 of 11 experiments found no significant impact on honey bee survival. It is possible that, by applying meta-analysis to a set of poorly replicated studies, a more reassuring picture may emerge. Of course, it is also possible that a meta-analysis will draw out small but potentially biologically important effects that went undetected by any individual study.

GMO Safety: What were the topics and outcomes of your meta-analyses?

Michelle Marvier: A meta-analysis of field studies found that Bt crops are generally more benign for non-target invertebrates than insecticides. A second meta-analysis of lab studies found no harmful effects of Cry proteins - the toxins produced by Bt crop - for honey bees.

GMO Safety: You promote large open-access databases on risk assessment studies. What is the idea behind this?

Michelle Marvier: In order to facilitate the execution of meta-analyses the scientific community needs easy access to data from the numerous biosafety studies on GM crops. Yet there are so many scientists doing so many different experiments and risk assessments that the information all too easily overwhelms decision-makers or causes the debate to zig-zag around. Meta-analysis has the potential to move the debate about the safety of genetically modified crops beyond a situation in which competing sides argue that “study X shows this” only to be countered with “yes, but studies y and z show the opposite.” Indeed, no single study should, by itself, be taken too seriously until other studies have confirmed the findings.

If meta-analyses and large databases of completed studies were to become a routine part of risk assessment, then there would not be the distraction of single experiments capturing media attention and inappropriately alarming or comforting the public and policy-makers. An investment in the creation and maintenance of risk assessment databases will have high payoff in terms of improved transparency, increased public confidence in the process, and more rapid advancement of scientific understanding.

GMO Safety: At the moment the database you developed together with your colleague Peter Kareiva only contains data on non-target effects of insecticide-resistant Bt crops. Do you intend to expand your database with data on other traits and GM crops?

Michelle Marvier: Actually, we set up the database as a model, in hopes that regulatory agencies - such as the US Environmental Protection Agency, which funded our research - would adopt a similar approach. It is a lot of work to extract data from published papers and unpublished reports and track down details by corresponding directly with authors. It would be way easier to keep a database up to date and complete if the researchers entered the data themselves, potentially as a condition for registration and deregulation.

GMO Safety: How do you imagine the set-up of a comprehensive database containing all relevant studies of GMO environmental risk assessment? Who will be responsible for that?

Michelle Marvier: I hope that that an internationally active organisation such as the International Society of Biosafety Research will be in charge of data collection, quality control and coordination. I would love to see a single publically-accessible database, detailing the methods and results for all of the completed studies. It would be an enormous service to science, to the public, and to industry.

GMO Safety: What is your personal view on the future challenges of biosafety research?

Michelle Marvier: Scientists and farmers were very familiar with Bt before Bt toxins were incorporated into transgenic crops. Other types of “plant-incorporated-protectants” that are currently under development and testing are less familiar, and we will likely find surprises along the way as these next generation transgenic insect-resistant plants are tested. I believe it was helpful that Bt was the first to go through this type of testing, so we could learn how to perform risk assessment and we can now apply what we have learned to crop traits that are less familiar.

GMO Safety: Thank you for talking to us.