What is science?

We learned today from an NIH announcement about a new plan by the federal agency to relocate and transfer all of the NIH-owned chimpanzees to the federally-funded sanctuary, Chimp Haven, by 2021 or later. The announcement was quickly the subject of announcements and proclamations of victory by PETA, HSUS, and some associated with Chimp Haven.

For others, in light of the concerns raised about the death of 9 of 13 chimpanzees transferred to Chimp Haven recently and subsequent calls for a thoughtful examination of these cases—at least a review of what might be done to minimize future risk—the announcement was troubling.

It does appear that NIH either shared, or was at least responsive to the need to address, the concerns that were expressed about the consequences of relocation on the chimpanzees’ health and welfare. That is evidenced by the fact that NIH did undertake an analysis of mortality rates at Chimp Haven and the research centers that house NIH chimpanzees.  That is as it should be – scientists use data to inform decisions.  No problem there. NIH conducted the analysis on the basis of data requested from each of the centers. It also appears that they referenced the findings of the analysis in their decision.  So what’s the problem?

It appears that the only evidence of the mortality analysis is a non- reviewed paper that was posted just yesterday to a website (Biorxiv) by the study author, NIH’s Dr. Michael Lauer. That paper may be viewed here. After even a cursory review and analysis of the Lauer paper, many questions are raised about both the methods and the conclusions drawn from the results.  Just a few of the issues or potential problems that an academic reviewer might raise are listed below. Others may read the paper and have different impressions or questions.The data includes 764 chimpanzees; 314 died during the 7 year median follow-up. The author states that: “The analyses were conducted to inform NIH’s plans to retire its surviving chimpanzees.”

To see NIH use data from an unpublished, non-peer reviewed manuscript as a basis for their decisions is incredibly disheartening. It defies the very premise and basis on which the scientific process works. Science doesn’t accept as fact those data and findings that are presented on the internet and that have not been properly vetted through the peer review process.  But before turning to questions about the paper, let’s be clear on a critical point:  The questions and critiques raised here would be raised regardless of the conclusions of the paper and regardless of the direction of NIH’s decision. The questions raised here are at the heart of how science is used to inform decisions and judgments.  

In other words, what would we conclude if NIH had used a non-reviewed paper to suggest that relocation was a threat to chimpanzee well-being and that the chimpanzees should be retired in place?  The same criticism would hold.  The issue is about the conduct of science and how it should be shared and viewed in decision-making. In this case, it is particularly important because of the close relationship between the findings and decisions that have immediate and real impact. Furthermore, in a time where scientific rigor and reproducibility are the subject of a great deal of concern and discussion, it is even more troubling to see that the results of an unreviewed paper posted only yesterday in public  view are the basis for an announcement made today.

That means that there was no opportunity for a broader public consideration, for thoughtful analysis, for viewing the data, for asking questions about the approach, methods, analysis, interpretation of results, and conclusions.  Thus, we post here some initial questions and comments about the unpublished and unreviewed paper from several scientific reviewers. We hope others read the paper (here) and offer their comments, or offer additional insight into the approach, analysis, and conclusions.

To be clear, these comments are not designed to advocate for or against the transfer of chimpanzees to Chimp Haven. Nor are they designed to judge the quality of care the animals receive after arriving there. Rather, they are designed to illustrate the fact that decisions about the welfare of captive chimpanzees that are being made by NIH appear to be based on data and analyses that are arguably flawed, at least as presented in the current draft of the Lauer paper.  Dr. Lauer might have excellent responses and answers to these critiques, which may then validate the claims in the current paper.

And, that is the point: the data will then have been subjected to critical peer review, the bedrock of the scientific method. It is disappointing, and frankly, stunning, that NIH appears to have accepted these results without proper peer review. Making captive chimpanzee retirement and movement decisions based on these findings seems premature and foolish.  Sadly, that may lead to unnecessary deaths of chimpanzees. NIH is clearly committed to sending their chimpanzees to Chimp Haven; if that is the mandate, then why try to justify the decision based on methods and analyses that have not been subjected to the normal scientific peer review process?  That ultimately raises more questions than answers and stands to further confuse the public view of how science works and how claims should be evaluated.

Finally, we would also note that the data does not appear to be publicly available. In other words, while the un-reviewed article is in public view and its conclusions appear to have informed the decisions the data is not in view and cannot be evaluated or analyzed by scientists or others who are independent of the decisions and the centers involved.

Below are just a few issues, or potential problems, that any reviewer might point out.

• The first part of the study was aimed at addressing mortality rates in chimpanzees housed at Chimp Haven compared to other facilities (Bastrop, Southwest Foundation and Alamogordo Primate facility). The author reported that age and sex had strong effect on mortality rates whereas location had only moderate effects. In point of fact, the influence of location was not a trivial effect based on the results presented in Table 2 but rather a statistically significant one.  The author seems to want to minimize the significance of the location effect because the overall p-value (p=.0173) was close to the significance level adopted in their analysis.  The effect for Chimp Haven was far below that, at p=.005. The problem, however, is that the argument for adjusting alpha as reported by the author was because they had 6 predictor variables, they therefore they increased alpha to control for possible Type I error. There are a number of issues here. First, it is not clear how the authors dummy coded the location variable. Second, even if there were 6 predictor variables, there were also more than 700 subjects in the study and thus whether the author had adequate power to guard against Type I error (and thus needed to adjust alpha below the traditional < .05) is not entirely clear without presentation of effect size or further rationale. In turn, to state that sex and age had strong effects and location had a moderate effect on mortality is simply not supported by any statistics other than the p-levels.

  Maynard at MD Anderson.

  The paper reports: “The strongest predictor, by far, of mortality was
  age (as calculated to be on January 1, 2005), followed by male sex and location. Older age predicted higher mortality (adjusted hazard ratio comparing animals 30 years versus 17 years 2.23, 95% CI 1.91 to 2.61); males also had higher mortality (adjusted hazard ratio 1.50, 95% CI 1.20 to 1.88). Location was only marginally associated with mortality (Wald c2=10, df=3, P=0.017). Compared to Chimp Haven, mortality was lower at APF (adjusted hazard ratio 0.65, 95% CI 0.48 to 0.88), while it was similar at Bastrop (adjusted hazard ratio 0.84, 95% CI 0.60 to 1.16) and almost identical at SNPC (adjusted hazard ratio 1.00, 95% CI 0.71 to 1.39).”

• The second part of the paper was designed to examine the influence of relocation/transfer on mortality rates in the different chimpanzee populations. This aspect of the study is likely in response to recent reports of higher-than-normal rates of mortality in chimpanzees transferred to Chimp Haven but sadly neither the design nor data analysis allow for any meaningful conclusions to be drawn.  Specifically, there is no control or comparison group. According to the author, at Chimp Haven (CH), chimpanzees die as they get older and this isn’t due to factors such as when they arrived at Chimp Haven, the season of year, etc….but these analyses are irrelevant. What one would want to know is what the mortality rate is for chimpanzees that get transferred to CH compared to either: 1) chimpanzees that stay at their original facility and don’t transfer; or 2) mortality in chimpanzees that transfer INTO CH from a lab compared to mortality rates of chimpanzees that are transferred FROM CH to another facility or 3) mortality rates of chimpanzees that have been transferred INTO another sanctuary (e.g., Save the Chimps).  The second situation does not occur; however, the 3^rd situation could be analyzed. Furthermore, there is a 4) mortality rates of chimpanzees that transferred FROM other facilities and INTO Bastrop.  None of these comparisons were made in the paper though they are necessary to make inferences about the effect of transfers on the quality of care and mortality. Thus, this entire part of the paper addressing the effects of transfer on mortality is fundamentally flawed. Of course, it is also recognized that in addition to the analyses, appropriate balancing of covariates that relate to the mortality for each of these four comparisons may be difficult, post hoc; however, the alternatives and limitations should be a feature of a carefully considered conclusion and discussion.

• For the Chimp Haven sample, why were non NIH-owned chimpanzees excluded from the mortality analyses? Chimp Haven has taken chimpanzees from other facilities such as Ohio State, Yerkes and New Iberia. If the question is not about mortality rates at a given facility, but rather the effect of transferring individuals from established housing conditions, why exclude any individuals? Further, were the non NIH-owned chimpanzees included in the sample size census within Chimp Haven? In other words, in Table 1, it indicates that the Chimp Haven had 273 chimpanzees. Is that all the chimpanzees at CH, or only those that are NIH-owned? A reviewer might guess is that these numbers are based on the entire sample of chimpanzees at Chimp Haven but it is far less clear on the mortality numbers (107). Moreover, peer review would surely point out that the methods are not sufficient for reproducibility.

• In Table 2, the most relevant comparison (at least in relation to the current issue, the transfer of NIH chimpanzees to CH) is starkly missing. Specifically, for location, Chimp Haven needs to be the reference group, so that comparisons of transfer from all other sites can be made. This is particularly strange as the text lists Chimp Haven as the reference group and interprets the data in this regard. If the point of this analysis is to inform the decision to transfer the animals to CH, vs retire-in-place, then the comparisons should be made with CH as the reference group so that we can see how it truly stacks up against leaving the animals where they are.