Research Highlight: In Pointing Out Flaws in Climate Change Data, Two Studies Show How Science Works

Causing a controversy that he would rather have avoided, Ian Eisenman published a study this summer that uncovered an error in a high-profile climate change record.

Eisenman’s study was published on July 22 in the European Geosciences Union journal The Cryosphere. Before long, dozens of science writers and bloggers around the world had latched onto the story.

A climate scientist at Scripps Institution of Oceanography, UC San Diego, Eisenman identified an unexpected discrepancy between two estimates of the expansion of sea ice off the coast of Antarctica. Antarctic sea ice expansion has been the subject of intense public interest and scientific discussions because it departs from standard expectations for a warming globe. The Antarctic expansion contrasts starkly with its northern counterpart, Arctic sea ice, which has been rapidly shrinking.

The two estimates, which were derived from satellite measurements, formed the basis for the accounts of sea ice changes that appeared in consecutive reports from the Intergovernmental Panel on Climate Change (IPCC), a United Nations-sponsored entity that produces a comprehensive assessment of the latest findings of climate science roughly twice per decade. In their study, Eisenman and coauthors found that the estimates could not both be correct. At least one, they concluded, contains a substantial error.

Eisenman’s was one of two studies by Scripps researchers that were released within days of each other in July that offer an object lesson in how science sometimes faces very human barriers on the way to discovering empirical truths.

While Eisenman’s path involved publicly questioning a widely trusted observational dataset, Scripps atmospheric scientist Amato Evan acknowledged in another paper that the computer simulations upon which scientists like him rely to forecast the effect of one climate variable were inaccurate. Those, too, Evan pointed out, were incorporated into recent IPCC reports, though with caveats added.

“In the long run, science is self-correcting, because future research can show, as in these cases, that some previous research results may need to be modified, or sometimes even abandoned as incorrect,” said Scripps Distinguished Professor Emeritus Richard Somerville, who is a noted commentator on the communication of science to the public. “Thus, some aspects of sea ice extent and the climate effects of dust – topics being studied by Eisenman and Evan – are among today's areas of active research, where this kind of careful investigation and incremental scientific progress is still going on.”

In Evan’s paper, he reviewed the performance of 23 computer models that simulate the climate effects of dust, which originates in places like the Sahara Desert and blows in large quantity over continents to influence monsoons and other significant climate patterns. He and his co-authors found that none of them were capable of accurately simulating the size and mass of dust storms among other variables.

The scientific community responded to Evan’s paper with appreciation tinged with disappointment. Evan said the shortcomings in representing the influence of dust in one region do not change the overall assessment that climate change is taking place worldwide.

“It’s one part of a much bigger story,” he said. “Climate change is still happening and my little bit about dust doesn’t stop that.”

Typically the accuracy of dust processes in these models relies in part on strong groundings in actual observations. Evan points out in his paper, which appeared July 4 in the online version of the journal Geophysical Research Letters, that there is a dearth of this kind of observed data when it comes to African dust.

“What we need is regional data, but we’re not there yet in terms of observations,” said Evan. “It’s where everyone wants to go in the next 10 or 20 years.”

Among the obstacles, he said, are conditions in the places to which scientists would like to travel to conduct field research. For instance, Evan and colleagues in France and Algeria want to install GPS stations at various locations in the Sahara desert. If installed, those stations would communicate with satellites and provide valuable information just by the way they relay data. The amount of water vapor in the air can cause a slight but measurable lag in the transmission of data.

Evan would like to collect that water vapor data to understand how much heat it adds to the Sahara region, since it is a powerful contributor to greenhouse warming of the atmosphere. How much heat is in the region influences local phenomena including a stationary cyclone that regularly forms over north Africa and distributes dust and rainfall to surrounding regions.

Evan’s efforts to gather the data, however, have been curtailed because of a host of political problems in pockets of each country in which he would hope to install GPS stations. Stations in Algeria and Mali would need to be guarded from al Qaeda cells in both countries. Scientists would put their lives in danger even by traveling to certain locations there. The militaries of Tunisia and Morocco collect potentially useful GPS data but are wary of sharing it, especially in the aftermath of the Arab Spring. To install equipment in Mauritania at the western edge of the Sahara requires an escort from expensive military convoys to installation sites and an unaffordable sequence of bribes to various officials, Evan said. In all those countries, a strong presence of bandits adds to the risk.

----------------------------------------------------------

Eisenman initially noticed that estimates of how rapidly sea ice is expanding off the coast of Antarctica had risen substantially during recent years. The IPCC’s Fourth Assessment Report in 2007 reported that the Antarctic sea ice cover remained nearly constant between 1979 and 2005, whereas the Fifth Assessment Report in 2013 reported that between 1979 and 2012 it increased at a rate of between 13,000 and 20,000 square kilometers (5,000 to 7,700 square miles) per year. Each IPCC report reflected the scientific literature at the time it was compiled. It had been widely assumed within the scientific community that the difference was the result of adding several more years to the observational record.

Suspicious that the numbers didn’t add up, Eisenman acquired the record himself and checked. What he found was a substantial discrepancy: the two reports had actually based their estimates on very different datasets.

The datasets used in the two reports were widely believed to be nearly identical. In 2007, between the times of the two reports, a minor update had been made to the algorithm used to generate the dataset from satellite measurements, and the entire record was reprocessed with the updated algorithm. But the update had not been expected to have any noticeable influence on the long-term rate of expansion.

Eisenman’s analysis showed that this update actually caused a four-fold increase in the rate of expansion between 1979 and 2006.

“Errors often occur and get corrected in science, and I was hesitant to write a paper about an error in someone else’s dataset,” Eisenman said. But, he added, it’s important that changes like this are “documented and thoroughly addressed.”

When Eisenman first identified the issue, he contacted the group at NASA's Goddard Space Flight Center who created the dataset. Josefino Comiso, a senior research scientist at NASA, leads the group. Comiso is a prolific researcher who has always made a point of making his data widely available to other scientists. However, the change in trend caused by the update was so unexpected that Eisenman was initially unable to convince Comiso that his analysis of the discrepancy was correct, so Eisenman and colleagues further analyzed the datasets. They traced the discrepancy to a difference in the way the measurements were calibrated during a transition from one satellite sensor to another in 1991.

“Our results imply that the estimates reported in the Fourth and Fifth Assessment Reports of the IPCC can’t both be correct: we find that the data used in one of the reports must contain a significant error,” Eisenman said. “But we have not yet been able to identify which one contains the error.”

Eisenman said that since the update was not expected to cause this change, more analysis would be necessary to discern whether an error was inadvertently corrected or introduced when the dataset was reprocessed.

An open letter in defense of climate science signed by 255 members of the U.S. National Academy of Sciences in 2010 explained how errors are pointed out and corrected in a scientific process that is “inherently adversarial.” Eisenman’s study, to some extent, followed this pattern.

After Eisenman’s paper and the accompanying press coverage, further discussions between Comiso and Eisenman and his co-authors took place. These discussions continue, as do further analyses of the data. In an e-mail to Eisenman and several journalists, Comiso agreed that Eisenman’s paper brought up a real discrepancy that “nobody noticed” previously. In response to Eisenman’s paper, he said he intends to publish an erratum for the old dataset.

Asked for comment, Comiso said that he is confident that the error is in the old dataset, and that the new dataset is accurate. He said that the old dataset "was not noticed as erroneous because the values were not too far from other published values." He believes the new dataset is correct because it is "based on a data set that has been enhanced and quality checked and the results are consistent with results from other investigators as well."

 “I expect that [Comiso] is correct that the error is in the old dataset rather than the new one, since he knows these datasets better than anyone,” Eisenman said. “But naturally I'll want to pore through the details of his analysis.”

In so doing, Eisenman will be adhering to standard scientific practice, said Somerville.

“The recent work of Ian Eisenman and Amato Evan illustrates a key aspect of scientific research, and it’s one that many non-scientists may not be fully aware of,” Somerville said. “Science is a collaborative enterprise, and researchers spend a lot of effort in checking each other’s work. When any of us publish an account of our research in a scientific journal, we expect other qualified scientists in our specialty to study it carefully and to do more research in order to learn whether ours is O.K. or has any flaws.”

– Robert Monroe

Related Image Gallery: In Pointing Out Flaws in Climate Change Data, Two Studies Show How Science Works