The Correlation Coefficient R and the reduction of range.

 Those who followed my series on climate change might remember that I would take some area I could define as a rectangle in longitude and latitude and track the average temperature by year, comparing equivalent seasons. (To be precise, if a longitude/latitude area include either the North or South Pole, it would be more like a slice of pie than a rectangle.) I then split the time span from 1955 to 2010 into four eras based on the El Niño/La Niña cycles, these in particular each starting and ending in a strong La Niña year.

This particular region and season, Siberia in Spring, has a clearly increasing trend of the median temperature, the dotted red line moving up in four separate steps. The lowest temperature registered only moves upward twice and the maximum average temperature takes a step down in the era of 1999-2010, as the highest temperature was registered back in the 1990s.

Another method would be to add a line of regression, also known as a trendline or predictor line or the line of least squares. Excel has an option which gives the equation of the line and the variance R². R is called the correlation coefficient and it varies between -1 and 1. R² must be between 0 and 1 and is sometimes thought of as a proportion. In this case, the .3655 would be the proportion we would assign to the general increase we see in the temperatures, while the fluctuations are about .6345 of the influence.

This doesn't sound very convincing, but if R² = .3655, R in this case would be +.6046, a value that by nearly every standard of correlation is considered high, though it doesn't meet the Rule of Thumb criteria for very high, which would be over .8.

This is one of the many reasons I don't love using the predictor line and the correlation coefficient. The statements of confidence seem arbitrary - I know of three different systems and they disagree radically on whether an R score is strong or not - but also there is a way to cherry pick data in both directions, either to show more correlation or less.
 

Generally though not always, taking a subset of a sample by restricting the range will result in R and R² being reduced. For example, if we look at the first Consistent Oceanic Niña Interval from 1955 to 1975, we see somewhat less overall increase (here we check the number multiplying x, which went from .0417 to .0346) and a drastic drop in R² from .3655 to .05097. Here I can say without fear of contradiction that the correlation is not impressive.

In our second interval, R² is stronger at .19237, but still well below the larger set's value of .3655. It could be considered moderately strong by some measures, but notice that here the trend shows the region cooling. (It really is coincidence that the first year of this era shows a large jump in temperature over the previous.)

Here again we see a downward sloping trendline and an extremely weak R² value of .01592, which is to say nearly no correlation.

Yet again, a small downward slope and a low R² value.

In my view, the problem is cherry picking in both directions. People who wish to downplay or deny warming temperatures can take smaller samples, but when they do, the R² value will often give little confidence in the trend they try to show. On the other hand, people wanting to show strong evidence of warming have a natural advantage of generally higher R² scores in larger data sets. To be fair, in this particular set it is impossible to create a subset longer than thirty years that doesn't show a warming trend, though it can be minimized and so can the correlation coefficient.

If anyone is coming to the blog for the first time, you should know that I am not a denier of the general warming trend in temperatures around the globe in my lifetime, which started in the Strong La Niña year of 1955. What I hope for is a discussion where both sides can agree on terms and methods and avoid cherry picking at all costs. I realize this hope may very well be in vain, and yet I hold on to it.

Historical Carbon Dioxide data 1958 to 2012

The data I'm using for today's post comes from the website CO2Now.org. If some "skeptic" wanders by and says this is data from an interested party and therefore not unbiased, bite me.

In fact, unlike temperature data which swings wildly from year to year, average yearly CO2 levels are increasing steadily and there is no contrarian position.

My question is how steadily?

This graph is a manipulation of a data set taken from the Mauna Loa CO2 readings. Let me explain my process.

1. Take the average yearly levels from 1959 to 2012, the first and last years with a full twelve months of data.

2. Starting in 1968, subtract the yearly level from ten years prior (example: 1968 level - 1959 level) and divide by 10 to get the average yearly change over the previous decade.

What this says is not that CO2 levels are increasing, but that the rate of change is increasing as well. The increase is not exactly linear, but you can see how close the black trend line is to the jagged red line with the white dots. The R² value of 0.88... means the fit to the line is very good. If the rate of increase is exactly linear, the graph on the CO2 levels would be the increasing part of a parabola. We call this quadratic growth.

The good news is this is not exponential growth, for all the good that does us. Quadratic growth is faster than linear growth, and even if it flattened out at current levels, CO2 levels are increasing about twice as fast as they did in 1970. (In comparison, world population has increased by a factor of 1.8 over the same period.) Given the relatively steady growth of the rate, the amount of change in the last 40 year span should only take next 30 years.

CO2 matters. It's a natural part of our environment, but like everything in nature, too much of it is not good. CO2 does a lot of things, many of them positive for the environment, but it is a greenhouse gas, which means it helps trap the heat from the Sun. The greenhouse effect is about as controversial as gravity. More CO2 in the long run means higher temperatures in the long run. This can mean big changes in the environment, positive changes for some and negative changes for others. Overall, it does not look like a zero sum game, with much more pain than gain.

The so-called Serious People In Washington are convinced the economy needs austerity. We must curb spending or the next generation will inherit a mess. On the other hand, these Serious People are not fully on board about us changing the way we spend energy. Austerity means lower taxes for the rich, so they are in favor. Changing energy habits means less money in their pockets and possibly regulatory limits on their styles of living. For anyone saying this is cynical, let me use Lily Tomlin's apt quote from years ago, "No matter how cynical you become, it's never enough to keep up."

Right now, people are paying attention because of breaking the 400 ppm barrier. As a mathematician, I know that most people are impressed by round numbers and ignore the rest of them. We may be decades away from 450 ppm, but we can't wait for that news story. We may not be able to put on the brakes, but we sure as hell should take our foot off the gas pedal, both literally and figuratively.

Think about your use of carbon, which for the most part means how you use energy. Think of ways you can cut back. For the people who argue that we shouldn't have to limit ourselves unless China, India, Brazil and others also set limits, my counterargument is this.

How old are you? Seven?

If you love anyone who is younger than you are, that is reason enough. We may not be able to give them a better world than we have, but we shouldn't consign them a hellhole.

U.S. vs. the Rest of the World: Climate change as measured from the 1988-1999 to 1999-2010 eras

Here's the map of consistent weather stations in the United States - with a few scattered in southern Canada - that reported at least one temperature every season from 1988 to 2010. If you go on comment boards around the Internet, there are people who post with confidence that global warming has stopped and the world has not gotten warmer since 1998.

The data shown here does not agree with that statement. I use the method of comparing two eras that start and end in strong La Niña years with a strong El Niño in between. I could switch and use strong El Niño years and the endpoints with a strong La Niña in between, but that would not bring us as close to present day as this method does.

What can be said is that the warming trend slowed down using this measurement system. Here are the numbers from the United States stations compared to the data from the rest of the world.

Total stations: 
U.S.A.: 3094
Rest of world 3363

Warming stations vs. cooling stations
U.S.A.: 71.5% warming, 28.5% cooling
Rest of world: 79.0% warming, 21.0% cooling

Average temperature change:
U.S.A.: 0.19° C
Rest of world: 0.29° C


This is still a warming trend, but not as fast as we saw comparing 1975-1988 to 1988-1999. If this continues, this would mean about a degree rise Celsius in the U.S.and a degree and half everyplace else. This is under the two degree per century threshold which climate scientists consider catastrophic.

Notice that the rest of the world has finally caught up and surpassed the United States in total consistent weather stations. A regularly reporting weather station is a useful thing, but not a vital part of the infrastructure. It's something of a luxury and back in 1955 the United States far outstripped the rest of the planet in being able to afford such luxuries, the aftereffects of World War II ten years earlier still being felt by the countries where the battles were fought.  As so many statistical measurements tell us, we are no longer number one in the world in quality of life.


Another number that jumps out at me is that no matter how many stations report in the United States and how many stations report every place else, the ratio of the temperature change stays relatively constant, the rest of the world warming a little more than 50% faster than the U.S. This could just be coincidence, but it would be interesting to find out why. If there is a cause other than random chance, it might mean the United States is doing something other countries could emulate to bring their warming trends down.

After several weeks work, I am convinced climate change is real. The simplest way to explain it is the planet is getting warmer in general, but certainly not at the same rate everywhere. Just looking at the temperatures does not give any answer to the question of whether mankind's actions have any effect on the increase we are seeing; that takes modeling and modeling such a complex system has to be inexact, even on the incredibly powerful computers we have in the 21st Century. But if there is a man-made cause for temperatures rising slower in the United States than everywhere else, it would be in our best interest financially and environmentally to export that cause to the rest of the world.

U.S. vs. the Rest of the World: Climate change as measured from the 1975-1988 to 1988-1999 eras

Yesterday, we looked at the eras 1955-1975 compared to 1975-1988. Moving forward, now we look at 1975-1988 compared to 1988-1999.

In the United States, the warming trend increased quite a bit compared to the earlier era differences. 80% of weather stations that reported every season showed a warming trend and the average increase was 0.3° C compared to 0.09° C, and this increase takes place over a shorter period of time.

Here are the numbers of the U.S. vs, the rest of the world.

Total stations: 
U.S.A.: 3534
Rest of world 3046

Warming stations vs. cooling stations
U.S.A.: 80.4% warming, 19.6% cooling
Rest of world: 86.2% warming, 13.8% cooling

Average temperature change:
U.S.A.: 0.30° C
Rest of world: 0.47° C


These changes are much more extreme than similar comparisons between the earlier eras reported yesterday. Climate scientists consider about two degrees Celsius in a century to be catastrophic warming. Since this is a twenty five year time span, the United States rate stretched to over a century would not be alarming, but the rest of the world would be.


Tomorrow, the 1988-1999 era compared to 1999-2010 era, a time when some global warming skeptics say the earth began cooling.

U.S. vs. the Rest of the World:Climate change as measured from the 1955-1975 to 1975-1988 eras

Unless I get a brilliant idea, I'm going to post three more climate data entries and then I will go back to other topics in math. Here are my reasons.

1. I am now convinced that anyone who talks about "global cooling" is a liar.
2. The data I work with can only verify if the general climate is warming, cooling or staying the same. I can't make any statement about human cause and distrust predictions about the future, even when they were done with mathematical modeling. Since the climate shows itself to be warming over so many regions, that is the only question of interest I can answer.
3. My hope was to become part of the conversation and to get people on both sides to agree on ways to discuss the data that does not involve cherry picking. If I magically gained some kind of influence, I think I could talk to climate scientists. Warming denialists are a completely different kettle of fish, and a right putrid one at that.

Here is the first data from my last idea, unless some brilliant clue strikes me. By using the consistent weather stations around the world that report every season between two La Niña peaks that have El Niño peaks between them, I can look at the worldwide data trends.  If I start in 1955, the United States has a huge advantage over the rest of the world. Having a consistently reporting weather station is a trifle in the developed world but a major cost in the undeveloped world. Even though World War II is ten years in the past and places like Canada, Australia and New Zealand come out as nearly unscathed as the U.S. does, the United States has more consistent weather stations than the rest of the world combined.  Here is how the two groups compare.

Total stations: 
U.S.A.: 3130
Rest of world 1938

Warming stations vs. cooling stations
U.S.A.: 60.9% warming, 39.1% cooling
Rest of world: 64.5% warming, 35.5% cooling

Average temperature change:
U.S.A.: 0.09° C
Rest of world: 0.14° C 

This time period shows warming and the differences between the U.S. and the rest of the world are significant. Neither would be catastrophic is continued for a century, a rise of a less than a half degree Celsius.

Tomorrow, we move forward to 1975-1988 compared to 1988-1999. 
 

Distribution of temperature changes, U.S. vs. Worldwide

I published this map of weather station in the U.S. that reported consistently from Winter of 1955 to Fall of 2010. The color of the dots represent the difference in the era averages of 1999 to 2010 minus the average from 1955 to 1975. The darkest dots show severe warming, the hollow red dots show warming of less than a degree Celsius and the blue dots show stations that cooled.

Red dots outnumber blue dots, but by how much?

Here are the numbers put on a bar chart. The bar chart looks to be somewhat normally distributed and the average is in the moderate warming range. The severe warming stations slightly outnumber the stations that are cooling.

The worldwide differences still show the average to be in the moderate range, but there are more stations showing severe warming and less showing cooling trends.

In a set of numbers like this, two questions should be asked.

1. Are we measuring something important or not?
2. Could the differences we see just be random chance or is there an underlying reason?

If the measurement is important and it's not just random variation, it could mean that something the United States has been doing for the last half century or so might help slow down the warming trend seen in so many places.

Consistent weather station map:Northern Hemisphere 1955-2010

I apologize for not posting yesterday. This blog takes thought sometimes and the news got in the way of concentration.

The dots on this map represent weather stations that reported at least one temperature every season from the Winter of 1955 to the Fall of 2010 in the Northern Hemisphere. The right side of the horizontal axis is the Prime Meridian, the left side is 180 degrees East and 180 degrees West.  It's a little hard to make out shapes except for that clump in the low middle, which is obviously the United States of America.

The start of the era is very significant in terms of how many weather stations around the world will be reporting. In 1955, the list of true First World countries is USA. Europe is still digging out from the rubble of the war that ended ten years before and likewise Japan and much of Asia.

Let's take a closer look at the data from the United States.
 

There are four different kinds of marks on this map.

1. Dark with red outline. Strong warming stations. Here the average temperature increase from the 1955-1975 era to the 1999-2010 era is more than 1 degree Celsius. That's about 30% of all the stations.

2. Empty with red outline. Warming, but less than a degree Celsius. 60% of all stations.

3. Empty with blue outline. Cooling by less a degree Celsius. 10% of all stations

4. Dark blue with black outline.  There are only a few on this map and they are hard to see at this magnification. A total of 5 out of 2,860, they don't even round to a percent.

Simply put, the large majority of the stations measured are showing warming, but by no means all. The average increase is 0.7 degrees Celsius with a standard deviation of 0.57 degrees. That's a lot of deviation, which means a lot of uncertainty. Climate denialists would play up that uncertainly and try to ignore the warming.  As an observer outside the bar fight that is climate science right now, I wish someone would look at the cooling stations and figure out why. Maybe is just randomness, but maybe many of these stations have something in common that we might exploit to start a man-made global cooling effect that offsets what we do with out burning of fossil fuels.

I despair of the Culture of Constant Conflict that so much of human discourse has become. Climate science is one of those areas of discourse and the split it between the center left and the far right. (Conservatives including Margaret Thatcher and Arnold Schwarzenegger are with the current scientific consensus, as was Newt Gingrich before someone offered him more cash.)I'm just a beginner in the field, but I do have some ideas for research I'd like to see explored and this is one of them.

A new idea:  What happens if we make the early era 1975-1988? All consistent stations in this set will still be consistent, but several others should join the ranks and the differences in the two averages cannot be the same.

More on this later this week.

New data method: Consistent weather station map

I had a new idea for how to look at the big data set made available by Berkeley Earth Surface Temperature. While I now understand the effects of La Niña and El Niño are not worldwide, they do make a difference over a huge amount of the earth's surface so I will continue to consider eras that start and end with a strong La Niña years (or conversely with a strong El Niño years) to be time periods that are worth comparing.

I took the data set and wrote a C program that is interested only in the weather stations worldwide that reported a temperature for every season from 1955 to 2010, both of which are strong La Niña years.  The earliest La Niña to La Niña era is 1955 to 1975, while the most recent spans from 1999 to 2010.  Complete data for 2011 and 2012 is dribbling in, but neither is measured as strong La Niña or strong El Niño, so these are the time periods I use to avoid cherry picking the data, which will often mean the data is not completely up to date.

Tomorrow, I will start showing the results for these consistent weather stations, looking at the difference of the averages of the early and late eras. Splitting the data into the two hemispheres, the northern hemisphere is much better covered than the south, completely unsurprising given the differences in both land mass and population. There are many ways to look at a data set this big and as the week progresses we will go from the simplest ideas to the more complex.

Climate Data: Siberia 1955-2010

The split between Siberia and the rest of Russia is usually defined as the Ural Mountains. Since I wanted to make it a rectangle is longitude and latitude, I chose 60 degree East for the leftmost cutoff point.

We think of Siberia as "frozen wasteland" and "place of exile", but I have read several Russian writer who say it has a meaning similar to the West in United States' cultural memory, a big, mostly empty place of challenge and opportunity.

Whichever it is, it certainly is not wanting for weather stations to measure its progress. Very good coverage from east to west and north to south.

The temperatures in Winter vary drastically. While the warmest Winters have spiked pretty close to each other all through the 1955-2010 era, the coldest Winters are getting generally warmer and the median keeps sneaking upward era by era.

Spring does not show as much fluctuation and the sequences show more steady increase.

Summer fluctuates even less and every measurement method shows consistent increase, but the total increase of the median from the 1955-1975 era to the 1999-2010 era is less than a degree Celsius, the mildest increase of all four seasons.

Like Winter, Fall temperatures jump around a lot. The warmest Fall are only slightly warmer, but the median has climbed over a degree and a half.

Confidence level of warming from time interval to time interval: 99.99%

Confidence level of the trend showing increasing warming: 65.8%

Average seasonal change in the medians of 1955-1975 to 1999-2010: 1.555° C.

1.555° C in 56 years is the number that matters. This is way too fast. More than that, the Siberian permafrost, like the Winter ice in the Arctic Circle, can become a really scary feedback loop. There's a lot of permafrost in Siberia, and most importantly there is a contiguous permafrost about the size of France and Germany combined. When permafrost melts, it changes the albedo, absorbing heat instead of reflecting it. If that wasn't bad enough, there is a huge amount of CO2 and methane trapped in the permafrost that will be released when it melts, things will only get worse.

I've only been looking at climate data for about ten weeks now, but I am pretty close to convinced it's real and it does not have to speed up for it to be a serious problem for humans, if not in my lifetime then in the lifetime of kids growing up today. Tomorrow, I will introduce another way to look at the data. I have not done the programming yet, so I don't know if the data will alarming or not, but it is an attempt to look at patterns worldwide instead of getting useful data region by region.

Climate Data: Sonoran Desert 1955-2010

The Sonoran Desert spans much of Northwestern Mexico and parts of California and Arizona. The best known cities north of the border are Phoeniz and Tuscon.

The coverage by consistent weather stations is much stronger in the United States than it is in Mexico. Those grid marks in the lower right hand corner are from the tip of Baja California.
 

While the record high temperature for Winter average is now nearly 20 years old, the other ways of measuring - the median in the red dotted line and the coldest winter, marked by the lower black line - are showing marked increase.

The Spring data is increasing using every measurement system shown here.

And the Summer is as well.

More than just steady increase, every season has shown a median increase from the 1955-1975 era to the 1999-2010 era of more that a degree Celsius, which is the simplest number that marks the Big Damn Deal cut-off point.

Confidence level of warming from time interval to time interval: 99.999%

Confidence level of the trend showing increasing cooling or warming: 50%

Average seasonal change in the medians of 1955-1975 to 1999-2010: 1.38° C.

The data says it's getting warmer but the trend is not currently on the upswing. This is some good news but not a lot, as an increase of 1.38° C in 56 years is already fast enough to be considered catastrophic. 

Climate Data: Atacama Desert 1955-2010

The Atacama Desert in South America is said to be the driest place on earth. Much of it high in the mountains, there are regions so desolate they look like they could be on the moon.
 

It is possible to have weather stations that are not manned, but it is rare. The inhospitable locations and climate of the Atacama means there is only one weather station that has reported consistently from 1955 to 2010.
 

Because this station is in the Southern Hemisphere, the beginning of the year is Summer. There is no strong trend, with highs and lows bouncing around randomly. The most recent era's median is slightly lower than the 1955-1975 median.

Change in 1955-1975 median to 1999-2010 median: -0.04° C. 

The Fall temperatures from this station show a steep cooling trend this century after slight warming last century.

Change in 1955-1975 median to 1999-2010 median: -0.38° C.  

Winter is like Fall, but with a much steeper recent drop.

Change in 1955-1975 median to 1999-2010 median: -1.47° C. 

The Spring data also shows the recent drop.

Change in 1955-1975 median to 1999-2010 median: -1.31° C. 

Confidence level of cooling from time interval to time interval: 63%

Confidence level of the trend showing increasing cooling: 88.9%

Average seasonal change in the medians of 1955-1975 to 1999-2010: -0.80° C.

As stated earlier, this is really just the data from a single station, not from an entire region. That said, this station shows a cooling trend over our 56 year only matched by a part of eastern Antarctica in our earlier worldwide survey.

This gives me an idea for way to search the data, looking for completely consistent stations worldwide over the 1955 to 2010 time period and checking the difference in the first era defined by the Consistent Oceanic Nino Intervals (1955 to 1975) and the most recent era (1999 to 2010).

But for now, I will continue to look at certain deserts. Tomorrow, the Sonoran Desert, which straddles the border of the western United States and Mexico.

Climate Data: Mongolia 1955-2010

During my four weeks of climate data, the regions I selected followed a set pattern. Right now, I am hopping around the map with no consistent rhyme or reason. I chose the Sahara and Australia because they are both dominated by desert. I chose China because I wanted to see what was happening in a very populated area. I choose Mongolia today because it fits nicely in a longitude/latitude rectangle, it was part of the China map yesterday and it contains a large part of the Gobi Desert, continuing with the theme of deserts started earlier.

Mongolia is much more sparsely populated than China and not as well covered by weather stations. Even so, there were 23,627 seasonal reports between 1955 to 2010, so it is nothing like the paucity of information we had in the polar regions.

Average Winter temperatures are all over the place, from less than 2° C to nearly 10° C. These temperatures are much colder than China and much more variable. The general trend is upward, though there was a drop in the low temperature and median at the beginning of this century compared to the end of the last.

Change in 1955-1975 median to 1999-2010 median: 1.6925° C. 

Spring shows steadily climbing in all our measurement standards, though the amount of increase is slowing down.

Change in 1955-1975 median to 1999-2010 median: 1.877° C. 

Summer shows increasing temperatures and the rate of warming is going up.

Change in 1955-1975 median to 1999-2010 median: 1.46° C.

Fall shows a temperature dip between 1988-1999 to 1999-2010, but these temperatures are still over a degree higher than what was seen in the 1955-197 era.

Change in 1955-1975 median to 1999-2010 median: 1.35° C.

===
Confidence level of warming from time interval to time interval: 99.98%

Confidence level of the trend showing increasing warming: 65.8%

Average seasonal change in the medians of 1955-1975 to 1999-2010: 1.59° C.

We are very confident that this is a warming trend and not just random variation. We are not confident at all that the warming is speeding up, but at 1.59° C in 56 years, we don't have to be.  This number is definitely in the "hair on fire" range. Of the regions I've looked at, this is the worst increase of anyplace outside the polar regions.

This gets me interested in deserts in general.  Tomorrow: the Atacama desert in South America.

Climate Data: China and surrounding areas 1955-2010

China is not like the Sahara or Australia. It's farther away from the equator, it's not a mono-climate and it fits rather clumsily in a Mercator rectangle. It's not warming as fast as the Sahara, but it is showing much more warming than Australia. Most climate scientist I've talked to think of 2 degrees Celsius in a century as a "hair on fire number" .82 degrees in 56 years would average out to about 1.5 degrees in 100 year if this keeps up.

The region is very well covered by weather stations.

Of all the seasons, Winter is showing the greatest increase in median temperature from the 1955-1975 era to the 1999-2010 era, about 1.02 degrees Celsius. (The choice of the beginning and ending numbers of these eras is based on Strong La Niña years, for those of you who are new to my posts on climate.)

The Spring red dotted line jumps up by 0.94 degrees Celsius from the first era to the last.

Summer temperatures dipped in the time period from 1975 to 1988, but other than that we see an increasing trend.

Fall temperature also have just one dip, but this one is from the end of last century to the beginning of this one.

Confidence level of warming from time interval to time interval: 99%

Confidence level of the trend showing increasing warming: 94.87%

Average seasonal change in the medians of 1955-1975 to 1999-2010: 0.82° C.