The parable of the histogram

I must be some kind of heretic. I'm a statistician, and here I am pointing out the problems in yet another common statistical tool.

We'll see how the histogram, which is a very popular way of displaying the distributional shape of a set of data, must be viewed with a good deal of caution.

Even though histograms are often found in the media, the problems with histograms are almost unknown among the general public. Indeed, most places that teach statistics at university completely fail to mention them.

I'd like to say that the problems are well known among professional statisticians, but that might be too strong. Certainly problems have been pointed out in the literature, and many statisticians are aware of the problems, but it seems many still are not, and the appropriate cautions are not always explained.

I'm going to show you a simple example.

Here's some data (40 observations in this sample), which I'm going to draw a histogram of. I have rounded the numbers off to two decimal places.

3.15 2.28 2.06 3.43 4.85 3.22 4.01 4.43 
5.46 3.12 5.53 5.51 5.56 5.52 5.31 4.96 
3.28 4.10 5.19 2.54 1.89 1.84 2.56 1.90 
4.20 3.42 2.39 3.64 4.84 4.31 5.11 5.60 
1.98 3.91 1.88 4.33 5.74 2.01 2.58 1.92

I give the numbers so you can (if you are so inclined) confirm for yourself what I will tell you in my little parable.
(Edit added Feb 2012: I noticed that the results didn't quite reproduce in R - three observations in the original data set I gave occurred exactly on bin boundaries for some situations. This was either a problem caused by rounding, or possibly by different conventions of different packages for handling observations at bin boundaries;  I have accordingly altered those three observations by tiny amounts to move them off boundaries and avoid the issue, whatever its source. There is R code at the end of the post that works.)

The parable

This data set was given to a student, Annie. She constructs her histogram of the data by counting the number of values between 0 and 1 (but not including exactly 1), between 1 and 2, and so on, and then drawing a series of boxes each of whose base covers the subset of values that the count came from and whose height is the count for that range of values. Annie's histogram is shown in the top-left of the picture below.

She obtains a histogram whose shape corresponds to a distribution that is skewed to the left. See, for example, this description of using histograms to assess distributional shape here (edit: broken link replaced with an alternative) - that's pretty much precisely the way many elementary books on statistics describe the way to assess the shape of a distribution (and usually it's going to give you the right sort of impression).

Note that I could remove the scale and I could still describe the shape - I don't need to know the numbers on the scale in order to arrive at my description.

Three of Annie's friends, Brian, Chris and Zoe (Hah! Psych!) also get data sets with 40 observations, and they all do exactly as Annie did. Their histograms are given below (Annie's data is V1, Brian's is V2 and so on).

(click pic for a larger image)

Correspondingly, Brian describes his distribution as symmetric (and he might add "uniform"). Chris describes his as skewed to the right. Zoe describes hers as symmetric and bimodal (it has two main peaks).

So far so good - this is exactly how the books tell you it all works.

So while they're comparing their histograms, Annie idly starts looking at Brian's actual numbers. She realizes something odd is going on. She quickly places all their data sets side-by-side.

"Look, Chris!" Annie says, "all Brian's values are smaller than mine by 0.25. All yours are a quarter smaller than Brian's, and Zoe's are a quarter smaller than yours!"

They all confirm that she is correct - each set of values is the same, but with its origin merely shifted a little. Their data sets are identical in shape, but the resulting histograms are not.

That is to say, assessment of distributional shape in histograms can be dramatically affected by choice of scale (specifically, by the choice of the origin and width of the histgram bins). Here ends the parable.

It usually isn't this dramatic, of course, but the fact is, if one can generate a seemingly innocuous set of numbers whose histogram will look completely different (and for which many people will assert the distributional shape is completely different) every time we merely add or subtract a quarter, it can happen with real data too. And it does happen. Mostly the difference in impression is more modest... but not always.

So if you see a histogram, just keep in the back of your mind that it's perfectly possible that a different choice of bin boundaries would yield a somewhat different impression of the data.

Imagine I want to show some students that I write "easy" tests (I don't know why this should be such an object of fascination for students since they all do the same test, and marks are generally scaled, but it is). In preparation, I draw a histogram and it turns out to like Chris' - it looks like most students score below the middle of the range of marks. But lo, I discover with a bit of fiddling around that if I make my bin centres where the edges were (and so on), the completely opposite impression is given - just like Annie's histogram. Yay, "easy test" ... and many fewer worried queries from students in the run-up to the test, because they tend to feel there's a good chance of scoring "above the middle".

Did I lie? No. Did I fudge the data? Well, no. I did something, though. Or rather, I didn't do something.

This is a sin of omission. I fail to explain what the data would have looked like given a different choice of bin location.

Clearly, when circumstances are right, the ability to choose the location and width of the bins can give us the opportunity to somewhat alter the impression given by a histogram. Without fudging the numbers themselves, we can sometimes fudge the impression they give.

What do statisticians do? Well, there are other ways to look at distributional shape. Kernel density estimates are popular, and they completely get rid of the "bin-location" issue, though there's still the equivalent of a "bin-width" issue (choice of bandwidth, also called the "window"), which is often dealt with by looking at more than one choice of width (usually a width that gives a nice smooth result and then one that is smaller, giving a "rougher" result, in order that we can see there's nothing unsual hiding away - like the blue and green curves in the graph at top left right** at the wikipedia link a few lines up). But there are a variety of other tools that might be used (which I don't plan on going into here).

**(did I ever mention that I have trouble with correctly attributing the words "left" and "right"? - well as you see, sometimes I do. But not when describing the shape of a distribution, isn't that odd?)


What can you do? Well, assuming you don't have anything more sophisticated that a basic histogram tool, at the least (with continuous data, anyway), try shifting your bin starts forward or back a fraction of a bin-width (if you're lazy, maybe try something near a half, otherwise maybe try a couple of values). Also try a narrower bin width. If you do a few different histograms that all give the same general impression, it doesn't matter much which one you use. And if they don't give the same impression, you better either say so, show more than one, or find some other way to convey the information.

[Or you can do a kernel density estimate readily enough - many packages (including some free ones) will do them; there are pages online that can draw them if you just paste in some data. Implementing a kernel density estimate of your own is fairly straightforward - you can compute one in a spreadsheet easily enough - if anything, it's probably slightly simpler to compute one than it is to compute counts for a histogram, which is in itself pretty straightforward. ]


Caveat Emptor
___

Added in edit in Feb 2012: 

Here is some R code to create the data:

histdata <- c(3.15,5.46,3.28,4.2,1.98,2.28,3.12,4.1,3.42,3.91,2.06,5.53
,5.19,2.39,1.88,3.43,5.51,2.54,3.64,4.33,4.85,5.56,1.89,4.84,5.74,3.22
,5.52,1.84,4.31,2.01,4.01,5.31,2.56,5.11,2.58,4.43,4.96,1.9,5.6,1.92)

Here is some R code to generate the histograms:

opar<-par()
par(mfrow=c(2,2))
hist(histdata,breaks=1:6,main="Annie",xlab="V1",col="lightblue")
hist(histdata-0.25,breaks=1:6,main="Brian",xlab="V2",col="lightblue")
hist(histdata-0.5,breaks=1:6,main="Chris",xlab="V3",col="lightblue")
hist(histdata-0.75,breaks=1:6,main="Zoe",xlab="V4",col="lightblue")
par(opar)

Here is some R code to generate some density estimates:

opar<-par()
par(mfrow=c(2,2))
plot(density(histdata,bw=.2),main="Annie")
plot(density(histdata-.25,bw=.2),main="Brian")
plot(density(histdata-.5,bw=.2),main="Chris")
plot(density(histdata-.75,bw=.2),main="Zoe")
par(opar)

Here is some R code to generate some other informative displays:
First - the sample cumulative distribution function

plot(ecdf(histdata))  

Second, a stripchart that shows the positions of the individual observations as they move back.

x<-c abline="" c="" each="40)" g="" histdata-.25="" histdata-.5="" histdata-.75="" histdata="" pch="|" rep="" stripchart="" v="(2:5),col=6,lty=3)</pre" x="">


end edit

Fighting Mathiness

'I know what you're thinking about,' said Tweedledum: 'but it isn't so, nohow.'
'Contrariwise,' continued Tweedledee, 'if it was so, it might be; and if it were so, it would be; but as it isn't, it ain't. That's logic.'

'Can you do Addition?' the White Queen asked. 'What's one and one and one and one and one and one and one and one and one and one?'
'I don't know,' said Alice. 'I lost count.'
'She can't do Addition,' the Red Queen interrupted. 'Can you do Subtraction? Take nine from eight.'
'Nine from eight I can't, you know,' Alice replied very readily: 'but-'
'She can't do Subtraction,' said the White Queen. 'Can you do Division? Divide a loaf by a knife-what's the answer to that?'

Jordan Ellenberg defined mathiness as "a series of fervent gestures that gives the impression that mathematical ideas are being expressed, but doesn’t actually deliver the goods".

Let us examine some examples of mathiness, and some examples where honest attempts to deal with mathematical situations have foundered, and try to understand how we can be led astray by mathematical arguments.


Skewness

I recently wrote about how in statistics, the measure that is often called skewness doesn't really mean what popular lore holds it to mean, and that it is often misused - for example, when people assert that zero skewness implies symmetry. I later pointed to several sites that made the kinds of errors I was talking about. In the brief time since then, new instances of the same issue have come up on some mathematics-related blogs. It's a case where the verbal "description" of the situation is not in agreement with the mathematical tools being used - mathematical ideas appear to be expressed, but the goods are not being delivered.


Misleading Graphics

In another vein, bad statistical graphics, such as this


lie in graphical form


can mislead us, whether by accident, or as in this case, by design.
(via Andrew Gelman at Statistical Modeling, Causal Inference, and Social Science; there's other good examples to be found there.)

Examples abound in the media. Here's one from the NYT (via the Gallery of Data Visualization’s Missed Opportunities and Graphical Failures - click image for bigger version):

The top plot there is a graph of happiness against GNP-per-capita for a number of countries. The NYT has circled the countries in the top left hand corner, noting that many countries "had higher ... happiness than their economic situation would predict". This is the cardinal sin of treating inherently nonlinear relationships as linear - as they point out at the Gallery, an appropriate transformation - in this case looking at log-GNP, not raw GNP, makes these supposed "outliers" seem much more in keeping with the rest, and the apparent relationship more linear - and indeed, if anything, some entirely different points don't fit the general pattern. We seem to find nearly-linear relationships easier to understand, so transformation is often a useful strategy.

I have discussed the same issues - both the danger of treating nonlinear relationships as linear and the value of transformations in understanding relationships better in another context - relationships involving percentages. It's so easy to fall into the rut of linear thinking that we should consider taking advantage of the tendency to think that way and use transformation to reduce nonlinearity.

A common "nonlinear effect treated as linear" is when people try to average miles per gallon (or miles per hour, or a variety of other rates) - such as "I got 15 mpg going up and 45 mpg coming back, so I averaged 30 mpg overall" (when it's actually 22.5). In terms of transformations - the reciprocal (gallons per mile) - is linear and can be averaged.


Relying on a False Premise

Seemingly mathematical arguments may just be based on bad premises (such as one requiring selecting from the positive integers with equal probability - an impossibility that completely sinks the argument that relies on it).

That "infinity" thingy can be tricky - it seems to cause problems for journalists as well because they tend to underestimate how big it is.


Adding percentiles

Treating percentiles of distributions as if they were additive is unfortunately extremely common. In the case of official estimates of total oil reserves, it means that we probably have a fair bit more oil that we think.


What's the square root of that?

Or, sometimes, it seems, mathiness comes in because someone has no clue what the heck they're talking about, so we can be told that the Maya knew how to take the square root of a rectangle.


Mathematical arguments can feel unsually convincing, even unassailable, and we're awash in them for precisely that reason. It's too easy to forget that just because something seems to be laid out mathematically, it's not necessarily true - or even meaningful at all. Mathiness, like truthiness, is all around us. Even among skeptics, it's possible to put too much store in an argument couched in mathematical terms. We should be at least as skeptical of mathematical arguments - and in basically the same kinds of ways - as any other kinds of arguments, because we're all too often misled by them.

Unfortunately, it seems that we sometimes accept the (often implicit) conclusions of a mathematical argument without even realizing that an argument was being made.

If we fail to treat these arguments with the skepticism they deserve, we're open to being deceived by charlatans.