How to mislead others for the sake of a zippier story

In which I spank Gallup's shiny arse a little, and maybe USA Today's

I wrote a much more detailed version of this post a couple of weeks ago, but my lameness resulted in me losing the whole damn file, and I didn't have time to rewrite... until now. This is not the same post. [Couldn't remember The Greatest Post in the World, no, no. This is a tribute...]

A couple of weeks ago, Gallup published a media release about its annual Honesty and Ethics Ratings of Professions survey, USA Today had an article on it, and Hemant Mehta expressed puzzlement at the fact that while overall approval for clergy had gone down, it went up amongst the non-religious. In his words, "What. The. Hell?".

Hemant quotes the USA Today article:

Ratings dropped year-over-year among Catholics and Protestants, as well as among regular and occasional churchgoers. However, they rose in one category: among those professing "no religion." Last year, 31% rated clergy honesty high or very high; in 2009, that figure inched up to 34%.

That came from Gallup's media release, where they published this graph, which deliberately sets out the comparison that was made in the USA Today article:


So what gives? Why would it go up for the non-religious?

The most likely explanation is sampling variation. Gallup mention sampling variation, but in this case, that's just not enough.

Gallup says, hidden away down the bottom:

"Results are based on telephone interviews with 1,017 national adults, aged 18 and older, conducted Nov. 20-22, 2009. For results based on the total sample of national adults, one can say with 95% confidence that the maximum margin of sampling error is ±4 percentage points."

Even with this disclaimer, this doesn't make sufficiently clear the magnitude of the problem. That 4% is a little more than the 3% figure for the difference, so maybe we should be a little bit cautious about the three percent being real (and caution is all it would suggest, since if we scale back from 95% confidence to say 75% confidence, it would go below the three percent difference).

But it's much, much worse.

First thing to note is that the 4% sampling variation figure that Gallup give only applies to the overall figures. [By the usual calculations, I get 3% rather than 4%. I assume Gallup is inserting some additional margin of caution there, but it's nowehere near enough, as we'll see.]

The 4% they give does NOT apply to percentages of subgroups.

There were about a thousand interviewed (1017). The proportion that are willing to give “no religion” for a question about their religion on a phone interview varies a bit, but it's generally around 10-15%. I can't tell what it was here, so let’s be generous to Gallup and say 15%.

That’s around 150 with no religion. The sampling variation for that subgroup is more than 2.5 times as big as it is for the original sample (sqrt(1017/150) times as big), or roughly 10% by Gallup’s reckoning of 4% for the original survey (their 4% is very rough so I am not worrying about being too precise - and I will err in Gallup's favour at each point).

Now, when you compare TWO surveys (31% vs 34%), the margin of error is bigger – if we can assume independence of the responses in the two surveys, you actually use good old Pythagoras’ theorem here.

So the margin of error on the change between two surveys on this subgroup is around 14%.

14%!

We have an increase of 3% give or take 14%.

There is no reason to assume anything happened at all. Maybe it went up, maybe it went down. We have NO clue. No way to tell if anything happened at all.

Yet Gallup clearly invite precisely the comparison USA Today made, and Hemant ran with.

It was irresponsible of Gallup not to point out that the comparison they made in the graph above had such a high margin of error that such comparison was meaningless. They should have pointed it out, or not made the comparison at all.

The disclaimer at the end is entirely insufficient. (And USA Today should have at least realized that even with a 4% margin of error their own comparison was at least a little dodgy, but you know, it's the media we're talking about. Probably didn't even read all the way to the bottom of the Gallup release. Take a look at approval figures for journalists some time.)

Pew online Political News IQ survey

Pew have a Political News Quiz up, here, and you can see how you went compared to a national US survey.

I did okay on that:

Here's Your Score: You correctly answered 12 of the 12 possible questions along with approximately 2% of the public. You did better than 98% of the general public.

If you do it, once you get a graph like the above, there are some further links on the left with more information; the link How you did, question by question takes you to a list of which questions you got right or not and the percentage of correct responses on each question from the previous survey.

Tea? Coffee?

Recent research shows that
(i) caffeine prevents and even reverses the effects of Alzheimers, and
(ii) tea consumption reduces the risk of Parkinsons

So do I drink coffee and get Parkinsons or tea and get Alzheimers?

I guess tea has a little caffeine, so I could drink like 15 cups a day.

Who knows - if I stop putting milk in it, all that peeing might even help with kidney stones. Or then again, maybe it will power my car.

The Chaser and right-wing censorship

"The Chaser's War on Everything" is a satirical TV show in Australia. (The Chaser itself was a satirical newspaper - the nearest US equivalent I can think of would be The Onion). The Chaser's TV show (which is on ABC-TV - owned by the Australian Broadcasting Corporation, a public-owned broadcaster) has been through various incarnations, but relies mainly on satirical sketch comedy and various other bits.

The Chaser boys regularly go over the top - some of their skits can be challenging to watch. And they don't always hit the mark - sometimes they're just not funny. None of that matters, except to their ratings.

They regularly offend the right wing, and talk radio goes completely berserk - the last time they went off quite this much was when they did a less-than-kind (but not completely inaccurate) song about people who had died. This time it was for a thing they did about the Make-a-wish foundation (a charity for sick children) - so you can imagine the furore from the "won't someone think of the children!" brigade. In both the earlier case and the current one, I actually watched the piece that caused all the fuss when it went to air - and in both cases the reaction was so over the top as to be pathetic.

Look, it's simple. If you find these guys offensive, don't bloody watch them, okay?

If they break the law - well, that's what police are for. If they break guidelines for broadcast material, there are regulations and penalties in place there too. If they contravene ABC's own internal procedures, there are penalties there as well. None of that happened. There may be a question of whether the internal review was sufficient, but that's an internal matter for the ABC. To take them off the air for two weeks to pacify the right wing - when they followed ABC procedure (the piece was reviewed and approved) - was ludicrous.

The weird thing is, it was relatively speaking, the cases that the right wing decide to go nuts over are not extreme compared to other stuff that they do. It's generally no worse in offense terms than South Park or Drawn Together (except it's live, rather than animated).

Occasionally, *I* find something they've done offensive. That's not bad, it's valuable. We need our comfortable middle-class existence shaken up a bit. Nobody and nothing should be above ridicule, and it can be very valuable to step back from yourself and try to examine why you find something offensive.

If a parent of a dying child watched the piece and was offended (and I can certaonly imagine some would), *that* parent has every right to take it up with the ABC. But all the people screaming about the possibility that someone might be offended? You offend me. Get a fucking life.

The ABC is moving to curtail the way satirical programming is reviewed - which means less satire. That's a terrible loss to our society.

You have a right to find things offensive. There is, however, no right to never be offended - and there should not be.