I wrote a much more detailed version of this post a couple of weeks ago, but my lameness resulted in me losing the whole damn file, and I didn't have time to rewrite... until now. This is not the same post. [Couldn't remember The Greatest Post in the World, no, no. This is a tribute...]
A couple of weeks ago, Gallup published a media release about its annual Honesty and Ethics Ratings of Professions survey, USA Today had an article on it, and Hemant Mehta expressed puzzlement at the fact that while overall approval for clergy had gone down, it went up amongst the non-religious. In his words, "What. The. Hell?".
Hemant quotes the USA Today article:
Ratings dropped year-over-year among Catholics and Protestants, as well as among regular and occasional churchgoers. However, they rose in one category: among those professing "no religion." Last year, 31% rated clergy honesty high or very high; in 2009, that figure inched up to 34%.
That came from Gallup's media release, where they published this graph, which deliberately sets out the comparison that was made in the USA Today article:
So what gives? Why would it go up for the non-religious?
The most likely explanation is sampling variation. Gallup mention sampling variation, but in this case, that's just not enough.
Gallup says, hidden away down the bottom:
"Results are based on telephone interviews with 1,017 national adults, aged 18 and older, conducted Nov. 20-22, 2009. For results based on the total sample of national adults, one can say with 95% confidence that the maximum margin of sampling error is ±4 percentage points."
Even with this disclaimer, this doesn't make sufficiently clear the magnitude of the problem. That 4% is a little more than the 3% figure for the difference, so maybe we should be a little bit cautious about the three percent being real (and caution is all it would suggest, since if we scale back from 95% confidence to say 75% confidence, it would go below the three percent difference).
But it's much, much worse.
First thing to note is that the 4% sampling variation figure that Gallup give only applies to the overall figures. [By the usual calculations, I get 3% rather than 4%. I assume Gallup is inserting some additional margin of caution there, but it's nowehere near enough, as we'll see.]
The 4% they give does NOT apply to percentages of subgroups.
There were about a thousand interviewed (1017). The proportion that are willing to give “no religion” for a question about their religion on a phone interview varies a bit, but it's generally around 10-15%. I can't tell what it was here, so let’s be generous to Gallup and say 15%.
That’s around 150 with no religion. The sampling variation for that subgroup is more than 2.5 times as big as it is for the original sample (sqrt(1017/150) times as big), or roughly 10% by Gallup’s reckoning of 4% for the original survey (their 4% is very rough so I am not worrying about being too precise - and I will err in Gallup's favour at each point).
Now, when you compare TWO surveys (31% vs 34%), the margin of error is bigger – if we can assume independence of the responses in the two surveys, you actually use good old Pythagoras’ theorem here.
So the margin of error on the change between two surveys on this subgroup is around 14%.
14%!
We have an increase of 3% give or take 14%.
There is no reason to assume anything happened at all. Maybe it went up, maybe it went down. We have NO clue. No way to tell if anything happened at all.
Yet Gallup clearly invite precisely the comparison USA Today made, and Hemant ran with.
It was irresponsible of Gallup not to point out that the comparison they made in the graph above had such a high margin of error that such comparison was meaningless. They should have pointed it out, or not made the comparison at all.
The disclaimer at the end is entirely insufficient. (And USA Today should have at least realized that even with a 4% margin of error their own comparison was at least a little dodgy, but you know, it's the media we're talking about. Probably didn't even read all the way to the bottom of the Gallup release. Take a look at approval figures for journalists some time.)
3 comments:
Another thing that needs to be pointed out is that "nonreligious" probably includes the New Age "spiritual but not religious" types as well as atheists and agnostics.
One should expect odd looking results when lumping such divergent groups together.
Even if that's the case we are still left with a mystery -
For the overall "no religion" figure to have actually GONE UP, some subset of that group must have had a substantial increase while every other religious group and every age group show a decrease in approval for clergy.
Let's assume it's spiritual-but-not-religious* people lifting the figure up.
WHY would they have an increase in approval of clergy, against a solid downward trend elsewhere?
*(but in fact these people usually do seem to identify a religion when asked for one in these surveys)
And the answer is the same as the one I gave before - in fact it's probably just sampling variation, there's likely nothing going on.
Another hypothesis: the nonreligious view of clergy bottomed out faster following all the sex abuse scandals and is inching up slightly as the outrage becomes less fresh.
The religious, on the other hand, have been slower to accept the reality of just how extensive the abuse has been and, for them, its still falling.
I agree though that your original hypothesis is the most likely one (not that any of these hypotheses are mutually exclusive---all three could be at work).
Post a Comment