Answers in Genesis BUSTED! points to New Scientist's reporting of the interesting story of contingent evolution in e. coli.
Fortunately, nowadays we have the resource of actual scientist-bloggers rather than being restricted to what passes for science reporting these days, because the New Scientist story drives me nuts. As much as I love being able to hold a magazine in my hands to read, this reminds me why I decided a few years back that New Scientist was too expensive for what I was getting. (I have boxes and boxes of the things. Is there a good charity that needs a whole lot of old science magazines that I won't go broke getting them to?)
Frankly, it's entirely typical of science reporting, and another example of why science reporting sucks, and why blogging by scientists beats it hands down.
Aside from the fact that the New Scientist report has the emphasis wrong, it's just a generally typical sucky example of science reporting, with distortions of fact and silliness. (To be fair, here and there it does an okay job of explaining some bits, but really, I'm no expert, and I could tell it sucked before I'd read about it anywhere else.)
[Actually, what used to really drive me to screaming fits of rage was whenever they let Ian Stewart start talking about statistics. The guy may be a reasonable mathematician, but his explanations of statistics sucked dogs balls.]
I highly recommend reading the Pharyngula
discussion.
For goodness sake, even the abstract of the paper he quotes (jargon aside) is in many places better written than the New Scientist article.
Let me quote a paragraph from the New Scientist report:
"In the meantime, the experiment stands as proof that evolution does not always lead to the best possible outcome. Instead, a chance event can sometimes open evolutionary doors for one population that remain forever closed to other populations with different histories."
Gah!
"doors ... that remain forever closed to other populations" is one of the more ridiculous statements I've seen from New Scientist, and they've had some doozys.
forever closed? forever??
Bullshit. If the "tricky part" - the enabling mutation - evolved once, in only 20000 generations (which then further evolved several times to produce the ability to metabolize citrate), it's obviously possible for it to happen again. The probability of it evolving is not zero.
The fact that Blount, Borland & Lenski couldn't get it to evolve independently a second time just means that probability is likely very low. Maybe extremely low.
But forever is actually a pretty long while. You might think it's a long time 'til lunch, but forever is such a really long time that any mathematically nonzero probability of it evolving in a finite amount of time (as was observed to occur) means that by forever, it will have evolved an infinite number of times. And, if you're not real good with the counting thing, take it from me that infinity times is just a tad bit bigger than zero.
And what's with that bit about "evolution does not always lead to the best possible outcome"? Which cryptotheist retarded version of evolution do they think they are arguing with there? All you have to do is take a look at a human eye (heck, just get a small light and grab a handy animal with a spine) to know that. The blood vessels of the eye lie on top of the rods and cones! They're in the way! Anyone can see them! And the nerves, too - that's why we have a blindspot. And, funnily enough, the light sensitive bits point backwards, away from the light. Our eyes are arranged ass-backwards!
(That it needn't have happened this way is obvious from squid and related creatures, whose eyes are not arranged "backwards" like ours. And there are hundreds of little examples of this sort of thing.)
Evolution is completely cobbled-together (i.e. contingent), and it's already completely obvious that it's so - to anyone with eyes. Well, and maybe a brain.
[based on a comment I made at AIGBusted.]
Wednesday, June 11, 2008
Tuesday, June 10, 2008
The problem of suffering and free will
There's a standard apologetic response to what's usually called "the problem of evil". That response is, basically, that evil is a consequence of free will.
However, that utterly fails to deal with suffering that has nothing to do with free will. Disasters, painful disease and so on have nothing whatever to do with free will, and so the apologetic response does not apply.
There's a second response I've seen (usually from the more fundamentally minded), which is that these things are either "caused by the devil" or "caused by the fall".
I could easily deal with both of these point by point, but they're directly covered by Epicurus' 33AD argument (slightly recast):
"Is God willing to prevent suffering, but not able?
Then he is not omnipotent.
Is he able, but not willing?
Then he is malevolent.
Is he both able and willing?
Then whence cometh suffering?
Is he neither able nor willing?
Then why call him God?"
Either God is able to stop the devil or not. If not, "caused by the devil" is answered by the first pair of lines, but if God could stop the devil, it's answered by the second pair of lines. If it's caused by the fall, then (second pair) God is punishing the innocent because "hundreds of generations ago" someone didn't believe him when he lied (it's right there in the bible) about the fruit on the tree.
(Yep, the snake told the truth and God lied, and he condemns humans to floods and earthquakes and painful deaths forever - nice guy. Oh, and he pulls the legs off talking snakes that point out he's a big shifty fibber. Niice. And then yells at everyone to get the hell off his lawn.)
So we get stuck - the problem of suffering is not answered by free will. If it's the devil, God is either not omnipotent or malevolent, and if it's the fall, God is plainly malevolent. As well as a big fat liar.
However, that utterly fails to deal with suffering that has nothing to do with free will. Disasters, painful disease and so on have nothing whatever to do with free will, and so the apologetic response does not apply.
There's a second response I've seen (usually from the more fundamentally minded), which is that these things are either "caused by the devil" or "caused by the fall".
I could easily deal with both of these point by point, but they're directly covered by Epicurus' 33AD argument (slightly recast):
"Is God willing to prevent suffering, but not able?
Then he is not omnipotent.
Is he able, but not willing?
Then he is malevolent.
Is he both able and willing?
Then whence cometh suffering?
Is he neither able nor willing?
Then why call him God?"
Either God is able to stop the devil or not. If not, "caused by the devil" is answered by the first pair of lines, but if God could stop the devil, it's answered by the second pair of lines. If it's caused by the fall, then (second pair) God is punishing the innocent because "hundreds of generations ago" someone didn't believe him when he lied (it's right there in the bible) about the fruit on the tree.
(Yep, the snake told the truth and God lied, and he condemns humans to floods and earthquakes and painful deaths forever - nice guy. Oh, and he pulls the legs off talking snakes that point out he's a big shifty fibber. Niice. And then yells at everyone to get the hell off his lawn.)
So we get stuck - the problem of suffering is not answered by free will. If it's the devil, God is either not omnipotent or malevolent, and if it's the fall, God is plainly malevolent. As well as a big fat liar.
Government in SF - a failure of imagination?
(Within this post, I'm going to take the stance that SF stands for "speculative fiction" rather than science fiction. My main reason is that SF these days rarely contains much actual "sciency" stuff, and the category has become so broad that the term "science fiction" is not even vaguely true of most of it. It is also possible that it serves my argument. )
I've long had a problem with how government is represented in SF, particularly film and television, but it also happens in written SF - that the governments almost always seem to be of only two very narrow types:
(i) near-absolute monarchy (though possibly called an emperor or something, often also with a noble class);
(ii) US-style presidential democracy.
I have no doubt that in many stories, these are plausible models. However, I simply cannot fathom why so many writers assume that if it's not the first, it must be the second. Sometimes the US political system is reproduced to an astonishing level of detail, including a near duplicate constitution and everything.
According to wikipedia:
- there are 123 democratic countries
- there are 72 parliamentary democracies
- in addition, there are a large number of "hybrid" models (particularly in Eastern Europe)
That is, a US-style democracy is by far in the minority. On earth, parliamentary-style democracies are the rule. Yet in SF, they're almost unheard of.
So why in visual-media SF (the majority of whose writers are from the US), if you don't have some form of monarchy or monarchy analogue, will it be (apart from a very few exceptions), a US-clone democracy, often down to its boot-buckles? Is this a failure of imagination? Are US film and TV writers largely unable to imagine anything else? Are they unaware of anything else? Or do they imagine that a US audience won't accept anything else (and everyone else will, presumably)? If it's the last, that, too, is a failure of imagination - I certainly don't believe US fans of SF are that unimaginative and unwilling to think about different ideas.
Obviously, people tend to write about what they're familiar with, but if you only ever write about only what's in your back yard, your fiction hardly deserves the name "speculative".
Come on, people, have a look at the world around you. Try a different model for government once in a while!
I've long had a problem with how government is represented in SF, particularly film and television, but it also happens in written SF - that the governments almost always seem to be of only two very narrow types:
(i) near-absolute monarchy (though possibly called an emperor or something, often also with a noble class);
(ii) US-style presidential democracy.
I have no doubt that in many stories, these are plausible models. However, I simply cannot fathom why so many writers assume that if it's not the first, it must be the second. Sometimes the US political system is reproduced to an astonishing level of detail, including a near duplicate constitution and everything.
According to wikipedia:
- there are 123 democratic countries
- there are 72 parliamentary democracies
- in addition, there are a large number of "hybrid" models (particularly in Eastern Europe)
That is, a US-style democracy is by far in the minority. On earth, parliamentary-style democracies are the rule. Yet in SF, they're almost unheard of.
So why in visual-media SF (the majority of whose writers are from the US), if you don't have some form of monarchy or monarchy analogue, will it be (apart from a very few exceptions), a US-clone democracy, often down to its boot-buckles? Is this a failure of imagination? Are US film and TV writers largely unable to imagine anything else? Are they unaware of anything else? Or do they imagine that a US audience won't accept anything else (and everyone else will, presumably)? If it's the last, that, too, is a failure of imagination - I certainly don't believe US fans of SF are that unimaginative and unwilling to think about different ideas.
Obviously, people tend to write about what they're familiar with, but if you only ever write about only what's in your back yard, your fiction hardly deserves the name "speculative".
Come on, people, have a look at the world around you. Try a different model for government once in a while!
Monday, June 9, 2008
The Consolations of Probability
There's little doubt that many people find comfort in the presence of an all-powerful, all-knowing being watching over them, in knowing that everything happens for a reason, even if they don't know what those reasons might be. People who don't believe in such things cannot have such consolation. After all, what consolation is there if things just happen?
I'm not just talking about the fact that there's a lot of random stuff that happens that we aren't responsible for, or that we're incredibly lucky to be here. Those pieces of understanding are major consolations, no doubt (or at least they are to me). Greta Christina has touched on these, and similar issues several times, and written about them better than I could.
Probability and statistics, or at least an understanding of them, are incredibly useful things to have. I make a living out of them, so they're obviously useful in that sense, but I am talking here about ordinary everyday getting-through-life stuff.
For example, they're essential tools when responding to the news, because they help you identify the difference between a real issue and a likely manufacturoversy.
The tale of the traffic blackspot
Some time ago I saw a headline about how a particular busy intersection had become a pedestrian "black-spot" because there had been a "50 percent increase" in pedestrian deaths and serious injuries last year; it was suggested that it may have been caused by increased traffic from changes several intersections back along the same road. The relevant state minister was grilled about his "lack of action" and induced to promise some very expensive changes at the intersection.
My initial, emotional response was the one the journalist was doubtless trying to induce - horror at an apparently huge increase in deaths and injuries. But that nice round "50 percent" figure worried me. Was that number so rounded off from say 48 or 52 percent, or was it exactly 50, suggesting the numbers were very small? So I looked through the story for the figures. The number of deaths and serious inuries had gone from 8 the previous year to 12 in the last year. A quick calculation in my head showed that, even if the underlying probability of incidents had not changed, such a change in actual numbers could easily have happened. The fuss over the change from 8 to 12 may well be a big fuss over nothing but randomness. It could well be that the figure of 8 was itself too high - maybe something should have been done years ago - but the controversy highlighted by the article was misplaced; there was no particular reason to believe anything had changed. The journalist had managed to induce the minister to reallocate funds that may well have been much better left where they had been before.
[Since the details of the mathematics is not the point of this post, I haven't included them - in another post at a later time, I may explore a little of the specific calculations in these (real) examples. ]
If you're watching for it, this kind of thing comes up a lot. H.G. Wells once wrote "Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write". Is he overselling the case? When government policy is increasingly driven by media attention to unimportant detail rather than substantive policy, I do think it's become increasingly important. We need to be able to understand the flood of information, and if necessary, debunk the nonsense it contains. Can the broader awareness suggested by Wells' comment come to pass?
It seems that educators are responding.
In the part of the world where I live, statistics and probability are now taught throughout school (K-12). My six-year-old has already learned to record data and produce simple graphs of it on the computer. My nine-year-old has learned some basic probability. By the end of year 12, they will have a pretty solid grounding in the basic ideas. They might not end up with all the tools to do the calculations I used, but they will at least be able to ask the right questions.
While important, this sort of understanding of what's going on in the world - certainly a form of comfort - is perhaps a relatively abstract one. However, the comforts can also be much more personal.
The tale of the overtested driver
Some years ago, a fellow was asking a group of statistically-minded people (of which I was one) whether he was being victimized at work. His employer, who owned a business that had many employees driving trucks, had instituted random drug testing. Every week a small number of employees were selected and tested.
This particular driver felt he was being targetted, because he had been tested seven times in 18 months, while he knew of several others who had not been tested at all. He was worried that someone must have it in for him; he thought that maybe another employee he'd had a falling out with may have said that he was taking drugs in order to make trouble for him. He was wondering if he should confront that person.
I pointed out that for it to be an effective deterrent, drug testing had to be done at random - you can't test someone once and then not again until everyone else had been done, because if someone were inclined to take drugs while driving trucks, once they'd been tested, they'd know they were "safe" for a while. There always had to be a risk you would be tested next month. Random testing in turn meant that there would always be some people tested substantially more than others. Since he was asking, rather than someone else, it was likely that he was one of the most heavily tested people in that company, probably the most heavily tested - but someone had to be the most tested person.
The question was - if testing really was being done randomly, would seven tests in 18 months be an unusually high number for the most-tested person? Of course, you need to look at the number of people in the company and the number being tested each week. You'd also want to ask whether you'd really expect to find some people still untested at the same time.
It turned out that after 18 months, it was reasonably likely someone at the company was going to be tested seven times (around a 20% chance - a higher chance than picking up a die and rolling a six), and at the same time there would be a fair number of still-untested people (enough so that he'd likely know several). So there was no particular reason to think he was being targetted. No reason, based on the numbers so far, to expect the person he was worried about was making trouble for him.
Such knowledge was indeed, a consolation. Sure, the drug testing was a nuisance, but there wasn't any reason to think it was personal, or that anybody had it in for him. A significant source of stress in his life was lifted.
And understanding not just that things can happen at random, but of the specific consequences of how randomness works can show us that nobody may be to blame. It can show us that we are not a victim.
Probability can be a consolation, sometimes quite a personal one.
I'm not just talking about the fact that there's a lot of random stuff that happens that we aren't responsible for, or that we're incredibly lucky to be here. Those pieces of understanding are major consolations, no doubt (or at least they are to me). Greta Christina has touched on these, and similar issues several times, and written about them better than I could.
Probability and statistics, or at least an understanding of them, are incredibly useful things to have. I make a living out of them, so they're obviously useful in that sense, but I am talking here about ordinary everyday getting-through-life stuff.
For example, they're essential tools when responding to the news, because they help you identify the difference between a real issue and a likely manufacturoversy.
The tale of the traffic blackspot
Some time ago I saw a headline about how a particular busy intersection had become a pedestrian "black-spot" because there had been a "50 percent increase" in pedestrian deaths and serious injuries last year; it was suggested that it may have been caused by increased traffic from changes several intersections back along the same road. The relevant state minister was grilled about his "lack of action" and induced to promise some very expensive changes at the intersection.
My initial, emotional response was the one the journalist was doubtless trying to induce - horror at an apparently huge increase in deaths and injuries. But that nice round "50 percent" figure worried me. Was that number so rounded off from say 48 or 52 percent, or was it exactly 50, suggesting the numbers were very small? So I looked through the story for the figures. The number of deaths and serious inuries had gone from 8 the previous year to 12 in the last year. A quick calculation in my head showed that, even if the underlying probability of incidents had not changed, such a change in actual numbers could easily have happened. The fuss over the change from 8 to 12 may well be a big fuss over nothing but randomness. It could well be that the figure of 8 was itself too high - maybe something should have been done years ago - but the controversy highlighted by the article was misplaced; there was no particular reason to believe anything had changed. The journalist had managed to induce the minister to reallocate funds that may well have been much better left where they had been before.
[Since the details of the mathematics is not the point of this post, I haven't included them - in another post at a later time, I may explore a little of the specific calculations in these (real) examples. ]
If you're watching for it, this kind of thing comes up a lot. H.G. Wells once wrote "Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write". Is he overselling the case? When government policy is increasingly driven by media attention to unimportant detail rather than substantive policy, I do think it's become increasingly important. We need to be able to understand the flood of information, and if necessary, debunk the nonsense it contains. Can the broader awareness suggested by Wells' comment come to pass?
It seems that educators are responding.
In the part of the world where I live, statistics and probability are now taught throughout school (K-12). My six-year-old has already learned to record data and produce simple graphs of it on the computer. My nine-year-old has learned some basic probability. By the end of year 12, they will have a pretty solid grounding in the basic ideas. They might not end up with all the tools to do the calculations I used, but they will at least be able to ask the right questions.
While important, this sort of understanding of what's going on in the world - certainly a form of comfort - is perhaps a relatively abstract one. However, the comforts can also be much more personal.
The tale of the overtested driver
Some years ago, a fellow was asking a group of statistically-minded people (of which I was one) whether he was being victimized at work. His employer, who owned a business that had many employees driving trucks, had instituted random drug testing. Every week a small number of employees were selected and tested.
This particular driver felt he was being targetted, because he had been tested seven times in 18 months, while he knew of several others who had not been tested at all. He was worried that someone must have it in for him; he thought that maybe another employee he'd had a falling out with may have said that he was taking drugs in order to make trouble for him. He was wondering if he should confront that person.
I pointed out that for it to be an effective deterrent, drug testing had to be done at random - you can't test someone once and then not again until everyone else had been done, because if someone were inclined to take drugs while driving trucks, once they'd been tested, they'd know they were "safe" for a while. There always had to be a risk you would be tested next month. Random testing in turn meant that there would always be some people tested substantially more than others. Since he was asking, rather than someone else, it was likely that he was one of the most heavily tested people in that company, probably the most heavily tested - but someone had to be the most tested person.
The question was - if testing really was being done randomly, would seven tests in 18 months be an unusually high number for the most-tested person? Of course, you need to look at the number of people in the company and the number being tested each week. You'd also want to ask whether you'd really expect to find some people still untested at the same time.
It turned out that after 18 months, it was reasonably likely someone at the company was going to be tested seven times (around a 20% chance - a higher chance than picking up a die and rolling a six), and at the same time there would be a fair number of still-untested people (enough so that he'd likely know several). So there was no particular reason to think he was being targetted. No reason, based on the numbers so far, to expect the person he was worried about was making trouble for him.
Such knowledge was indeed, a consolation. Sure, the drug testing was a nuisance, but there wasn't any reason to think it was personal, or that anybody had it in for him. A significant source of stress in his life was lifted.
And understanding not just that things can happen at random, but of the specific consequences of how randomness works can show us that nobody may be to blame. It can show us that we are not a victim.
Probability can be a consolation, sometimes quite a personal one.
Sunday, June 8, 2008
What I think of when sports stars thank God
16000 children died of hunger today, but God made your football team win.
God must really like football.
God must really like football.
Thursday, June 5, 2008
Right change
NEW ORLEANS, Louisiana -- Senator John McCain portrayed himself as the candidate of "right change". "The choice is between the right change and the wrong change," he said.
"No matter what you're buying, you should always count your change. Senator McCain thinks it's very important that you get the right amount of money back," an aide explained. "Also, check down the back of your sofa. I found a dollar fifty three last week. When elected, Senator McCain plans to check every sofa in the White House. Do you have any idea how much the last Hundred Years War cost? And wars are apparently more expensive nowadays. If everyone in America found a dollar fifty three every week, umm... how many people live in America? Let's say a trillion? Okay. Oh wow, is that a penny on the ground? Senator McCain! Senator McCain! I found another one!"
"No matter what you're buying, you should always count your change. Senator McCain thinks it's very important that you get the right amount of money back," an aide explained. "Also, check down the back of your sofa. I found a dollar fifty three last week. When elected, Senator McCain plans to check every sofa in the White House. Do you have any idea how much the last Hundred Years War cost? And wars are apparently more expensive nowadays. If everyone in America found a dollar fifty three every week, umm... how many people live in America? Let's say a trillion? Okay. Oh wow, is that a penny on the ground? Senator McCain! Senator McCain! I found another one!"
Sunday, June 1, 2008
Time again for the triple quote
Hmm. I've almost hit the 8 month mark with this blog. This will be my 71st post, so that's...say (1/5)/(2/3)... 0.3 posts per day on average. (Which is better than I originally thought I'd manage.)
I haven't done the "Three quotes" thing in a while (and sometimes it's not three, anyway - it's not like it's a rule). So I guess it's time to dust it off.
In honour of the just-posted Carnival of Elitist Bastards #1, here's three quotes relating to knowledge and truth ... and their lack.
"I have observed that the world has suffered far less from ignorance than from pretensions to knowledge." - Daniel Boorstin
"The truth may be puzzling. It may take some work to grapple with. It may be counterintuitive. It may contradict deeply held prejudices. It may not be consonant with what we desperately want to be true. But our preferences do not determine what’s true." –Carl Sagan
"If I look over my life, every single step of maturing for me, every single one, has had the exact same common denominator, and that was accepting what was true over what I wished were true." –Julia Sweeney
I haven't done the "Three quotes" thing in a while (and sometimes it's not three, anyway - it's not like it's a rule). So I guess it's time to dust it off.
In honour of the just-posted Carnival of Elitist Bastards #1, here's three quotes relating to knowledge and truth ... and their lack.
"I have observed that the world has suffered far less from ignorance than from pretensions to knowledge." - Daniel Boorstin
"The truth may be puzzling. It may take some work to grapple with. It may be counterintuitive. It may contradict deeply held prejudices. It may not be consonant with what we desperately want to be true. But our preferences do not determine what’s true." –Carl Sagan
"If I look over my life, every single step of maturing for me, every single one, has had the exact same common denominator, and that was accepting what was true over what I wished were true." –Julia Sweeney
Thursday, May 29, 2008
Relationships between percentages are not linear (usually)
Friendly Atheist had some graphs relating to biblical literalism and some other variables, which in turn came from several posts by razib at Gene Expression (e.g.).
I objected in comments to the fact that the relationships had been fitted by straight lines when the relationships would necessarily be curved, because of the fact that percentages are bounded by 0 and 100%. Sure enough, the curvature that we would expect to see was plainly there in some of the graphs.
Now, leaving aside a whole lot of other issues (some of which are addressed in the Friendly Atheist thread, some of which aren't), one way to make the relationships easier to see is to transform the percentages in order to stretch out the values close to the boundaries. One popular transform for proportions is the logit transform (log-odds-ratio).
Let's see its effect on the graph of the relationship between percentage of people with postgrad education and percentage who take the bible literally.
Original:
The R2 value is the square of the linear correlation between the two variables. But the relationship is strongly nonlinear! There's little value in this number.
As I said over in the Friendly Atheist comment thread:
The original post even refers to the "Roman Catholic" point as an outlier. It isn't!. It only looks like an outlier if you're crazy enough to fit a straight line. If you look at it as a curved relationship, it fits in just fine.
Instead of using a linear correlation, we could measure a nonparametric correlation - one that measures the monotonic association between the two variables. That is, something that measures the extent to which one variable increases while the other decreases. There are many such quantities - two common ones are the Kendall measure of concordance (Kendall's tau) and the rank correlation (Spearman's rho).
Because there is a strong relationship - just not a linear one - the monotonic association is higher than the linear correlation for this data. The linear correlation is -0.86, while the Spearman measure of the monotonic association is -0.92.
However, I think the main issue is simply a better display, so let's return to the graph of the suggested transformation.
Transformed:
I did this one in a bit of a hurry, so it's a little rough, but it gives the idea. In order that we can see the relationship more clearly I have omitted the labels for the individual points, but they could be included. Notice that the "Catholic" point (the one to the right of the "10%" tickmark on the vertical axis) is clearly not an unusual point - it fits the pattern nicely.
Note that since the logit transformation is monotonic, this transformation doesn't alter the nonparametric correlations at all. So, for example, the Spearman measure of monotonic association is still -0.92.
The linear correlation on this transformed scale is, however, changed from what it was before, because now the relationship is now almost linear (it's now -0.90). I still don't advocate drawing a line on the plot, however (though a line is now a pretty good description); if a relationship must be drawn in, any of a number of standard nonparametric smoothers could be used. I think we can see the relationship just fine on the second plot.
As to whether the relationship means much of anything, that's another issue, but at least we can now clearly see it, without some distracting straight line (and equation, and r-squared value) on the graph, mis-relating the raw percentages.
I think both graphs provide valuable information - Ideally, I'd be tempted to display both, side-by-side, and since the table of data is small, to give that as well.
Update:
Here's a plot with an added smooth, done on the original raw percentage scale. This one was generated by an old version of SPlus, but R can also generate stuff like this (as can numerous other packages). The smooth here is just the default spline smooth, though the supersmoother was about as good, and the loess smoother would probably work fine if I tweaked its parameters a bit (the default is too local).
This smooth (and the others I mentioned) done on the original scale don't recognize the inherent restrictions I discuss above; I think a better way to smooth would be to transform the data (like the second plot above), smooth that and then if desired, take that smooth back to the original scale. Of course, it's no longer estimating a mean, but that's not such a huge deal - we're just trying to describe a relationship.
Further Update:
Here's a graph of what happens when you smooth on the transformed scale and transform back to the original (percentage) scale. The blue curve is the smooth curve shown above, while the more strongly bent green curve is the smooth done on the logit-logit scale and then brought back. On the transformed scale the default spline smooth was somewhat curved (though much less so than a spline smooth on the original scale), and of course, when we come back, it's definitely curved on the percentage scale.
We shouldn't extrapolate any of these relationships outside the range of the data, but at least within the range of the data, the smooth curves above are not implausible descriptions of how the variables are related to one another.
Also fixed first two links.
I objected in comments to the fact that the relationships had been fitted by straight lines when the relationships would necessarily be curved, because of the fact that percentages are bounded by 0 and 100%. Sure enough, the curvature that we would expect to see was plainly there in some of the graphs.
Now, leaving aside a whole lot of other issues (some of which are addressed in the Friendly Atheist thread, some of which aren't), one way to make the relationships easier to see is to transform the percentages in order to stretch out the values close to the boundaries. One popular transform for proportions is the logit transform (log-odds-ratio).
Let's see its effect on the graph of the relationship between percentage of people with postgrad education and percentage who take the bible literally.
Original:
The R2 value is the square of the linear correlation between the two variables. But the relationship is strongly nonlinear! There's little value in this number.
As I said over in the Friendly Atheist comment thread:
These variables should NOT be having straight lines fitted to their relationships, unless someone really thinks percentages can go outside 0-100!
Look at the fitted “percentage with postgrad education” for the “Church of God In Christ”. It’s NEGATIVE! That makes no sense at all.
At the very least, a functional relationship that at least obeys the a priori facts about the situation (those fractions being bounded to [0,1], for example) should be used. In the first graph, the IQs also have a lower bound of 0, but we’re so many s.d.s from zero it doesn’t matter quite so much for that variable (there’s still the issue that there’s no a priori reason to expect linearity, though).
It really does matter for the percentages, because they approach their limits in this data. Notice the actual relationship from the points is curved in the second graph? That’s because the boundaries force it to be curved. Why is a straight line being fitted to a relationship that is plainly not (and worse, pretty obviously won’t be before we even see data)?
The linear equation, the R-squared and so on are all nonsense - worse than useless! (Indeed, since neither variable is necessarily thought to be causative, why use a technique - regression, whether linear or not - that treats one variable as the predictor and the other as the response?)
The original post even refers to the "Roman Catholic" point as an outlier. It isn't!. It only looks like an outlier if you're crazy enough to fit a straight line. If you look at it as a curved relationship, it fits in just fine.
Instead of using a linear correlation, we could measure a nonparametric correlation - one that measures the monotonic association between the two variables. That is, something that measures the extent to which one variable increases while the other decreases. There are many such quantities - two common ones are the Kendall measure of concordance (Kendall's tau) and the rank correlation (Spearman's rho).
Because there is a strong relationship - just not a linear one - the monotonic association is higher than the linear correlation for this data. The linear correlation is -0.86, while the Spearman measure of the monotonic association is -0.92.
However, I think the main issue is simply a better display, so let's return to the graph of the suggested transformation.
Transformed:
I did this one in a bit of a hurry, so it's a little rough, but it gives the idea. In order that we can see the relationship more clearly I have omitted the labels for the individual points, but they could be included. Notice that the "Catholic" point (the one to the right of the "10%" tickmark on the vertical axis) is clearly not an unusual point - it fits the pattern nicely.
Note that since the logit transformation is monotonic, this transformation doesn't alter the nonparametric correlations at all. So, for example, the Spearman measure of monotonic association is still -0.92.
The linear correlation on this transformed scale is, however, changed from what it was before, because now the relationship is now almost linear (it's now -0.90). I still don't advocate drawing a line on the plot, however (though a line is now a pretty good description); if a relationship must be drawn in, any of a number of standard nonparametric smoothers could be used. I think we can see the relationship just fine on the second plot.
As to whether the relationship means much of anything, that's another issue, but at least we can now clearly see it, without some distracting straight line (and equation, and r-squared value) on the graph, mis-relating the raw percentages.
I think both graphs provide valuable information - Ideally, I'd be tempted to display both, side-by-side, and since the table of data is small, to give that as well.
Update:
Here's a plot with an added smooth, done on the original raw percentage scale. This one was generated by an old version of SPlus, but R can also generate stuff like this (as can numerous other packages). The smooth here is just the default spline smooth, though the supersmoother was about as good, and the loess smoother would probably work fine if I tweaked its parameters a bit (the default is too local).
This smooth (and the others I mentioned) done on the original scale don't recognize the inherent restrictions I discuss above; I think a better way to smooth would be to transform the data (like the second plot above), smooth that and then if desired, take that smooth back to the original scale. Of course, it's no longer estimating a mean, but that's not such a huge deal - we're just trying to describe a relationship.
Further Update:
Here's a graph of what happens when you smooth on the transformed scale and transform back to the original (percentage) scale. The blue curve is the smooth curve shown above, while the more strongly bent green curve is the smooth done on the logit-logit scale and then brought back. On the transformed scale the default spline smooth was somewhat curved (though much less so than a spline smooth on the original scale), and of course, when we come back, it's definitely curved on the percentage scale.
We shouldn't extrapolate any of these relationships outside the range of the data, but at least within the range of the data, the smooth curves above are not implausible descriptions of how the variables are related to one another.
Also fixed first two links.
Monday, May 26, 2008
Who's in the lead?
Percentages don't always do what you expect
I was reading an article at Dailykos (with the ironic title "Montana's huge black population gives Obama the edge"), which is mostly debunking the myth that Obama has a problem with blue-collar whites. (I am not normally given to reading Dailykos, but I do pop in occasionally.)
At one point in the article I was reminded of Simpson's Paradox, and I thought I'd discuss it here, since it lets me talk about elections and about simple mathematics at the same time. Note that the issue doesn't occur in the story to my knowledge - it just served to remind me of it.
Imagine the following situation. Let's say you're looking at how much likely voters approve a particular candidate - let's imagine people are asked a question about exactly one candidate - whether they agree with a statement something like "the candidate would make a good president".
[NB: These numbers are completely made up! They're not real, but are just there to illustrate a point.]
Imagine we ended up with the following table (of percentages):
Note that in this case, different people are being asked about Clinton and McCain; this would not usually be the case with a typical survey. Also, these figures don't have to add to 100% - all candidates might have 80% approval, or all might have 30%.
So on these figures Clinton is slightly leading McCain in approval (maybe not outside the margin of error, but let's ignore that issue). Let's say you want to figure out whether McCain's biggest problem is with males or females. Fortunately, it turns out that this information is available.
Hang on a minute! McCain leads Clinton on men and on women!? Did we make a mistake somewhere?
Actually, no, this is possible. Look at these numbers (counts of people):
If you check those numbers out (unless I made a mistake somewhere), they give the percentages I quote above (to the nearest whole percent).
This is called Simpson's Paradox. It's worth knowing about if you're comparing rates or percentages.
What's going on? Well, as it turns out in our sample, women were less approving overall than men, and more women were asked about McCain than were men.
So which figure matters? Well, actually, if the proportion within each of the sub-samples are representative of the voting population, you'd need to weight each figure by the proportion in the population ... and then McCain would tend to come out looking better.
I was reading an article at Dailykos (with the ironic title "Montana's huge black population gives Obama the edge"), which is mostly debunking the myth that Obama has a problem with blue-collar whites. (I am not normally given to reading Dailykos, but I do pop in occasionally.)
At one point in the article I was reminded of Simpson's Paradox, and I thought I'd discuss it here, since it lets me talk about elections and about simple mathematics at the same time. Note that the issue doesn't occur in the story to my knowledge - it just served to remind me of it.
Imagine the following situation. Let's say you're looking at how much likely voters approve a particular candidate - let's imagine people are asked a question about exactly one candidate - whether they agree with a statement something like "the candidate would make a good president".
[NB: These numbers are completely made up! They're not real, but are just there to illustrate a point.]
Imagine we ended up with the following table (of percentages):
Percentage of people surveyed that agree that
candidate would make a good president
McCain (R) 38
Clinton (D) 41
Note that in this case, different people are being asked about Clinton and McCain; this would not usually be the case with a typical survey. Also, these figures don't have to add to 100% - all candidates might have 80% approval, or all might have 30%.
So on these figures Clinton is slightly leading McCain in approval (maybe not outside the margin of error, but let's ignore that issue). Let's say you want to figure out whether McCain's biggest problem is with males or females. Fortunately, it turns out that this information is available.
Percentage of people surveyed that agree that
candidate would make a good president
Men Women
McCain (R) 46 36
Clinton (D) 43 34
Hang on a minute! McCain leads Clinton on men and on women!? Did we make a mistake somewhere?
Actually, no, this is possible. Look at these numbers (counts of people):
Number of people surveyed that agree that
candidate would make a good president
Men Women Total
McCain Approved 80 189 269
Number Asked 174 526 700
Clinton Approved 233 55 288
Number Asked 540 160 700
If you check those numbers out (unless I made a mistake somewhere), they give the percentages I quote above (to the nearest whole percent).
This is called Simpson's Paradox. It's worth knowing about if you're comparing rates or percentages.
What's going on? Well, as it turns out in our sample, women were less approving overall than men, and more women were asked about McCain than were men.
So which figure matters? Well, actually, if the proportion within each of the sub-samples are representative of the voting population, you'd need to weight each figure by the proportion in the population ... and then McCain would tend to come out looking better.
Saturday, May 24, 2008
Fame! Fortune! ... well, neither, really
... but on the latest Monday Math Madness - they used one of my solutions. (solutions not linked directly so as not to spoil it for anyone)
Friday, May 23, 2008
My take on...
My take on Ray Comfort's banana shtick:
Jesus said of the false prophets, "By their fruits shall ye know them."
Jesus said of the false prophets, "By their fruits shall ye know them."
Thursday, May 22, 2008
A couple of relevant quotes.
"What kind of peace do we seek? Not a Pax Americana enforced on the world by American weapons of war. Not the peace of the grave or the security of the slave. I am talking about genuine peace, the kind of peace that makes life on earth worth living, the kind that enables men and nations to grow and to hope and to build a better life for their children - not merely peace for Americans but peace for all men and women - not merely peace in our time but peace for all time." - JFK
"Let us never negotiate out of fear. But let us never fear to negotiate." - JFK
Contrasting these sentiments with the current Republican lackwit talking points is left as an exercise for the reader.
"Let us never negotiate out of fear. But let us never fear to negotiate." - JFK
Contrasting these sentiments with the current Republican lackwit talking points is left as an exercise for the reader.
Tuesday, May 20, 2008
Celebrating the intellect
The underlying message of the Intelligent Design movement, and most especially Expelled is "Scientists are too dumb to understand science". It is an attack on science, masquerading as an attack on elitism (and various other attacks). It is an eerie echo of the usual attack on experts by sociopolitical elites: "Don't listed to smart people. They don't understand you."
In every case, they're asking you to stop thinking.
Viva eggheads!
We need smart people. We need experts. We need people who are prepared to devote years of their lives to studying a subject. These days, human knowledge is vast, and increasingly complex.
And why is it that it takes a lot of work to become an expert these days? Why is our knowledge vast and growing? Because we build on the work of other experts. In the 1930s, vast numbers of people - many of them children - died or were crippled by polio.
Where are they now? Why don't we have thousands of crippled children? Science. Experts. Eggheads.
Smallpox used to kill thousands every year and disfigure thousands more. Where are all the new victims of smallpox?
In the early decades of last century, and all the years that preceeded it, simple infections killed millions. If it weren't for antibiotics, I'd be dead, bedridden or at best so sickly as to be barely able to work. So would many of my friends and family. If it weren't for other forms of medical treatment, almost everyone I know would be dead, or at best, suffering. One of my heroes is in the Carnival banner over on the right.
We fight an ongoing war with bacteria, which (via evolution) gradually become able to deal with antibiotics. Experts - scientists - work to keep a step ahead, saving countless lives.
Without spectacles, I'd be so blind as to be useless for most tasks (these days, I could even choose to have laser surgery).
We need to celebrate intellectualism - we need some tiny bastion against the permanent Carnival of the Stupid that those who would seek to impose their will on ours would like us to have.
What would be the ultimate consequences of this growth industry of anti-intellectualism, this mantra of whatever you want to be true is true?
Stuff like teachers who know the toothpick trick getting fired for practising wizardy. Gee, I hope something that stupid never happens.
The deadly danger of anti-knowledge rhetoric
Currently the position being held when people are decrying "elites" is in reality, anti-expert rhetoric. It's not even listening to the opinions of people who have devoted their working lives to the study of a particular subject.
Now, I'm not suggesting that they will have all the answers. But experts are experts because they know their area. It would pay to understand why they're saying what they're saying. I am not suggesting that their opinions can never be assailed. Not so; if you learn what they know, then you can take them on in their arena. Who knows, maybe you'll even come up with a good idea. But more likely, everything you think of will have already been thought of and either debunked or already tried. The experts will be able to tell you who has tried it, when and how it turned out, or they will know who debunked it, how they debunked it, and what you'll have to overcome to get it to work.
To just dismiss almost universal opinions of the knowledgeable as "elitist" is a ridiculous ploy. It's appealing to the lowest common denominator - "you're just as good as they are" is the implication. Well, that's true - human lives are all valuable. But the additional implication - "a completely ignorant opinion is just as good as any experts" is ludicrous. The hidden claim is that "smart people are stupid". Well, that is stupid.
The subtext of the dismissal of expert opinion as elitist is "believe what you want - it's just as true as reality". That's not only ludicrous, it's dangerous.
What does it get us?
To exploit this kind of anti-expert rhetoric for cheap (and temporary) advantage... that deserves our most vehement contempt. Politicians - at least ones worthy of our vote - should be leading by example, be encouraging us to be smart, be discussing difficult issues and trusting us to realize that they're not going to be solved by platitudes and ignorant policies. They should not be exploiting ignorance, and we should not be rewarding them for treating us like morons. They should be denounced, loudly. If the media won't do it, then we must.
Beware the wealthy and influential masquerading as populists. They're liars, and their favourite lie is to slander someone smart and hardworking with the label "elite". Don't trust them, because they think you're stupid and can be exploited if they just push a few buttons.
In every case, they're asking you to stop thinking.
Viva eggheads!
We need smart people. We need experts. We need people who are prepared to devote years of their lives to studying a subject. These days, human knowledge is vast, and increasingly complex.
And why is it that it takes a lot of work to become an expert these days? Why is our knowledge vast and growing? Because we build on the work of other experts. In the 1930s, vast numbers of people - many of them children - died or were crippled by polio.
Where are they now? Why don't we have thousands of crippled children? Science. Experts. Eggheads.
Smallpox used to kill thousands every year and disfigure thousands more. Where are all the new victims of smallpox?
In the early decades of last century, and all the years that preceeded it, simple infections killed millions. If it weren't for antibiotics, I'd be dead, bedridden or at best so sickly as to be barely able to work. So would many of my friends and family. If it weren't for other forms of medical treatment, almost everyone I know would be dead, or at best, suffering. One of my heroes is in the Carnival banner over on the right.
We fight an ongoing war with bacteria, which (via evolution) gradually become able to deal with antibiotics. Experts - scientists - work to keep a step ahead, saving countless lives.
Without spectacles, I'd be so blind as to be useless for most tasks (these days, I could even choose to have laser surgery).
We need to celebrate intellectualism - we need some tiny bastion against the permanent Carnival of the Stupid that those who would seek to impose their will on ours would like us to have.
What would be the ultimate consequences of this growth industry of anti-intellectualism, this mantra of whatever you want to be true is true?
Stuff like teachers who know the toothpick trick getting fired for practising wizardy. Gee, I hope something that stupid never happens.
The deadly danger of anti-knowledge rhetoric
Currently the position being held when people are decrying "elites" is in reality, anti-expert rhetoric. It's not even listening to the opinions of people who have devoted their working lives to the study of a particular subject.
Now, I'm not suggesting that they will have all the answers. But experts are experts because they know their area. It would pay to understand why they're saying what they're saying. I am not suggesting that their opinions can never be assailed. Not so; if you learn what they know, then you can take them on in their arena. Who knows, maybe you'll even come up with a good idea. But more likely, everything you think of will have already been thought of and either debunked or already tried. The experts will be able to tell you who has tried it, when and how it turned out, or they will know who debunked it, how they debunked it, and what you'll have to overcome to get it to work.
To just dismiss almost universal opinions of the knowledgeable as "elitist" is a ridiculous ploy. It's appealing to the lowest common denominator - "you're just as good as they are" is the implication. Well, that's true - human lives are all valuable. But the additional implication - "a completely ignorant opinion is just as good as any experts" is ludicrous. The hidden claim is that "smart people are stupid". Well, that is stupid.
The subtext of the dismissal of expert opinion as elitist is "believe what you want - it's just as true as reality". That's not only ludicrous, it's dangerous.
What does it get us?
- Global Warming denialists
- Overfishing of crucial marine resources
- Creationism in classrooms
- The hell-hole that Iraq has been turned into
- The post-Katrina mess
- Anti-vaxers playing footsie with polio... and risking everyone else's health
- Flagrantly stupid energy policies that will lead to much worse problems down the track, like higher energy prices as the fuel runs out even faster, and no reserves to deal with temporary shortages.
To exploit this kind of anti-expert rhetoric for cheap (and temporary) advantage... that deserves our most vehement contempt. Politicians - at least ones worthy of our vote - should be leading by example, be encouraging us to be smart, be discussing difficult issues and trusting us to realize that they're not going to be solved by platitudes and ignorant policies. They should not be exploiting ignorance, and we should not be rewarding them for treating us like morons. They should be denounced, loudly. If the media won't do it, then we must.
Beware the wealthy and influential masquerading as populists. They're liars, and their favourite lie is to slander someone smart and hardworking with the label "elite". Don't trust them, because they think you're stupid and can be exploited if they just push a few buttons.
God of the non-gaps
Evolved and Rational's latest post is another delightful snark, with which I am nevertheless going to take issue.
Perhaps uncharacteristically, Evolved is being far too kind this time, I'm afraid.
She says: "Behe's theology is one where god resides in the gaps of human knowledge [...] his criteria for detecting design is merely what has not been explained by science at the time. "
I think the first part of that quote is what Behe was aiming for. Unfortunately, he missed - Behe's examples of gaps, his cases of "irreducible complexity", his examples of non-evolution, are wrong.
On HIV, Behe was wrong. He was thoroughly spanked by erv (erv is now at scienceblogs), pursued at Panda's Thumb (e.g. ), and eventually forced to admit that he was wrong. A sucky admission, but still, he finally noticed he'd been pwned.
On the bacterial flagellum, Behe was wrong. Epically wrong.
On the immune system and much else besides, Behe was wrong
These are just examples, there are lots of others.
I'm not aware of a single instance where Behe managed to pick an actual legitimate gap.
[It's not like it's hard. As the cryptotheist evolution-denialists are so fond of pointing out, just like building a drystone wall, every time a gap is filled in, there are more (but much tinier) gaps around it - so these days there are plenty of little gaps to go around, if you have even the slightest clue about where to look. Of course the big problem for the cryptotheist is that to actually find the genuine gaps you have to know enough science to be able to read a few papers - that is, to display a basic competence one might expect from a beginning science postgrad.]
While trying to construct a God-of-the-gaps (a trivial exercise for even the barely competent), Behe - doubtless inspired by God - constructed a God-of-the-non-gaps, a God-of-the-reducibly-complex, a God-of-the-already-explained-by-science.
Behe's work is a masterpiece of epic fail. Perhaps God is trying to tell you something, Michael. Like how much you suck at science. (As Abbie pointed out, you also suck at Google.)
(I won't even bother with Dembski. His work is more incompetent, contentless and pathetic than that of Behe. And he's an unrepentant - and incompetent - thief. Why pussyfoot about, let's just go the whole hog. Dembski just generally sucks.)
Perhaps uncharacteristically, Evolved is being far too kind this time, I'm afraid.
She says: "Behe's theology is one where god resides in the gaps of human knowledge [...] his criteria for detecting design is merely what has not been explained by science at the time. "
I think the first part of that quote is what Behe was aiming for. Unfortunately, he missed - Behe's examples of gaps, his cases of "irreducible complexity", his examples of non-evolution, are wrong.
On HIV, Behe was wrong. He was thoroughly spanked by erv (erv is now at scienceblogs), pursued at Panda's Thumb (e.g. ), and eventually forced to admit that he was wrong. A sucky admission, but still, he finally noticed he'd been pwned.
On the bacterial flagellum, Behe was wrong. Epically wrong.
On the immune system and much else besides, Behe was wrong
These are just examples, there are lots of others.
I'm not aware of a single instance where Behe managed to pick an actual legitimate gap.
[It's not like it's hard. As the cryptotheist evolution-denialists are so fond of pointing out, just like building a drystone wall, every time a gap is filled in, there are more (but much tinier) gaps around it - so these days there are plenty of little gaps to go around, if you have even the slightest clue about where to look. Of course the big problem for the cryptotheist is that to actually find the genuine gaps you have to know enough science to be able to read a few papers - that is, to display a basic competence one might expect from a beginning science postgrad.]
While trying to construct a God-of-the-gaps (a trivial exercise for even the barely competent), Behe - doubtless inspired by God - constructed a God-of-the-non-gaps, a God-of-the-reducibly-complex, a God-of-the-already-explained-by-science.
Behe's work is a masterpiece of epic fail. Perhaps God is trying to tell you something, Michael. Like how much you suck at science. (As Abbie pointed out, you also suck at Google.)
(I won't even bother with Dembski. His work is more incompetent, contentless and pathetic than that of Behe. And he's an unrepentant - and incompetent - thief. Why pussyfoot about, let's just go the whole hog. Dembski just generally sucks.)
Friday, May 16, 2008
Sex and Drugs and guys who wear Prada with their dresses...
The Pope recently came out and says that sex can be like a drug.
I find it kind of amusing, to be honest, but I guess I should be used to it by now - religious leaders talking at length about things of which they presumably have no experience, as though they knew exactly what they were talking about (what God thinks about playing video games, what God thinks about pollution, what God thinks about who should be President...).
The Pope's statement brings to mind a vision of some really great parties at the Vatican involving snorting cocaine off trays on the heads of dwarves while they fiddle with... but you know, like the parties they're always having. Apparently.
But rather than a long diatribe, I'm going to leave you with a little bit of Blackadder, from season 1 (the season where Edmund Blackadder and Percy are in the last days of pre-Tudor England). Richard Curtis says it better than I could.
Percy: You know, they do say that the Infanta’s eyes are more beautiful than the famous Stone of Galveston.
Blackadder: Mm? ... What?
Percy: The famous Stone of Galveston, My Lord.
Blackadder: And what’s that, exactly?
Percy: “Well, it’s a famous blue stone, and it comes... from Galveston.”
Blackadder: “I see. And what about it?”
Percy: “Well, My Lord, the Infanta’s eyes are bluer than it, for a start.”
Blackadder: “I see. And have you ever seen this stone?”
Percy: “No, not as such, My Lord, but I know a couple of people who have, and they say it’s very, very blue indeed.”
Blackadder: “And have these people seen the Infanta’s eyes?”
Percy: “No, I shouldn’t think so, My Lord.”
Blackadder: “And neither have you, presumably.”
Percy: “No, My Lord.”
Blackadder: “So, what you’re telling me, Percy, is that something you have never seen is slightly less blue than something else... you have never seen.”
I find it kind of amusing, to be honest, but I guess I should be used to it by now - religious leaders talking at length about things of which they presumably have no experience, as though they knew exactly what they were talking about (what God thinks about playing video games, what God thinks about pollution, what God thinks about who should be President...).
The Pope's statement brings to mind a vision of some really great parties at the Vatican involving snorting cocaine off trays on the heads of dwarves while they fiddle with... but you know, like the parties they're always having. Apparently.
But rather than a long diatribe, I'm going to leave you with a little bit of Blackadder, from season 1 (the season where Edmund Blackadder and Percy are in the last days of pre-Tudor England). Richard Curtis says it better than I could.
Percy: You know, they do say that the Infanta’s eyes are more beautiful than the famous Stone of Galveston.
Blackadder: Mm? ... What?
Percy: The famous Stone of Galveston, My Lord.
Blackadder: And what’s that, exactly?
Percy: “Well, it’s a famous blue stone, and it comes... from Galveston.”
Blackadder: “I see. And what about it?”
Percy: “Well, My Lord, the Infanta’s eyes are bluer than it, for a start.”
Blackadder: “I see. And have you ever seen this stone?”
Percy: “No, not as such, My Lord, but I know a couple of people who have, and they say it’s very, very blue indeed.”
Blackadder: “And have these people seen the Infanta’s eyes?”
Percy: “No, I shouldn’t think so, My Lord.”
Blackadder: “And neither have you, presumably.”
Percy: “No, My Lord.”
Blackadder: “So, what you’re telling me, Percy, is that something you have never seen is slightly less blue than something else... you have never seen.”
Thursday, May 15, 2008
Possible banner for the Carnival of Elitist Bastards
Since I have an anti-"anti-elitism" post set for the Carnival of Elitist Bastards (and this too), I thought I'd have a go at a banner. I stole one of the banners here and modified it. Because, you know, a bastard elitist egghead like me won't admit that there's something he couldn't do if he wanted.
So, anyway, it's a bit rough, but here we go:
(Edit: I just noticed a minor issue, which I am not going to fix. So there.)
So, anyway, it's a bit rough, but here we go:
(Edit: I just noticed a minor issue, which I am not going to fix. So there.)
US elections
So Edwards has declared for Obama, and polls show both Obama and Clinton with substantial leads over McCain (47-40 for Obama, 46-41 for McCain). If you look at the rate superdelegates are declaring for Obama, Clinton could win every race 70-30 (or even higher) and she would still be falling behind in total delegates faster than before, and looking at the poll results, one of her main arguments for remaining in the race (that she would do better in the race against McCain) doesn't hold water.
To be honest, I think a Clinton presidency would, in many respects, be a reasonably good one. I don't think that's enough. A curate's egg presidency? - when there's another choice?
Of course that's just my opinion - it's not like I can vote. I got my chance at that months ago.
To be honest, I think a Clinton presidency would, in many respects, be a reasonably good one. I don't think that's enough. A curate's egg presidency? - when there's another choice?
Of course that's just my opinion - it's not like I can vote. I got my chance at that months ago.
Tuesday, May 13, 2008
Selecting a random uniform positive integer
This post is edited together from some comments arising out of comments at Pharyngula, where some commenters were discussing randomly selecting a positive integer (with equal probability on every positive integer), in a discussion of creationists abuse of mathematics.
The problem is, you can't do it.
It is simply not possible to actually select an integer at random where each one has the same probability of being chosen.
In order to be able to construct something which could have a uniform distribution over a countably infinite set, you'd have to drop Kolmogorov's axiom of countable additivity.
Such a thing is possible to do, but when people deal with countably infinite sample spaces they do use the axiom, so whatever "probability" you end up with by dropping it will not be like the probability that people use.
(In measure theory terms, it would be put something like: The union of a countable number of measurable sets cannot have measure 1 with all the sets having equal measure.)
What does that mean? It means you either have to give up probabilities adding to 1, or you have to give up equal probability on your countably infinite set.
A probabilist called de Finetti (and facist, as it happens, but that has nothing to do with whether his work was any good) claimed it was possible to have a notion that was effectively a uniform distribution over a countably infinite set by dropping the aforementioned axiom.
That is, in effect, he dropped the "probabilities add to 1" part that would otherwise be a consequence of the axiom. In effect, he said each probability was zero. The uncomfortable consequence is that the probability that some number is chosen is not 1, but 0.
Other probabilists have tried making each probability > 0. The uncomfortable consequence there is that of course, the probability that some number is chosen becomes infinite.
No actual mechanism for producing a random number from these somewhat unintuitive constructs exists. Any process that gives you a number will be inevitably weighted toward the small numbers (give small numbers generally higher probability than sufficiently large numbers).
Now, if you're only using such a construct to represent something like a state of knowledge**, and not actually requiring to be able to actually observe a value from it, it is sometimes possible to work with such things. That's the sort of thing that's effectively being done when Bayesians work with flat priors over the positive integers (they refer to things like that as "improper priors") - with appropriate normalization, you can get proper posteriors to come out the other end.
** Some people regard the uniform over the positive integers as representing a complete lack of knowledge, but to my mind it actually represents the dramatically strong assumption that no matter how large a value you consider, you effectively assume that essentially all values are larger than it.
A fair number of people object to this unsual kind of construct, insisting that subjective probability and actual realizable probability ought to have the same features. They also like to point to the paradoxes that come along with not doing so, with which I am not presently sufficiently familiar to discuss right now.
Anyway, the short version of all that is "You can't actually select an element from a countably infinite set (such as the positive integers, all the integers, the rationals, or the differences of the square roots of non-negative integers) with equal probability on all elements."
Indeed, it's stronger than that. For example, not only does the probability need to generally decrease as the values become arbitrarily large, the probability can't decrease very slowly. For example, it can't even decrease at a rate proportional to 1/k -- though a finite number of values need not be required to follow the general decrease.
The problem is, you can't do it.
It is simply not possible to actually select an integer at random where each one has the same probability of being chosen.
In order to be able to construct something which could have a uniform distribution over a countably infinite set, you'd have to drop Kolmogorov's axiom of countable additivity.
Such a thing is possible to do, but when people deal with countably infinite sample spaces they do use the axiom, so whatever "probability" you end up with by dropping it will not be like the probability that people use.
(In measure theory terms, it would be put something like: The union of a countable number of measurable sets cannot have measure 1 with all the sets having equal measure.)
What does that mean? It means you either have to give up probabilities adding to 1, or you have to give up equal probability on your countably infinite set.
A probabilist called de Finetti (and facist, as it happens, but that has nothing to do with whether his work was any good) claimed it was possible to have a notion that was effectively a uniform distribution over a countably infinite set by dropping the aforementioned axiom.
That is, in effect, he dropped the "probabilities add to 1" part that would otherwise be a consequence of the axiom. In effect, he said each probability was zero. The uncomfortable consequence is that the probability that some number is chosen is not 1, but 0.
Other probabilists have tried making each probability > 0. The uncomfortable consequence there is that of course, the probability that some number is chosen becomes infinite.
No actual mechanism for producing a random number from these somewhat unintuitive constructs exists. Any process that gives you a number will be inevitably weighted toward the small numbers (give small numbers generally higher probability than sufficiently large numbers).
Now, if you're only using such a construct to represent something like a state of knowledge**, and not actually requiring to be able to actually observe a value from it, it is sometimes possible to work with such things. That's the sort of thing that's effectively being done when Bayesians work with flat priors over the positive integers (they refer to things like that as "improper priors") - with appropriate normalization, you can get proper posteriors to come out the other end.
** Some people regard the uniform over the positive integers as representing a complete lack of knowledge, but to my mind it actually represents the dramatically strong assumption that no matter how large a value you consider, you effectively assume that essentially all values are larger than it.
A fair number of people object to this unsual kind of construct, insisting that subjective probability and actual realizable probability ought to have the same features. They also like to point to the paradoxes that come along with not doing so, with which I am not presently sufficiently familiar to discuss right now.
Anyway, the short version of all that is "You can't actually select an element from a countably infinite set (such as the positive integers, all the integers, the rationals, or the differences of the square roots of non-negative integers) with equal probability on all elements."
Indeed, it's stronger than that. For example, not only does the probability need to generally decrease as the values become arbitrarily large, the probability can't decrease very slowly. For example, it can't even decrease at a rate proportional to 1/k -- though a finite number of values need not be required to follow the general decrease.
Subscribe to:
Posts (Atom)
