(budding ... becaue that's how mathematicians reproduce, of course.)
My daughter, who is in year three, was given the opportunity to enter the Australian Mathematics Competition (which presently has kids from over 40 countries competing in it).
Not everyone gets to enter; at her school it was only offered to the kids in the extension class.
She got a distinction, which places her in the top 15% of her division (the year 3 and 4 students that entered). Pretty good going, I thought.
Edit: Turns out she was in the top 6% among the entrants in the competition, for her year and the year above.
---
I believe I actually competed in an early incarnation of the same competition way back when I was a school kid, in around year 10 - probably the first year it was offered in NSW, when it was called the Wales awards. (Assuming it's the same competition; in any case I got $40 in an account from the Wales bank, now Westpac. For me at the time that was a lot of money and I used it carefully - I didn't actually finish spending it until I went to university, nearly two and a half years later. I don't know if they still offer any cash.)
Showing posts with label mathematics. Show all posts
Showing posts with label mathematics. Show all posts
Wednesday, November 3, 2010
Thursday, June 25, 2009
Erdős number - do published books count?
In the past I've tried tracking my Erdős number through a particular coauthor that I think is likely to have the smallest number of all my coauthors. Today I tried doing it a different way, through someone else who I know well but don't have a direct publication with (who I just realized would have a low number)... and found a link to him.
But the question is, is the series of links I found legit?
Do coauthors of published books (which are not only citeable but cited) count as links toward Erdős number?
If so, mine is at most 5. If I look hard enough I can probably get it down to papers only, but if books count I don't need to spend the time looking. (Later edit: found a papers-only link that's of length 6. Considering its convolutedness I expect there's a shorter one to be found somewhere.)
Judging by the discussion later in the wikipedia article, the answer is that the book counts. So E ≤ 5.
One day I'm going to take some time and try to find a shorter link.
But the question is, is the series of links I found legit?
Do coauthors of published books (which are not only citeable but cited) count as links toward Erdős number?
If so, mine is at most 5. If I look hard enough I can probably get it down to papers only, but if books count I don't need to spend the time looking. (Later edit: found a papers-only link that's of length 6. Considering its convolutedness I expect there's a shorter one to be found somewhere.)
Judging by the discussion later in the wikipedia article, the answer is that the book counts. So E ≤ 5.
One day I'm going to take some time and try to find a shorter link.
Monday, March 2, 2009
It's times like this I wish...
... that I had more than high school physics.
I was solving a nifty little optimization problem which occurred to me as a continuous version of a discrete problem that used to come up in an old computer game I used to play long ago. The problem boiled down to finding a best route of travel given two different speeds in different kinds of terrain.
Anyway, after a page of scribbling around, I came up with a formula for a simple case of the problem.
Then I realized that my simple case was (in a modified form) essentially going to be solved by Snell's Law (also called the Law of Sines). And sure enough, my simple formula was Snell's law (but changed about a bit so it wasn't instantly obvious, like having a ratio of cosecants of angles to the normal instead of sines - which is just a matter of inverting both sides...).
This is essentially what I was doing, but less formally and with a lot more faffing about and a few false starts. (It's easy to find on-line if you already know what to look for, eh?)
I felt like a bit of a dope. On the other hand, at least I was definitely on the right track. Since I'd had a headache all day, that was all I could manage in the time available, so I didn't go on to derive the problem I was actually interested in (the shape made by traveling as far as you could in a given time, given a particular boundary between the two regions), but with Snell's Law it should become a somewhat more straightforward calculation for the situations I was playing with (since it tells me "where to head" after striking a smooth boundary, so for the simpler cases it's a matter of computing where you end up given each boundary point).
I was solving a nifty little optimization problem which occurred to me as a continuous version of a discrete problem that used to come up in an old computer game I used to play long ago. The problem boiled down to finding a best route of travel given two different speeds in different kinds of terrain.
Anyway, after a page of scribbling around, I came up with a formula for a simple case of the problem.
Then I realized that my simple case was (in a modified form) essentially going to be solved by Snell's Law (also called the Law of Sines). And sure enough, my simple formula was Snell's law (but changed about a bit so it wasn't instantly obvious, like having a ratio of cosecants of angles to the normal instead of sines - which is just a matter of inverting both sides...).
This is essentially what I was doing, but less formally and with a lot more faffing about and a few false starts. (It's easy to find on-line if you already know what to look for, eh?)
I felt like a bit of a dope. On the other hand, at least I was definitely on the right track. Since I'd had a headache all day, that was all I could manage in the time available, so I didn't go on to derive the problem I was actually interested in (the shape made by traveling as far as you could in a given time, given a particular boundary between the two regions), but with Snell's Law it should become a somewhat more straightforward calculation for the situations I was playing with (since it tells me "where to head" after striking a smooth boundary, so for the simpler cases it's a matter of computing where you end up given each boundary point).
Tuesday, November 18, 2008
A little tidbit I should have already known
I was reading online about a game I bought while in the US and which I have only just had time to take a peek at. Someone made a point about many parts of the game being based on the fact that a right triangle with sides of 7 and 4 units has a hypotenuse with length very close to 8.
Well, 72 + 42 = 82 + 1
so the hypotenuse is close to 8, as suggested: √(72 + 42) ≅ 8.
In fact, I knew √65 to be very close to 8 1/16
(if x is not too small, √(x2 + 1) ≅ x + 1/2x).
(Note that (x + 1/2x)2 = x2 + 1 + 1/4x² , and if x >> 1, the final term is quite small )
So that's an error of around 1/128, or about 0.8%; pretty good, since the game aims for much less accuracy than that in general.
But then I thought about the fact that 16 in the denominator was a bit too small, and I wondered about how much. I realized straight away that it was in fact about a sixteenth too small. That is, it occurred to me that √65 is very close to 8 1/(16 + ¹/16).
A little light went off in my head, so I hauled out my calculator.
Try this with me, if you have a calculator handy:
Take the square root of 65. (You should see 8.06225...)
Now subtract 8 (the bit we know).
Take the reciprocal (¹/x). You get 16 and a bit.
Subtract 16 and take the reciprocal. Looks like you get the same number back...
What is this number? A tiny bit of algebra shows it's 8 + √65.
So far, that may seem like a trivial curiosity. But this happens all over.
For example, you get the same thing with any positive integer, x;
√(x2 + 1) + x is a number like that "16 and a bit", where
you can keep subtracting that integer part and taking the reciprocal.
That is, expressions like 8 1/(16 + ¹/(16+ ...)) come up lots of times (and recognizing that I'd hit one of these objects was what made the light go off).
Take √10 for example - it's 3 1/(6 + ¹/(6+...))
And you don't just get it with roots of 1 more than a perfect square. As I said before, it happens all over.
We've hit continued fractions. They come up a fair bit in mathematics, and they appear in numerous places where rational approximation comes in - I remember playing with them when dealing with asymptotic approximations in statistics, for example. There's a much nicer notation (see the wikipedia article), so if you're playing with them you're not stuck with endless layers of fraction running down the page.
So, for example, the sequence 8, 8 1/16, 8 1/(16 + ¹/16), ... 8 1/(16 + ¹/(16+ 1/16...)) would be rendered as:
8, [8; 16], [8; 16, 16], ... [8; 16, 16, ...]
Similarly, √10 is [3; 6, 6, 6, ...].
The well known continued fraction for √2 falls into this class: [1; 2, 2, 2...].
Compute a few terms in that sequence with me:
1, 1.5, 1.4, 1 5/12 = 1.416666... , ...
already we're quite close - and it continues to jump about either side of √2, getting closer and closer.
For larger numbers, the convergence is much faster. The general continued fraction for √(x2 + 1) is [x; 2x, 2x, 2x, ...].
Try seeing if you can work out what is going on with square roots with different offsets from a perfect square.
So anyway not only is there a handy way of computing square roots that are close to perfect squares, there's a handy way to improve the calculation if it wasn't as accurate as you needed.
There are many beautiful things related to continued fractions. Take a look over at MathWorld if you've a mind for some boggling factoids.
What fun.
(Two posts in one day! OMFFSM)
Well, 72 + 42 = 82 + 1
so the hypotenuse is close to 8, as suggested: √(72 + 42) ≅ 8.
In fact, I knew √65 to be very close to 8 1/16
(if x is not too small, √(x2 + 1) ≅ x + 1/2x).
(Note that (x + 1/2x)2 = x2 + 1 + 1/4x² , and if x >> 1, the final term is quite small )
So that's an error of around 1/128, or about 0.8%; pretty good, since the game aims for much less accuracy than that in general.
But then I thought about the fact that 16 in the denominator was a bit too small, and I wondered about how much. I realized straight away that it was in fact about a sixteenth too small. That is, it occurred to me that √65 is very close to 8 1/(16 + ¹/16).
A little light went off in my head, so I hauled out my calculator.
Try this with me, if you have a calculator handy:
Take the square root of 65. (You should see 8.06225...)
Now subtract 8 (the bit we know).
Take the reciprocal (¹/x). You get 16 and a bit.
Subtract 16 and take the reciprocal. Looks like you get the same number back...
What is this number? A tiny bit of algebra shows it's 8 + √65.
So far, that may seem like a trivial curiosity. But this happens all over.
For example, you get the same thing with any positive integer, x;
√(x2 + 1) + x is a number like that "16 and a bit", where
you can keep subtracting that integer part and taking the reciprocal.
That is, expressions like 8 1/(16 + ¹/(16+ ...)) come up lots of times (and recognizing that I'd hit one of these objects was what made the light go off).
Take √10 for example - it's 3 1/(6 + ¹/(6+...))
And you don't just get it with roots of 1 more than a perfect square. As I said before, it happens all over.
We've hit continued fractions. They come up a fair bit in mathematics, and they appear in numerous places where rational approximation comes in - I remember playing with them when dealing with asymptotic approximations in statistics, for example. There's a much nicer notation (see the wikipedia article), so if you're playing with them you're not stuck with endless layers of fraction running down the page.
So, for example, the sequence 8, 8 1/16, 8 1/(16 + ¹/16), ... 8 1/(16 + ¹/(16+ 1/16...)) would be rendered as:
8, [8; 16], [8; 16, 16], ... [8; 16, 16, ...]
Similarly, √10 is [3; 6, 6, 6, ...].
The well known continued fraction for √2 falls into this class: [1; 2, 2, 2...].
Compute a few terms in that sequence with me:
1, 1.5, 1.4, 1 5/12 = 1.416666... , ...
already we're quite close - and it continues to jump about either side of √2, getting closer and closer.
For larger numbers, the convergence is much faster. The general continued fraction for √(x2 + 1) is [x; 2x, 2x, 2x, ...].
Try seeing if you can work out what is going on with square roots with different offsets from a perfect square.
So anyway not only is there a handy way of computing square roots that are close to perfect squares, there's a handy way to improve the calculation if it wasn't as accurate as you needed.
There are many beautiful things related to continued fractions. Take a look over at MathWorld if you've a mind for some boggling factoids.
What fun.
(Two posts in one day! OMFFSM)
Monday, October 20, 2008
Statistics as philosophy
This post is related to a point I often try to make (not that I am completely of a mind with the author, but much of what he's saying I identify with).
Fundamentally, statistics is different from mathematics, though it uses the tools of mathematics. Mathematics helps with the "what" (such as "given I want to measure this, what do I do?", but the "why" (such as in the sense of "why work that out, rather than something else") is somewhere other than mathematics.
This point is often lost on otherwise highly competent people. I have seen many good mathematicians come a cropper on it. Some of the worst explanations of statistics I have ever seen come not from people who have trouble with the mathematics in it, but from people who have no trouble with the mathematics at all. (I could name names, but I am feeling generous today.)
And it's often like that with students - I see it a lot. As a student, I had a similar experience to the poster I linked to - I was - manipulation-wise - reasonably competent at statistics. I could do the calculations, if it was reasonably clear what calculations were required. But I did two years of it and still didn't comprehend it. I didn't even comprehend that there was something to comprehend - after all, I could pass the subjects okay, so I must have 'got it' okay, even though it seemed sort of wishy-washy to me. But actually I didn't get it at all. It wasn't until I was some doing third-year subjects that it eventually clicked. I suddenly understood what all the previous subjects had been about. I got it. The material I had learned wasn't a bunch of different stuff all lumped together that was done the way it was purely by convention (though there are no shortage of conventions) - there was, in fact, a coherence to it all. It was all of a thing, it fitted together; the stuff I'd learned was the result of a limited collection of concepts applied to different problems. I could actually begin applying my understanding outside my direct learning, to problems I'd not seen before. I had a framework within which each new piece of knowledge fitted in with everything else.
I'm not certain how to even convey this understanding, though I try. Students recognise that I'm passionate, at least (or so they tell me), though I'm not sure that the "why" always comes across to more than a very few of them.
Fundamentally, statistics is different from mathematics, though it uses the tools of mathematics. Mathematics helps with the "what" (such as "given I want to measure this, what do I do?", but the "why" (such as in the sense of "why work that out, rather than something else") is somewhere other than mathematics.
This point is often lost on otherwise highly competent people. I have seen many good mathematicians come a cropper on it. Some of the worst explanations of statistics I have ever seen come not from people who have trouble with the mathematics in it, but from people who have no trouble with the mathematics at all. (I could name names, but I am feeling generous today.)
And it's often like that with students - I see it a lot. As a student, I had a similar experience to the poster I linked to - I was - manipulation-wise - reasonably competent at statistics. I could do the calculations, if it was reasonably clear what calculations were required. But I did two years of it and still didn't comprehend it. I didn't even comprehend that there was something to comprehend - after all, I could pass the subjects okay, so I must have 'got it' okay, even though it seemed sort of wishy-washy to me. But actually I didn't get it at all. It wasn't until I was some doing third-year subjects that it eventually clicked. I suddenly understood what all the previous subjects had been about. I got it. The material I had learned wasn't a bunch of different stuff all lumped together that was done the way it was purely by convention (though there are no shortage of conventions) - there was, in fact, a coherence to it all. It was all of a thing, it fitted together; the stuff I'd learned was the result of a limited collection of concepts applied to different problems. I could actually begin applying my understanding outside my direct learning, to problems I'd not seen before. I had a framework within which each new piece of knowledge fitted in with everything else.
I'm not certain how to even convey this understanding, though I try. Students recognise that I'm passionate, at least (or so they tell me), though I'm not sure that the "why" always comes across to more than a very few of them.
Monday, October 6, 2008
Happy Happy Joy Joy
I am one happy blogger. I have completely sorted out a little mathematical problem that's been plaguing me for ages. It's one where I already knew the result, but all the proofs I could construct were either too embarrassingly clunky to use (I mean, really, really awful), or elegant but handwavy in one place.
I realized last week we really needed this result for something I'm working on with my research student. I sat and thought about it for a while today and finally noticed that the one remaining bit of argument we needed to make was obvious if you just recast the whole problem as a count from a thinned Poisson process. The crazy thing was my old handwavy argument was in effect already doing that, I just had failed to recognize it for what it was. Now that it's been set up in the right way, all the handwavy aspects drop away, and a nice clean half-page argument based on already-known results establishes the result we need.
This is one of those moments when after the fact everything is so obvious that I feel inadequate for not having seen it much earlier, but for the moment the joy is undiminished, because what this small step gets us to is something dramatic (assuming showing a bunch of well-known-in-their-application-area people that what they've been saying and doing is completely wrong is dramatic).
I realized last week we really needed this result for something I'm working on with my research student. I sat and thought about it for a while today and finally noticed that the one remaining bit of argument we needed to make was obvious if you just recast the whole problem as a count from a thinned Poisson process. The crazy thing was my old handwavy argument was in effect already doing that, I just had failed to recognize it for what it was. Now that it's been set up in the right way, all the handwavy aspects drop away, and a nice clean half-page argument based on already-known results establishes the result we need.
This is one of those moments when after the fact everything is so obvious that I feel inadequate for not having seen it much earlier, but for the moment the joy is undiminished, because what this small step gets us to is something dramatic (assuming showing a bunch of well-known-in-their-application-area people that what they've been saying and doing is completely wrong is dramatic).
Wednesday, August 27, 2008
Captaining the Titanic
Bad arithmetic can leave us like the captain of the Titanic - convinced we're unsinkable while we confidently steam toward the iceberg.
The inability of the Clinton advisors to perform basic number crunching cost them dearly in their primary campaign.
After the expensive loss in Iowa the Clinton campaign focused on states with a primary, such as Texas, pouring an enormous amount of resources into winning them while Obama racked up win after win in states they weren't even running polling in... only to discover that the time, money and effort had gained them little advantage in several of the contests that they focused on. Clinton won the Texas primary, but it didn't translate into a big advantage in delegates. A little number-crunching (which plenty of people pointed out well before the Texas contest took place) would have shown that a big effort in Texas would have conferred a relatively minor advantage, given the way that Texas' system works. But their campaign apparently didn't understand the issue - in spite of the fact that the issue was well understood by others - until too late; the Clinton camp started whining about it a few days before the primary, but it is not like Texas' circumstances were a secret before then.
Clintons' campaign paid "millions of dollars to consultants who offered up dubious advice".
These "experts" then managed to make further, even simpler elementary mathematical mistakes (by applying a calculation suitable for districts with 6 delegates to districts with different numbers of delegates), which meant that time after time, Clinton must have been mispending money, by allocating resources where they would be certain to be wasted and failing to allocate them where they could make a real difference, in effect multiplying Obama's financial advantage many-fold.
[Mathematics was not the only problem in the Clinton campaign, by any means - but it was a very important one that should never have arisen at all.]
What is it that causes monumental errors on the scale of using a calculation based on six delegates - that the target should be 7/12 of the vote ("the magic number is 59%") - for other districts?
Might it be the Dunning-Kruger effect? Is it just arrogance? Is it getting so focused on things like spin and sound-bites that you can't even remember the rules of the game?
Perhaps the Dunning-Kruger effect might also explain why the California Supreme Court have ruled that courts, not statisticians, will decide which calculations are to be used in cases involving so-called "cold-hit" DNA-matches. Statistical experts are to be reduced to "calculators", performing court approved calculations, whether the circumstances merit the calculations or not.
Innumeracy is what lets a political leader spend a trillion dollars on a largely futile and deadly war, and at the same time veto spending a million dollars on an essential education program, with a stunningly small backlash, partly because many voters don't realize the first is a million times as large as the second. Trillion, billion and million all just become different ways of saying "gazillion", and even the most implausible justifcation can be made to sound iron-clad.
A citizen needs enough mathematics to understand such differences in scale, and a political advisor certainly needs at least enough to be able to formulate a sensible strategy. Without it, we're all in for some very painful and expensive lessons.
The inability of the Clinton advisors to perform basic number crunching cost them dearly in their primary campaign.
After the expensive loss in Iowa the Clinton campaign focused on states with a primary, such as Texas, pouring an enormous amount of resources into winning them while Obama racked up win after win in states they weren't even running polling in... only to discover that the time, money and effort had gained them little advantage in several of the contests that they focused on. Clinton won the Texas primary, but it didn't translate into a big advantage in delegates. A little number-crunching (which plenty of people pointed out well before the Texas contest took place) would have shown that a big effort in Texas would have conferred a relatively minor advantage, given the way that Texas' system works. But their campaign apparently didn't understand the issue - in spite of the fact that the issue was well understood by others - until too late; the Clinton camp started whining about it a few days before the primary, but it is not like Texas' circumstances were a secret before then.
Clintons' campaign paid "millions of dollars to consultants who offered up dubious advice".
These "experts" then managed to make further, even simpler elementary mathematical mistakes (by applying a calculation suitable for districts with 6 delegates to districts with different numbers of delegates), which meant that time after time, Clinton must have been mispending money, by allocating resources where they would be certain to be wasted and failing to allocate them where they could make a real difference, in effect multiplying Obama's financial advantage many-fold.
[Mathematics was not the only problem in the Clinton campaign, by any means - but it was a very important one that should never have arisen at all.]
What is it that causes monumental errors on the scale of using a calculation based on six delegates - that the target should be 7/12 of the vote ("the magic number is 59%") - for other districts?
Might it be the Dunning-Kruger effect? Is it just arrogance? Is it getting so focused on things like spin and sound-bites that you can't even remember the rules of the game?
Perhaps the Dunning-Kruger effect might also explain why the California Supreme Court have ruled that courts, not statisticians, will decide which calculations are to be used in cases involving so-called "cold-hit" DNA-matches. Statistical experts are to be reduced to "calculators", performing court approved calculations, whether the circumstances merit the calculations or not.
Innumeracy is what lets a political leader spend a trillion dollars on a largely futile and deadly war, and at the same time veto spending a million dollars on an essential education program, with a stunningly small backlash, partly because many voters don't realize the first is a million times as large as the second. Trillion, billion and million all just become different ways of saying "gazillion", and even the most implausible justifcation can be made to sound iron-clad.
A citizen needs enough mathematics to understand such differences in scale, and a political advisor certainly needs at least enough to be able to formulate a sensible strategy. Without it, we're all in for some very painful and expensive lessons.
Wednesday, August 20, 2008
Too much mathematics homework
Recently I commented over at En Tequila Es Verdad, saying in part that I thought too much mathematics homework was a bad thing, education wise.
The response to headlines about US falling behind in education (say, like this one) is usually to increase homework.
Well, a paper in Econometrics Journal apparently concludes that for average students (about half of them, speaking roughly), lots of mathematics homework is not productive.
[Of course, this is looking at relatively short term effects. What will be the effects of too much homework five years down the line? My guess is that long term it will probably be unproductive for an even larger percentage.]
The linked news article says: According to Henderson, the learning process needs to remain a rich, broad experience.
Which is one of the main points I was getting at in my lengthy comment over at Dana's blog. Nice to see I'm not talking complete bullshit.
Think of it this way:
Imagine art class consisted of having to practice drawing a duck, over and over, until you could produce a good outline of a few very particular kinds of duck, drawn just so. You would do half an hour of ducks every night for homework. Then back to school the next day for more ducks. Then you'd move on to chickens. The generalization to all birds would be sort of handwaved, because the curriculum is kind of packed. It's time to move on to drawing fish! If you didn't learn to draw ducks, you would get even more work on drawing ducks. Some aspects of what you learned in drawing ducks could be used in drawing fish, but the relationships aren't very intuitive, and anyway, there's just so many bits to remember and it's all so confusing and WTF, now I have to go home and do fish for an HOUR?
And then suddenly you're drawing battleships, and while drawing kind of made sense before, suddenly it makes no sense. You never quite got the hang of ducks and now you're trying to catch up that, fish and now battleships? How on earth are you ever going to remember all the parts of a battleship? And god forbid you should draw the parts in the wrong order!
If art was like that, most people would hate it.
Imagine Rembrandt at a party, who desperately wants to convey something of the beauty and importance of chiaroscuro. What would he hear, over and over, as he brought up the topic of art?
"Art? I was never any good at that. I always hated art! My worst subject. All those ducks! You must be very strange."
Most people - if you forced them - would be able to draw a fairly reasonable-looking duck, but there'd be precious little art in their lives. They'd certainly have no sense that it could be moving and beautiful - or indeed that it was about anything other than ducks and fish, and maybe something painful about battleships.
The response to headlines about US falling behind in education (say, like this one) is usually to increase homework.
Well, a paper in Econometrics Journal apparently concludes that for average students (about half of them, speaking roughly), lots of mathematics homework is not productive.
[Of course, this is looking at relatively short term effects. What will be the effects of too much homework five years down the line? My guess is that long term it will probably be unproductive for an even larger percentage.]
The linked news article says: According to Henderson, the learning process needs to remain a rich, broad experience.
Which is one of the main points I was getting at in my lengthy comment over at Dana's blog. Nice to see I'm not talking complete bullshit.
Think of it this way:
Imagine art class consisted of having to practice drawing a duck, over and over, until you could produce a good outline of a few very particular kinds of duck, drawn just so. You would do half an hour of ducks every night for homework. Then back to school the next day for more ducks. Then you'd move on to chickens. The generalization to all birds would be sort of handwaved, because the curriculum is kind of packed. It's time to move on to drawing fish! If you didn't learn to draw ducks, you would get even more work on drawing ducks. Some aspects of what you learned in drawing ducks could be used in drawing fish, but the relationships aren't very intuitive, and anyway, there's just so many bits to remember and it's all so confusing and WTF, now I have to go home and do fish for an HOUR?
And then suddenly you're drawing battleships, and while drawing kind of made sense before, suddenly it makes no sense. You never quite got the hang of ducks and now you're trying to catch up that, fish and now battleships? How on earth are you ever going to remember all the parts of a battleship? And god forbid you should draw the parts in the wrong order!
If art was like that, most people would hate it.
Imagine Rembrandt at a party, who desperately wants to convey something of the beauty and importance of chiaroscuro. What would he hear, over and over, as he brought up the topic of art?
"Art? I was never any good at that. I always hated art! My worst subject. All those ducks! You must be very strange."
Most people - if you forced them - would be able to draw a fairly reasonable-looking duck, but there'd be precious little art in their lives. They'd certainly have no sense that it could be moving and beautiful - or indeed that it was about anything other than ducks and fish, and maybe something painful about battleships.
Monday, August 18, 2008
Wednesday, August 13, 2008
Emotion and mathematics
It would be easy for people outside of mathematical areas to assume that the exercise of mathematics is an austere and unemotional activity, and that as a result mathemamaticians are, whether by nature or by habit, cold and disinclined to emotion.
Having observed many people (including myself) doing mathematics and discussing mathematically-related topics, this is far from the case.
I have had many congenially heated arguments with colleagues, and I have even caught myself grinning in delighted anticipation of going another round with a valued fellow-traveller.
(I've been called crazy a lot of times - but more times in mathematically-related discussions than anywhere else - and with unstinting good humour to boot. "You're crazy! You can't do that." "No, really, it's right. You can do it here..." "No, no, it's nuts to do it that way even if it's right." -- and so on back and forth; in fact, I think that's how a lot of mathematical arguments get polished)
Even as a solitary activity, mathematics is for me, intensely emotional, even visceral. Many times, equations I have worked with have various kinds of symmetry, and many of those symmetries will carry through the equations as the argument develops.. this is, I presume, what produces a strong sense of rightness that I often feel as the steps progress. There's also a corresponding sense that there is some mistake - for me a feeling something like that moment on a roller coaster as it begins to descend, though it is sometimes even stronger than that - before being aware of exactly what is wrong, or precisely where it lies.
If you work with particular kinds of expressions a lot, you built up a sense of what they "should" look like, and it becomes easier to recognize that something is wrong before you can say precisely what the problem is; because the intellectual cognition is behind the pattern-recognition, it has an emotional quality.
A really clever manipulation (I can't help but think of them as "tricks") or an inspired substitution that makes a difficult problem easy can produce a tingling sensation up the back of my neck and head. A particularly beautiful piece of mathematics can, on occasion, move me almost to tears.
Then there's joy and delight. On occasion I have had the fortune to look at some neat, if modest, just-derived result and wonder if perhaps I am the first to have ever seen it (it is, obviously, rarely the case that I am - it is not unusual to find that my result has been tucked away in some mathematical corner for many decades ... on one occasion I found I had been beaten by Gauss - but the thrill of discovery is there all the same).
There's also what I call the "stupid feeling". When I'm working on something new or unfamiliar (or even, on occasion on what ought to be familiar), I can spend long periods - days, weeks, or even, shamefully, months - where I feel intensely incompetent, like I'm reaching around in the dark for something that I know is right there, but can't seem to locate it - and then there's a fleetingly brief moment of joy as I see how to do it (often barely long enough to say "Yes!"). Then quickly after, in retropspect (sometimes as I see an even better way to do it), it is all so utterly obvious, so agonizingly plain, that the prior feeling of incompetence seems, if anything, far too mild.
For me, that feeling is occasionally so intense I cannot even bear to write it up properly, or sometimes even to mention it, because the whole thing is so painfully facile. (I doubt that most people feel this quite so keenly; I'd be curious to know.)
Mathematicians don't discuss emotion much; a kindly supervisor might have a few words on dealing with the disappointments that naturally come with trying to get some result to come out, or those that come with trying to get something published. But outside of that, the preference is almost always to talk about the mathematics itself.
But just because we don't talk about our feelings with each other doesn't mean we're not feeling them.
Having observed many people (including myself) doing mathematics and discussing mathematically-related topics, this is far from the case.
I have had many congenially heated arguments with colleagues, and I have even caught myself grinning in delighted anticipation of going another round with a valued fellow-traveller.
(I've been called crazy a lot of times - but more times in mathematically-related discussions than anywhere else - and with unstinting good humour to boot. "You're crazy! You can't do that." "No, really, it's right. You can do it here..." "No, no, it's nuts to do it that way even if it's right." -- and so on back and forth; in fact, I think that's how a lot of mathematical arguments get polished)
Even as a solitary activity, mathematics is for me, intensely emotional, even visceral. Many times, equations I have worked with have various kinds of symmetry, and many of those symmetries will carry through the equations as the argument develops.. this is, I presume, what produces a strong sense of rightness that I often feel as the steps progress. There's also a corresponding sense that there is some mistake - for me a feeling something like that moment on a roller coaster as it begins to descend, though it is sometimes even stronger than that - before being aware of exactly what is wrong, or precisely where it lies.
If you work with particular kinds of expressions a lot, you built up a sense of what they "should" look like, and it becomes easier to recognize that something is wrong before you can say precisely what the problem is; because the intellectual cognition is behind the pattern-recognition, it has an emotional quality.
A really clever manipulation (I can't help but think of them as "tricks") or an inspired substitution that makes a difficult problem easy can produce a tingling sensation up the back of my neck and head. A particularly beautiful piece of mathematics can, on occasion, move me almost to tears.
Then there's joy and delight. On occasion I have had the fortune to look at some neat, if modest, just-derived result and wonder if perhaps I am the first to have ever seen it (it is, obviously, rarely the case that I am - it is not unusual to find that my result has been tucked away in some mathematical corner for many decades ... on one occasion I found I had been beaten by Gauss - but the thrill of discovery is there all the same).
There's also what I call the "stupid feeling". When I'm working on something new or unfamiliar (or even, on occasion on what ought to be familiar), I can spend long periods - days, weeks, or even, shamefully, months - where I feel intensely incompetent, like I'm reaching around in the dark for something that I know is right there, but can't seem to locate it - and then there's a fleetingly brief moment of joy as I see how to do it (often barely long enough to say "Yes!"). Then quickly after, in retropspect (sometimes as I see an even better way to do it), it is all so utterly obvious, so agonizingly plain, that the prior feeling of incompetence seems, if anything, far too mild.
For me, that feeling is occasionally so intense I cannot even bear to write it up properly, or sometimes even to mention it, because the whole thing is so painfully facile. (I doubt that most people feel this quite so keenly; I'd be curious to know.)
Mathematicians don't discuss emotion much; a kindly supervisor might have a few words on dealing with the disappointments that naturally come with trying to get some result to come out, or those that come with trying to get something published. But outside of that, the preference is almost always to talk about the mathematics itself.
But just because we don't talk about our feelings with each other doesn't mean we're not feeling them.
Saturday, August 2, 2008
Pentagonal Tiling...
I was reading Julie Rehmeyer's current column (I like Julie's writing and have been following MathTrek for more than almost a year now) at Science News, which is on quasicrystals.
But this column I think she said something other than what she intended... I'd have commented there but you have to register, which I'm not going to do just to leave one small comment, and anyway, posting it here gives me a change to prattle on and point at pictures and such.
To quote:
"That's why you've never seen a bathroom tiled with pentagons - it'd be impossible to cover the whole surface with no gaps."
Now the problem is, this statement is wrong... in fact, here's a counterexample I knocked up in a few seconds (it's a bit rough, but you can see what's going on easily enough).
This is called the Cairo pentagonal tiling. It's one of fourteen known tilings of pentagons (it's probably not obvious, but "Type 4" on that page is the same type of tiling). Another favourite of mine is the the Floret pentagonal tiling. (Go take a look at those two named ones, they're pretty.)
What Julie meant was "...tiled with regular pentagons". Cos, yeah, that doesn't work.
Anyway, aside from that hiccup, it's a good article; worth a read.
But this column I think she said something other than what she intended... I'd have commented there but you have to register, which I'm not going to do just to leave one small comment, and anyway, posting it here gives me a change to prattle on and point at pictures and such.
To quote:
"That's why you've never seen a bathroom tiled with pentagons - it'd be impossible to cover the whole surface with no gaps."
Now the problem is, this statement is wrong... in fact, here's a counterexample I knocked up in a few seconds (it's a bit rough, but you can see what's going on easily enough).
This is called the Cairo pentagonal tiling. It's one of fourteen known tilings of pentagons (it's probably not obvious, but "Type 4" on that page is the same type of tiling). Another favourite of mine is the the Floret pentagonal tiling. (Go take a look at those two named ones, they're pretty.)
What Julie meant was "...tiled with regular pentagons". Cos, yeah, that doesn't work.
Anyway, aside from that hiccup, it's a good article; worth a read.
Wednesday, July 23, 2008
The parable of the histogram
I must be some kind of heretic. I'm a statistician, and here I am pointing out the problems in yet another common statistical tool.
We'll see how the histogram, which is a very popular way of displaying the distributional shape of a set of data, must be viewed with a good deal of caution.
Even though histograms are often found in the media, the problems with histograms are almost unknown among the general public. Indeed, most places that teach statistics at university completely fail to mention them.
I'd like to say that the problems are well known among professional statisticians, but that might be too strong. Certainly problems have been pointed out in the literature, and many statisticians are aware of the problems, but it seems many still are not, and the appropriate cautions are not always explained.
I'm going to show you a simple example.
Here's some data (40 observations in this sample), which I'm going to draw a histogram of. I have rounded the numbers off to two decimal places.
I give the numbers so you can (if you are so inclined) confirm for yourself what I will tell you in my little parable.
(Edit added Feb 2012: I noticed that the results didn't quite reproduce in R - three observations in the original data set I gave occurred exactly on bin boundaries for some situations. This was either a problem caused by rounding, or possibly by different conventions of different packages for handling observations at bin boundaries; I have accordingly altered those three observations by tiny amounts to move them off boundaries and avoid the issue, whatever its source. There is R code at the end of the post that works.)
The parable
This data set was given to a student, Annie. She constructs her histogram of the data by counting the number of values between 0 and 1 (but not including exactly 1), between 1 and 2, and so on, and then drawing a series of boxes each of whose base covers the subset of values that the count came from and whose height is the count for that range of values. Annie's histogram is shown in the top-left of the picture below.
She obtains a histogram whose shape corresponds to a distribution that is skewed to the left. See, for example, this description of using histograms to assess distributional shape here (edit: broken link replaced with an alternative) - that's pretty much precisely the way many elementary books on statistics describe the way to assess the shape of a distribution (and usually it's going to give you the right sort of impression).
Note that I could remove the scale and I could still describe the shape - I don't need to know the numbers on the scale in order to arrive at my description.
Three of Annie's friends, Brian, Chris and Zoe (Hah! Psych!) also get data sets with 40 observations, and they all do exactly as Annie did. Their histograms are given below (Annie's data is V1, Brian's is V2 and so on).
(click pic for a larger image)
Correspondingly, Brian describes his distribution as symmetric (and he might add "uniform"). Chris describes his as skewed to the right. Zoe describes hers as symmetric and bimodal (it has two main peaks).
So far so good - this is exactly how the books tell you it all works.
So while they're comparing their histograms, Annie idly starts looking at Brian's actual numbers. She realizes something odd is going on. She quickly places all their data sets side-by-side.
"Look, Chris!" Annie says, "all Brian's values are smaller than mine by 0.25. All yours are a quarter smaller than Brian's, and Zoe's are a quarter smaller than yours!"
They all confirm that she is correct - each set of values is the same, but with its origin merely shifted a little. Their data sets are identical in shape, but the resulting histograms are not.
That is to say, assessment of distributional shape in histograms can be dramatically affected by choice of scale (specifically, by the choice of the origin and width of the histgram bins). Here ends the parable.
It usually isn't this dramatic, of course, but the fact is, if one can generate a seemingly innocuous set of numbers whose histogram will look completely different (and for which many people will assert the distributional shape is completely different) every time we merely add or subtract a quarter, it can happen with real data too. And it does happen. Mostly the difference in impression is more modest... but not always.
So if you see a histogram, just keep in the back of your mind that it's perfectly possible that a different choice of bin boundaries would yield a somewhat different impression of the data.
Imagine I want to show some students that I write "easy" tests (I don't know why this should be such an object of fascination for students since they all do the same test, and marks are generally scaled, but it is). In preparation, I draw a histogram and it turns out to like Chris' - it looks like most students score below the middle of the range of marks. But lo, I discover with a bit of fiddling around that if I make my bin centres where the edges were (and so on), the completely opposite impression is given - just like Annie's histogram. Yay, "easy test" ... and many fewer worried queries from students in the run-up to the test, because they tend to feel there's a good chance of scoring "above the middle".
Did I lie? No. Did I fudge the data? Well, no. I did something, though. Or rather, I didn't do something.
This is a sin of omission. I fail to explain what the data would have looked like given a different choice of bin location.
Clearly, when circumstances are right, the ability to choose the location and width of the bins can give us the opportunity to somewhat alter the impression given by a histogram. Without fudging the numbers themselves, we can sometimes fudge the impression they give.
What do statisticians do? Well, there are other ways to look at distributional shape. Kernel density estimates are popular, and they completely get rid of the "bin-location" issue, though there's still the equivalent of a "bin-width" issue (choice of bandwidth, also called the "window"), which is often dealt with by looking at more than one choice of width (usually a width that gives a nice smooth result and then one that is smaller, giving a "rougher" result, in order that we can see there's nothing unsual hiding away - like the blue and green curves in the graph at topleft right** at the wikipedia link a few lines up). But there are a variety of other tools that might be used (which I don't plan on going into here).
**(did I ever mention that I have trouble with correctly attributing the words "left" and "right"? - well as you see, sometimes I do. But not when describing the shape of a distribution, isn't that odd?)
What can you do? Well, assuming you don't have anything more sophisticated that a basic histogram tool, at the least (with continuous data, anyway), try shifting your bin starts forward or back a fraction of a bin-width (if you're lazy, maybe try something near a half, otherwise maybe try a couple of values). Also try a narrower bin width. If you do a few different histograms that all give the same general impression, it doesn't matter much which one you use. And if they don't give the same impression, you better either say so, show more than one, or find some other way to convey the information.
[Or you can do a kernel density estimate readily enough - many packages (including some free ones) will do them; there are pages online that can draw them if you just paste in some data. Implementing a kernel density estimate of your own is fairly straightforward - you can compute one in a spreadsheet easily enough - if anything, it's probably slightly simpler to compute one than it is to compute counts for a histogram, which is in itself pretty straightforward. ]
Caveat Emptor
___
Added in edit in Feb 2012:
Here is some R code to create the data:
Here is some R code to generate the histograms:
Here is some R code to generate some density estimates:
Here is some R code to generate some other informative displays:
First - the sample cumulative distribution function
Second, a stripchart that shows the positions of the individual observations as they move back.
We'll see how the histogram, which is a very popular way of displaying the distributional shape of a set of data, must be viewed with a good deal of caution.
Even though histograms are often found in the media, the problems with histograms are almost unknown among the general public. Indeed, most places that teach statistics at university completely fail to mention them.
I'd like to say that the problems are well known among professional statisticians, but that might be too strong. Certainly problems have been pointed out in the literature, and many statisticians are aware of the problems, but it seems many still are not, and the appropriate cautions are not always explained.
I'm going to show you a simple example.
Here's some data (40 observations in this sample), which I'm going to draw a histogram of. I have rounded the numbers off to two decimal places.
3.15 2.28 2.06 3.43 4.85 3.22 4.01 4.43 5.46 3.12 5.53 5.51 5.56 5.52 5.31 4.96 3.28 4.10 5.19 2.54 1.89 1.84 2.56 1.90 4.20 3.42 2.39 3.64 4.84 4.31 5.11 5.60 1.98 3.91 1.88 4.33 5.74 2.01 2.58 1.92
I give the numbers so you can (if you are so inclined) confirm for yourself what I will tell you in my little parable.
(Edit added Feb 2012: I noticed that the results didn't quite reproduce in R - three observations in the original data set I gave occurred exactly on bin boundaries for some situations. This was either a problem caused by rounding, or possibly by different conventions of different packages for handling observations at bin boundaries; I have accordingly altered those three observations by tiny amounts to move them off boundaries and avoid the issue, whatever its source. There is R code at the end of the post that works.)
The parable
This data set was given to a student, Annie. She constructs her histogram of the data by counting the number of values between 0 and 1 (but not including exactly 1), between 1 and 2, and so on, and then drawing a series of boxes each of whose base covers the subset of values that the count came from and whose height is the count for that range of values. Annie's histogram is shown in the top-left of the picture below.
She obtains a histogram whose shape corresponds to a distribution that is skewed to the left. See, for example, this description of using histograms to assess distributional shape here (edit: broken link replaced with an alternative) - that's pretty much precisely the way many elementary books on statistics describe the way to assess the shape of a distribution (and usually it's going to give you the right sort of impression).
Note that I could remove the scale and I could still describe the shape - I don't need to know the numbers on the scale in order to arrive at my description.
Three of Annie's friends, Brian, Chris and Zoe (Hah! Psych!) also get data sets with 40 observations, and they all do exactly as Annie did. Their histograms are given below (Annie's data is V1, Brian's is V2 and so on).
(click pic for a larger image)Correspondingly, Brian describes his distribution as symmetric (and he might add "uniform"). Chris describes his as skewed to the right. Zoe describes hers as symmetric and bimodal (it has two main peaks).
So far so good - this is exactly how the books tell you it all works.
So while they're comparing their histograms, Annie idly starts looking at Brian's actual numbers. She realizes something odd is going on. She quickly places all their data sets side-by-side.
"Look, Chris!" Annie says, "all Brian's values are smaller than mine by 0.25. All yours are a quarter smaller than Brian's, and Zoe's are a quarter smaller than yours!"
They all confirm that she is correct - each set of values is the same, but with its origin merely shifted a little. Their data sets are identical in shape, but the resulting histograms are not.
That is to say, assessment of distributional shape in histograms can be dramatically affected by choice of scale (specifically, by the choice of the origin and width of the histgram bins). Here ends the parable.
It usually isn't this dramatic, of course, but the fact is, if one can generate a seemingly innocuous set of numbers whose histogram will look completely different (and for which many people will assert the distributional shape is completely different) every time we merely add or subtract a quarter, it can happen with real data too. And it does happen. Mostly the difference in impression is more modest... but not always.
So if you see a histogram, just keep in the back of your mind that it's perfectly possible that a different choice of bin boundaries would yield a somewhat different impression of the data.
Imagine I want to show some students that I write "easy" tests (I don't know why this should be such an object of fascination for students since they all do the same test, and marks are generally scaled, but it is). In preparation, I draw a histogram and it turns out to like Chris' - it looks like most students score below the middle of the range of marks. But lo, I discover with a bit of fiddling around that if I make my bin centres where the edges were (and so on), the completely opposite impression is given - just like Annie's histogram. Yay, "easy test" ... and many fewer worried queries from students in the run-up to the test, because they tend to feel there's a good chance of scoring "above the middle".
Did I lie? No. Did I fudge the data? Well, no. I did something, though. Or rather, I didn't do something.
This is a sin of omission. I fail to explain what the data would have looked like given a different choice of bin location.
Clearly, when circumstances are right, the ability to choose the location and width of the bins can give us the opportunity to somewhat alter the impression given by a histogram. Without fudging the numbers themselves, we can sometimes fudge the impression they give.
What do statisticians do? Well, there are other ways to look at distributional shape. Kernel density estimates are popular, and they completely get rid of the "bin-location" issue, though there's still the equivalent of a "bin-width" issue (choice of bandwidth, also called the "window"), which is often dealt with by looking at more than one choice of width (usually a width that gives a nice smooth result and then one that is smaller, giving a "rougher" result, in order that we can see there's nothing unsual hiding away - like the blue and green curves in the graph at top
**(did I ever mention that I have trouble with correctly attributing the words "left" and "right"? - well as you see, sometimes I do. But not when describing the shape of a distribution, isn't that odd?)
What can you do? Well, assuming you don't have anything more sophisticated that a basic histogram tool, at the least (with continuous data, anyway), try shifting your bin starts forward or back a fraction of a bin-width (if you're lazy, maybe try something near a half, otherwise maybe try a couple of values). Also try a narrower bin width. If you do a few different histograms that all give the same general impression, it doesn't matter much which one you use. And if they don't give the same impression, you better either say so, show more than one, or find some other way to convey the information.
[Or you can do a kernel density estimate readily enough - many packages (including some free ones) will do them; there are pages online that can draw them if you just paste in some data. Implementing a kernel density estimate of your own is fairly straightforward - you can compute one in a spreadsheet easily enough - if anything, it's probably slightly simpler to compute one than it is to compute counts for a histogram, which is in itself pretty straightforward. ]
Caveat Emptor
___
Added in edit in Feb 2012:
Here is some R code to create the data:
histdata <- c(3.15,5.46,3.28,4.2,1.98,2.28,3.12,4.1,3.42,3.91,2.06,5.53 ,5.19,2.39,1.88,3.43,5.51,2.54,3.64,4.33,4.85,5.56,1.89,4.84,5.74,3.22 ,5.52,1.84,4.31,2.01,4.01,5.31,2.56,5.11,2.58,4.43,4.96,1.9,5.6,1.92)
Here is some R code to generate the histograms:
opar<-par() par(mfrow=c(2,2)) hist(histdata,breaks=1:6,main="Annie",xlab="V1",col="lightblue") hist(histdata-0.25,breaks=1:6,main="Brian",xlab="V2",col="lightblue") hist(histdata-0.5,breaks=1:6,main="Chris",xlab="V3",col="lightblue") hist(histdata-0.75,breaks=1:6,main="Zoe",xlab="V4",col="lightblue") par(opar)
Here is some R code to generate some density estimates:
opar<-par() par(mfrow=c(2,2)) plot(density(histdata,bw=.2),main="Annie") plot(density(histdata-.25,bw=.2),main="Brian") plot(density(histdata-.5,bw=.2),main="Chris") plot(density(histdata-.75,bw=.2),main="Zoe") par(opar)
Here is some R code to generate some other informative displays:
First - the sample cumulative distribution function
plot(ecdf(histdata))
Second, a stripchart that shows the positions of the individual observations as they move back.
x<-c abline="" c="" each="40)" g="" histdata-.25="" histdata-.5="" histdata-.75="" histdata="" pch="|" rep="" stripchart="" v="(2:5),col=6,lty=3)</pre" x=""> end edit
Sunday, July 20, 2008
Fighting Mathiness
'I know what you're thinking about,' said Tweedledum: 'but it isn't so, nohow.'
'Contrariwise,' continued Tweedledee, 'if it was so, it might be; and if it were so, it would be; but as it isn't, it ain't. That's logic.'
'Can you do Addition?' the White Queen asked. 'What's one and one and one and one and one and one and one and one and one and one?'
'I don't know,' said Alice. 'I lost count.'
'She can't do Addition,' the Red Queen interrupted. 'Can you do Subtraction? Take nine from eight.'
'Nine from eight I can't, you know,' Alice replied very readily: 'but-'
'She can't do Subtraction,' said the White Queen. 'Can you do Division? Divide a loaf by a knife-what's the answer to that?'
Jordan Ellenberg defined mathiness as "a series of fervent gestures that gives the impression that mathematical ideas are being expressed, but doesn’t actually deliver the goods".
Let us examine some examples of mathiness, and some examples where honest attempts to deal with mathematical situations have foundered, and try to understand how we can be led astray by mathematical arguments.
Skewness
I recently wrote about how in statistics, the measure that is often called skewness doesn't really mean what popular lore holds it to mean, and that it is often misused - for example, when people assert that zero skewness implies symmetry. I later pointed to several sites that made the kinds of errors I was talking about. In the brief time since then, new instances of the same issue have come up on some mathematics-related blogs. It's a case where the verbal "description" of the situation is not in agreement with the mathematical tools being used - mathematical ideas appear to be expressed, but the goods are not being delivered.
Misleading Graphics
In another vein, bad statistical graphics, such as this
lie in graphical form
can mislead us, whether by accident, or as in this case, by design.
(via Andrew Gelman at Statistical Modeling, Causal Inference, and Social Science; there's other good examples to be found there.)
Examples abound in the media. Here's one from the NYT (via the Gallery of Data Visualization’s Missed Opportunities and Graphical Failures - click image for bigger version):
The top plot there is a graph of happiness against GNP-per-capita for a number of countries. The NYT has circled the countries in the top left hand corner, noting that many countries "had higher ... happiness than their economic situation would predict". This is the cardinal sin of treating inherently nonlinear relationships as linear - as they point out at the Gallery, an appropriate transformation - in this case looking at log-GNP, not raw GNP, makes these supposed "outliers" seem much more in keeping with the rest, and the apparent relationship more linear - and indeed, if anything, some entirely different points don't fit the general pattern. We seem to find nearly-linear relationships easier to understand, so transformation is often a useful strategy.
I have discussed the same issues - both the danger of treating nonlinear relationships as linear and the value of transformations in understanding relationships better in another context - relationships involving percentages. It's so easy to fall into the rut of linear thinking that we should consider taking advantage of the tendency to think that way and use transformation to reduce nonlinearity.
A common "nonlinear effect treated as linear" is when people try to average miles per gallon (or miles per hour, or a variety of other rates) - such as "I got 15 mpg going up and 45 mpg coming back, so I averaged 30 mpg overall" (when it's actually 22.5). In terms of transformations - the reciprocal (gallons per mile) - is linear and can be averaged.
Relying on a False Premise
Seemingly mathematical arguments may just be based on bad premises (such as one requiring selecting from the positive integers with equal probability - an impossibility that completely sinks the argument that relies on it).
That "infinity" thingy can be tricky - it seems to cause problems for journalists as well because they tend to underestimate how big it is.
Adding percentiles
Treating percentiles of distributions as if they were additive is unfortunately extremely common. In the case of official estimates of total oil reserves, it means that we probably have a fair bit more oil that we think.
What's the square root of that?
Or, sometimes, it seems, mathiness comes in because someone has no clue what the heck they're talking about, so we can be told that the Maya knew how to take the square root of a rectangle.
Mathematical arguments can feel unsually convincing, even unassailable, and we're awash in them for precisely that reason. It's too easy to forget that just because something seems to be laid out mathematically, it's not necessarily true - or even meaningful at all. Mathiness, like truthiness, is all around us. Even among skeptics, it's possible to put too much store in an argument couched in mathematical terms. We should be at least as skeptical of mathematical arguments - and in basically the same kinds of ways - as any other kinds of arguments, because we're all too often misled by them.
Unfortunately, it seems that we sometimes accept the (often implicit) conclusions of a mathematical argument without even realizing that an argument was being made.
If we fail to treat these arguments with the skepticism they deserve, we're open to being deceived by charlatans.
Wednesday, July 9, 2008
The Riemann hypothesis - why does it matter?
Mathematicians regard the Riemann hypothesis as very important, not least because a fair number of important theorems have been shown, given the assumption that the Riemann hypothesis is correct, so as soon as it's shown to be true, a whole pile of other interesting stuff is also true. Its impact is both practical and theoretical.
But when I was talking to William earlier today, he summed it up well:
"If it isn't true, math is a lot weirder than we think it is."
[Oh, and in case you are wondering what the situation with Li's proof is - while I was away on vacation for a few days, the paper was withdrawn. It appears Li could not rescue the proof. ]
But when I was talking to William earlier today, he summed it up well:
"If it isn't true, math is a lot weirder than we think it is."
[Oh, and in case you are wondering what the situation with Li's proof is - while I was away on vacation for a few days, the paper was withdrawn. It appears Li could not rescue the proof. ]
Friday, July 4, 2008
Riemann not proved. Not yet, anyway.
Well, I was right, there are holes (no prizes for making an obvious guess).
Isabel reports on it; I can't do better than direct you to her links, where Terry Tao and Alain Connes discuss problems.
Tao said: "It unfortunately seems that the decomposition claimed in equation (6.9) on page 20 of that paper is, in fact, impossible; it would endow the function h (which is holding the arithmetical information about the primes) with an extremely strong dilation symmetry which it does not actually obey. It seems that the author was relying on this symmetry to make the adelic Fourier transform far more powerful than it really ought to be for this problem."
Connes said: "I dont like to be too negative in my comments. Li's paper is an attempt to prove a variant of the global trace formula of my paper in Selecta. The "proof" is that of Theorem 7.3 page 29 in Li's paper, but I stopped reading it when I saw that he is extending the test function h from ideles to adeles by 0 outside ideles and then using Fourier transform (see page 31). This cannot work and ideles form a set of measure 0 inside adeles (unlike what happens when one only deals with finitely many places)."
Isabel reports on it; I can't do better than direct you to her links, where Terry Tao and Alain Connes discuss problems.
Tao said: "It unfortunately seems that the decomposition claimed in equation (6.9) on page 20 of that paper is, in fact, impossible; it would endow the function h (which is holding the arithmetical information about the primes) with an extremely strong dilation symmetry which it does not actually obey. It seems that the author was relying on this symmetry to make the adelic Fourier transform far more powerful than it really ought to be for this problem."
Connes said: "I dont like to be too negative in my comments. Li's paper is an attempt to prove a variant of the global trace formula of my paper in Selecta. The "proof" is that of Theorem 7.3 page 29 in Li's paper, but I stopped reading it when I saw that he is extending the test function h from ideles to adeles by 0 outside ideles and then using Fourier transform (see page 31). This cannot work and ideles form a set of measure 0 inside adeles (unlike what happens when one only deals with finitely many places)."
Thursday, July 3, 2008
Riemann proved??
A Proof of the Riemann Hypothesis by Xian-Jin Li. (link now fixed)
This is no crackpot. He's a mathematician who has his PhD in the right area.
I am not qualified (in the sense that I havenolittle* familiarity with this area of mathematics) to judge, but the mathematical world will be all over this one. [* Edit: well, actually, having read through it, it doesn't look to be as tough a paper as I first thought. I can understand some of it. I expect this claimed result will be either confirmed or disconfirmed in short order. If it's right, I suspect that it can be made even simpler (and if its wrong, it can be made as simple as you like!)]
My guess as an outsider... it will probably turn out to have a few holes in it. But if he's claiming it, he's probably really got something; if it's not there yet, it will probably turn out to be an important step, and if the holes are small, they'll be plugged soon enough. The fact that its a short paper increases the chance that Li hasn't missed something.
h/t Isabel Lugo
This is no crackpot. He's a mathematician who has his PhD in the right area.
I am not qualified (in the sense that I have
My guess as an outsider... it will probably turn out to have a few holes in it. But if he's claiming it, he's probably really got something; if it's not there yet, it will probably turn out to be an important step, and if the holes are small, they'll be plugged soon enough. The fact that its a short paper increases the chance that Li hasn't missed something.
h/t Isabel Lugo
Monday, June 30, 2008
Examples of the genre
In my previous post I said:
That last site approaches farce, and it's trying to teach statistics!. This is often what happens when people whose own area is not statistics get put in charge of teaching it. (For some reason mathematicians are among the worst offenders.)
There were more examples. The above ones are pretty standard.
Unfortunately, based only on the information that the third central moment was zero, many people would in fact describe it as symmetric.I've seen many examples over the years, so I figured it shouldn't be hard to find one or two. A quick google search reveals some examples:
- This page on using a TI83 calculator has:
- "Skewness measures the departure from symmetry" (it defines skewness as the third moment measure I mentioned, so it is saying that 0 third moment implies symmetry)
- it goes on to suggest a test statistic for symmetry based on this, and concludes that discussion with "If that fraction is between −2 and 2, you can’t say whether the population is symmetric (skewness = 0) or skewed."
- This paper on brain evolution has:
- "SK, subclade skewness (- negative skew; 0, symmetric distribution; + positive skew)"
- "the system is probably passive if average subclade skew is neutral (symmetric distribution) or negative"
- This set of notes for a university* subject called "Introduction to Statistics" has the following complete howlers:
- "If the skewness is approximately zero, the histogram (distribution) for the data is symmetric and usually normal"
- "'varB' has a skewness close to zero so that its distribution should be normal and mean and median should be similar."
That last site approaches farce, and it's trying to teach statistics!. This is often what happens when people whose own area is not statistics get put in charge of teaching it. (For some reason mathematicians are among the worst offenders.)
There were more examples. The above ones are pretty standard.
Sunday, June 29, 2008
Not fooling ourselves (I) - the unmeasuring of asymmetry
In order to impose some kind of structure on our understanding of complex phenomena, we tend to use simple terms to describe what may be surprisingly nonsimple.
In order to make progress, we often attempt to quantify those descriptives. This is not just useful, it's often unavoidable, but the act carries with it a special danger, because we then (almost universally) invest the nonunique quantification of the concept with a "reality" that it doesn't merit - and that can lead to nonsense.
[It may be this is another version of the common phenomenon of believing mathematical models when at the beginning we knew them to be at best a rough approximation. Models tend to take on a life of their own, and their conclusions are often treated with a respect we did not accord the original model when it was first tentatively adopted. We need to step back and remember the model was never exactly the thing it was used to describe.]
I think this may be the flip side of what Blake Stacey was talking about when he was discussing the confusions that come in when an inherently mathematical concept is translated into a nonmathematical description.
Let me take a concrete example with which I am familiar. It relates to descriptions of probability distributions.
We begin with something that is inherently mathematical - symmetry. Symmetry is an extraordinarily useful, almost universal concept in mathematics. For what I'll be talking about you can just use the more common senses of reflection symmetry and rotational symmetry.
In the case of symmetry of distributions, there are several ways to define it (if you have a continuous or a discrete random variable, you can define it in the usual "mirror symmetry" sense), but the more general definition would have something like "m is the centre of symmetry if Prob(X≤ m-a) = Prob(X≥ m+a) for all a" (which effectively corresponds to rotational symmetry of the distribution function about the points (m, 1/2), if you tidy up details the right way).
Anyway, "symmetry" is both easily understood and fairly easy to pin down. Of course, almost all distributions are not symmetric (but symmetry arises in a natural way in particular circumstances, so it's far from a useless concept).
Aside from "unimodal" (one "hump"), or the overused and badly abused "bell-shaped", one of the most common descriptions of a distributional shape that is applied would probably be "skewed". An elementary book might have a diagram like the top one below (see for example, the diagram at Wikipedia's entry on skewness). The corresponding distribution function is underneath.
Diagram of a right (positive) skewed density and distribution function.
If you flipped the above density left-to-right (or rotated the distribution about the median), it would have left (negative) skewness.
Notice that "right skewed" means the long tail is to the right (this is often the opposite of a beginner's intuition about what the term should mean).
The problem comes when this seemingly clear but actually vague notion is quantified. There are numerous quantities that have been called "skewness". By far the most popular is the standardized third moment (or, equivalently, the standardized third cumulant) - so much so that it is frequently called "the" skewness. Equivalent sample statistics are used for samples.
Now for a distribution like the one above, this quantity is positive (when it exists). If you flip that density shape left to right, the skewness measure is negative. Importantly, when the density is symmetric and the first three moments exist, the skewness is 0. So far so good - left and right skewness and symmetry in pictures generally correspond to negative, positive and zero quantities on the measurement.
However, the problem comes when interpreting a standardized third moment back in terms of the density.
A positive number will lead people to call the distribution "right skew" without looking at it. A number near zero will cause people to call the distribution "symmetric". In the first case, the distribution will be asymmetric but it may not appear to be skewed with a tail to the right. And the value can be exactly zero without the distribution being symmetric. While symmetry implies zero third moment (if it exists), the implication does not go back the other way.
Consider a fair six-sided die with the following labels on its faces: 0, 0, 5, 5, 5, 9. The distribution of outcomes is not symmetric, yet its "skewness" measure is zero. Unfortunately, based only on the information that the third central moment was zero, many people would in fact describe it as symmetric.[Added later: here's one that most people would say on inspection was "right skew" - an ordinary die labelled 0, 0, 5, 5, 7, 10. But the third central moment is zero.]
Other examples, both continuous and discrete, abound. Continuous asymmetric distributions exist for which all odd central moments are zero.
Other measures of skewness (and there have been many) also have their problems, though some are very useful.
The sort of problem described here is essentially unavoidable - there are so many ways a distribution may be asymmetric that a single measure of asymmetry cannot possibly suffice, except perhaps within the framework of a particular family of distributions.
This cautionary tale is not an argument against using measures like the third central moment to attempt to capture something of deviations from symmetry - it's an argument against investing them with more than the limited meaning than they posses.
In order to make progress, we often attempt to quantify those descriptives. This is not just useful, it's often unavoidable, but the act carries with it a special danger, because we then (almost universally) invest the nonunique quantification of the concept with a "reality" that it doesn't merit - and that can lead to nonsense.
[It may be this is another version of the common phenomenon of believing mathematical models when at the beginning we knew them to be at best a rough approximation. Models tend to take on a life of their own, and their conclusions are often treated with a respect we did not accord the original model when it was first tentatively adopted. We need to step back and remember the model was never exactly the thing it was used to describe.]
I think this may be the flip side of what Blake Stacey was talking about when he was discussing the confusions that come in when an inherently mathematical concept is translated into a nonmathematical description.
Let me take a concrete example with which I am familiar. It relates to descriptions of probability distributions.
We begin with something that is inherently mathematical - symmetry. Symmetry is an extraordinarily useful, almost universal concept in mathematics. For what I'll be talking about you can just use the more common senses of reflection symmetry and rotational symmetry.
In the case of symmetry of distributions, there are several ways to define it (if you have a continuous or a discrete random variable, you can define it in the usual "mirror symmetry" sense), but the more general definition would have something like "m is the centre of symmetry if Prob(X≤ m-a) = Prob(X≥ m+a) for all a" (which effectively corresponds to rotational symmetry of the distribution function about the points (m, 1/2), if you tidy up details the right way).
Anyway, "symmetry" is both easily understood and fairly easy to pin down. Of course, almost all distributions are not symmetric (but symmetry arises in a natural way in particular circumstances, so it's far from a useless concept).
Aside from "unimodal" (one "hump"), or the overused and badly abused "bell-shaped", one of the most common descriptions of a distributional shape that is applied would probably be "skewed". An elementary book might have a diagram like the top one below (see for example, the diagram at Wikipedia's entry on skewness). The corresponding distribution function is underneath.
If you flipped the above density left-to-right (or rotated the distribution about the median), it would have left (negative) skewness.
Notice that "right skewed" means the long tail is to the right (this is often the opposite of a beginner's intuition about what the term should mean).
The problem comes when this seemingly clear but actually vague notion is quantified. There are numerous quantities that have been called "skewness". By far the most popular is the standardized third moment (or, equivalently, the standardized third cumulant) - so much so that it is frequently called "the" skewness. Equivalent sample statistics are used for samples.
Now for a distribution like the one above, this quantity is positive (when it exists). If you flip that density shape left to right, the skewness measure is negative. Importantly, when the density is symmetric and the first three moments exist, the skewness is 0. So far so good - left and right skewness and symmetry in pictures generally correspond to negative, positive and zero quantities on the measurement.
However, the problem comes when interpreting a standardized third moment back in terms of the density.
A positive number will lead people to call the distribution "right skew" without looking at it. A number near zero will cause people to call the distribution "symmetric". In the first case, the distribution will be asymmetric but it may not appear to be skewed with a tail to the right. And the value can be exactly zero without the distribution being symmetric. While symmetry implies zero third moment (if it exists), the implication does not go back the other way.
Consider a fair six-sided die with the following labels on its faces: 0, 0, 5, 5, 5, 9. The distribution of outcomes is not symmetric, yet its "skewness" measure is zero. Unfortunately, based only on the information that the third central moment was zero, many people would in fact describe it as symmetric.[Added later: here's one that most people would say on inspection was "right skew" - an ordinary die labelled 0, 0, 5, 5, 7, 10. But the third central moment is zero.]
Other examples, both continuous and discrete, abound. Continuous asymmetric distributions exist for which all odd central moments are zero.
Other measures of skewness (and there have been many) also have their problems, though some are very useful.
The sort of problem described here is essentially unavoidable - there are so many ways a distribution may be asymmetric that a single measure of asymmetry cannot possibly suffice, except perhaps within the framework of a particular family of distributions.
This cautionary tale is not an argument against using measures like the third central moment to attempt to capture something of deviations from symmetry - it's an argument against investing them with more than the limited meaning than they posses.
Friday, June 20, 2008
The tale of "The square root of a rectangle"
Wow. Just wow.
I was just flicking through channels and caught a little of "Engineering an Empire" on the History Channel. The episode was on the Maya.
The narrator said something about the Maya knowing how to "compute the square root of a rectangle".
I was so boggled, I concluded I must have misheard - but no, it was repeated a minute or so later. It was emphasized; apparently we were expected to find this an impressive feat.
The Maya, apparently, knew how to compute the square root of a rectangle. That's pretty clever of them, because that, as far as I can see, is utter nonsense.
As Inigo Montoya might have put it: I do not think it means what you think it means.
I wondered if they meant "compute the hypotenuse of a right angled triangle" (which is equivalent to the diagonal of a rectangle). I also wondered if perhaps they meant "the geometric mean of two numbers" (since that would be the square root of the area of a rectangle). I also wondered if they possibly intended something else.
I concluded they probably meant the first thing (since that's a very valuable thing for a civilization to be able to do, crucial to surveying land - and then they had a shot of a Mayan guy squinting across the top of a stick as if he was indeed surveying). So that would kind of make sense.
But what the heck is someone who isn't comfortable with mathematics supposed to make of it?
The show was no cheap-and-nasty affair. They made some pretty fancy graphics. They had actors dress up in costumes, and they had some pretty fancy sets and location shots. This was a pretty involved documentary. Why the hell couldn't they have had the script looked over by anyone with even a modest bit of mathematics? Say, a mathematics undergrad? (Heck, I could have told them that was obvious nonsense before I was 16.)
That bit of (repeated, emphasized) mathematical nonsense is enough to make the entire program suspect - because if they were that lax with the mathematics, who knows what care was taken with the rest of it?
I was just flicking through channels and caught a little of "Engineering an Empire" on the History Channel. The episode was on the Maya.
The narrator said something about the Maya knowing how to "compute the square root of a rectangle".
I was so boggled, I concluded I must have misheard - but no, it was repeated a minute or so later. It was emphasized; apparently we were expected to find this an impressive feat.
The Maya, apparently, knew how to compute the square root of a rectangle. That's pretty clever of them, because that, as far as I can see, is utter nonsense.
As Inigo Montoya might have put it: I do not think it means what you think it means.
I wondered if they meant "compute the hypotenuse of a right angled triangle" (which is equivalent to the diagonal of a rectangle). I also wondered if perhaps they meant "the geometric mean of two numbers" (since that would be the square root of the area of a rectangle). I also wondered if they possibly intended something else.
I concluded they probably meant the first thing (since that's a very valuable thing for a civilization to be able to do, crucial to surveying land - and then they had a shot of a Mayan guy squinting across the top of a stick as if he was indeed surveying). So that would kind of make sense.
But what the heck is someone who isn't comfortable with mathematics supposed to make of it?
The show was no cheap-and-nasty affair. They made some pretty fancy graphics. They had actors dress up in costumes, and they had some pretty fancy sets and location shots. This was a pretty involved documentary. Why the hell couldn't they have had the script looked over by anyone with even a modest bit of mathematics? Say, a mathematics undergrad? (Heck, I could have told them that was obvious nonsense before I was 16.)
That bit of (repeated, emphasized) mathematical nonsense is enough to make the entire program suspect - because if they were that lax with the mathematics, who knows what care was taken with the rest of it?
Sunday, June 15, 2008
Adding percentiles means there's probably more oil than we thought
Isabel Lugo has an interesting post up that points out that because "proven reserves" figures for individual oil fields are given as the tenth percentile (that is, the amount of oil in a field is estimated to have a 90% chance of being larger than the stated amount), you can't just add up the individual "proven reserve" figures and have a figure that means the same thing. But apparently that's exactly what many bodies do with the figures!
(She refers to this New Scientist story.)
Isabel gives an example where adding two tenth percentiles gives much lower than the tenth percentile of the estimate of the total oil for both (with the implication that if you add enough of these things together, the true amount of oil is probably far, far larger).
[I pointed out in comments that the conservatism in the sum is not necessarily the case (but for readers here - it very likely is the case; the counterexample to conservatism that I give is not a likely situation).]
(She refers to this New Scientist story.)
Isabel gives an example where adding two tenth percentiles gives much lower than the tenth percentile of the estimate of the total oil for both (with the implication that if you add enough of these things together, the true amount of oil is probably far, far larger).
[I pointed out in comments that the conservatism in the sum is not necessarily the case (but for readers here - it very likely is the case; the counterexample to conservatism that I give is not a likely situation).]
Subscribe to:
Posts (Atom)
