Monday, October 20, 2008

Statistics as philosophy

This post is related to a point I often try to make (not that I am completely of a mind with the author, but much of what he's saying I identify with).

Fundamentally, statistics is different from mathematics, though it uses the tools of mathematics. Mathematics helps with the "what" (such as "given I want to measure this, what do I do?", but the "why" (such as in the sense of "why work that out, rather than something else") is somewhere other than mathematics.

This point is often lost on otherwise highly competent people. I have seen many good mathematicians come a cropper on it. Some of the worst explanations of statistics I have ever seen come not from people who have trouble with the mathematics in it, but from people who have no trouble with the mathematics at all. (I could name names, but I am feeling generous today.)

And it's often like that with students - I see it a lot. As a student, I had a similar experience to the poster I linked to - I was - manipulation-wise - reasonably competent at statistics. I could do the calculations, if it was reasonably clear what calculations were required. But I did two years of it and still didn't comprehend it. I didn't even comprehend that there was something to comprehend - after all, I could pass the subjects okay, so I must have 'got it' okay, even though it seemed sort of wishy-washy to me. But actually I didn't get it at all. It wasn't until I was some doing third-year subjects that it eventually clicked. I suddenly understood what all the previous subjects had been about. I got it. The material I had learned wasn't a bunch of different stuff all lumped together that was done the way it was purely by convention (though there are no shortage of conventions) - there was, in fact, a coherence to it all. It was all of a thing, it fitted together; the stuff I'd learned was the result of a limited collection of concepts applied to different problems. I could actually begin applying my understanding outside my direct learning, to problems I'd not seen before. I had a framework within which each new piece of knowledge fitted in with everything else.

I'm not certain how to even convey this understanding, though I try. Students recognise that I'm passionate, at least (or so they tell me), though I'm not sure that the "why" always comes across to more than a very few of them.


Anonymous said...

Hmmm…I certainly agree that statistics is different from what most mathematicians do, but it seems to me that it's more a matter of the fact that statistics attempts to answer questions about the real world. Granted that I haven't done much statistics (or any applied math), but it seems to me that the distinction is pretty much one of "pure" vs. "applied" math.

In my interests I'm about as pure a mathematician as they come--I love abstraction and I could care less how my math relates to the real world. But I can still understand the basic statistics I've studied, and it doesn't seem to be fundamentally different from other aspects of applied math, in that you're developing and using mathematical tools to answer concrete questions.

Valhar2000 said...

I had a similar problem with probability in school. I could do all the problems and got good grades, but I knew that there was a deeper understanding the eluded me, and that if I had understood it properly I could have done much more interesting things with it (and the problems we got in school would have been absurdly easy).

However, I went on to do other things and now my math is rusty.