On proper care and feeding of ideas

I'm writing several papers with an able student of mine (who is presently working as a research assistant with me).

We're working in a financially-related area. It's not the area my PhD is in - I'm a statistician, but one of my undergrad degrees was in this area, and the problems we're looking at are essentially forecasting-related and many of the published papers (at least the better ones) are at heart applied statistics, with a few twists that arise from the particular application.

One thing that has come up over and over again as we work is just how careless some of even the best-known work in this particular sub-area is. Many papers that pass peer review display a fundamental ignorance of the work that they themselves refer to (that is, they appear not to have actually read much of their references, and missed important information contained in them). Their algorithms have not been carefully checked, and fail to meet even fairly basic "reasonableness checks". They notice and then effectively ignore major clues about what's wrong (describing ways to avoid problems that should have acted as large flashing neon warning signs). They often make unsupported assertions that might seem plausible but which are in fact (if you check carefully) untrue. Even the better papers we're looking at contain many subtle errors.

It's a problem outside of the narrow area we're currently working in, and seems to infest a much broader swathe of literature.

There's a distinct lack of scholarship. Little intellectual rigor, little curiosity. Editors, referees and authors are often woefully ignorant.

Much of the best work is flawed and the more mediocre efforts are so laughable that I'd be unable to pass an undergrad for doing better work than they can get published. Some of these obviously ludicrous efforts win prizes.

When I act as a reviewer, I am frequently met with astonishment from editors at my thoroughness. I am in fact not a particularly thorough reviewer, but I do at least try to be somewhat familiar with the necessary background before trying to review work, I try to read the paper all the way through (and the substantive parts more than once) and as far as possible understand it and where feasible check what it says, make certain the arguments hang together and make sense. I do reasonableness checks where I can. That is, I do a fairly elementary level of checking that the work is not total garbage (less than with my own work).

The fact that even this fairly basic process can take me many weeks of solid work and result in a referees report longer than the original paper is not an indication that I'm obsessive, but it's an indication of just how careless most of the work in this (broader) area is.

And it's more general than that. I have helped people out in other areas (I could list something approaching a dozen, some widely divergent from the ones I've mostly worked in) where even the "classic" papers seem to be full of half-baked nonsense mingled with good ideas, joined up with death-defying leaps of logic and half-understood references.

I posted recently about how making mistakes is not necessarily a bad thing. The ability to make mistakes is an important part of getting things right. But you have to be willing to look for your errors and try to correct them. The error-filter can't be left out! And if you're writing supposedly academic papers, you should try to do it before they're published.