Academic Articles on the Performance of New Methodology
When we started our work to release our first release of CART (a 1993 command line version running very nicely on UNIX), I was startled by some (now long forgotten) articles claiming to describe a new technology that was more accurate, or faster, on some class of analytic problem. At the time, I assumed that such articles needed to be taken seriously because they represented peer-reviewed, solidly-researched scientific advances.
Every time such an article or book chapter appeared I would be on the phone with one of the four co-creators of CART asking for their opinion and advice. In the early days, such conversations were frequently with Richard Olshen and Leo Breiman, and later more frequently with Jerry Friedman and Chuck Stone, but I had no compunction soliciting the thoughts of all four. I was always impressed first by how calm they always were in the face of something dramatically new and "better" and by their even-keeled and I now see justified skepticism. The interesting thing is: there is a non-stop flow of articles about something newer and better, and there is a non-stop increase in the number of ignored and forgotten new methodologies. So what can we make of this?
There is quite a bit to say about this topic and I won't cover most of it in this blog entry. Instead, I want to convey something Jerry Friedman told me just last week when I had another such conversation with him concerning yet another "better" method. I will be paraphrasing Jerry and I hope I don't interject anything he won't agree with. Jerry first observed that of course an article introducing a new methodology always has as its primary conclusion that the author's new method is better or faster because this is the raison d'etre of the article. The performance results displayed in various tables reporting Monte Carlo experiments or results on well-trodden archives of public data sets always show the new method is at least usually if not always best. As this goes without saying, Jerry recommended ignoring this part of the report. This does not mean, however, that we should ignore the rest of the paper! Jerry's insight is that typically the authors of such papers are usually reasonably objective about any method that is not theirs. Hence, the reports on the relative performance of the other (older) methods have the possibility of being interesting.
So now we have Friedman's rule on such academic articles: Ignore any results on the so-called "winner" and focus on the results reported for all the other methods!