Prediction models are used to forecast future events on the basis of past performance data. One of the most frequently used and simplest prediction models is one that posits that performance of a given system in the future will resemble performance in the past, apart from random variation. This form of prediction modeling is thus based on past performance evaluation, with the past performance projected into the future. The system that is evaluated may be a process, technology, strategy, treatment, organization or individual, and, for ease of reference, is referred to as an "entity." Examples of such entities, along with the type of entity and criteria that might be used to evaluate the entity, are given as follows:
______________________________________ ENTITY TYPE EVALUATION CRITERION ______________________________________ Mutual Fund Organization, % Return, Process, or % Return per Unit Risk Strategy Athlete Individual Batting Avg., ERA Sports Team Organization Winning % Airline Organization % Return on Stock Pharmaceutical Treatment Survival Rate, Remission Rate Chemical Reaction Process, Technology Yield Rate Noise Reduction Process, Technology Gain in Signal-to-Noise Method Controller Process, Technology Tracking Error Stock Portfolio Process, Strategy % Return ______________________________________
The above-referenced patent application recognizes that it is problematic to ascertain whether a successful forecast is due to luck, or due to the effectiveness of the prediction model used to generate the forecast. To address that problem, the above-referenced patent application provides a system for testing the effectiveness of prediction models.
Relatedly, I have further recognized that it is problematic to ascertain whether an apparently good past performance is due to luck, or due to the effectiveness of the entity generating the performance. As an example, if a mutual fund performs well, it might be that the successful performance was a result of investment skill on the part of the fund manager, but it is also possible that the good performance was actually a matter of luck. For example, the fund managers might have switched from stocks to bonds just before a market crash, but for reasons other than their anticipation of the crash. In such a circumstance, the mutual fund is not "good," only lucky.
Nevertheless, it might be evaluated under such circumstances that the performance of the entity is truly good and may be relied upon in the future, based on a lucky outcome. Such lucky outcomes can arise easily when, as is common, a plurality of comparable entities are evaluated for purposes of attempting to find the best of the group compared. This use of the performance data is a form of "data mining." When one engages in such mining there is thus a significant danger that lucky results will be mistaken for good results.
The above-referenced patent application addresses a way to avoid the adverse consequences of so-called data "snooping" by providing an indication of the statistical significance of a prediction model's performance. As recognized herein, it is also possible to avoid the adverse consequences of data mining by providing an indication of the statistical significance of an entity's performance. As further recognized herein, one way to measure the statistical significance of an entity's performance is to compare the entity's performance with the performance of a benchmark, often one that is standard or straightforward. To use the mutual fund analogy, a benchmark against which the performance of various mutual funds might be compared is "always hold the Standard & Poor's 500."
As still further recognized herein, however, it is desirable to understand the statistical significance of performance outcomes vis-a-vis a benchmark performance in the context of more than a single entity, that is, in comparing a group of entities, to avoid the adverse consequences of data mining. That is, the present invention recognizes that it is desirable to generate performance evaluations for each of a group of comparable entities and then determine the statistical significance of the best of the entities relative to the benchmark. Such consideration of the performance of a plurality of entities is called a "group comparison" and is a form of data mining. Stated differently, a statistic that represents the statistical significance of a "best" entity vis-a-vis a benchmark can be misleading, unless a complete group comparison is reflected in the statistic. By accounting for a group comparison in the statistics, incorrectly positive evaluations of the effectiveness of an entity can be avoided.
The present invention accordingly recognizes the need to provide a computer method for evaluating the statistical significance of the best of a plurality of comparable entities, vis-a-vis a benchmark by computing an estimate of a p-value for a test of the formal null hypothesis that a best entity has expected performance no better than that of a benchmark, where the p-value is the probability of wrongly rejecting the null hypothesis on the basis of the evidence provided by the data. Another object of the present invention is to provide a method for performance evaluation in the context of a group of comparable entities. Still another object of the present invention is to provide a method for performance evaluation that is easy to use and cost-effective.