At least partial automation is being applied in a growing range of fields, improving the efficiency of industrial processes and raising productivity dramatically. The improvement has not been fully matched in the service sector, however, in part because many of the tasks carried out in the services industries are perceived to require intelligent reasoning and skills that are not easily emulated by machines.
Lack of automation is notable in areas traditionally associated with complexity and/or intuition. For example, although computers have greatly aided the process of processing financial information, it is still principally a human-based activity to interpret technical analysis charts and predict company performance. While automated algorithmic trading systems are becoming more popular, these systems are not true predictive systems since they rely, ultimately, on heuristic rules that cannot have global validity over time in a constantly changing financial market.
In many other fields, a relatively high level of automation has been achieved, such as in information storage, retrieval and communication. At the level of predictive analysis, however, automation is very rudimentary. The increasing demand in recent years for “predictive analytics” has been chiefly addressed with large analytic engines, such as those of SAS and SAP. Such analytics engines are essentially large, well equipped statistical tool boxes, but require significant training to use. Additionally, a user has to decide which statistical model is most appropriate for a given problem. When confronted by a choice of thousands of possible models, die user must either use expert human advice or develop the relevant skills to make a suitable judgment.
Furthermore, a model that is good for one aspect of analysis may be poor for another. For instance, neural networks can be very useful non-parametric statistical estimators that can lead to good predictive accuracy in a large variety of problems. Their outputs, however, may be opaque and therefore difficult to understand from, for example, a business perspective. On the other hand, association rules can be quite transparent, but may be suboptimal in terms of the precision of their predictions.
Such deficiencies are more than academic. In today's complex business world, problems are multi-faceted, addressing important prediction tasks such as—Who are my best customers? Where should a new sales office be located? When is a customer likely to change vendors? Associated with these different questions there is another—Why? An answer to each of the above however, gives only a partial solution to the overall problem of increasing the company's ROI or profit margins.
Each of the above is then a sub-problem, associated with a particular perspective, of the overall problem. Hence, solutions should be multi-faceted and multi-perspective, with solutions to sub-problems being combined together to form solutions at an aggregate level. This is precisely how teams of humans working together function—outputs from different sub-teams demonstrating expertise in a particular area being integrated together to give an overall solution.
Up to now, such high level cooperation and integration has been an exclusively human domain. For example, for a typical company, experts in marketing try to advertise a particular product line to stimulate demand, without necessarily having good predictive tools to understand how a particular marketing campaign can translate into demand. Meanwhile, another group has to plan the production that will satisfy the demand. Yet another group has to plan how to sell the product through different channels while another integrates all these different perspectives at a corporate level to assure that all the sub-teams are functioning within the framework associated with the company's overall goals. However, these goals have to be adjusted and adapted according to the constant feedback from the different sub-teams. Perhaps the production team cannot satisfy the demand generated by the marketing team for instance.
Artificial Intelligence is a relative newcomer to the field of predictive modeling and has held out the hope of providing automated systems that may one day substitute, at least partially, some of the high level tasks normally associated with humans. However, although there are now many systems available for prediction which may contain sophisticated elements such as neural networks and evolutionary algorithms, these systems may apply highly non-linear analysis and use computationally complex processes whose results can be highly unstable. Additionally, they do not necessarily offer predictions a priori but need to be “tuned” or “trained” by the user who almost inevitably is not an expert in artificial intelligence and, therefore, likely to produce unreliable results. Additionally, training of neural networks or optimization of genetic algorithm parameters, if done correctly, tend to be computationally intensive processes requiring computational resources and resources of time from the client that could better be dedicated to other tasks. Furthermore, typically, artificial intelligence applications have as their goal the solution of a very specific (sub)-problem. For instance, the IBM computer Deep Blue can play chess at the very highest level. However, it cannot do anything else. As emphasized above, in the real world, global “solutions” to real problems often require the simultaneous solution of many different sub-problems. On the forefront of artificial intelligence research are intelligent artificial agent systems which are now opening new avenues for productivity increases in areas where humans are carrying out repetitive intelligent tasks. This sets the stage for a new technological revolution that will change the way in which many services are rendered. Commercial applications of intelligent agents have essentially been restricted to “data mining” where a more intelligent search of databases is carried out. In fact many such systems are no more sophisticated than standard web search engines.
The use of analytical models may be complicated by factors such as a lack of data, too much data, and/or difficulty of determining a correlation between inputs and outputs. As to systems having a lack of data, this condition may arise due to a failure to properly record activity or difficulty of quantifying activity. For example, without detailed records of the behavior of financial instruments, it would be difficult to make any predictions regarding future trends. As another example, an advertising agency may have difficulty quantifying such concepts as brand loyalty, whether an ad is humorous, and/or the like. Accordingly, it may be difficult for an advertising agency to replicate past successes.
Analysis may be complicated by the sheer volume of data. For instance, a variety of predictive functions may be employed to produce an estimate of future activity. As the volume of data increases, it may become cumbersome to manually determine estimates based on the predictive algorithms.
In addition, in some systems correlations may be difficult to determine. For instance, accurately predicting whether a baseball prospect will be a future Hall of Frame player may be complicated by the difficulty of isolating indicia of future ability. As another example, predicting whether a condominium complex will be successful in a given area may be exceedingly difficult given the variety of factors relating to success.
While prediction is an increasingly analytical rather than intuitive process, even relatively quantitative systems of prediction may have substantial shortcomings. While sports scouts may make a living by predicting the future success of athletes based on a combination of statistical analysis and experience, the number of failed former first round draft picks would suggest that conventional scouting methods are not necessarily robust. In addition, that corporations frequently suffer decreased share prices due to overestimated earnings suggests that the predictive tools employed by even the biggest corporations do not operate with optimal accuracy.