Maintaining And Optimizing Dependencies Between Statistical Calculations

Statistical calculations involve iterating a (possibly very large) dataset one or more times. The designer of a statistical analysis tool wants to ensure that no more iterations than necessary are performed. Whereas, on a case by case basis, a statistical calculation can be optimised by inspection this is not practical in a general purpose statistics tool where a set of several statistical calculations are to be determined and the elements of the set are, at time of design, unknown. This presentation shows how caching, and a dependency graph can be used to determine the optimal number and order of iterations. An implementation is presented, which demonstrates how the use of lisp can obviate the need for the programmer to maintain the dependency relationships. Instead, they are extracted from the implicit information contained within the program itself.