As mentioned in the introduction, the evaluation of reactive scheduling systems is not so easy. We want to evaluate our approach by using a simulation environment in which unanticipated events such as new orders, canceled orders, late completion of processes, bad product quality, break-down of machines, and other events occur randomly. Since no adequate tool exists for our purpose we will develop a simulation framework with which production process simulations can be easily built. The framework allows the description of production capabilities of a certain plant and some distribution function for the occurrence of unanticipated events. Although many simulation tools already exist, none matches our requirements exactly. If we would use a tool, as for example SIMAN, the effort of interfacing it with a reactive scheduler would be about the same effort as building our own tool. Furthermore, the proposed approach will use the same object descriptions as the scheduling system. This simulation environment should communicate like a real application with the scheduler. It must pass the scheduler such events such as start and finish of an operation and all unanticipated events.
With this simulation environment, we test the reactive scheduler as to whether it will always react in time, the robustness of the produced schedules, and the quality of the schedules. These measurements shall be compared with alternative approaches. Since no public benchmarks exist for reactive scheduling there are no other results we can compare our approach with. Therefore, the system shall be compared to a system applying simple dispatching rules, and to the OPIS-system of Carnegie-Mellon University. A system which applies only dispatching rules to allocate orders to availabe resources, can be relatively easily realized. However, this system can not be compared in terms of the robustness since no predictive schedule is changed by dispatching rules. We can only compare the quality of the solution. The comparison to the OPIS-system is more interesting. If our approach can compete with the OPIS-approach, our system will be benefitial independent of domain-specific repair heuristics. To enable a comparison with OPIS, we must realize our scheduling scenarios in OPIS and interface OPIS with our simulation environment. The Robotics Institute has shown its interest in this comparison.