Tackling the Debugging Challenge of Rule Based Systems

Valentin Zacharias
FZI Research Center for Information Technology, Karlsruhe 76131, Germany zach@fzi.de http://www.fzi.de

Abstract. Rule based systems are often presented as a simpler and more natural way to build computer systems - compared to both imperative programming and other logic formalisms. However, with respect to finding and preventing faults this promise of simplicity remains elusive. Based on the experience from three rule base creation projects and a survey of developers, this paper identifies reasons for this gap between promise and reality and proposes steps that can be taken to close it. An architecture for a complete support of rule base development is presented. Keywords: Rule based systems, F-logic, Knowledge acquisition, Rule based modeling, Knowledge modeling, Verification, Anomaly detection.

1 Introduction
Rule based systems are often presented as a simpler and more natural way to build computer systems - compared to both imperative programming and other logic formalisms such as full first order logic or desciption logics. It seems natural to assume that this simplicity also applies to finding and preventing faults; to software quality assurrance in general. However, with respect to debugging of rule bases this promise of simplicity remains elusive. Indeed a survey found [1] that most developers of rule based systems think that the debugging of rule based systems is more difficult than the debugging of ’conventional’ - object oriented and procedural programs. An observation that the authors also found corroborated in three rule base development projects they participated in. 1.1 The Debugging Challenge The assumption of rule based systems as simpler than ’conventional’ programs rests on three observations: – Rule languages are (mostly) declarative languages that free the developer from worrying about how something is computed. This should decrease the complexity for the developer. – The If-Then structure of rules resembles the way humans naturally communicate a large part of knowledge. – The basic structure of a rule is very simple and easy to understand.
J. Filipe and J. Cordeiro (Eds.): ICEIS 2008, LNBIP 19, pp. 144–154, 2009. c Springer-Verlag Berlin Heidelberg 2009

The authors found this problem of fault detection corroborated in three different projects he participated in. chemistry and biology. This project was performed by a junior developer who had little prior experience with rule based system.2 Overview This paper is an attempt to reconcile the promise of simplicity of rule based systems with reality. – The goal of the project F-verify was to create a rule base as support for verification activities. 2 Experiences In addition to the survey in [1] and literature research. As part of the second phase of Project Halo six domain experts were employed for 6 weeks each to create rule bases representing domain knowledge in physics. Towards this end an existing reporting application in a human resource department was re-created as rule based system. The results showed. rank it as the issue most hindering the development of rule based systems.com . in a different question.projecthalo. – Project Halo1 is a multistage project to develop systems and methods that enable domain experts to model scientific knowledge. The next two sections then discuss overall principles to better support the creation of rule bases and an architecture of tool support for the development process. reuse and understandability that were expected from rule based (and declarative) knowledge and program specification. that these developers indeed see the advantages in ease of change. 1 http://www. – Online Forecast was a project to explore the potential of knowledge based systems with respect to maintainability and understandability. This paper’s structure is as follows: section two gives a short overview of the three projects that form the input for this analysis. Section three discusses what kinds of problems were encountered. 1. However. It models (mostly heuristic) knowledge about anomalies in rule bases. Approximately 5 person months went into this application. participants of the survey saw debugging of rule based systems as actually more difficult than the debugging of conventional software. a survey of developers of rule based systems [1] included a question that asked participants to compare rule base development to the development of procedural and object oriented programs. at the same time a majority of rule base developers sees rule base development as inferior with respect to debugging. And. The analysis presented in this paper is based on the experience from three projects. This project was done by a developer with experience in creating rule bases and took 3 month to complete. It consists of anomaly detection rules that work on the reified version of a rule base.Tackling the Debugging Challenge of Rule Based Systems 145 To see whether this simplicity assumption holds in practice. It identifies why rule bases are not currently simple to build and proposes steps to address this.

it still stands in a marked contrast to the often repeated assertion that creating rule bases is simpler. – That one rule understands a parent relation biologically. methods and tools presented in this paper are geared towards this kind of small. F-verify used Ontostudio together with prototypical debugging and editing tools. – That one rule processes a number as inch. F-logic is very similar to some of the rule languages under discussion for the Semantic Web [4] and it is based on normal logic programs probably the prototypical rule language. in particular when performed by domain experts or junior programmers.as is to be expected for such relatively small projects. The analysis. 3 Analyis In the three projects sketched above the authors found that the creation of working rule bases was an error prone and tiresome process. The editing tools differed between the projects: Project Halo used the high level. domain expert/end user programmer driven projects. the inference engine obviously combines the rules only based on how the user has specified the rules. because they are true in broader contexts than program statements can be used [5]. In particular for rules in a rule set to work together.in the creation of rules in a way that they can work together . Declarative statements such as rules are often assumed to be easier reusable and more modular. all rules need to be consistent in the domain formalization and the use of terminology. The authors examined literature and tools to ensure that tool support identified as missing is indeed missing from rule based systems and not only from systems supporting F-logic. Developers with prior experience in imperative programming languages said that developing rule bases was more difficult then developing imperative applications. We identified six reasons for this observed discrepancy. All of these projects have been created using relatively lightweight methods that focus more on the actual implementation of rules than on high level models . 2 http://www. The use of only one rule formalism obviously restricts the generality of any statements that can be made.ontoprise. For example it must not happen: – That one rule uses a relation part while another one uses part of . Based on these sentiments the one rule fallacy goes as follows: One rule is simple to create and the combination of declarative rules is handled automatically by the inference engine hence entire rule bases are simply to create. However. While part of these observations can be explained by longer training with imperative programming. while another processes it as millimeter. Zacharias All of these projects used F-logic [2] as representation language and the Ontobroker inference engine [3].146 V. while another understands it socially.de . Online Forecast used a simple version of Ontoprise’s2 OntoStudio.that most errors get made. and it is here . graphical tools developed in the project. – The One Rule Fallacy and the Problem of Terminology. In all cases the developers complained that creating correct rule bases takes too long and in particular that debugging takes up too much time. However.

because it becomes hard. Because rule interactions are managed by the inference engine. however.should be part of(Chili. The problem of terminology makes the rule base harder to understand. another that it leads to a rule being tried in unexpected circumstances that then lead to errors due to the wrong usage of built-ins. Not having an overall view on the rule base mainly affects the ability of developers to correctly edit the rule base. More typical than this example. At the same time. In the authors experience failed interactions between rules are the most important source of errors during rule base creation. for example.is absolutely unrelated. – The Problem of Interconnection. user managed modularization. this depends on ensuring the consistent formalization and use of terminology across the rule base. to see the consequences of changes or to gain confidence in the completeness of a rule base.Tackling the Debugging Challenge of Rule Based Systems 147 Rule based systems hold the promise to allow the automatic recombination of rules to tackle even problems not directly envisioned during the creation of the rule base. This complicates error localization for the user. .from the user’s point of view .X) . At the same time this points to the Problem of Terminology as one explanation for the difficulty in debugging rule bases. As a very simple example consider the following rule base: hot(X) ← part of (Chile. are cases that do not lead to wrong conclusions but rather merely lead the inference engine to try one path that ends in a rule cycle or in some kind of error. shown in the source code and the subject of visualizations. everything is potentially relevant to everything else3 . This stands in contrast to imperative programming. The Problem of Opacity also affects the developer’s ability to identify faults in rule bases. part of (Chile. – The Problem of Opacity. where the relations between the entities and the overall structure of the program are explicitly created by the developer. makes it harder to identify. One example would be that a rule causes a whole rule base to become unstratisfiable. SouthAmerica) In this example a rule that deduces that something is hot based on it having chili as part. however. avoid and correct mistakes based on only a local view of the rule base. that is opaque to the user. because errors appear in seemingly unrelated parts of the rule base. Hence a part of the gap between the expected simplicity of rule base creation and the reality can be explained by naive assumptions about rule base creation. The only common way to explicitly show these interactions between the rules is in a prooftree of a successful query to the rule base. This simple error then introduces a connection to a part of the rule base that deals with countries and that . is changed by a typo to deduce that something is hot if the country Chile is part of it.. it is this interaction between rules that is commonly not shown. However. X) # fault . 3 At least in the absence of robust.

By far the most common symptom of an error in the rule base was a query that (unexpectedly) did not return any result . This debugging paradigm is well known from the world of imperative programming and characterized by the concepts of breakpoints and stepping. – The Problem of Procedural Debugging.hence the order of debugging is based on the evaluation strategy of the inference engine. In this way procedural debugging of rule bases breaks the declarative paradigm . Doing these design and architecture decisions ’on the fly’ reduces the risk of costly requirements. .it forces the developer to learn about the inner structure of the inference engine. Zacharias – The No-Result Case and the Problem of Error Reporting. not on the expected scope of the final application. The development of rule based systems cannot take full advantage of the declarative nature of rules. The first corollary of this observation is that the verification support for these systems cannot rely on the availability of formal specifications of any kind. Agile and iterative methods were used in more than 50% of the projects with more than 10 person months of effort. however. at the same time it means that many of these mistakes are then made during implementation.they are now widely believed to be superior to waterfall-like models [7]. In such a case most inference engines give no further information that could aid the user in error localization. Both imperative and rule based systems sometimes show bugs by behaving erratically. Recent years also saw the increasing adoption and acceptance of iterative and evolutionary development methodologies [6] . With many of these methods architecture and design decisions are taken on the fly during the implementation of software. In addition. A further 23% of the projects worked without following any specific methodology. The execution steps of a procedural debugger are then for instance: the inference engine tries to prove a goal A or it tries to find more results for another goal B.hence the order of debugging is based on the evaluation strategy of the inference engine. when debugging is done on the procedural nature of the inference engine. All deployed debugging tools for rule based programs known to the authors are based on the procedural or imperative debugging paradigm.the no-result case. an impediment for many static analysis methods. A second likely consequence is a higher prevalence of implementation mistakes. because agile methodologies encourage developers to make these decisions based on the scope of the current iteration. as a declarative representation a rule base does not define an order of execution . but only rule based systems show the overwhelming majority of bugs by terminating without giving any help on error localization. The user can then look at the current state of the program and give commands to execute a number of the successive instructions. these design and architecture decisions are often repeated and changed. A procedural debugger offers a way to indicate a rule/rule part where the program execution is to stop and has to wait for commands from the users. A high prevalence of agile and iterative methods was also found in the developer survey [1]. However.148 V. design and (up-front) architecture mistakes. Iterative and Lightweight Methods. – Agile. However a rule base does not define an order of execution . This is unlike many imperative languages that often produce a partial output and a stack trace. This stands in marked contrast to the idea that rule bases free the developer from worrying about how something is computed.

try-and-error process of rule base creation by allowing trying out a rule as it is created. Within this group an increasing role is played by end user programmers . End user programmers usually can’t justify making an investment in programming training comparable to that of professional programmers. End user programmers are particularly important for web related development. This discrepancy is a direct consequence from the fact that in recent years the percentage of applications built with imperative programming languages was much larger than those built with rule languages. – Less refined tools support.11]. Tools that embodied interactivity proved to be very popular and successful in the three projects under discussion. For the US it is estimated that there are at least four times as many end user programmers as professional programmers [8]. These principles where conceived either by generalizing from tools that worked well or as direct antidote to problems encountered repeatedly. The amount of software in society increases continually. because web developers are known to have an even higher percentage of end user programmers [10. Slightly more than half of the projects included at least one domain expert that created rules. 4 This was mostly due to very long reasoning times or due to technical problems with the inference engine. . To create tools in a way that they give feedback at the earliest moment possible. this was reflected immediately in erroneous rules and unmotivated developers. that the average developer is not trained as well as the knowledge engineers that created the first expert systems.5 domain experts that create rules themselves and 1. but has now become a set of skills possessed to some degree by tens of millions of people. – Interactivity. To support an incremental. The survey of developers [1] also found that the average rule base development projects includes 1.Tackling the Debugging Challenge of Rule Based Systems 149 – End User Programmers and Domain Experts. 4 Core Principles The authors have identified four core principles to guide the building of better tool support for the creation of rule bases.people that are usually trained for a non-programming area and just need a program. Tools such as fast graphical editors for test queries. The creation of software artifacts used to be the very specialized profession of a few thousand experts worldwide.7 domain experts that were involved in verification and validation. with estimates for the number of end user programmers ranging from 11 million [8] up to 55 million [9]. those for the development of rule bases often lack refinement. Compared to tools available as support for the development with imperative languages. This rise in the number of end user programmers means. and more and more people are involved in its creation. script or spreadsheet as a tool for some task. In cases where quick feedback during knowledge formulation could not be given4. text editors that automatically load their data into the inference engine or simple schema based verification during rule formulation where the most successful tools employed.

– Visibility.20]) we have identified (and largely implemented) an architecture of tool support for the development of rule bases.150 V. in particular for those geared towards end user programmers [12.17] and will not be further discussed in this paper. The declarititivity principle is a direct response to the problem of procedural debugging described in the previous section.1 Test. Zacharias Interactivity is known to be an important success factor for development tools. Interactivity as a principle addresses many of the problems identified in the previous section by supporting faster learning during knowledge formulation. [14. An editor can automatically display that (given the state of the rule that is being created. Debug and Rule Creation as Integrated Activity In order to truly support the interactivity principle the test. To show the hidden structure of (potential) rule interactions at every opportunity. create test and debug if necessary. This principle is a direct consequence of the problem of interconnection. Immediate feedback after small changes also helps to deal with the problem of interconnection and the problem of error reporting. – Declarativity. the contents of the rule base and the test data) the current rule allows to infer such and such. Based on our experience from the three projects described earlier and on best practices as described in literature (eg [18. This allows the developer to get instant feedback on whether her intuition of what a rule is supposed to infer matches reality. However.16. Declaritivity of all development tools is a prerequisite to realize the potential of reduced complexity offered by declarative programming languages. Based on the test data an editor can give feedback on the inferences made possible by a rule while it is created. 5.13].as mandated by the interactive principle.19. Each of the main parts will be described in more detail in the following paragraphs. To create tools in a way that the user never has to worry about the how. At its core it shows test. These activities are supported by test coverage.even in the absence of actual test queries.g. about the procedural nature of the computation. An overview of this integrated activity is shown in figure 2. debug and rule creation as an integrated activity . This stands in contrast to the usual sequence of: create program part. 5 Supporting Rule Base Development The principles identified in the previous section need to be embodied in concrete tools and development processes in order to be effective. – Modularization. modularization of rule bases has been extensively researched and implemented (e. Visibility is included as a direct counteragent to the problems of opacity and interconnection. Testing has been broken up into two separate activities: the creation/identification of test data and the creation and evaluation of test queries. anomaly detection and visualization. A high level view of this architecture is shown in figure 1. This has been done because test data can inform the rule creation process .15. When the . To support the structuring of a rule base in modules in order to give the user the possibility to isolate parts of the rule base and prevent unintended rule interactions. debug and rule creation activities need to be closely integrated.

Examples for anomalies are rules that use concepts that are not in the ontology or rules deducing relations that are never used anywhere. 5. Test. debug and rule creation as integrated activities inferences a rule enables do not match the expectations of the user she must be able to directly switch to the debugger to investigate this. The overall architecture of tool support for the rule base development Fig. anomalies detection partly addresses the problem of opacity. Of the problems identified in section 3. debug and rule creation because unlike in traditional development: – test data is used throughout editing to give immediate feedback – debugging is permanently available to support rule creation and editing. Anomalies give feedback to the rule creation process by pointing to errors and can guide the user to perform extra tests on parts of the rule base that seem problematic. The developer can debug rules in the absence of test queries. Anomaly detection heuristics focus on errors across rules that would otherwise be very hard to detects. 2.Tackling the Debugging Challenge of Rule Based Systems 151 Fig. Anomaly detection is a well established and often used static verification technique for the development of rule bases. interconnection and error reporting by finding some of the errors related to rule interactions based on static analysis. In this way testing and debugging become integrated. . even without a test query being present. Hence we speak of integrated test.2 Anomalies Anomalies are symptoms of probably errors in the knowledge base [21]. 1. even without an actual test query. Its goal is to identify errors early that would be expensive to diagnose later.

thereby violated the interactivity paradigm. however. Zacharias We deployed anomaly detection heuristics within Project Halo. development. The anomaly heuristics integrated into the editor performed well and were well received by the developers. It enables the user to use all her knowledge to quickly find the problem and to learn about the program at the same time. 5. the more complex heuristics. Unlike in procedural debuggers the debugging process is not determined by the procedural nature of the inference engine but by the user who can use the logical/semantic relations between the rules to navigate. . and weren’t accepted by the users. It enables the user to check which inferences a rule enables. Explorative Debugging puts the focus on rules.4 Debugging In section 3 the current procedural debugging support was identified as unsuited to support the simple creation of rule bases. Not being able to get an overview of the entire rule base and being lost in the navigation of the rule base was a frequent complaint in the three projects described in the beginning. validation and maintenance by facilitating a better understanding of a computer program/model by the developers. The overview visualization of the entire rule base is a response to the problem of opacity. Other logical connections are formed by the prooftree that represents the logical structure of a proof that lead to a particular result. both as very simple heuristics integrated into the rule editor and more complex heuristics as a separate tool. – Visualizes the logical/semantic connections between rules and how rules work together to arrive at a result. The goals for such visualizations are the same as for UML class diagrams and other overview representations of programs and models: to aid teaching. To address this shortcoming we have developed and implemented the Explorative Debugging paradigm for rule-based systems [23]. Explorative Debugging works on the declarative semantics of the program and lets the user navigate and explore the inference process. how it interacts with other parts of the rule base and what role the different rule parts play. show which other parts are affected by a change. An Explorative Debugger is not only a tool to identify the cause of an identified problem but also a tool to try out a program and learn about its working. debugging. for example. detailed descriptions can be found in [22].152 V. independent of the answer to any particular query. 5 Logical connections between rules are static dependencies formed for example by the possibility to unify a body atom of one rule with the head atom of another. It supports the navigation along these connections5. 5. took a long time to be calculated. An explorative debugger is a rule browser that: – Shows the inferences a rule enables. To address this we created a scalable overview representation of the static and dynamic structure of a rule base. To. It was established that debuggers needs to be declarative in order to realize the full potential of declarative programming languages.3 Visualization The visualization of rule bases here means the use of graphic design techniques to display the overall structure of the entire rule base.

Journal of the ACM 42. Such a complete support for the development of rule bases is an important prerequisite for Semantic Web rules to become a reality and for business rule systems to reach their full potential.Tackling the Debugging Challenge of Rule Based Systems 153 – It allows to further explore the inference a rule enables by digging down into the rule parts. MIT Sloan Management Review.R. Larman. V. Lausen. J.. R..) RuleML 2005. In: Database Semantics: Semantic Issues in Multimedia Systems. Fensel. pp. visibility and modularization can guide the instantiation of this architecture in concrete tools.: Estimating the numbers of end users and end user programmers. declarativity. Her Majesty’s Stationary Office. Heidelberg (2005) 5.. C. Basili. 207–214 (2005) 9. Heidelberg (2008) 2. 351–369 (1999) 4. S. In: Bassiliades... G.. vol.a survey of developers. Governatori.. pp. pp.: Development and verification of rule based systems . References 1. LNCS. B..: Ontobroker: Ontology-based access to distributed and semi-structured unformation. B. 47–56 (June 2003) 7.: Product-development practices that work. 741–843 (1995) 3. London. Boley. vol. Studer. M. M. visualization and anomaly detection heuristics can help tackle this challenge. N... A. Shaw. V. Springer. Stoutenburg. S.) Rule ML 2008. Tabet. In: L/HCC 2005: Proceedings of the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing. Fensel. D. J. IEEE Computer. In: Proceedings of the Teddington Conference on the Mechanization of Thought Processes.. Scaffidi. Paschke. pp. Kifer.: Software Cost Estimation with COCOMO II. M.. Boehm.. Decker.: Logical foundations of object-oriented and frame-based languages. Zacharias. 75–84 (2001) 8. S. pp. (eds. D. Prentice Hall PTR. Kifer. Wu. 17–29.. LNCS. Springer..: Iterative and incremental development: A brief history. G. A. McCarthy. 6–16. MacCormack. 3791. H.: Programs with common sense. – Is integrated into the development environment and can be started quickly to try out a rule as it is formulated. The principles of interactivity. Englewood Cliffs (2000) . 5321. de Bruijn. J. The interested reader finds a complete description of the explorative debugging paradigm and one of its implementation in [23].: A realistic architecture for the semantic web. Erdmann. In: Adi. 6 Conclusions Current tools support for the development of rule based knowledge based systems fails to address many common problems encountered during this process. 75–91 (1959) 6. C. The novel paradigm of explorative debugging together with techniques from the automatic debugging community can form the robust basis for debugging in this context. A. debugging and testing supported by test coverage metrics. An architecture that combines integrated development. (eds. M. Myers.

Shneiderman. 202–212 (1999) 21. Vignollet. Rosson..: Evaluation of verification tools for knowledge-based systems. Froscher.: The dangers of end-user programming. Harrison. Zhang. L.: Revealing the structure of rule based systems. Stud. Burnett. (1991) 18. Phalgune.: Verification and validation of knowledge-based systems.: Explorative debugging for rapid rule base development. J. Nash..: Validation and verification of knowledge-based systems: a survey. 47...: Foundation and application of knowledge base verification. Ruthruff.. Int. Beckwith. V.. Final Rep.B. U.: Rewarding good behavior: End-user debugging and rewards. Preece.. In: Proceedings of the 2004 IEEE Symposium on Visual Languages and Human Centric Computing (2004) 11. In: 1st Workshop on End-User Software Engineering (WEUSE 2005) at ICSE 2005 (2005) 13. International Journal of Expert Systems (1994) 17. T. J. M. Journal of Applied Intelligence. V.: Visualization of rule bases . Tsai.: A software engineering methodology for rule-based systems. IEEE Transactions on Knowledge and Data Engineering 11. L. Simon. Burnett. International Journal of Intelligent Systems 9.the overall structure. Ruthruff. Baroff.. J. J.. A. Gokulchander. A. IEEE Transactions on Knowledge and Data Engineering 2. NASA Contract NAS1-18585.D. B. Abecker. R. 173–189 (1990) 15. J. Zacharias. In: VL/HCC 2004: IEEE Symposium on Visual Languages and Human-Centric Computing (2004) 14.. 5–7 (July/August 2004) 12.. Preece. D.-Comput. 683–701 (1994) 22. Mehrotra. Grossner. 629–658 (1997) 19. Hum. Balling. S.. In: Proceedings of the 3rd Workshop on Scripting for the Semantic Web at the ESWC 2007 (2007) .: Rule groupings: a software engineering approach towards verification of expert systems. Shinghal. H. R.T. Gupta.. IEEE Software.: Six challenges in supporting end-user debugging.. Zacharias 10. R. pp. W. M. W. Vishnuvajjala. In: Proceedings of the 7th International Conference on Knowledge Management ... Talbot. Gilman. C.: Everyday programming: Challenges and opportunities for informal web development. A. F. R..: Direct manipulation user interfaces for expert systems. J.. Preece. A. 99–125 (1987) 16. Zacharias.Special Track on Knowledge Visualization and Knowledge Discovery (2007) 23. A.154 V. M. P. Radhakrishnan. 343–363 (1993) 20. Technical report. Jacob. M..