General approximation for the size of the search space tree

In order to provide a general approximation for the time complexity of a given search plan (i.e., the size of the state space being traversed during pattern matching), we may apply the cost estimation of Sec. 7.3.1. Ifp,r,nandldenote the number ofProcesses,Resources,nextedges andreleaseedges in the model, respectively, then the search plan of Fig. 4.4(b) expectedly hitsr+r_r^l + (r_r^l)ⁿ_p matching combinations as (i) there arer different choices for fixing pattern nodeR, (ii) on average _r^l release

edges lead out of each resource resulting inr_r^l for the approximation on the number of possible partial matchings, which already bind pattern nodesRandP1, and (iii) for each such partial matching there are on averageⁿ_p extension possibilities by navigating alongnextedges. As a result of a similar calculation process, the search plan of Fig. 4.4(d) expectedly traversesp+pⁿ_p + (pⁿ_p)_p^l combinations.

The above formulae can be simplified tor+l+^ln_p andp+n+^ln_p, respectively. By comparing these expressions, it can be determined which search plan is better for a model, which containspprocesses, rresources,nnextedges, andlreleaseedges.

4.3 Conclusion

In this chapter, a general purpose, graph pattern matching algorithm has been presented first, in which mapping candidate calculation and matching check tasks could be customized providing a framework for analyzing both the theoretical and practical complexity of all existing and upcoming heuristics.

Then, the technique of search plan driven pattern matching has been discussed by defining the concepts of search graphs and search plans, by extending its heuristic description facilities towards constraint satisfaction based algorithms, by formally specifying the exact pattern matching process, and by pre-senting a Java implementation for the technique.

CHAPTER

5

Benchmarking Framework for Graph Transformation

In this chapter, I present a benchmarking framework for graph transformation tools to be able to quan-titatively assess, compare and analyze the run-time performance of (i) already existing approaches, (ii) their optimization strategies, and (iii) also all the techniques and algorithms that are going to be presented in later chapters.

5.1 Motivation for benchmarking

The aim of benchmarking is to systematically measure the performance of a system under different and precisely defined circumstances (i.e., by using several parameter combinations and data sets for these measurements). Such measurements help system engineers in decision making, i.e., when a choice has to be made between different alternatives by providing a proper assessment on the system characteris-tics. Although the graph transformation community has several specification examples for determining the expressiveness of approaches, it lackssystematic benchmarks for measuring the performance of different tools.

Related work

A good theoretical overview on software engineering benchmarks is provided by [123], which presents the preconditions and the main consequences of a successful benchmarking process by using a case study of the reverse engineering community.

There is a large variety of benchmarks and facilities supporting experiment design in different fields of computer engineering.

• Artificial intelligence. In case of agent-based systems, the Common Lisp Analytical Statistics Package (CLASP) [28] provides a formal, model-based methodology with powerful statistical techniques for justifying the decisions made during the design of the agent architecture.

• Relational databases. TPC-C [137] is a benchmark issued by the Transaction Processing Per-formance Council for measuring the perPer-formance of on-line transaction processing (OTLP) sys-tems. Its current version comprises of a mix of five concurrent transactions and nine types of tables with a wide range of record and population sizes.

51

52 CHAPTER 5. BENCHMARKING FRAMEWORK FOR GRAPH TRANSFORMATION

• Rule-based expert systems. [20] presents a collection of test suites that can be used for as-sessing the performance of rule-based expert systems. Among these benchmarks, Manners [74]

describes a depth-first search solution to the problem of seating arrangement for guests at a din-ner party, Waltz is a diagram labeling problem, which gives a 3-dimensional interpretation of 2-dimensional lines, the Aeronautical Route Planner searches for the lowest cost route between two points, and Weaver [68] is a VLSI channel and box routing algorithm.

Several specification examples (mappings such as UML-to-XMI in QVT [109], object-relational [109], UML-to-EJB [78], UML-to-XSD [24]) exist for graph transformation approaches, but their main goal is to demonstrate theexpressivenessof the given approach, and they omit the performance aspects of graph transformation tools.

Objectives

Thus, I propose a systematic method for quantitative benchmarking in order to evaluate the performance of graph transformation tools. Typical features of the graph transformation paradigm and various opti-mization strategies exploited in different tools are identified and categorized. Moreover, the speed-up effects of these optimization strategies are measured and compared.

Since MOF metamodels and models are frequently formalized as graphs and graph transformation is a popular technique for capturing model transformations as indicated by a large variety of tools pre-sented in Sec. 3.3, conclusions being drawn from experiments on GT tools are expected to be valid for model transformation tools as well. This statement is validated by [116], which proposes graph trans-formation to be used for defining the semantics of QVT. Further considerations about the applicability and the limitations of the benchmarking framework can be found in Section 5.6.

My objectives in the current chapter are the following.

• I define terms and concepts for graph transformation specific benchmarking in Section 5.2.1.

• I determine the most common features of graph transformation problems and tools in Sec-tions 5.2.2 and 5.2.3. Based on tool-specific properties I identify various optimization strategies that are used in several graph transformation tools.

• I design benchmarks for different application scenarios for model transformations. In this sense, I propose a benchmark for simulating the dynamic behaviour of a system defined in a visual language in Section 5.3. In addition, I specify a benchmark for a model refactoring scenario in Section 5.4.

• In Section 5.5, measurements are executed on the benchmark of Sec. 5.3 by using carefully selected parameter settings and combinations of optimization strategies, and measurement results are presented. Then I shortly analyze the effects of optimization methods.

• Section 5.6 concludes this chapter with summarizing its theoretical and practical relevance.