INVESTIGATING TEXT-BASED DOMAIN-SPECIFIC MODELING
TECHNIQUES
SZ ¨ OVEG ALAP ´ U SZAKTER ¨ ULETI MODELLEZ´ ESI M ´ ODSZEREK
VIZSG ´ ALATA
Ph.D. Dissertation
Ferenc Attila Somogyi
Advisor:
Gergely Mezei, Ph.D.
Associate Professor
Budapest, 2022
Models have been present in software development for a long time. The underlying structure of a model is usually represented in a graph-based form. Using the text- based form of models is less frequent, with most approaches focusing on the visual, graph-based representation. There are many research areas in the field of model- based software development such as defining, analyzing, or unifying concepts used by the many different approaches, managing the versioning of graph-based models, supporting model-based methodologies with text-based approaches, or processing models via model (graph) transformations. Furthermore, this research field is con- stantly evolving, with new ideas and methodologies often iterating on older ones.
One such example is multi-level modeling, an emerging modeling paradigm that aims to reduce or even eliminate accidental complexity that is the usual by-product of conventional model design. Accidental complexity refers to parts of the solution that are required to express the modeled (or multi-level) nature of the solution, instead of focusing on the domain itself.
This work focuses on investigating text-based domain-specific modeling tech- niques. The first main contribution of this work is the proposal of a text-based model differencing and merging algorithm. The algorithm is aimed to be used dur- ing the version control of models in practice. The algorithm is analyzed based on several aspects: i) soundness, ii) completeness, iii) generality, and iv) time complex- ity. To reinforce the motivations and main ideas behind the algorithm, an in-depth Systematic Literature Review (SLR) is also presented, focusing on the categoriza- tion of model matching approaches and on open questions in the field of model differencing and merging. The SLR was conducted following rigorous, scientifically proven and repeatable methodologies.
The second main contribution revolves around the goal of supporting scientific experimentation in the field of multi-level modeling. A formalism for specifying the structure and semantics of multi-level modeling constructs is proposed, showing how the definition of multi-level concepts based on different theoretical foundations can be described in a unified way. A textual language - based on the underly- ing formalism - with the goal of supporting the concise definition of multi-level modeling constructs is also proposed. Moreover, the practical application of the re- sults is demonstrated: the Multi-Level Modeling Playground (MLMP), a scientific playground where researchers can experiment with ideas in multi-level modeling is presented, based on the previously mentioned formalism. The Dynamic Multi-Layer Algebra (DMLA) is used as the foundation of MLMP. DMLA is a multi-layer mod- eling framework that is actively being worked on by our research group. MLMP is also evaluated in practice, highlighting its advantages and disadvantages compared to other approaches.
A modellek r´eg´ota jelen vannak a szoftverfejleszt´esben. A modellek strukt´ur´aj´at
´
altal´aban valamilyen gr´af form´aban ´abr´azolj´ak. A modellek sz¨oveges form´aj´anak haszn´alata ritk´abb, a legt¨obb m´odszer a vizu´alis, gr´af alap´u ´abr´azol´asra ¨osszpontos´ıt.
A modell alap´u szoftverfejleszt´es ter¨ulet´en sz´amos tov´abbi kutat´asi ter¨ulet l´etezik, p´eld´aul a sokf´ele megk¨ozel´ıt´es ´altal haszn´alt fogalmak meghat´aroz´asa, elemz´ese vagy egys´eges´ıt´ese, gr´af alap´u modellek verzi´okezel´es´enek menedzsel´ese, modell alap´u m´odszertanok sz¨oveges t´amogat´asa, vagy modellek feldolgoz´asa modell (gr´af) transzform´aci´ok r´ev´en. Ez a kutat´asi ter¨ulet folyamatosan fejl˝odik, az ´ujabb
¨
otletek ´es m´odszertanok gyakran ´ep¨ulnek a kor´abbiakra. Ilyen p´eld´aul a t¨obbszint˝u modellez´es, egy kialakul´oban l´ev˝o modellez´esi paradigma, amelynek c´elja az ´un.
”accidental complexity” cs¨okkent´ese vagy kik¨usz¨ob¨ol´ese, amely a hagyom´anyos modelltervez´es szok´asos mell´ekterm´eke. Az ”accidental complexity” a megold´as azon r´eszeire vonatkozik, amelyek annak modellezett jelleg´enek kifejez´es´ehez sz¨uks´egesek, ´es nem azokra, amik mag´at a c´el domaint k´epzik le.
A disszert´aci´om a sz¨oveg alap´u domain-specifikus modellez´es vizsg´alat´at helyezi f´okuszba. Az els˝o kontrib´uci´o egy sz¨oveg alap´u modell ¨osszehasonl´ıt´o
´
es ¨osszef´es¨ul˝o algoritmus, ami a gyakorlatban modellek verzi´okezel´es´eben alkalmazhat´o. Az algoritmust t¨obb szempont alapj´an elemzem: i) helyess´eg, ii) teljess´eg, iii) ´altal´anoss´ag ´es iv) id˝obeli komplexit´as. Az algoritmus m¨og¨ott megh´uz´od´o motiv´aci´ok ´es f˝obb gondolatok meger˝os´ıt´ese ´erdek´eben egy m´elyrehat´o Systematic Literature Review (SLR) is bemutat´asra ker¨ul, amely a modellilleszt´esi megk¨ozel´ıt´esek kategoriz´al´as´ara, valamint a modellek
¨
osszehasonl´ıt´asa ´es ¨osszef´es¨ul´ese ter´en l´ev˝o nyitott k´erd´esekre f´okusz´al. Az SLR szigor´u, tudom´anyosan megalapozott ´es megism´etelhet˝o m´odszerek alapj´an lett elv´egezve.
A disszert´aci´o m´asodik f˝o kontrib´uci´oja a t¨obbszint˝u modellez´es ter¨ulet´en v´egzett tudom´anyos k´ıs´erletez´es t´amogat´as´ara ir´anyul. Egy olyan formalizmust javaslok, ami a t¨obbszint˝u konstrukci´ok szerkezet´enek ´es szemantik´aj´anak megad´as´ara alkalmas. A formalizmus seg´ıts´eg´evel egys´egesen specifik´alhat´ok a k¨ul¨onb¨oz˝o elm´eleti ´es gyakorlati alapokon nyugv´o konstrukci´ok. Egy sz¨oveges nyelv is bemutat´asra ker¨ul, amely a m¨og¨ottes formalizmuson alapul. A nyelv c´elja, hogy t´amogassa a t¨obbszint˝u modellez´esi konstrukci´ok min´el egyszer˝ubb ´es t¨om¨orebb le´ır´as´at. Az eredm´enyek gyakorlati alkalmaz´as´anak ismertet´esek´ent bemutat´asra ker¨ul a Multi-Level Modeling Playground (MLMP), egy tudom´anyos k´ıs´erletez˝o keretrendszer, ami a kor´abban javasolt formalizmusra ´es nyelvre ´ep¨ul. Itt a kutat´ok t¨obbszint˝u modellez´esi ¨otleteikkel k´ıs´erletezhetnek. A Dynamic Multi-Layer Algebra (DMLA) az MLMP alapja. A DMLA egy t¨obbr´eteg˝u modellez´esi keretrendszer, amelyen kutat´ocsoportunk akt´ıvan dolgozik. Az MLMP-t a gyakorlatban is ki´ert´ekelem, kiemelve annak el˝onyeit ´es h´atr´anyait m´as megk¨ozel´ıt´esekhez k´epest.
Declaration
The content of this dissertation is a product of the author’s original work except where explicitly stated otherwise.
Nyilatkozat
Alul´ırott Somogyi Ferenc kijelentem, hogy ezt a doktori ´ertekez´est magam k´esz´ıtettem, ´es abban csak a megadott forr´asokat haszn´altam fel. Minden olyan r´eszt, amelyet sz´o szerint, vagy azonos tartalomban, de ´atfogalmazva m´as forr´asb´ol
´
atvettem, egy´ertelm˝uen, a forr´as megad´as´aval megjel¨oltem.
Budapest, 2022. 03. 03. ...
Vannak, akik n´elk¨ul ez a dolgozat nem, vagy csak nagyon nehezen k´esz¨ult volna el.
Els˝ok´ent kor´abbi t´emavezet˝omet, Asztalos M´arkot emeln´em ki, aki egyengette az utamat a PhD k´epz´es sor´an. Biztat´asai a nehezebb id˝oszakokban sokat seg´ıtettek a dolgozat l´etrej¨ott´eben.
M´asodszor k¨ul¨on kiemeln´em Mezei Gergelyt, a jelenlegi t´emavezet˝omet. Szakmai
´
es nem szakmai k´erd´esekkel egyar´ant mindig fordulhattam hozz´a, mindig nagyon k´eszs´egesen seg´ıtett, b´armi is volt a probl´ema. Az ˝o vezet´ese ´es ir´anymutat´asa n´elk¨ul a t¨obbszint˝u metamodellez´essel kapcsolatos kutat´asi eredm´enyeim nem j¨ottek volna l´etre.
Szeretn´em tov´abb´a megk¨osz¨onni a DMLA kutat´ocsoport tov´abbi tagjainak is, akik szint´en fontos szerepet j´atszottak a dolgozat m´asodik fel´eben taglalt kutat´omunk´am elk´esz¨ult´eben. Theisz Zolt´annak els˝osorban az MLMP kezdeti
¨
otlet´enek felvet´es´e´ert ´es az azzal kapcsolatos visszajelz´esei´ert. B´acsi S´andornak els˝osorban a cikk´ır´asban ny´ujtott seg´ıts´eg´e´ert ´es az MLMP-vel kapcsolatos visszajelz´esei´ert. Palatinszky D´anielnek els˝osorban az´ert, hogy mindig sz´ıvesen seg´ıtett, amikor valamilyen technikai probl´em´aba ¨utk¨oztem.
K¨osz¨onettel tartozok m´eg Vajk Istv´annak ´es Lengyel L´aszl´onak a dolgozat
´
atn´ez´es´e´ert, ´es az ez alapj´an adott hasznos visszajelz´eseik´ert. Vajk Istv´annak k¨ul¨on k¨osz¨on¨om, hogy visszajelz´eseivel seg´ıtett a t´ezisek pontos kimond´as´aban, valamint a t´ezisf¨uzet elk´esz´ıt´es´eben is.
1 Introduction 1
1.1 Statement of Contribution . . . 2
1.2 Outline . . . 4
2 Model Differencing and Merging 5 2.1 Background and Related Work . . . 5
2.2 Systematic Literature Review . . . 11
2.2.1 Planning the Survey . . . 11
2.2.2 Conducting the Survey . . . 17
2.2.3 Results and discussion . . . 19
2.3 A Novel Text-Based Model Differencing and Merging Algorithm . . . 24
2.3.1 Overview and Requirements . . . 24
2.3.2 AST Matching Phase . . . 27
2.3.3 Conflict Detection Phase . . . 29
2.3.4 Merge Phase . . . 33
2.4 Analysis of the Algorithm . . . 38
2.4.1 Soundness and Completeness . . . 39
2.4.2 Generality and Configuration Cost . . . 41
2.4.3 Time Complexity . . . 45
3 Playground for Multi-Level Modeling 49 3.1 Background and Related Work . . . 49
3.1.1 Introduction to Multi-Level Modeling . . . 49
3.1.2 Multi-Level Modeling Approaches . . . 53
3.1.3 The Dynamic Multi-Layer Algebra (DMLA) . . . 56
3.1.4 Abstract State Machines . . . 60
3.2 A Novel Approach for Experimenting with Multi-Level Constructs . . 62
3.2.1 Formal Specification of Multi-Level Constructs . . . 63
3.2.2 Requirements of Selected Constructs . . . 73
3.2.3 Mapping of Selected Constructs . . . 75
3.3 Language to Specify Multi-Level Constructs . . . 83
3.3.1 Language Proposal and Expressive Power . . . 84
3.3.2 Construct Specification with MLSL . . . 89
3.4 Multi-Level Modeling Playground (MLMP) . . . 91
3.4.1 Requirements . . . 91
3.4.2 Architecture . . . 92
3.4.3 Modules and domain models . . . 93
3.4.4 The underlying platform of MLMP . . . 96
3.5 Practical Evaluation of MLMP . . . 97
3.5.1 Eclipse Modeling Framework . . . 98
3.5.2 Level-adjuvant multi-level solutions . . . 104
3.5.3 Summary and Results . . . 107
4 Conclusions 109 5 Appendix 112 A.1 Domain Model (Formalism) . . . 112
A.1.1 NamedNode . . . 113
A.1.2 ProductType . . . 113
A.1.3 Book . . . 114
A.1.4 DVD . . . 114
A.1.5 MobyDick Book . . . 115
A.1.6 MobyDick DVD . . . 116
A.1.7 Inheritance . . . 116
A.1.8 BasedOn at Level 1 . . . 117
A.1.9 BasedOn at Level 0 . . . 117
A.2 Classic Potency - Semantics (MLSL) . . . 118
A.3 Domain Model (MLSL) . . . 120
1.1 A two-level modeling example (inspired by related work [KS07]). . . . 3
2.1 An overview of text-based modeling. . . 5
2.2 Model matching illustrated on a simple example. . . 6
2.3 Change tracking illustrated on the previous example. . . 9
2.4 Overview of the study selection process. . . 16
2.5 Screening process for a study. . . 17
2.6 Summary of the focus of graph-based algorithms. . . 20
2.7 Summary of matching techniques used by algorithms. . . 20
2.8 Correlations between matching type and the focus of the algorithm. . 21
2.9 The main focus of operation-based and state-based algorithms. . . 22
2.10 Number of included primary studies per year. . . 23
2.11 Building an AST from the text. . . 24
2.12 Differencing text and AST-s. . . 25
2.13 Overview of the algorithm. . . 26
2.14 Matching AST-s - an example. . . 27
2.15 Different Text Conflict (DTC) - example. . . 30
2.16 New tree conflict (NTC) - example. . . 31
2.17 Move Conflict (MC) recognition - example. . . 31
2.18 Merging AST-s - example. . . 34
2.19 Merging textual representations - example. . . 34
2.20 Different interactions that can occur between subtrees. . . 36
2.21 Determining whether an interaction affects a subtree. . . 37
2.22 An MDM example based on VMTS models. . . 43
2.23 EMF-based example of context-sensitive unique identifiers. . . 44
2.24 Separating subtrees into disjoint sets. . . 45
3.1 The architecture of the Meta-Object Facility (MOF). . . 50
3.2 A multi-level model (inspired by related work [KS07]). . . 51
3.3 The Orthogonal Classification Architecture (OCA). . . 52
3.4 The main building blocks of DMLA. . . 57
3.5 The entities of the Bootstrap. . . 58
3.6 Level L2 of the example from Figure 3.2. . . 82
3.7 Requirements of a multi-level modeling playground. . . 91
3.8 The architecture of DMLA injected with multi-level concepts. . . 92
3.9 The modular architecture of MLMP. . . 93
3.10 The structural mapping of classic potency in EMF. . . 98
3.11 An interface-based solution for handling annotation-like concepts. . . 102
3.12 An alternative solution for handling annotation-like concepts. . . 103
3.15 An alternative definition of Inheritance in Melanee. . . 106
A.1 The multi-level model from Figure 3.2. . . 112
List of Tables
2.1 The PICOC criteria used in the survey. . . 132.2 Summary of the study selection process. . . 18
2.3 Extracted data structure with possible values. . . 19
3.1 Built-in structural meta types in MLSL. . . 95
Introduction
Domain-specific modeling aims to model a problem domain (e.g., a factory product line or an insurance process) as concisely and accurately as possible [KT07]. Us- ing methodologies such as Model-Driven Engineering [Sch06,SVC06] in the software development process can boost productivity, increase the comprehension of the prob- lem domain, and even foster understanding between developers and domain experts or stakeholders [MA07]. Models can also be used for other purposes like formal analysis, design space exploration, requirements elicitation, or source code gener- ation [KT07]. Model transformation is the automated process of transforming a source model to a target model based on different rules. Model transformations are at the core of model-driven methodologies as they are crucial in a lot of modeling tasks [BDH+15,CH06,SK03]. Even model management operations can be described using model transformations [SZK04].
Most model-driven methodologies, including Model-Driven Engineering (MDE) or the Model-Driven Architecture (MDA) of the Object Management Group (OMG) [Fra02a] use graph-based models as the main artifacts during software development.
The structure of a model is usually represented as a graph [TELW10], since this form makes it easier to visualize the model, which is a very prevalent approach to modeling [BG05]. Text-based approaches in modeling are often only delegated to constraint checking (using a language like the Object Constraint Language [WK03]) or gener- ating source code. However, the models themselves can also be described using a textual notation, which comes with its own advantages and disadvantages [GKR+14].
Many argue that using the textual and graphical notations in conjunction is the ideal solution, as we can keep the advantages of both [GKR+14, PAdLG08].
Model-driven methodologies are slowly becoming more prevalent in the industry, although problems like model evolution management and model versioning impede their further spread [ABK+09, BE08]. Model evolution management is a difficult task and is an active research field today [PMR16]. Version Control Systems [Bab86, Spi05] can also support model evolution. When used properly, they can greatly improve the efficiency of traditional software development [BA02]. Applying them to model-driven methodologies can also boost efficiency. However, differencing and merging graph-based models is inherently different from text-based source code, thus, they require different treatment during version control [ASW09, BKL+12].
Managing the version control of models in their textual form provides an alternative that comes with its own challenges, advantages, and drawbacks [Pub10].
Multi-level modeling [AK01] is a modeling paradigm aiming to solve the short- 1
comings of classic (meta-)modeling approaches such as OMG’s four-level Meta- Object Facility (MOF) [MOF05]. Multi-level modeling allows for an unlimited number of instantiation levels as opposed to the four levels prescribed by OMG.
One of the main goals of multi-level modeling is to reduce accidental complexity, which refers to parts of the solution needed only to express its modeled (multi-level) nature, instead of describing the domain in question [AK08]. A good example of this is the application of the Item Description [Coa92] pattern to describe multiple do- main levels in object-oriented languages. Compared to classic modeling approaches, multi-level models are often reduced in size, are more compact, and describe the target domain more accurately. Therefore, many problem domains can benefit sig- nificantly from using multi-level modeling.
One of the most prevalent enablers of multi-level modeling are the notions of clabject and potency [AK01]. They were introduced to handle challenges caused by instantiation spanning multiple levels (often referred to as deep instantiation). Over the years, several variations of the potency notion emerged, each of them adding to, modifying, or improving upon the original concept. While there have been recent advances made to unify multi-level modeling concepts [Fra18, K¨18], there are still many different concepts behind multi-level modeling that are more or less related to each other. For example, some approaches do not rely on the potency notion at all [LWM+13, FAGC18,CA18, UMT17], while others use one or more of its variations [Ger11,AG16, dLG10, NSJS18, RdLG+12,IGSS18,MRS+18]. However, it is often difficult to compare or evaluate these approaches since they are usually based on different foundations, formalism, principles, or even toolchains.
As a demonstration of accidental complexity, let us consider the two-level model described in Figure 1.1. We are modeling products (with a price) and product types (with a tax rate). Since we only have two levels, the isOfType association betweenProduct andProductType is a case of accidental complexity, because instead of expressing that a product is a product type, we use a workaround solution to express this. Thus, theProduct class represents accidental complexity in the model.
Moreover, if we want to introduce abasedOn relationship between books and DVD-s, then theBook andDVD classes have to be in the meta-model instead of the instance model. This could be done via inheritance (fromProductType), but then the instance model would contain instances of Book and DVD, which is again redundant. Also, arguably, the meta-model should be domain-independent, although this constraint is arbitrary. A multi-level solution is presented later in Figure 3.2.
The goal of this work is to investigate and provide efficient - and if possible, text-based - solutions to the aforementioned problems related to domain-specific modeling. More specifically, the main problems this work focuses on are i) text- based model differencing and merging, and ii) specifying and experimenting with multi-level constructs in a unified way.
1.1 Statement of Contribution
This work focuses on two main research fields. The first is the field of Model Differ- encing and Merging (MDM), more specifically, focusing on text-based approaches and the problem of generality and accuracy. Regarding MDM, this work makes the following main contributions:
• A summarized background on the field of MDM is provided, drawing from
Figure 1.1: A two-level modeling example (inspired by related work [KS07]).
existing work. Key concepts like the generality and accuracy of an MDM algorithm are also identified and defined.
• The results of a Systematic Literature Review (SLR) are presented. The SLR was conducted according to rigid and repeatable scientific methodologies. The SLR was aimed at discovering the state of MDM approaches regarding aspects like generality, accuracy, or whether the algorithm is text-based or graph- based. The survey was also carried out to discover the main open questions in this research field, for which the results are also presented.
• A novel MDM algorithm that is text-based and focuses on generality while also maintaining accuracy is proposed. This comes at a configuration cost, which is not unusual in the literature, however, text-based algorithms of this nature are much more rare.
• The proposed algorithm is analysed based on several aspects: soundness, com- pleteness, generality (configuration cost), and time complexity in order to as- certain its feasibility.
The second focused research field is Multi-Level Modeling, on how different multi- level constructs can be described and experimented with in a unified way. Regarding multi-level modeling, this work makes the following main contributions:
• A comprehensive introduction to multi-level modeling is presented by dis- cussing the most prevalent concepts in this research field. Some of the most prominent approaches to multi-level modeling are also briefly discussed.
• A formalism that can be used to experiment with multi-level constructs relying
on different formalism and underlying theory is introduced. Using the formal- ism, it is possible to specify both the structure and semantics of a multi-level construct. The formalism is aimed to be the foundation of a general multi-level modeling playground, where researchers can experiment with existing and new concepts.
• In order to prove the feasibility of the approach, several existing multi-level constructs from the literature are specified, along with a related domain model.
• A formal language is proposed that can be used to specify multi-level concepts in a compact way, based on the proposed formalism. The language is devised with practical application in mind, so that researchers can easily use it to experiment with different concepts and ideas.
• The expressive power of the language is shown to be similar to that of the formalism. Moreover, the previous multi-level constructs and domain model that were specified using the formalism are specified again, using the language.
• A practical application of the results is presented in the form of the Multi-Level Modeling Playground (MLMP), a general playground capable of specifying and experimenting with multi-level constructs, built on the aforementioned formalism. MLMP is also evaluated in practice; its capabilities are compared to similar solutions that could be used as a basis of a playground.
1.2 Outline
Note that due to the distinctly different nature of the two main topics, there is no single chapter delegated to the background of this research. Instead, there are sepa- rate background sections (also discussing the mathematical concepts and formalism used) for the two main topics in order to better focus the discussion. Moreover, the conducted Systematic Literature Review (SLR) in the field of MDM also serves as an extension of the background, in addition to being a part of the contribution.
Thus, the rest of this work is organized as follows:
• Chapter 2 first introduces the motivation behind and the main concepts of MDM, along with related work. Then, it discusses the results related to Thesis I, starting with the Systematic Literature Review. Next, a novel, text-based MDM algorithm is proposed. The main phases of the algorithm are introduced, discussed and analyzed separately. Analyses include completeness, soundness, generality, and time complexity.
• Chapter 3 starts with a background on multi-level modeling, which - similarly to the previous chapter - introduces the motivations, main concepts and the most important related work in this research field. Afterwards, the results related to Thesis II are presented, proposing a novel formalism to describe and experiment with multi-level modeling constructs. A textual language is also proposed in order to better express the concepts of the formalism. Sev- eral existing multi-level constructs from the literature are described using this approach in order to prove its feasibility. Based on the theoretical results, the Multi-Level Modeling Playground (MLMP), a general playground for exper- imenting with multi-level constructs, is also introduced. MLMP is evaluated in practice via a comparison to other approaches.
• Chapter 4 summarizes this work and outlines future work. It also contains the exact Thesis Structure of this work.
Model Differencing and Merging
In this Chapter, we first discuss the background on model differencing and merging (MDM), introducing the main concepts and definitions in this research field. Next, a Systematic Literature Review (SLR) that we have conducted is presented, high- lighting some of the major challenges in this field in addition to establishing our motivation for developing a text-based MDM algorithm. The SLR also serves as an extended discussion of related work. Afterwards, we present a novel text-based MDM algorithm that aims to be as general as possible while maintaining accuracy.
Finally, we analyze the algorithm based on several aspects: soundness, completeness, generality, and time complexity.
2.1 Background and Related Work
Figure 2.1: An overview of text-based modeling.
Differencing and merging models are important processes in model-based ver- sion control [ABK+09,CW98,Men02,LZG04]. Model matching is usually the first step of model differencing and merging. The goal of model matching is to identify pairs of model elements in the input models that are considered to be matching, based on various considerations. There are several approaches to model matching, with most of them focusing on different goals like accuracy or generality. As the next step in the MDM process, differences between models can be established using the matched pairs. Most matching approaches work directly with the graph-based
5
structure of the models. However, there exist solutions that approach the problem differently, for example, by working with the textual representations of the mod- els. The textual notation can be mapped to the model in many ways, for example, by using a formal language and a parser [ASU86] or by specifying it via a textual language like Alloy [Jac02]. Describing models with a textual notation can have certain advantages over using the graphical notation, including i) the off-line editing of models, ii) better focus on details and thus easier editing - especially in the case of large models, iii) and using a textual notation can offer a more comprehensible serialization compared to XML-based approaches [GKR+14,HKC11,Pet95, SR01].
Disadvantages compared to the graphical approach include i) no clear overview of the model, ii) worse readability, and iii) the lack of simulation support.
The textual representation of a model can be described in multiple ways. A common approach is using a formal language [ASU86]. The language essentially describes the mapping between the graph-based model and the text via the model- to-text (M2T) and text-to-model (T2M) transformations. In practice, this requires more effort to realize if the modeling environment does not already support it.
However, this can be worthwhile long-term as there are similar approaches to text- based modeling with industrial applications [EB10]. Figure 2.1 depicts an overview of a process that from now on we refer to as text-based modeling. Some argue that using the graphical and textual approaches in conjunction is the ideal solution as we can keep the advantages of both [GKR+14,PAdLG08]. Doing so however, introduces the problem of keeping the two notations synchronized with each other. There is existing research on the synchronization problem [ALC08, vRWS+13], however, it is out-of-scope for this work.
Figure 2.2: Model matching illustrated on a simple example.
To demonstrate the problem of model matching, Figure 2.2 contains a simple example. Here, we try to match modelsModel 1 andModel 2 using a model match- ing algorithm. Let us assume that the algorithm matches the following model el- ements with their named counterparts: Node A, Node C, Edge BA, Node B, and field B2. The output of the matching algorithm is the list of pairs containing the aforementioned model elements from the two input models. Let us assume that the algorithm achieved these results by matching nodes and fields based on their
names, and matching edges based on their names, types, and the nodes they con- nect. We can see even in this simple example that model matching is not a trivial task. For example, a different matching algorithm might have matched Edge CA with its counterpart on the basis that the type of the edge is not as important as the nodes it connects. Let us assume that the correct decision in the case of Edge CA in modeling language L1 would be to ignore the match, but in modeling language L2, the correct decision would be to perform the match. If the matching algorithm focuses on generality, then its goal is to work with both modeling languages. In this case, the algorithm might not always achieve the correct result with a particular language. If the algorithm focuses on accuracy, then it aims to work with either L1
or L2. If the algorithm also aims for accuracy in addition to generality, then this could be achieved by making the algorithm configurable in order to get the correct result for multiple languages. This of course introduces some configuration cost.
Definition 2.1.1. ModelM is graph-based and consists of a set of model elements.
A model element me that is a part of model M is notated as follows: me ∈M. A model element can be a node, edge, or field. The underlying formalism (graph rep- resentation) of the model can be arbitrary, along with the notation used to describe it.
Definition 2.1.2. Given models M1 and M2, a matching pair is a pair of model elements (m1,m2) found matching by an MDM algorithm based on arbitrary criteria, wherem1 ∈M1andm2 ∈M2. Theset of matching pairsM P between modelsM1 and M2 consists of every matching pair discovered by an MDM algorithm. The only restriction is that for ∀m1 ∈M1 :@(m1, mx)∈M P ∧(m1, my)∈M P :mx 6=my. Definition 2.1.3. The input of a model differencing and merging (MDM) algorithm consists of models M1 and M2 and the output consists only of merged model Mmerged. Differences between M1 and M2 are represented by a conflict set (defined later).
Definition 2.1.4. The goal of amodel matching algorithmis to find a matching pair (m1, m2) for ∀m1 ∈M1, m2 ∈M2. The output of model matching is the set of matching pairs M P between M1 and M2.
We consider a representation of a model text-based if i) the model is described in a textual form and ii) the representation is mapped to the model via a well- defined mechanism (e.g., a formal language with a parser [ASU86]). It is important to note that since models are inherently graph-based, their structure eventually has to be taken into account during the matching process. Therefore, we uphold the assumption that raw text differencing tools [KDi03] that are often used with source code should not be used (on their own) with models [BKL+12]. By the term
”raw text differencing”, we refer to the problem of differencing text only by itself (without context), or in other words, solving the Longest Common Subsequence problem [PD94]. In contrast to source code, the textual representation of a model forms another layer on which the MDM process can be conducted. It is worth noting that we do not consider the XML-based standard representation of a model (e.g., XMI [XMI07]) to be a textual representation unless an approach relies heavily on its textual nature. It is also worth mentioning that there are existing AST- based differencing approaches that are not tailored for modeling [FMB+14,GFP09].
Such tools usually focus on source code differencing or on a specific programming language. These approaches may or may not work when differencing models, since algorithms tailored for modeling usually take the structure of the model into account (directly or indirectly).
Definition 2.1.5. A model matching algorithm is text-basedif it works with the textual representations of the models during the matching process. Otherwise, it is graph-based.
Some of the main advantages of using text-based model matching algorithms are as follows:
• Compared to raw text differencing, we can better identify semantic differences between models (e.g., field changed, node removed).
• We can achieve better comprehensibility by using the textual notation during synchronization instead of standards like XMI.
• Text-based MDM algorithms can support synchronization by recognizing dif- ferences that occurred between two editing sessions.
• Text-based MDM algorithms can be used to differentiate between semantic and non-semantic (e.g., comments, formatting) information in the text.
• Raw text differencing can always be used as a fall-back plan should the text- based MDM algorithm fail.
Kolovos et al. [KDRPP09] analyzed existing model matching approaches and separated them into four categories. Their work is one of the earliest takes on categorizing model matching algorithms, with many later studies being influenced by it [vdBHVP11,vdBPV10,AKA14,BKL+12]. The categories are as follows, with the definitions taken from the aforementioned research work [KDRPP09]:
1. Static Identity-Based Matching is based on persistent and non-volatile unique identifiers assigned to the model elements. This typically means a uni- versally unique identifier (UUID). Static approaches are always accurate since a matching pair can always be determined. However, they are more difficult to generalize as the reliance on unique identifiers may cause the algorithm to not work with more complex modeling languages.
2. Signature-Based (Dynamic) Matching approaches use a subset of the features of model elements in order to match them. This is usually preceded by a configuration phase, during which a user-defined function is created to calculate the signature of model elements dynamically. Contrary to Static Identity-Based Matching, individually, the various features do not have to be unique. The downside of this form of matching is the configuration cost of specifying the signature calculating functions.
3. Similarity-Based Matching differs from previous categories in that these algorithms do not give a binary ”yes or no” answer to the question of whether two model elements are matching. Instead, a similarity degree is calculated, based on arbitrary features of the model elements. Different weighting val- ues can be assigned to different features. If the similarity degree is above a certain (configurable) threshold, then the model elements are considered to be matching. Similarly to Signature-Based (Dynamic) Matching, the features and weighting can usually be configured by a user-defined function.
4. Custom Language-Specific Matching algorithms are tailored for a par- ticular modeling language. They heavily rely on the particular semantics of
the language during matching, thus, the process is always accurate. However, since they are tailored for a specific language, these approaches are not general at all.
Regarding differencing, a key difference between algorithms is the way they han- dle changes (differences / conflicts, the terminology differs) between the input mod- els. In the literature [KHL+10, KHS09, HK10, K¨08], there exist two main ap- proaches:
• State-basedapproaches calculate changes by matching model elements in the input models with each other. In this case, model matching occurs. State- based approaches usually store changes as instances of the meta-model of the input models, or as instances of a difference meta-model [CDRP08, RV08].
While there is often a significant computational complexity overhead (mostly due to the graph isomorphism problem [KST93]), these approaches are usually tool-independent.
• Change-basedapproaches record changes as they occur instead of calculating differences later. In this case, there is no need for model matching. The changes can often be run as model transformations on the initial model(s). In this case, we talk about operation-based approaches. These algorithms are usually easier to develop and comprehend [KHL+10] but are more dependent on the underlying technology.
Figure 2.3 demonstrates state-based and operation-based change tracking on the example from Figure 2.2. In the case of a state-based approach, the algorithm would store the differences using an instance model of the meta-model or a difference meta-model, as seen in the left side of the figure. In the case of an operation-based approach, the algorithm records changes as they occur - in our example, we assume that Model 1 is the initial model. Afterwards, it produces a change log that can be applied to a model, as seen in the right side of the figure.
Figure 2.3: Change tracking illustrated on the previous example.
In the following, we present further definitions of basic concepts related to MDM algorithms, including the quantification of accuracy and generality. Note that some definitions might not be fully precise (for example, conflict, or the models them- selves), as graph-based models can be represented in a variety of ways in practice.
Therefore, these definitions serve as a more general introduction to basic MDM concepts.
Definition 2.1.6. Given models M1 and M2, a conflict represents a syntactic or semantic difference found by an MDM algorithm between model elements (me1, me2), whereme1 ∈M1 andme2 ∈M2. Aconflict setbetweenM1 andM2 contains every conflict recognized by an MDM algorithm.
Remark. Conflicts are recognized by an MDM algorithm, based on arbitrary con- siderations. The model elements in a conflict can also form a matching pair, but a conflict can also occur for non-matching pairs as well.
Definition 2.1.7. Given models M1 andM2, the input of themergeprocess is the conflict set between M1 and M2. The output of the merge process is the merged model Mmerged. The merged model contains the solution of every conflict in the conflict set between M1 and M2. The resolutions are determined by the MDM algorithm.
The focus of a model matching algorithm can vary based on the goals the algo- rithm aims to achieve. We consider accuracy and generality to be the main goals a matching algorithm can focus on, although other aspects like performance also exist. Some argue that striking a balance between accuracy and generality greatly benefits the version control of models [Bro09].
Definition 2.1.8. The generality of an MDM algorithm refers to the cardinality of set G where G consists of every modeling language the algorithm is compatible with. A modeling language is compatible with an MDM algorithm if the algorithm aims to work with any given input models M1 and M2 in said language. G can be a finite or an infinite set.
Remark. Defining the concept of generality in a formal and precise way is difficult if not impossible, as it is more of a philosophical concept. We realize our definition is imprecise in this regard, nevertheless, it still specifies a somewhat refined meaning of generality.
Definition 2.1.9. Thematching accuracyof an MDM algorithm between models M1 and M2 is the vector AM = [|M|Mc|
f ull|,|M|Mi|
f ull|], where Mf ull is the set consisting of
every matching pair between M1 and M2, Mc is the set of correctly matched (true positive) pairs by the algorithm, and Mi is the set of incorrectly matched (false positive) pairs.
Definition 2.1.10. The differencing accuracy of an MDM algorithm between models M1 and M2 is the vector AD = [|C|Cc|
f ull|,|C|Ci|
f ull|], where Cf ull is a conflict set consisting of every conflict betweenM1 andM2,Ccis the set of correctly recognized (true positive) conflicts by the algorithm, andCi is the set of incorrectly recognized (false positive) conflicts.
Remark. The ideal matching accuracy of an MDM algorithm is of course the vector AM = [1,0]which means that i) the algorithm matches every existing pair correctly and that ii) the algorithm does not match any non-existing, incorrect pairs. As we will see later, there are no universal metrics for measuring the accuracy of an MDM algorithm, as measurement varies. It is difficult to define an accurate metric and draw general conclusions from it. The definitions for accuracy listed here only apply to given models M1 and M2 and is not a universal indicator of an algorithm’s accuracy.
Definition 2.1.11. Given matching accuracy AM and differencing accuracy AD, an MDM algorithm focuses on accuracy if its main goal is maximizingAM0 and AD0, while minimizing AM1 and AD1 for ∀M1, M2 input models, even at the cost of minimizing generality, namely, the number of compatible languages (|G|). AM0 and AD0 refer to the first, while AM1 and AD1 refer to the second component of their respective vectors. An MDM algorithm focuses on generality if it focuses on maximizing |G|, even at the cost of minimizing accuracy.
2.2 Systematic Literature Review
We have conducted a Systematic Literature Review (SLR) in order to i) discover and categorize the differences between MDM approaches (with a focus on model matching), and to ii) motivate the text-based MDM algorithm that we present in Section 2.3. The survey was conducted using a rigorous, scientifically-proven methodology. This section summarizes this methodology, along with the main re- sults of the survey. For the full review, please refer to the relevant paper [Pub11].
Systematic Literature Reviews have a well defined methodology that they follow, originating from the field of medicine [PMG+04]. Our survey followed the recom- mendations proposed in the literature [BMNT05, KC07, KDJ04, PR06, ZBT11].
According to these recommendations, the process consisted of the following phases:
1. Planning phase: formulate the research questions, the search strategy, and define the inclusion and exclusion criteria.
2. Conducting phase: carry out the survey based on the planning phase, and extract relevant data from the selected primary studies.
3. Reporting phase: report and discuss the results of the survey and draw conclusions.
The survey was conducted in an iterative fashion. The planning phase was constantly refined based on the results of the conducting phase. For example, when we found a new synonym of a keyword, we added it to the search string that we have defined in the planning phase. Then, we examined the newly found studies found during the conducting phase, and if we found more keywords, we updated the search strings again. We conducted a similar iterative process when defining the inclusion and exclusion criteria.
2.2.1 Planning the Survey
Motivations
By studying the most relevant and comprehensive surveys we found in this research field [ASW09, Sel07, SC12, BKL+12], we have concluded that few research works
focus on the categorization of model matching techniques. Moreover, we have found none of them using a systematic approach to identify its key studies, and none focus- ing on text-based algorithms. Summarized, our main motivations behind conducting the survey were as follows:
Table 2.1: The PICOC criteria used in the survey.
Population Researchers and practitioners using model-driven methodologies.
Intervention Analysis of existing matching approaches, with an additional focus on text-based model matching.
Comparison Differences between different matching techniques, and the comparison of text-based and graph-based approaches.
Outcomes Better comprehension of the state-of-the-art, the identification of current research trends and possi- ble avenues for future research.
Context Academic research related to, and industrial ap- plications of model matching.
• Model matching is an important task in state-based model differencing and is useful in other areas as well, i.e., in model transformation testing.
• There exist few surveys that focus on the categorization of model matching algorithms, especially considering text-based algorithms.
• Discovering the reason why the number of text-based model matching algo- rithms is lower, and how they compare with graph-based algorithms.
• Analyzing the aforementioned topics can help researchers in identifying trends and open questions in the field of model matching.
Scope and Goals
Kitchenham et al. [KC07] proposed that the PICO guidelines be used in order to frame research questions and to define the scope of the survey. This system originates from the medical domain [PMG+04] and has been updated for other sciences [PR06]
like software engineering [BMNT05]. The result is the PICOC criteria system. This consists of the following (the definitions are taken from the work of Kitchenham et al. [KC07]) criteria:
• The population represents who or what benefits from the SLR. This can be a specific software engineering role, a category of software engineers, an application area, or an industrial group.
• Theintervention is a software methodology/tool/technology/procedure that addresses a specific issue.
• The comparisonsoftware methodology / tool / technology / procedure with which the intervention is being compared.
• Outcomes relate to factors of importance to practitioners such as improved reliability or reduced production cost.
• Thecontextin which the comparison takes place (e.g., academia or industry), the participants taking part in the study (e.g., practitioners or academics), and the tasks being performed (e.g., small scale or large scale).
Our PICOC criteria can be found in Table 2.1. Based on the criteria and the
motivations discussed earlier, the goals of this survey was as follows:
• Present the state-of-the-art of model matching algorithms.
• Discover the fundamental differences between matching techniques in order to identify research trends and future research directions.
• Conduct a survey on text-based model matching approaches, and compare them to graph-based ones.
• Identify open questions and challenges in the field of model matching.
Research Questions
Defining the research questions is arguably the most important task in the planning of a Systematic Literature Review. Research questions drive the survey process, with the goal of finding answers to the questions. Inclusion and exclusion criteria are also partially derived from research questions. Based on the goals and scope of the survey, we have formulated research questions that are as follows:
• RQ1: What is the state-of-the-art of model matching?
RQ1.1: How common are graph-based and text-based model matching approaches? How do they compare with each other?
RQ1.2: What model matching techniques are most often used? What are the main differences between them?
RQ1.3: How often are state-based and operation-based change tracking used? What are the main differences between the two, with regard to the focus of the algorithms using them?
• RQ2: How can model matching approaches be evaluated? What are the main considerations for evaluation?
• RQ3: What are the main open questions related to the field of model match- ing?
• RQ4: How has interest in model matching developed over time?
Inclusion and Exclusion Criteria
Inclusion and exclusion criteria are used to identify suitable primary studies and to exclude unsuitable ones during the study selection process. A candidate study always has to fulfill these criteria. Every inclusion criteria has to be fulfilled in order for a study to be included, but if one or more exclusion criteria are violated, the study is excluded. The list of inclusion and exclusion criteria are as follows:
• (Inclusion) Study is a peer-reviewed journal article, conference or workshop paper, or a tech report.
• (Inclusion) Study is related to model-driven methodologies.
• (Inclusion) Study contains research that is related to model matching. The presented algorithm must be able to support the matching of models that conform to the same meta-model. Alternatively, the study focuses on the evaluation of, or on open questions related to model matching.
• (Inclusion)Study contains research that answers at least one of our research questions.
• (Exclusion)Study is not a peer-reviewed journal article, conference or work- shop paper, or a tech report (e.g., books, masters theses, doctoral disserta- tions).
• (Exclusion) Study is not related to model-driven methodologies.
• (Exclusion) Study does not contain research related to model matching, or it presents an algorithm that does not support the matching of models that conform to the same meta-model. Moreover, it does not focus on the evaluation of, or on open questions related to model matching.
• (Exclusion) Study is not related to any of our research questions.
• (Exclusion) Study is not a primary study, for example, it is a survey. How- ever, references of related secondary studies are checked during the snowballing process.
• (Exclusion) Study is not written in English.
• (Exclusion) Study is not available on-line (for the authors of this survey).
Search Strategy
During the search process, we acquired relevant primary studies from the follow- ing sources: i) manual search of certain conference and workshop proceedings, ii) automatic search of popular digital libraries, and iii) the snowballing process.
The manual search was conducted on the proceedings of known conferences and workshops that contained relevant studies. The reason we conducted the manual search is to have an additional starting point besides the automatic search. We aimed to expand these starting points via the snowballing process, in order to get as many relevant papers screened as we could. During the manual search, we screened every paper in every year of the following conference and workshop proceedings:
• MODELS(International Conference on Model Driven Engineering Languages and Systems)
• ME @MODELS (Models and Evolution Workshop) and its predecessors MoDSE and MoDSE-MCCM
• MODELSWARD (International Conference on Model-Driven Engineering and Software Development)
• CVSM (Comparison and Versioning of Software Models)
During the automatic search, we used the ACM Guide to Computing Litera- ture and IEEE Xplore. They are the largest digital libraries, and the majority of surveys related to software engineering use these two [ZBT11]. The ACM Guide to Computing Literature covers most other digital libraries1, but the search strings can return more unrelated results. The results of IEEE Xplore can be better focused with more specialized search strings, while covering less studies. We have concluded that the two databases together, along with the previously mentioned manual search adequately covered most of the existing literature. We have used different search strings for the two digital libraries, which are as follows:
• ACM DL: (+text +model +diff )
OR ((keywords.author.keyword:(+model +match) OR keywords.author.keyword:(+model +merge) OR keywords.author.keyword:(+model +difference) OR keywords.author.keyword:(+model +compare) OR keywords.author.keyword:(+model +comparison) OR keywords.author.keyword:(+model +versioning) OR keywords.author.keyword:(+model +composition)
1The ACM Guide to Computing Literature contains 1,406,570 as of the writing of the survey:
https://libraries.acm.org/digital-library/acm-guide-to-computing-literature
OR keywords.author.keyword:(+model +union)) AND domain-specific model-driven model-based UML)
• IEEE Xplore: ((model NEAR/2 differencing OR model NEAR/2 comparing OR ”model comparison” OR model NEAR/2 matching OR model NEAR/2 merging OR ”model composition” OR model NEAR/2 versioning OR model NEAR/2 union) AND (”UML” OR ”model-driven” OR ”model-based engi- neering” OR ”domain-specific”))
Figure 2.4: Overview of the study selection process.
The search strings were formulated according to our research questions, using terms and keywords found in previously known primary studies. The search strings were also continuously refined during the search process. It is worth noting that due to the search term limit of IEEE Xplore, the related search string was split into multiple search strings when used in practice. Keywords like model merging or model composition were included because we have realized during the search process that research related to these topics often deal with model matching in some form.
The last component of our search strategy is the snowballing process. The goal of the snowballing process [Woh14] is to analyze the reference list of every included study from the manual and automatic search processes, and screen them using the same inclusion and exclusion criteria. The process is then repeated with the newly included studies, until we can no longer include new studies. The main goal of snowballing is to include related studies that our search strings might have missed, but the included primary studies reference them (e.g., other related studies, or follow-up studies).
2.2.2 Conducting the Survey
Figure 2.5: Screening process for a study.
Study Selection Process
The study selection (search) process was conducted based on the planning presented in Section 2.2.1. Figure 2.4 summarizes the process. First, the automatic and manual searches were conducted (they could be run independently), which resulted in 34 and 20 included studies, respectively. The snowballing process was started with these 54 studies, and finally resulted in an additional 65 included primary studies.
Snowballing was an essential part of our study selection process, as over half of the included studies were included during this process. The screening process for a candidate study is depicted in Figure 2.5. This process was applied to every study that we have found during the manual, automatic, and snowballing processes. First, we have read the title and abstract of the study, and excluded it if the inclusion and exclusion criteria were violated. After that, we have read the full paper, and if the study was a related secondary study (in other words, a survey), we have included it for snowballing, but not as a primary study. Otherwise, if, after reading the full paper, the study fulfilled our inclusion and exclusion criteria, we have included it as a primary study. Thus, the second inclusion and exclusion criteria check is there to ensure the study still fulfills the criteria after reading the paper.
Table 2.2 contains the detailed results of the entire study selection process. To sum it up, out of the 4693 studies examined, 1419 were duplicates, 2930 were ex- cluded based on their title and abstract, and 219 were excluded after analyzing the paper. At the end of the process, 119 studies were included, with an additional 6 papers categorized as surveys (secondary studies). These surveys are not classified as primary studies, but their references were all checked during the snowballing pro- cess. We would like to note that for the sake of clarity, references examined during the snowballing process does not include studies that are not available on-line2, are
2This criterion was applied in a total of 39 cases (of which 22 was during the snowballing process) and consisted mostly of situations when the paper in question was no longer available, as the website displaying it ceased to function. There were no significant major publication sources that were not included for this reason.
Table 2.2: Summary of the study selection process.
Automatic Manual Snowballing Overall
Total 1139 1179 2375 4693
Included 34 20 65 119
Surveys 0 1 5 6
Duplicates 95 15 1309 1419
Excluded (soft)
945 1102 883 2930
Excluded (hard)
65 41 113 219
not peer-reviewed journal articles, conference or workshop papers, or tech reports.
Extracted Data
For every study in the 119 included studies, we have extracted relevant data that we used to answer the research questions. Table 2.3 summarizes the structure of the extracted data together with the related research questions and the potential values each data type column can take. In addition to the values depicted there, every data type can also take on the values unknown (when we could not decipher the value based on the paper), other (when the value is not listed in the table, but otherwise valid) andnone. The detailed explanations for the potential values are as follows:
• Type (RQ1.1)- The main type of the matching algorithm, namely, whether it works using the graph-based structure or the textual representation of the models. The textual representation of a model was defined in Section 2.1.
• Matching (RQ1.2) - The matching technique used by the algorithm. We used the categorization proposed by Kolovos et al. [KDRPP09].
• Change tracking (RQ1.3)- State-based or operation-based change tracking.
• Focus (RQ1) - The goal and focus of the algorithm. We consider the main ones to be accuracy and generality (see Section 2.1 for details). If an algorithm focuses on both of them, the possible trade-off for doing so is also marked during data extraction. An algorithm can also focus on other aspects, like performance or scalability.
• Evaluation (RQ2) - Whether the study contains information on the evalu- ation of model matching approaches.
• Open Questions (RQ3)- Whether the study contains information regarding open questions. This excludes open questions that are only related to the presented algorithm in the paper, like specific improvements to said algorithm.
• Year (RQ4)- The year the study was published.
• Author(s) - The author(s) of the paper. Also used for identifying studies discussing the same research work.
• Version Control Systems- Whether the study focuses on model matching in the context of version control systems. This data was only needed to reinforce
Table 2.3: Extracted data structure with possible values.
Type (RQ1.1) graph / text
Matching (RQ1.2) static / dynamic / similarity / custom Change Tracking (RQ1.3) operation-based / state-based Focus (RQ1) accuracy / generality / both
Evaluation (RQ2) yes / no
Open Questions (RQ3) yes / no
Year (RQ5) -
Author(s) -
Version Control Systems yes / no
our conjuncture of version control systems being one of the primary motivating factors for researching model matching.
2.2.3 Results and discussion
In the following, the main results of the SLR are briefly discussed, highlighting our findings in the form of answers to the research questions. For a more thorough discussion of the results, see the full paper [Pub11].
State-of-the-art (RQ1)
Our first research question is aimed at identifying the state-of-the-art in model matching techniques. We split this question into multiple sub-questions, which we are answering here: i) comparison of text-based model matching to graph-based model matching, ii) discovery of differences between main matching techniques, and the iii) analysis of change tracking approaches, with regard to the focus of the algorithm.
Text-based Model Matching (RQ1.1)
RQ1.1: How common are graph-based and text-based model matching ap- proaches? How do they compare with each other?
Out of the 72 related primary studies, the number of text-based algorithms (8) is significantly lower than the number of graph-based algorithms (64). We believe that this 11% to 89% ratio is notable enough to conclude that text-based algorithms are significantly more rare than graph-based ones in the existing literature. Figure 2.6 depicts graph-based algorithms with their focus. Statistically, we can conclude that graph-based algorithms are more varied than text-based ones. In contrast to text- based algorithms, similarity-based approaches are also more common among graph- based algorithms.
Summarized, our key findings on comparing text-based model matching algo- rithms with graph-based ones are as follows:
Figure 2.6: Summary of the focus of graph-based algorithms.
• Lack of text-based algorithms. There is a lack of text-based model match- ing algorithms, and an even greater lack of general text-based algorithms.
• Motivations behind using text-based algorithms. The main motiva- tions behind text-based model matching we found are i) using the simplicity of raw text differencing and enhancing it to be suitable for models [BLF14, FBL13,vRvdS15], ii) aiming for better accuracy at a higher configuration cost [Pub9][RV08], and iii) supporting the textual notation of the models by solving a specific problem [BSCB09, vRvdS15].
• Generality comes with a configuration cost. Text-based model matching algorithms tend to focus either on accuracy, or on both accuracy and generality.
This often comes with a compromise in the form of an increased configuration cost of the algorithm and the mapping of the textual representation. Only when these costs are acceptable that text-based algorithms can be reliably used in a more general way in practice. Thus, they are either accurate (usually with a single language in mind), or have a higher configuration cost to be used generally. However, more research is needed on the analysis of the trade- off between the configuration cost and the generality / accuracy of the text- based algorithm, as no study we found discussed this problem in detail. As a final note, the idea of supporting both accuracy and generality at a higher configuration cost is also present in graph-based algorithms.
• Quantifying the configuration cost? What metrics can we use to express the effort required to use a text-based matching algorithm with a particular modeling language? Can it be expressed using standard metrics? The studies we found did not discuss these problems.
Figure 2.7: Summary of matching techniques used by algorithms.
Matching Techniques (RQ1.2)
RQ1.2: What model matching techniques are most often used? What are the main differences between them?
Figure 2.7 summarizes the matching techniques used by the algorithms we found, according to our previous categorization (Section 2.1). Note that the sum of the numbers on the figure is greater than the number of unique algorithms. This is due to the fact that algorithms using more than one matching technique are counted multiple times for every used technique. Similarity-based matching is the most popular, followed by static and dynamic matching. Operation-based algorithms usually use no matching technique (hence the category none on the figure).
Figure 2.8 depicts the different matching techniques along with the main focus of the algorithm. Static matching is often used in achieving both accuracy and generality, although the use of static identifiers usually severely limits the latter.
Similarity-based matching is often employed when the accuracy of the algorithm is the main focus. In the case of dynamic or custom matching, and other algorithms, we have found no distinguishable correlations.
Figure 2.8: Correlations between matching type and the focus of the algorithm.
Change Tracking (RQ1.3)
RQ1.3: How often are state-based and operation-based change tracking used?
What are the main differences between the two, with regard to the focus of the algorithms using them?
Of the 72 identified algorithms, the majority (75%) were state-based, some were operation-based (˜21%), while a small number of algorithms (˜4%) used both change tracking approaches. Figure 2.9 depicts the main focus of state-based and operation- based algorithms. Contrary to our preliminary expectations, the operation-based algorithms we found are split evenly with regard to the focus of the algorithm. State- based algorithms tend to focus more often on accuracy (50%). Thus, in general, we
Figure 2.9: The main focus of operation-based and state-based algorithms.
can draw two main conclusions from these results: i) state-based approaches are more commonly used, and ii) there are no significant correlations between the used change tracking method and the main focus of the algorithm.
Evaluation Techniques (RQ2)
RQ2: How can model matching approaches be evaluated? What are the main considerations for evaluation?
Summarized, our key findings on the evaluation of model matching algorithms are as follows:
• Measurement metrics are established. Matching accuracy is often used to evaluate matching algorithms. It can be measured with metrics borrowed from pattern recognition: precision, recall, and F-score [Bis06]. The accu- racy of conflict detection and resolution can also be measured using these metrics. Performance and scalability are also common factors, although large- scale models are somewhat neglected in this regard.
• Lack of automated benchmarking. Benchmarking is often mentioned as one of the main challenges in the field of model matching. We found that there is a need for the automatic generation of technology-independent benchmark- ing model datasets.
• Lack of empirical studies. Empirical user studies are rare, although they can be used to verify certain aspects of an algorithm (e.g., human effort re- quired to use) that would otherwise be difficult to quantify.
• Lack of technology-independent evaluation frameworks. There are some existing evaluation frameworks for model matching algorithms, but they are usually technology-dependent [GDGBSM+12, US13].
Open Questions (RQ3)
RQ3: What are the main open questions related to the field of model matching?
Summarized, some of the main open questions and challenges in the field of model matching we have identified are as follows:
• Relatively low industrial appliance. Low tool adoption of existing al- gorithms. Most algorithms have a working prototype, although industrial applications are rare.
• Inadequate change visualization. Although only loosely related to model matching, difference and conflict visualization is not adequate enough, the concrete syntax of the model should more often be used for this purpose.
• Intention-aware model versioning. Intention-aware model versioning is a possible future research direction where the result of a model merge is based on the intentions of the developers committing the changes.
• Semantic model matching. Semantic model matching is a relatively new research direction with several problems: algorithmic complexity, difference representation, and integration with syntactic matching.
• Lack of benchmarking datasets. Benchmarking datasets are uncommon, and are usually dependant on a particular technology. Automatically gener- ated and technology-independent datasets are needed in order to more objec- tively evaluate algorithms.
• Large-scale model matching. Large-scale model matching needs more re- search. The trade-off between performance and accuracy also merits further research.
• Further analysis of text-based algorithms. The investigation of the dif- ferences between text-based and graph-based algorithms, and identifying and quantifying the aspects that they differ in merits further research. This also includes the analysis of the correlation between the configuration cost and accuracy / generality of a text-based algorithm.
Interest in Model Matching (RQ4)
RQ4: How has interest in model matching developed over time?
Figure 2.10: Number of included primary studies per year.
Figure 2.10 illustrates every included primary study per year, from the earliest published paper (2001) to the latest one (2018). The apex of research related to model matching was between 2007 (16 studies) and 2009 (20 studies). From then, a steady decline in the frequency of published papers can be observed.
