June 22, 2012

Domain Specific Model-Based Testing


Domain-specific modeling (DSM) is a software engineering methodology for designing and developing systems, such as computer software. It involves systematic use of a domain-specific language (DSL) to represent the various facets of a system. DSM languages tend to support higher-level abstractions than general-purpose modeling languages; so they require less effort and fewer low-level details to specify a given system.

Domain-specific modeling is used as an approach where the developed system is modeled in terms of domain-specific concepts and these models are automatically transformed to other forms such as application code. A model is a computerized representation, where each element in their presentation corresponds to an element or concept in the domain. For many years, models have been important in defining how data is held within IT systems. Today, models are increasingly being used for other purposes, such as modeling of business processes, distributed service implementations, and physical datacenters.

Domain-specific modeling (DSM) often includes the idea of code generation: automating the creation of executable source code directly from the DSM models. Being free from the manual creation and maintenance of source code means DSM can significantly improve developer productivity [1]. The reliability of automatic generation compared to manual coding will also reduce the number of defects in the resulting programs thus improving quality.

 
Figure 1: Domain-specific Modeling
DSM differs from earlier code generation attempts in the CASE tools of the 1980s or UML tools of the 1990s. In both of these, the code generators and modeling languages were built by tool vendors. While it is possible for a tool vendor to create a DSM language and generators, it is more normal for DSM to occur within one organization. One or a few expert developers create the modeling language and generators, and the rest of the developers use them.
Having the modeling language and generator built by the organization that will use them allows a tight fit with their exact domain and needs. It also reduces the time needed for developers to learn the modeling language, since it can use familiar terms and concepts. Finally, since only one organization's requirements need be taken into account, it is easier for the modeling language to evolve in response to changes in the domain.
DSM languages can usually cover a range of abstraction levels for a particular domain. For example, a DSM language for mobile phones could allow users to specify high-level abstractions for the user interface, as well as lower-level abstractions for storing data such as phone numbers or settings. Likewise, a DSM language for financial services could permit users to specify high-level abstractions for clients, as well as lower-level abstractions for implementing stock and bond trading algorithms.
When comparing the benefits of DSM and traditional software development means, cases conducted in industrial settings constantly show 5-10 times productivity gains compared to the traditional software development [2]. In addition to the productivity gains, the DSM is expected to have a positive impact on the software quality [3].

 1.1.      DSM Basic Architecture

Productivity and quality gains are primarily caused by the three corner stone’s of DSM, i.e. Domain-Specific Modeling Languages (DSML), code generators and software frameworks which when combined are known as the DSM basic architecture [3][6].
o    A meta-model defines the syntax of a modeling language. In the case of DSMLs, meta-model mirrors the problem-space by providing modeling elements found directly from the problem domain. In practice, the-meta model also includes elements and restrictions of the target platform.

o    Code generators define the transformation rules on how to transform application models that are based on a meta-model to a source code representation.
o    The software framework abstracts low-level details of the target platform and functions as a platform on which a code generator generates source code. Sometimes no framework is required and the generated code directly accesses the services and functions of the target platform.
Actual applications are modeled based on the model elements and constraints of the developed meta-model. The models can be transformed into source code or any given representation with generators.


1.2.      Goals of The Domain-Specific Modeling Practice

o    Good coverage of the system under test (application interactions, interleaved executions)
o    Reusability of model components when UI changes (testing different UIs but similar applications)
o    Reusability of model components when configuration changes (testing devices with different set of applications)
o    Reusability of model components when test objective changes (The same test models can be used in smoke, requirement coverage and long period tests.)


Model-based testing (MBT) is a prominent black box software testing method that enables the creation of a comprehensive test suite by modeling the behavior of a system, where the models can then be transformed into a test suite by utilizing several test design algorithms [7]. This enables automating the generation of a test suite from models that are easier to keep in synch with the evolving software system. As the test suites are based on models, the maintenance effort of the test suite also decreases.
Figure 2: Model Based Testing
This is as an advanced test automation concept has been gaining increasing interest in the industry in recent years which produces test cases based on a model of the system under test. MBT can be a powerful approach in generating test cases to cover various aspects of the system under test from the test models. MBT automates the detailed design of the test cases and the generation of the traceability matrix based on an abstract model of the system under test. It is the most automated level of testing (ranging from manual, capture/replay, script-based, keyword-driven and model-based testing). A model-based testing tool generates a set of test cases from that model. MBT has been previously successfully applied on a variety of sectors and languages such as B, UML.
However, typically several different types of expertise are required to build useful test models and to effectively generate test cases from these models  

2.1.      Model Base Testing Process


The MBT process can be divided into three phases, namely modeling, test generation and test execution (Figure 1). The modeling stands for the modeling of system behavior, where the test generator then generates a test suite from these models allowing the test executor to conducts the test.
Figure 3: Model Based Testing Process

2.1.1.   Modeling

When developing a Model Base testing first uses the functional requirements of the software systems. These models appear to be somewhat similar behavior to the real product. The system requirements are also modeled with the MBT, in addition to realizing the requirements as an implementation exist two opinions of the behavior of the system. The differences between these can be viewed as errors. The model is required to have knowledge of the input and output data of the System Under Test (SUT). The input data is used for executing the tests and the output data is for validating the tests. The model can be made from an environmental or design viewpoint. The viewpoints are mirror images and are equally suitable for test generation. The design viewpoint for the MBT is similar to the modeling viewpoint for implementation purposes, but at a higher abstraction level. The implementation model can therefore be reused to model design viewpoint MBT model. The notation of the models can be graphical, textual or mixed, where the notation varies from general purpose to domain-specific.

2.1.2.   Test Generation

Test generation is based on a model, where several algorithms, called test design algorithms, are utilized for generating test cases from the model. There are three main categories of test design algorithms 2] : 
·         Requirement-based criteria i.e. the test generator strives to cover all the marked requirements,
·         Coverage criteria, i.e. a test suite is generated on the basis of covering a certain degree of the model
·         Walking algorithms, i.e. an algorithm determines how the model is traversed and the test suite is generated from that basis.

2.1.3.   Test Execution


A test execution can be performed either offline or online. The offline testing stands for generating tests first and then executing the tests separately. With online testing, one step at a time is generated based on the output of the SUT. The difference between the tests approaches is that in the offline approach, test generation is separated from test execution and therefore there is a possibility to utilize algorithms requiring heavy computation without the test execution suffering. In the case of online testing, the following test step is generated on the basis of the previous step output values of the SUT, therefore in order to have an immediate response to outputs, only algorithms requiring less computation can be utilized. Online testing also enables the ability to have infinite test suites and enables handling the non-deterministic behavior of the SUT
. 
This often means testing the DSM Basic Architecture by Utilizing Model-Based Testing.
Software testing, in the case of DSM, essentially means testing the primary sources of errors, i.e. testing the meta-model and the code generator. Although being the primary sources of errors, not much research have taken place to address these [3, 4, 5].
In the context of iterative and incremental software development processes, there is also concern about the test maintenance because the applications evolve. Additionally, code generators are under constant evolution, when the underlying platform evolves. Changes in the meta-model will also have an impact on the code generators, whereas changes in the meta-model and the code generators have an impact on the existing applications. When one of the three aspects evolves, one has to therefore have tests in order to make sure that the evolved versions function correctly.
Testing application models, meta-models and code generators cannot be performed separately since all of the three levels are intertwined tightly together. This is because modeling languages consist of syntax and semantics definitions. The meta-model describes the semantics of the model, but it cannot have an impact on how the code generator decides to produce the code. It is therefore the code generator that makes the ultimate decision for what is generated and how and thus the meta-model and the code generator, as a combination, define the semantics in practice. This speaks on behalf of the testing practice introduced in [3].
MBT can be seen as a prominent method for generating comprehensive test suites. This statement is based on the capability of the test design algorithms to produce a comprehensive test suite from the MBT models. A model being easier to maintain compared to a test suite, decreases the maintenance effort [6]. In addition to lightening maintenance, the test suite will always be in synch with the MBT model. 

4.1.      Advantages 

o    MBT reduces the overall test design time for specific test classes.
o    MBT can guarantee specific and complex coverage criteria.
o    MBT allows to use formal models only on the top part of a design process and rely on standard (test-based) development techniques for the rest of the tool chain

4.2.      Disadvantages 

o    Specific test classes, limited to what is captured by model
o    Model validation is required
o    Formal modeling skills are necessary
o    Need for an adaptation layer for running abstract test on a concrete environment
o    Technical limitations of tools limiting the size of the model or requiring one to write models with MBT in mind.  

There are two techniques for utilizing the MBT in testing the DSM basic architecture.
o    Generating a Test Case for the DSM Basic Architecture
o    Generating a Test Suite for the DSM Basic Architecture 

5.1.      Generating a Test Case for the DSM Basic Architecture

The MBT can be utilized for testing the applications developed with the DSM approach. However, testing the applications differs in the case of DSM, from the traditional MBT. In the traditional MBT, models and the implementation are derived from informal software specifications; therefore it is tested whether the application implementation follows the specifications. In the DSM approach, the implementation and the MBT are derived from the same model, thus it cannot be tested whether the application is implemented according to the specifications but it is tested whether the code generator produces a working application running on the software framework from the application model.
Thus, in the context of DSM, one application model can be seen as a test case for the whole DSM basic architecture. The technique for utilizing the MBT in developing a test case for the DSM basic architecture is illustrated in Figure 4. The left side of the figure follows the basic code generation process, whereas the right side follows the MBT process. Considering the MBT process, the difference between the introduction of the DSM to the MBT and the traditional MBT is in the source of the model. Whereas in the traditional MBT the model is derived from software specifications, in this case the model is derived from the same model that the DSM utilizes for the code generation. By doing so, both the code generator and the MBT tool always have the same conception of the model. Errors are detected if the code generator realizes the conception differently, thus the MBT model no longer matches the generated application when executed.
Figure 4: MBT Utilized for Testing Applications

5.2.      Generating a Test Suite for the DSM BasicArchitecture 

One test case does not test the whole DSM basic architecture thoroughly. By modeling of a set of applications, test coverage increases. However, the effort to maintain the test suite, i.e. a set of application models, can become an issue when the DSM basic architecture evolves. As in the case of the DSM, all applications of a certain domain are based on a meta-model, the test suite for the DSM basic architecture should also be based on this meta-model. In several papers it is argued that the MBT can be utilized for generating a comprehensive test suite of application models for the DSM basic architecture from a meta-model. This claim is based on the capability of the MBT to generate a test suite from models, of which meta-models essentially are. If the test suite is generated from a meta-model, it will also always be in synch with the evolving language, thus decreasing the maintenance effort of the test suite.
An overview of the technique for generating a test suite for the DSM basic architecture is depicted in Figure 5. It must be noted that it is the application models that are generated from the meta-model by the MBT tool. As the MBT tool utilizes the meta-model for generating the test suite, the meta-model has to be strictly defined, i.e. the meta-model should be defined in such away that only legal applications can be modeled. If not, the meta-model test suite also includes test cases, i.e. application models that are not legal for the target platform. With enough application models, the errors in the DSM basic architecture can be noticed.
Figure 5: Testing the Meta-model
Enabling the test suite generation requires a DSM tool providing facilities for exporting the meta-model into a format required by the MBT tool. In addition, the DSM tool has to enable importing models generated by the MBT tool, whereas the MBT tool has to enable exportation of the test suite, i.e. a set of application models, to the DSM tool, in addition to providing facilities for generating the test suite for the imported application models.


6.1.      Tools for the Model-Based Testing

Tool support for the MBT is extensive. However, many of the MBT tools are still immature but there also exists commercial tool vendors providing more mature tools and support when required. MaTeLo from All4Tec is for control oriented MBT. Reactis from Reactive Systems and T-Vec provides tools focused in embedded MBT testing. Test Designer from Smartesting and Conformiq Qtronic from Conformiq5 are general-purpose solutions for the MBT. Scrutinizing the MBT tool evaluation, presented in [10], reveals the Qtronic to be mature enough and provides open data formats for importing and exporting the models.
The Qtronic expects the input model to be either in
o    A similar format as the UML state machine diagram extended with a variant of Java, which is called QML, or
o    A textual representation, where the programming language is QML.
Both of the model types represent a model where the input and output pairs are defined. In addition, special requirements can be defined for the inputs and outputs that guide the test design algorithms in generating the test suite. The test suite generator can be implemented by utilizing the provided plug-in interface.

6.2.      Tools for the Domain-Specific Modeling

The Generic Modeling Environment 6 from Vanderbilt University and Metaedit+ 4.5 from Metacase7 are probably the most well-known language work benches. Microsoft also provides a DSM tool with Microsoft Visual Studio 2005 SDK. There are also open source tools available, such as the Generic Eclipse Modeling System.
Metaedit+ is our choice among the tools since, as far as we know, it is the only language work bench providing code generator facilities with a language dedicated only for developing code generators. This enables a rapid development of generators compared to a situation where the code generators are developed by utilizing the APIs of the modeling tools. In addition, Metaedit+ enables importing models in an XML format.
o    This approach is useful at engineering level to work through the mechanics of partitioning or integration or to decide how much functionality to support achieves maximum return.
o    DSM can help in developing future market models such as voice phones, audio phones, location devices, camera phones, personal organizers, smart phones, tablets, dongles and autonomous robots. 
o    DSM approach allowing product families to develop their product in a level that clearly differentiated functionally based cost and value metrics.  

Industrial cases constantly reveal 5-10 productivity gains when utilizing DSMLs in the software development. Although the gains witnessed so far are great, a concerned question raised is how to make sure the languages, i.e. meta-models and code generators, are correct especially in the cases where the languages are provided for the masses and for mission critical systems. Iterative and incremental software development processes bring an additional question of test suite maintenance. Although testing is an important means to detect errors in the traditional software engineering, it is not well known how to test the DSM basic architecture thoroughly.
The techniques presented from the research papers; do not strive for the testing of the layers of the DSM basic architecture in isolation, but to test the whole architecture. The test technique is based on generating a comprehensive test suite of application models by utilizing the means provided by the MBT plus generating test suites for the generated applications. As a combination, the DSM basic architecture will be tested thoroughly.

References

M Utting, A Pretschner, and B Legeard, "A taxonomy of modelbased modelbased testing," in University of Waikato, Department of Computer Science, Hamilton, 2006.
A Hartman, M Katara, and S Olvovsky, "Choosing a test modeling language: A survey," in Haifa Verification Conference, 2006, pp. 204-218.
O P Puolitaival, M Luo, and T Kanstren, "On the Properties and Selection of Model-Based Testing tool and Technique," in 1st Workshop on on Model-based Testing in Practice, Berlin,Germany, 2008.
L Nachmanson, M Veanes, W Schulte, N Tillmann, and Grieskamp, "Optimal strategies for testing nondetermistic Systems," in ACM SIGSOFT Internationa Symposium on SWTesting and Analysis, 2004, pp. 55-64.
[5]
J Merilinna, O Puolitaival, J Pärssinen,Towards Model-Based Testing of Domain-Specific Modelling Languages,  VTT Technical Research Centre of Finland,2010.
[6]
Schulz, S., Honkola, J. and Huima, A., "Towards Model-Based Testing with Architecture Models", Engineering of Computer Based Systems (ECBS ’07), pp. 495-502, 2007
  [7]
Utting, M. and Legeard, B., "Practical Model Based Testing: A Tools Approach", Morgan Kaufmann 1st ed., ISBN: 978-0123725011 , 456p., 2006
[8]
E. Bringmann& A. Kr¨amer (2008): Model-Based Testing of Automotive Systems. In: IEEE Int’l. Conf. on Software Testing, Verification and Validation (ICST2008), pp. 485–493
[9]
TeemuKanstr´en, Olli-PekkaPuolitaival (2012): Using Built-In Domain-Specific Modeling Support to Guide Model-Based Test Generation
[10]
W Prenningerand APretschner. Abstractions for model-based testing. In Electronic Notes in Theoretical Computer Science. Proceedings of the International Workshop on Test andAnalysis of Component Based Systems ( 2004), volume 116,pp 59–71, Los Alamitos, CA, USA, January 2005. ElsevierScience Publishers Ltd, 2004

No comments:

Post a Comment