Domain-specific
modeling (DSM) is a software engineering methodology for designing and
developing systems, such as computer software. It involves systematic use of a
domain-specific language (DSL) to represent the various facets of a system. DSM
languages tend to support higher-level abstractions than general-purpose
modeling languages; so they require less effort and fewer low-level details to
specify a given system.
Domain-specific modeling is used as an approach where the
developed system is modeled in terms of domain-specific concepts and these
models are automatically transformed to other forms such as application code. A
model is a computerized representation, where each element in their
presentation corresponds to an element or concept in the domain. For many
years, models have been important in defining how data is held within IT
systems. Today, models are increasingly being used for other purposes, such as
modeling of business processes, distributed service implementations, and
physical datacenters.
Domain-specific modeling (DSM) often includes the idea of
code generation: automating the creation of executable source code directly
from the DSM models. Being free from the manual creation and maintenance of
source code means DSM can significantly improve developer productivity [1]. The
reliability of automatic generation compared to manual coding will also reduce
the number of defects in the resulting programs thus improving quality.
Figure
1: Domain-specific Modeling
DSM differs from earlier
code generation attempts in the CASE tools of the 1980s or UML tools of the
1990s. In both of these, the code generators and modeling languages were built
by tool vendors. While it is possible for a tool vendor to create a DSM
language and generators, it is more normal for DSM to occur within one
organization. One or a few expert developers create the modeling language and
generators, and the rest of the developers use them.
Having the modeling
language and generator built by the organization that will use them allows a tight
fit with their exact domain and needs. It also reduces the time needed for
developers to learn the modeling language, since it can use familiar terms and
concepts. Finally, since only one organization's requirements need be taken
into account, it is easier for the modeling language to evolve in response to
changes in the domain.
DSM languages can usually
cover a range of abstraction levels for a particular domain. For example, a DSM
language for mobile phones could allow users to specify high-level abstractions
for the user interface, as well as lower-level abstractions for storing data
such as phone numbers or settings. Likewise, a DSM language for financial
services could permit users to specify high-level abstractions for clients, as
well as lower-level abstractions for implementing stock and bond trading
algorithms.
When comparing the
benefits of DSM and traditional software development means, cases conducted in
industrial settings constantly show 5-10 times productivity gains compared to
the traditional software development [2]. In addition to the productivity
gains, the DSM is expected to have a positive impact on the software quality
[3].
Productivity and quality
gains are primarily caused by the three corner stone’s of DSM, i.e.
Domain-Specific Modeling Languages (DSML), code generators and software
frameworks which when combined are known as the DSM basic architecture [3][6].
o A
meta-model defines the syntax of a modeling language. In the case of DSMLs,
meta-model mirrors the problem-space by providing modeling elements found
directly from the problem domain. In practice, the-meta model also includes
elements and restrictions of the target platform.
o Code
generators define the transformation rules on how to transform application
models that are based on a meta-model to a source code representation.
o The
software framework abstracts low-level details of the target platform and
functions as a platform on which a code generator generates source code.
Sometimes no framework is required and the generated code directly accesses the
services and functions of the target platform.
Actual applications are modeled based on the model elements and constraints of
the developed meta-model. The models can be transformed into source code or any
given representation with generators.
o Good
coverage of the system under test (application interactions, interleaved
executions)
o Reusability
of model components when UI changes (testing different UIs
but similar applications)
o Reusability
of model components when configuration changes (testing
devices with different set of applications)
o Reusability
of model components when test objective changes (The same
test models can be used in smoke, requirement coverage and long period tests.)
Model-based testing
(MBT) is a prominent black box software testing method that enables the
creation of a comprehensive test suite by modeling the behavior of a system,
where the models can then be transformed into a test suite by utilizing several
test design algorithms [7]. This enables automating the generation of a test
suite from models that are easier to keep in synch with the evolving software
system. As the test suites are based on models, the maintenance effort of the
test suite also decreases.
This is as an advanced
test automation concept has been gaining increasing interest in the industry in
recent years which produces test cases based on a model of the system under
test. MBT can be a powerful approach in generating test cases to cover various
aspects of the system under test from the test models. MBT automates the
detailed design of the test cases and the generation of the traceability matrix
based on an abstract model of the system under test. It is the most automated
level of testing (ranging from manual, capture/replay, script-based,
keyword-driven and model-based testing). A model-based testing tool generates a
set of test cases from that model. MBT has been previously successfully applied
on a variety of sectors and languages such as B, UML.
However, typically
several different types of expertise are required to build useful test models
and to effectively generate test cases from these models
The MBT process can be divided into three phases, namely modeling, test
generation and test execution (Figure 1). The modeling stands for the modeling
of system behavior, where the test generator then generates a test suite from
these models allowing the test executor to conducts the test.
Figure
3: Model Based Testing Process
When developing a Model
Base testing first uses the functional requirements of the software systems.
These models appear to be somewhat similar behavior to the real product. The
system requirements are also modeled with the MBT, in addition to realizing the
requirements as an implementation exist two opinions of the behavior of the
system. The differences between these can be viewed as errors. The model is
required to have knowledge of the input and output data of the System Under
Test (SUT). The input data is used for executing the tests and the output data
is for validating the tests. The model can be made from an environmental or
design viewpoint. The viewpoints are mirror images and are equally suitable for
test generation. The design viewpoint for the MBT is similar to the modeling
viewpoint for implementation purposes, but at a higher abstraction level. The
implementation model can therefore be reused to model design viewpoint MBT
model. The notation of the models can be graphical, textual or mixed, where the
notation varies from general purpose to domain-specific.
Test
generation is based on a model, where several algorithms, called test design
algorithms, are utilized for generating test cases from the model. There are
three main categories of test design algorithms 2] :
·
Requirement-based criteria i.e. the test
generator strives to cover all the marked requirements,
·
Coverage criteria, i.e. a test suite is
generated on the basis of covering a certain degree of the model
·
Walking algorithms, i.e. an algorithm
determines how the model is traversed and the test suite is generated from that
basis.
A test execution can be performed either offline or online. The offline testing
stands for generating tests first and then executing the tests separately. With
online testing, one step at a time is generated based on the output of the SUT.
The difference between the tests approaches is that in the offline approach,
test generation is separated from test execution and therefore there is a
possibility to utilize algorithms requiring heavy computation without the test
execution suffering. In the case of online testing, the following test step is
generated on the basis of the previous step output values of the SUT, therefore
in order to have an immediate response to outputs, only algorithms requiring
less computation can be utilized. Online testing also enables the ability to
have infinite test suites and enables handling the non-deterministic behavior
of the SUT .
This often means testing
the DSM Basic Architecture by Utilizing Model-Based Testing.
Software testing, in the
case of DSM, essentially means testing the primary sources of errors, i.e.
testing the meta-model and the code generator. Although being the primary
sources of errors, not much research have taken place to address these [3, 4,
5].
In the context of
iterative and incremental software development processes, there is also concern
about the test maintenance because the applications evolve. Additionally, code
generators are under constant evolution, when the underlying platform evolves.
Changes in the meta-model will also have an impact on the code generators,
whereas changes in the meta-model and the code generators have an impact on the
existing applications. When one of the three aspects evolves, one has to
therefore have tests in order to make sure that the evolved versions function
correctly.
Testing application
models, meta-models and code generators cannot be performed separately since
all of the three levels are intertwined tightly together. This is because
modeling languages consist of syntax and semantics definitions. The meta-model
describes the semantics of the model, but it cannot have an impact on how the
code generator decides to produce the code. It is therefore the code generator
that makes the ultimate decision for what is generated and how and thus the
meta-model and the code generator, as a combination, define the semantics in
practice. This speaks on behalf of the testing practice introduced in [3].
MBT can be seen as a
prominent method for generating comprehensive test suites. This statement is
based on the capability of the test design algorithms to produce a
comprehensive test suite from the MBT models. A model being easier to maintain
compared to a test suite, decreases the maintenance effort [6]. In addition to
lightening maintenance, the test suite will always be in synch with the MBT
model.
o
MBT reduces the overall test design time for
specific test classes.
o
MBT can guarantee specific and complex coverage
criteria.
o
MBT allows to use formal models only on the top
part of a design process and rely on standard (test-based) development
techniques for the rest of the tool chain
o
Specific test classes, limited to what is
captured by model
o
Model validation is required
o
Formal modeling skills are necessary
o
Need for an adaptation layer for running
abstract test on a concrete environment
o
Technical limitations of tools limiting the size
of the model or requiring one to write models with MBT in mind.
There are two
techniques for utilizing the MBT in testing the DSM basic architecture.
o
Generating a Test Case for the DSM Basic
Architecture
o
Generating a Test Suite for the DSM Basic
Architecture
The MBT can be utilized
for testing the applications developed with the DSM approach. However, testing
the applications differs in the case of DSM, from the traditional MBT. In the
traditional MBT, models and the implementation are derived from informal
software specifications; therefore it is tested whether the application
implementation follows the specifications. In the DSM approach, the
implementation and the MBT are derived from the same model, thus it cannot be
tested whether the application is implemented according to the specifications
but it is tested whether the code generator produces a working application
running on the software framework from the application model.
Thus, in the context of
DSM, one application model can be seen as a test case for the whole DSM basic
architecture. The technique for utilizing the MBT in developing a test case for
the DSM basic architecture is illustrated in Figure 4. The left side of the
figure follows the basic code generation process, whereas the right side
follows the MBT process. Considering the MBT process, the difference between
the introduction of the DSM to the MBT and the traditional MBT is in the source
of the model. Whereas in the traditional MBT the model is derived from software
specifications, in this case the model is derived from the same model that the
DSM utilizes for the code generation. By doing so, both the code generator and
the MBT tool always have the same conception of the model. Errors are detected
if the code generator realizes the conception differently, thus the MBT model
no longer matches the generated application when executed.
Figure
4: MBT Utilized for Testing Applications
One test case does not
test the whole DSM basic architecture thoroughly. By modeling of a set of
applications, test coverage increases. However, the effort to maintain the test
suite, i.e. a set of application models, can become an issue when the DSM basic
architecture evolves. As in the case of the DSM, all applications of a certain
domain are based on a meta-model, the test suite for the DSM basic architecture
should also be based on this meta-model. In several papers it is argued that
the MBT can be utilized for generating a comprehensive test suite of
application models for the DSM basic architecture from a meta-model. This claim
is based on the capability of the MBT to generate a test suite from models, of
which meta-models essentially are. If the test suite is generated from a
meta-model, it will also always be in synch with the evolving language, thus
decreasing the maintenance effort of the test suite.
An overview of the
technique for generating a test suite for the DSM basic architecture is
depicted in Figure 5. It must be noted that it is the application models that
are generated from the meta-model by the MBT tool. As the MBT tool utilizes the
meta-model for generating the test suite, the meta-model has to be strictly
defined, i.e. the meta-model should be defined in such away that only legal
applications can be modeled. If not, the meta-model test suite also includes
test cases, i.e. application models that are not legal for the target platform.
With enough application models, the errors in the DSM basic architecture can be
noticed.
Figure 5: Testing the
Meta-model
Enabling the test suite
generation requires a DSM tool providing facilities for exporting the
meta-model into a format required by the MBT tool. In addition, the DSM tool
has to enable importing models generated by the MBT tool, whereas the MBT tool
has to enable exportation of the test suite, i.e. a set of application models,
to the DSM tool, in addition to providing facilities for generating the test
suite for the imported application models.
Tool support for the
MBT is extensive. However, many of the MBT tools are still immature but there
also exists commercial tool vendors providing more mature tools and support
when required. MaTeLo from All4Tec is for control oriented MBT. Reactis from
Reactive Systems
and T-Vec provides tools focused in embedded MBT testing. Test
Designer from Smartesting
and Conformiq Qtronic from Conformiq5 are general-purpose
solutions for the MBT. Scrutinizing the MBT tool evaluation, presented in [10],
reveals the Qtronic to be mature enough and provides open data formats for
importing and exporting the models.
The Qtronic expects the
input model to be either in
o
A similar format as the UML state machine
diagram extended with a variant of Java, which is called QML, or
o A textual
representation, where the programming language is QML.
Both of the model types
represent a model where the input and output pairs are defined. In addition,
special requirements can be defined for the inputs and outputs that guide the
test design algorithms in generating the test suite. The test suite generator
can be implemented by utilizing the provided plug-in interface.
The Generic Modeling
Environment 6 from Vanderbilt University and Metaedit+ 4.5 from Metacase7 are probably the most
well-known language work benches. Microsoft also provides a DSM tool with
Microsoft Visual Studio 2005 SDK. There are also open source tools available,
such as the Generic Eclipse Modeling System.
Metaedit+ is our choice
among the tools since, as far as we know, it is the only language work bench
providing code generator facilities with a language dedicated only for
developing code generators. This enables a rapid development of generators
compared to a situation where the code generators are developed by utilizing
the APIs of the modeling tools. In addition, Metaedit+ enables importing models
in an XML format.
o
This approach is useful at engineering
level to work through the mechanics of partitioning or integration or to decide
how much functionality to support achieves maximum return.
o
DSM can help in developing future market
models such as voice phones, audio phones, location devices, camera phones,
personal organizers, smart phones, tablets, dongles and autonomous robots.
o
DSM approach allowing product families to
develop their product in a level that clearly differentiated functionally based
cost and value metrics.
Industrial cases
constantly reveal 5-10 productivity gains when utilizing DSMLs in the software
development. Although the gains witnessed so far are great, a concerned
question raised is how to make sure the languages, i.e. meta-models and code
generators, are correct especially in the cases where the languages are
provided for the masses and for mission critical systems. Iterative and
incremental software development processes bring an additional question of test
suite maintenance. Although testing is an important means to detect errors in
the traditional software engineering, it is not well known how to test the DSM
basic architecture thoroughly.
The techniques
presented from the research papers; do not strive for the testing of the layers
of the DSM basic architecture in isolation, but to test the whole architecture.
The test technique is based on generating a comprehensive test suite of
application models by utilizing the means provided by the MBT plus generating
test suites for the generated applications. As a combination, the DSM basic
architecture will be tested thoroughly.
|
|
M Utting, A Pretschner, and B Legeard, "A
taxonomy of modelbased modelbased testing," in University of Waikato,
Department of Computer Science, Hamilton, 2006.
|
|
|
A Hartman, M Katara, and S Olvovsky, "Choosing
a test modeling language: A survey," in Haifa Verification Conference,
2006, pp. 204-218.
|
|
|
O P Puolitaival, M Luo, and T Kanstren, "On the
Properties and Selection of Model-Based Testing tool and Technique," in 1st
Workshop on on Model-based Testing in Practice, Berlin,Germany, 2008.
|
|
|
L Nachmanson, M Veanes, W Schulte, N Tillmann, and
Grieskamp, "Optimal strategies for testing nondetermistic Systems,"
in ACM SIGSOFT Internationa Symposium on SWTesting and Analysis, 2004,
pp. 55-64.
|
|
[5]
|
J
Merilinna, O Puolitaival, J
Pärssinen,Towards Model-Based Testing of Domain-Specific Modelling
Languages, VTT Technical
Research
Centre of Finland,2010.
|
|
[6]
|
Schulz, S., Honkola,
J. and Huima, A., "Towards
Model-Based Testing with Architecture Models", Engineering of
Computer Based Systems (ECBS ’07), pp. 495-502, 2007
|
|
[7]
|
Utting, M. and
Legeard, B., "Practical Model
Based Testing: A Tools Approach", Morgan Kaufmann 1st ed., ISBN:
978-0123725011 , 456p., 2006
|
|
[8]
|
E. Bringmann& A.
Kr¨amer (2008): Model-Based Testing of
Automotive Systems. In: IEEE Int’l. Conf. on Software Testing,
Verification and Validation (ICST2008), pp. 485–493
|
|
[9]
|
TeemuKanstr´en,
Olli-PekkaPuolitaival (2012): Using
Built-In Domain-Specific Modeling Support to Guide Model-Based Test
Generation
|
|
[10]
|
W Prenningerand APretschner. Abstractions for model-based testing. In Electronic Notes in
Theoretical Computer Science. Proceedings of the International Workshop
on Test andAnalysis of Component Based Systems ( 2004), volume 116,pp 59–71,
Los Alamitos, CA, USA, January 2005. ElsevierScience Publishers Ltd, 2004
|
