Aerospace and Defense Digest - February 2004

Using Model Coverage Analysis To Improve The Controls Development Process

By William Aldrich
The MathWorks

Abstract

Graphical simulation tools improve the effectiveness and efficiency of embedded controls development by providing an abstract view of a component or system in a visual paradigm. These modeling tools have unambiguous semantics and can execute designs. Coverage analysis can indicate the completeness and consistency of a set of requirements. Recent advances in code synthesis allow very efficient code to be produced directly from the graphical models. When generated code is used without modification, model-based coverage analysis is more effective than the same analysis performed on generated code. Coverage analysis within a modeling tool should be consistent with the executable semantics of the tool and should be presented in a form that is easily associated with the graphical diagrams.

2. Introduction

The context of this paper is the development process used to create large control systems of software and hardware such as those found in aerospace and automotive applications. The typical design process for a large system consists of a sequence of design and implementation steps that rely on documents produced at the output of a preceding step. This process is frequently depicted on a V-shaped diagram to indicate the narrowing scope of the successive design steps followed by the increasingly wider scope of integration and testing steps, as shown in Figure 1.

This process requires a considerable effort before any results can be verified. As documented in 1, the goals of software testing, starting at the lower right of the V diagram, include verifying the consistency of system and high-level requirements. When high-level requirements change there can be a considerable change in the underlying software. Without techniques to test requirements early in the design process, errors propagate from one step to the next and become costly to fix.

Figure 1: A "V" diagram of system development. The shape indicates the increasingly narrow scope of the requirement and design phase followed by the increasingly wider scope of the integration and testing phases.

A variety of software and hardware tools aid these design steps. Requirements capture and traceability tools are designed to improve the accuracy of requirements and reduce the involved effort when they change. During software testing, code coverage tools provide a measure of test completeness. Finally, logic analyzers are used to unobtrusively capture the software behavior on the final target. This paper focuses on graphical modeling tools used in the heart of the design process and demonstrates how the roles of these tools are enhanced with integrated coverage analysis.

The rest of the paper is organized as follows. Section 3 describes block diagram and state diagram model representations. Section 4 describes how these models are used in the system development process. Section 5 introduces several types of code coverage that serve as the basis for model coverage. Section 6 describes exactly how coverage is implemented in a modeling tool and discusses some special considerations that can result. Section 7 demonstrates the ways that coverage analysis extends modeling tools and aids the development process. Finally, Section 8 investigates the relationship between code coverage and model coverage. This is particularly significant when code is automatically generated from the model.

3. Graphical Models

This section discusses two fundamental graphical modeling paradigms: block diagrams and state diagrams. Both of these paradigms are familiar to many programmers who have never used modeling tools since they are often used as a means of documentation and specification.

The block diagram examples in this paper were created in Simulink®, a block diagram tool for dynamic simulation. The state diagrams were created with Stateflow®, a state diagram tool that works with Simulink.

A diagram by itself has very limited value if its interpretation is not precise. In addition to diagrams, a useful modeling tool must have a set of semantics that precisely and uniquely determine how the diagram is interpreted and it must have the ability to execute the diagram to observe behavior. The models described in this paper are all executable and deterministic.

3.1. Block Diagrams

Block diagrams emphasize the flow of data between units of computation. The lines on a block diagram are signals that represent data. Each line shows the direction from a source block that assigns values to one or more destination blocks that read those values. Visually, a path through a set of signals and blocks indicates a sequence of computations and intermediate results. A representative block diagram for a P.I. controller is shown below in Figure 2.

Figure 2: A block diagram of a P.I. controller. This diagram shows how a set of input values are processed to compute an output.

The functional behavior of a block diagram is determined by the set of blocks that compose the diagram, the connections between those blocks, and the underlying semantics of the tool that determine the frequency and ordering of each block execution. The basic blocks available in the diagram should have clear and unambiguous functionality. Ideally, common block patterns should be encapsulated into a single block so a user can create elaborate functionality with a minimum of blocks. The key to a useful set of blocks is that each one should have an easily identifiable behavior based on its appearance. Some basic Simulink blocks are shown in Figure 3. To handle scalability, most block diagram tools allow groups of blocks and connections to be organized into a component that is represented as a single block, called a subsystem. This hierarchical representation improves clarity of a design by abstracting details at different levels.

Figure 3: Examples of atomic blocks used to create diagrams. A practical block diagram tool that can support sophisticated designs needs to provide control flow constructs. Examples of Simulink control flow constructs are shown below in Figure 4. In the first two rows are examples of explicit control flow where the execution of a subsytem causes all the contained blocks to execute. In the bottom row are examples of implicit control flow internal to each block.

Figure 4: Examples of blocks and constructs that incorporate control flow.

3.2. State Diagrams

State diagrams emphasize the logical behavior of a system. Traditionally, state diagrams have been used to explain how a system with a finite set of modes, or states, can change from one mode or state to another. In Figure 5 rectangles with rounded corners represent the states in a system. The directed lines from one state to another are called transitions. These indicate the ability to change from one state to another. Transitions are usually labeled with the conditions that must be satisfied before the transition can be taken. Several transitions can originate or terminate on the same state.

Figure 5: The basic elements of a state diagram.

State diagrams are useful for visualizing logical paths through a series of states. A state diagram can help to clarify the exact sequence of logic that is needed to change from one state to another, particularly when each state has a small number of transitions that originate or terminate on it. Actions associated with states and transitions enable the state diagram to interact with its external environment.

As a design tool, classic state diagrams are limited by scalability problems. Extended state diagrams, like those supported in Stateflow, overcome these limitations with constructs that handle hierarchy, parallelism, and transition re-use. Hierarchy allows states to be grouped together into a superstate so that common transitions only need to be drawn once. Parallelism allows the diagram to be partitioned into several parallel states, each with its own hierarchy of active substate(s). Parallelism prevents the state explosion that results when independent modes or attributes have numerous possible combinations.

A portion of an extended state diagram for an autopilot is shown in Figure 6. Hierarchy allows the states that represent the active modes of the controller to be grouped together in a natural manner. Parallelism allows this logic to be combined in the same diagram with the contents of another parallel state.

Figure 6: A state diagram showing the logic for enabling ILS glide-slope control.

4. Using Models within a Design Process

Models are used in a design process as executable specifications for the system being developed. Through simulation the model is used to predict behavior. Simulation is particularly useful when the observed behavior is the result of subtle interactions between software and hardware, as in control systems.

An executable specification is used to verify that a design will meet its requirements. These models can identify inconsistent requirements that occur when the model cannot satisfy two or more requirements simultaneously. Once the requirements are verified, the model can be used as a way to predict correct component outputs in the source code.

Useful graphical paradigms are a more natural medium for describing the design and are inherently easier to produce than textual languages. Development time is also reduced when the model is a simplification of the final system. Effective model-based specifications often ignore diagnostics, error handling, calibration support, process scheduling, and other target specific considerations.

Recent advances in code synthesis enable very efficient code to be produced directly from graphical models. When automatically generated code is used without modification the graphical model serves as an implementation. The graphical environment can be thought of as a language equivalent to any other programming language.

Using models as implementations is motivated by the advantages of graphical representations over source code. Rapidly generated specifications can be used as a starting point for developing model-based implementations. The process of successive model refinements and eventual automatic code generation represents a complete model-based development process.

5. Code Coverage Analysis

Coverage analysis is used to dynamically analyze the way that a program executes. It provides a measure of the completeness of testing based on the code structure, known as white box testing. A code fragment, shown in Figure 7 below, will clarify the meaning of each coverage metric.

The simplest form of coverage is called statement coverage, sometimes abbreviated as C1. Full statement coverage indicates that every statement in a program has executed at least once. The limitation of statement coverage is that it does not completely analyze the control flow constructs within a program. For example, when an if statement does not have a matching else, full coverage only requires that the if evaluate to true. In the example a single test case can achieve complete statement coverage. By setting x=15 and y=2 every statement executes.

x = 2* a;
z = 3;
if (x>5 && y<4) {
     z = 10;
}
out = 1;
if (y*x==30){
      out = 4;
}

Figure 7: A code fragment to illustrate coverage metrics.

A more rigorous form of coverage is decision coverage, sometimes abbreviated as C2. Full decision coverage indicates that each control flow point in a program has taken every possible outcome at least once. In a well-structured program, such as one written in a higher order programming language, decision coverage implies statement coverage3. In the example there are two decisions, x>5 & y<4, and y*x==30. Both of these decisions can be either true or false, so at least two test cases are needed for full decision coverage. One case sets both to true. The other sets both to false.

Decision coverage provides a good measure of completeness but it ignores the complications that result when a decision is determined by a logical expression containing the logical operators AND or OR. In this case, the Boolean inputs to logical expressions are analyzed using condition coverage. Full condition coverage indicates that every input to a logical expression, called a condition, has taken a true and a false value at least once. In the example there are three conditions, x>5, y<4 and y*x==30. Complete coverage can be achieved with two test cases by testing all the conditions as true and then all the conditions as false.

MC/DC coverage, or modified condition-decision coverage, provides an even more rigorous analysis of conditions. Full MC/DC coverage implies that each input to a logical expression has been shown to independently change the expression’s outcome while the other conditions are held constant. In the above example the line if (x>5 & y<4) will require three tests. One test will set both conditions as true, one test will just set x>5 as true and the last test will just set y<4 as true.

 

More

Subscribe:
To Subscribe to upcoming issues of The MathWorks Aerospace and Defense Digest, please send an email to digest@mathworks.com