A Embedded Software Testing Process Model

Hua-ming Qian College of Automation, Harbin Engineering University Harbin, China qianhuam@sina.com
AbstractThe characteristics of embedded systems is analyzed in detail, embedded software testing technology is introduced in host and target testing, a comprehensive embedded software testing process model is put forward with the shortcomings of software testing V-model. Software testing is imported in the stage of requirement, and the performance of the architecture of embedded software system is evaluated, the problem of performance, which is caused by the architecture and the design, will be avoided in the software realization later. At last, with the embedded software testing tools used, software testing automation is implemented, the coverage of system testing is collected, the dynamic allocation and free of memory is monitored, the memory leak is prevented. Keywords- embedded software; software testing process model; software testing; software testing automation

Chun Zheng College of Automation, Harbin Engineering University Harbin, China zhengchun@hrbeu.edu.cn 3) The different stages of development have the appropriate testing technology, a reasonable compromise is made between host-based testing and target-based testing. 4) Embedded software testing tools will be introduced in the model, embedded software test automation is realized to improve the efficiency of software testing. 5) After the defects are fixed, the domain affected should be analyzed, test cases are reasonably designed to carry out regression testing and to avoid the appearance of new defects. In view of the above requirements, this paper analyzes the characteristics of embedded systems, combined the V model of software testing process with embedded software testing technology and embedded software testing tools, the comprehensive embedded software testing process model is put forward and applied to the development of embedded electronic charts display information system, the practice shows that the model have certain practical value and five characteristics that a good embedded software testing process model should have. II.
CHARACTERISTICS OF EMBEDDED SYSTEMS

I.

INTRODUCTION

With the development of embedded software technology, the application of embedded software is continuously popular in the aviation, aerospace, marine and military security and is playing an increasingly important role in these areas. Usually, there is a strict quality requirement in these areas, once software failure happens, it will directly or indirectly affect human lives and property and the safety of the ecological environment. The correctness of embedded software functionality and performance plays a decisive role in the software quality, software testing is an important mean of software quality assurance. However, how accurately and as soon as possible to find defects in software will require an accurate software testing process model, it tells us how to carry out testing activities, what is the purpose of testing, what time to establish a test, and what is the basis to create a test, a good test model can guide our thinking, and a bad model allows us confused. A good model for embedded software testing process should have the following characteristics: 1)software testing is introduced in the phase of requirement to ensure that each of requirement is clearly and accurately described, and can be properly understood by everyone so that software defects in the requirement are found as soon as possible[1,4]. 2) the architecture of the system must be reviewed, in the process, system software performance engineering model is established to evaluate whether the system architecture of embedded software meets performance requirements (timeliness). Experience has shown that performance problem is usually caused by the basic architecture or design factors, and not caused by inefficient coding in most cases[2,3].

A. embedded characteristic Embedded software development environment is inconsistent with the runtime environment, which causes a lot of trouble to the embedded software testing. At the same time, considering the embedded software for the hardwaredependent, even if the test is fully carried out in the host environment, it can not ensure that the software will normally run in the target environment. Therefore, the embedded software is also faced with the test based on the target environment. It will not only increase the cost of testing, but also bring out the issue of the embedded software testing strategy, that is, which test should be distributed to the host environment, which test should be distributed to the target environment. B. timeliness Performance is an indicator whether the embedded software systems meet the requirement of timeliness. Response time is a required time for the request, it may be the time required to complete transaction for a single transaction; it should be endto-end time for the user tasks, the timeliness is one of the key factors to determine whether embedded software systems success.

978-1-4244-4507-3/09/$25.00 2009 IEEE

C. limited resources (memory usage) III.

SOFTWARE TESTING V MODEL AND SHORTAGE

A. software testing V model V model is the most typical test model, as shown in Fig. 1, V model was first used by Paul Rook in the late 1980's, V model is published in the UK National Computing Center literature, which aimed at improving the efficiency and effectiveness of software development. In the traditional development process, the testing process is the last stage after the requirement analysis, outline design, detailed design and coding, but this is not adequate. V model, which reflects the relationship between the test activities and analysis design, is a variant of the waterfall model of software development, from left to right, the basic development process and testing behavior are described, it clearly indicates that the testing process exists the different levels, and distinctly describes the correspondence between the testing phase and the various stages of the development process. V model illustrates that unit testing and integration testing are used to verify the program design, development personnel and testing group should validate whether the execution of program meet the requirements of software design; system testing should verify the system design and inspect whether the quality characteristics of system function and performance meet design indicators, software validate testing and acceptance testing are carried out to determine whether the realization of software meet user requirement by testing engineers and users[5,7] .

Software development is thought the demand, design, coding to be a series of serial activities in V model. Similarly, a linear fore-and-aft relationship is maintained in software development and testing, there is no round consideration for the parallelism of software development and software testing. The regression testing is not explicitly illustrated in V model, the strategy of regression testing isnt be established, how to determine the scope of regression testing to avoid the introduction of new defects.
THE TECHNOLOGY OF EMBEDDED SOFTWARE TESTING

IV.

The difference between embedded software testing and general software testing is the method of embedded software testing, embedded software testing include the test based on host environment and the test based on target machine environment. A reasonable compromise should be made on between host-based testing and target-based testing. The more funds and time will be consumed by the test based on the target machine, the smaller cost will be spent on host-based testing. The current trend is to shift more test to the host-based environment, and embedded software testing tools should be used to realize the automate testing, based on my practical work experience, which test should be distributed to the host environment and which test should be distributed to the target environment will be illustrated. A. host-based testing a) static analysis of embedded software Combined the level of security of embedded software systems with MISAC + +2008, AV C + + coding rules, a list of rules check is established, code rules inspection and quality evaluation are implemented in virtue of software static analysis tool, the statement violating the code rules will be modified. b) embedded software unit testing Unit testing is aiming at the inspection work of accuracy for the smallest unit of software design - program modules, The objective is to found a variety of defects that may exist in interior of modules, according to own practical experience in unit testing, during the coding, the code involved in arithmetic operations and logic operations should be packed in a function to improve the code testability, test engineers can develop automated test case script, driver programs and stubs will be automatically generated through the automated testing tool to automate unit testing and improve the efficiency of the test. c) embedded software integration testing In the basis of unit testing, all the modules having passed unit test will integrated through outline design specification and detailed design specification, the objective of integration testing is to identify the defect of data and message transmission between the modules and the logic error after the system is integrated.

Figure 1. software testing V model

B. shortage of software testing V model The test is thought as a last phase after requirement analysis, outline design, detailed design and coding, the mainly objective is to find the defects aimed at the procedures, nevertheless ,the problem hidden in the phase of requirement analysis can be found until acceptance testing, and the performance issues caused by the system architecture design can be found until system testing only , at this time ,it can only be resolved by optimizing the code, but ,optimizing the code to improve system performance is far less than architecture for the impact of performance.

B. target-based testing a) embedded software validation testing Based on the software requirement specification, test cases are designed to cover all the functionality and performance, and to verify whether software functionality and performance and other characteristics are consistent with the users requirements. b) embedded software system testing Embedded software is instrumented through the use of embedded software test tool, and compiled to download to the target machine to run, in a real system running environment, comprehensive test cases are designed to cover all the function and performance, with all the test cases implemented, the coverage of system testing is collected, the coverage is analyzed to verify whether the coverage for function and performance is complete, at the same time, the use of memory is dynamically monitored to prevent the memory leak. V. A COMPREHENSIVE EMBEDDED SOFTWARE TESTING PROCESS MODEL A good embedded software test model should have five characteristics described in the introduction, this paper presents a comprehensive embedded software testing process model, as shown in Fig. 2, the specific testing process is as follows: A. the implementation of embedded software testing model a) software requirement assessment Software testing activities should be started in requirement stage, the software requirement specification should be reviewed, every function must be described in software requirement specification in detail, every requirement must be clear and precise to ensure that everyone who reads it can understand in the same way, and the accuracy, completeness, consistency, testability, feasibility, necessity, accuracy and traceability of requirement is ensured, the defects hidden in the requirement are discovered as soon as possible, and at the same time, based on software requirement specification , the functional test cases are designed and evaluated. b) system architecture design review In the stage of system design architecture review, based on the system architecture ,software performance engineering model of pre-development system is established , a systematic, quantitative approach is used to verify whether the architecture design of the system meet performance requirement, experience has shown that performance issue is due to the basic architecture or design factors in most cases, and not caused by inefficient coding, the betterment made in the phase of architecture and design is far more than the realization phase for improving the embedded software performance. Once software achieved, it is extremely difficult to solve the problem due to the system architecture and design, and at the same time, based on system design specification, the system test cases are designed and evaluated.

c) software outline design review In the process of software outline design review, an application system is divided into a number of tasks and the responsibility of each task is defined, this is an art. If a embedded software application system is designed well, and the division of its tasks should have the following characteristics: the responsibility of each task is clear, the information transmission and signal synchronization mutex between tasks should be simple and clear, a reasonable communication mechanism is chosen, the logic relationship should be nature, the definition of priority should have sufficient reason. At the same time, based on software outline design specification, the integration test cases are designed and evaluated. d) software detailed design evaluation In the assessment of detailed design, it is simply impossible to ignore the details of the realization of programs and coding. If the wrong choice is made in the realization or coding, such as the choice of data structure and the realization of program is not the best, there will be many hidden dangers, which result in a good architecture and design having an adverse performance. At the same time, the unit test cases are designed and evaluated, automated unit test case script is developed. e) embedded software unit and integration testing In the phase of coding, automated unit testing environment is built, unit test cases script is called to automate unit testing, when the modules having passed unit testing will be integrated, checking whether the data through the module interface will be lost, whether the transmission of information between tasks is correct, whether the priority setting of interruption is reasonable, and whether the transmission of information between interruption of services and tasks is correct or not. f) embedded software validation and system testing Function test is implemented to verify whether software functionality and performance satisfy users requirement, system testing is carried out in the real system test environment, first of all, the source codes are instrumented through the use of software testing tool, the target file is the compiled and downloaded to the target machine via web, the functionality and performance test cases are implemented, system test coverage is collected to analyze whether the coverage data meet the testing standards, and cover all functionality and performance, additional test cases are designed to achieve the testing standards. At the same time, the memory usage is dynamically monitored to prevent the memory leak, memory leak has great impact on the performance of embedded software. g) embedded software regression testing Each of the defects, found in all testing phase, must be fixed, the domain affected by the modified code must be analyzed, the scope of regression testing must be determined, regression testing cases are designed and assessed until all the defects have been repaired[6]

User requirement

requirement anlysis

System architecture design

Outline design

Detailed design

Software requirement specification and evaluation

System architecture design performance evaluation

Software outline design specification and evaluation

Software Detailed design specification and evaluation

Coding realizition

Validation test case design and evaluation

system test case design and evaluation

integration test case design and evaluation

unit test case design / test case script development and evaluation

acceptance testing

Validation testing

System testing

integration testing

Unit testing

bug Target--based testing fixed CodeTEST The collection of System testing coverage and dynamically monitoring memory allocation Host-based testing

the domain affected by defects regression test case design and evaluation regression testing

Testbed/Tbrun Static analysis rule checker Automation unit and integration testing

closed

Figure 2. a comprehensive embedded software testing process model

B.

the advantages of embedded software testing process model

a) the defects hidden in software requirement can be found as soon as possible In the early stage of software life cycle, software requirement is assessed to ensure the accuracy, completeness, consistency and traceability of software requirement, software defects are found as soon as possible to reduce maintenance cost. b) software performance issues can be found as early as possible Software performance engineering model is established to quantitatively verify whether the system architecture design meet performance requirement, software performance issues can be exposed as early as possible. Experience has illustrated that, once software have realized, it is would be extremely difficult to solve the problem as a result of system architecture design. c) the domain of regression testing is analyzed The incidence for the entire system is analyzed due to the defects, the scope of regression testing must be determined, regression testing cases are designed and assessed until all the defects have been repaired, there is no new defect introduced.

d) embedded software test automation Embedded software testing tools CodeTEST and Testbed/Tbrun will be introduced into the test model to improve the efficiency and quality of software testing. VI.
THE PRACTICAL APPLICATION OF EMBEDDED SOFTWARE TESTING PROCESS MODEL

The model is used in the development of embedded electronic chart software system, which includes charts show, charts calculating, charts correction, marine operations, and marine planning function and so on, the host platform of development environment is Pentium D 3.0G/512M/120G, the target platform is Pentium 933M/64M/10G, the development tool is the Tornado 2.2 integrated development environment, the communication is through the Ethernet network, based on the tftp, the graphics development tool is the Zinc 6.0, the target machine operating system is Vxworks 5.4, the graphics user interface of system sees Fig. 3. First of all, the requirement is analyzed and verified, software performance engineering model is established to assess whether the system architecture is reasonable or not, the performance problem of the system is exposed as soon as possible, in the phase of outline design, the tasks is reasonably divided, the system is divided into six tasks, in the coding stage, automated unit testing and integration testing are implemented.

dynamically monitored, knowing well that which function the memory was assigned in, how much is the size of allocation, in order to check system memory leak, memory leak has seriously affected on the performance of the system. In this paper, with the testing process model used, the test results are as follows: a) two defects hidden in requirement were found in advance, the performance of system architecture is quantitatively analyzed. b) in the process of static analysis, unit and integration testing, a total of 307 defects seriously violate code rules, which mainly include the comparison of floating point number, divisor may is 0, the number extracted may be less than 0, the operator = = is falsely written = in the while cycle, array exceed and so on. c) in the process of validation testing, the 20 functional defects are discovered, which mainly include the defects of charts calculating function, the text input box without the protection of data, there is no difference between the standard display and the expansion shows and so on. d) in the process of system testing, with test cases implemented, the statement coverage is 88%, which did not meet the standard 100%, the reason is the existence of some redundant code and exception handling memory allocation and file operation. At the same time, the system memory leak is monitored, the concrete data is shown in Table 1.
MEMORY LEAK

The detailed test cases are designed to cover all the functionality and performance, the source codes are instrumented by the testing tool, and compiled to download to the target machine, the test cases about the functionality and performance are implemented, with CodeTEST and Testbed software testing tools used, the system test coverage is collected, Let us clearly know which codes have been implemented, which codes have not been covered, why are codes not covered? Is not redundant code, how to improve test cases to cover the codes not covered, in order to prevent the code not covered which exist hidden dangers and reduce the reliability of the system, at the same time, memory usage is
TABLE I.

Figure 3. embedded electronic chart software system graphics user interface

Name of module Memory leak Bytes

charts correction 63

marine planning 1881

marine operations 29

charts calculating 520

hydrological information 20120

marine datum 3600

VII. CONCLUSION The shortage of software testing process V model is analyzed in this paper, combined the characteristics of embedded systems with embedded software testing methods and testing tools, a new comprehensive embedded software test process model is put forward, which is successfully used to test embedded electronic chart software system. Practice shows that the test model is effective, with the test model used, the defects hidden in the requirement can be found as soon as possible. At the same time, in the stage of system architecture design, the performance of system software is assessed to prevent to solve the performance problems due to the architecture design when the system has been achieved, in order to speed up the software development cycle, at the same time, embedded software testing tools are successfully introduced, the code coverage implemented is clearly known, which code have been covered to ensure the safety of the code. The dynamic memory allocation and use are monitored to prevent the memory leak, and the quality of the software is further enhanced, the model has a practical value.
[1] [2]

REFERENCES Dustin ,Elfriede. Effective Software Testing [M].Boston, Pearson Education, pp. 150, 2003. Connie U. Smith, Lloyd G. Williams, Performance Solutions A Practical Guide to Creating Responsive, Scalable Software. Boston:Pearson Education,2003. Hu Jin, Liang-Yin Chen, Ling-Ming Zeng, Bao-Lin Li, Performance Testing Based on Time Complexity Analysis for Embedded Software, The 2008 International Conference on Embedded Software and Systems, pp.243-247, 2008. Li Yi, Chen Yiping An effective software testing model, Computer Engineering and Applications. No. 10 , pp. 114115, 2004. Liu Shuping, Pang Ling, The research of V model in testing embedded software, International Conference on Computer Science and Information Technology. pp. 463 466,2008. Phyllis G. Frankl , Elaine J. Weyuker, A Safe, Efficient Regression Test Selection Technique, ACM Transactions on Software Engineering and Methodology. Vol. 6, No. 2, pp. 173210 ,April 1997. Liu Chunlu, Huang Zihe, Chen Luping, Software evaluating engineering tutorial, Beijing: tsing Hua University Press pp. 1350, 2005.

[3]

[4]

[5]

[6]

[7]