(A) goal-oriented testing approach for system of systems using task models

Abstract

System of systems (SoS) is a collection of systems that integrates a number of independent self-contained systems to accomplish their SoS-level goals, which cannot be accomplished by a single constituent system. Because SoS involves multiple and complex interactions among constituent systems for accomplishing SoS-level goals, an effective goal-oriented testing approach for distinguishing SoS interactions is necessary. However, previous research on goal-oriented testing requires manual processes, and does not focus on the sequence of system interactions. Therefore, this paper presents an effective goal-oriented testing approach for SoS, based on message interactions among constituent systems. Our approach uses SoS message interaction trace as a test input and expected message interaction trace as an expected output. To generate the expected trace, we proposed a goal-oriented test oracle model along with the XML-based oracle generator to demonstrate our generation process systematically. To allow testers to determine the effectiveness and efficiency, we proposed three-level comparison criteria to compare the simulation trace to the expected trace differently. To support our approach, we also developed a simulation-based testing tool along with the goal-oriented test oracle generator that produces the expected trace. In the evaluation, we demonstrated the effectiveness and efficiency of using the supporting tool we developed, and we concluded that our approach is more effective and saves more time than manually testing goals and effectively distinguishes the SoS interaction faults. In addition, we demonstrated that the proposed three-level comparison criteria shows the difference in effectiveness and time efficiency, so we concluded that testers need to choose between the testing effectiveness and testing time as a tradeoff. As a case study, we applied a Mass Casualty Incident (MCI) management and response plan to demonstrate our approach.