7.6 - Dynamic Analysis and Testing

---

Definition

Unit testing is a software testing methodology performed by developers to validate that individual units of source code—typically functions, methods, or classes—function correctly in isolation.

These units represent the smallest testable parts of an application and are defined based on the programming language used. For instance, in object-oriented programming, a unit might be a class or a method, whereas, in procedural programming, it might be a function or procedure.

The primary objectives of unit testing are threefold.

1. It allows developers to identify problems early in the development process, reducing the cost and effort of fixing defects in later stages.
2. Unit tests can guide the design by encouraging modular and decoupled code, making the software easier to maintain and scale.
3. Comprehensive unit testing significantly increases code coverage, which is a measure of how much of the code is executed during testing, leading to better software reliability.

A notable challenge in unit testing is the dependence of units on other units. For example, if a function B relies on the behavior of another function A, testing B in isolation can be difficult. To address this, developers often use scaffolding techniques like mocking and stubbing, which simulate the behavior of the dependent units. By replacing the real dependencies with simulated ones, the unit under test can be exercised independently, ensuring that its behavior is evaluated without interference from other components.

This concept becomes critical in large systems, where unit dependencies are often intricate. Proper scaffolding ensures that unit tests are accurate representations of the unit’s functionality under controlled conditions, ultimately leading to more robust and defect-free software.

While unit testing focuses on isolated components, integration testing evaluates how different units or modules work together. This type of testing examines the interfaces and interactions between components, ensuring that they integrate seamlessly to form a functional system. Integration testing uncovers faults that may not be apparent during unit testing, such as:

1. Inconsistent parameter interpretation: A classic example is the Mars Climate Orbiter failure, where conflicting units (meters versus yards) caused catastrophic results.
2. Violation of assumptions about data domains: This includes issues like buffer overflows when input data exceeds the allocated memory space.
3. Unintended side effects: Parameters or shared resources can be inadvertently modified, leading to unpredictable outcomes, such as conflicts arising from temporary file usage.
4. Nonfunctional properties: Integration can also reveal unexpected performance issues, resource contention, or delays that impact the overall system.

By methodically testing the integration points, developers can identify and resolve issues early, improving the software’s robustness and reliability.

Example of Integration Testing: The Apache Web Server Bug

A real-world illustration of integration issues is found in the Apache Web Server version 2.0. The code snippet below demonstrates a problem in handling secure (HTTPS) server requests:

static void ssl_io_filter_disable(ap_filter_t *f) { 
    bio_filter_in_ctx_t *inctx = f->ctx; 
    inctx->ssl = NULL; 
    inctx->filter_ctx->pssl = NULL; 
}

Here, the function ssl_io_filter_disable disables the SSL filter for incoming secure connections. However, the code neglects to properly release the SSL structure, leading to resource leaks. This oversight caused issues when integrating the secure connection handling with the broader server functionality.

The issue was addressed in a later version with the following corrected code:

static void ssl_io_filter_disable(SSLConnRec *sslconn, ap_filter_t *f) { 
    bio_filter_in_ctx_t *inctx = f->ctx; 
    SSL_free(inctx->ssl); 
    sslconn->ssl = NULL; 
    inctx->ssl = NULL; 
    inctx->filter_ctx->pssl = NULL; 
}

In the repaired code, the SSL_free function is explicitly called to release allocated SSL resources, preventing memory leaks and ensuring proper cleanup. This fix illustrates how integration testing can uncover and resolve such critical issues.

Integration and Test Plan: Coordinating Development and Verification

The integration and test plan is a critical component of software development, typically outlined in the Design Document (DD). This plan specifies both the sequence of implementation (build plan) and the method for performing integration testing (test plan). Effective integration requires consistency between these two plans to ensure that testing is aligned with the development sequence.

The build plan determines the order in which modules are developed and integrated. The test plan complements it by detailing the approach for validating the integration, ensuring that each component interacts correctly with others. The following diagram represents the relationship between the system architecture, build plan, and test plan:

Integration Testing Strategies: From “Big Bang” to Incremental Approaches

Integration testing can be approached using various strategies, each with its strengths and limitations. The choice of strategy depends on factors like system complexity, development workflow, and resources available.

Big Bang Integration Testing

The big bang approach involves testing only after all modules have been integrated. However, this is often considered inadequate and unreliable for most modern software systems.

Advantages	Disadvantages
Minimal requirement for test scaffolding (stubs or drivers).	Limited observability: Faults in interactions between components are harder to detect.
Simplified setup compared to incremental methods.	Poor fault localization: Diagnosing the cause of errors is challenging when all components are integrated simultaneously.
	High cost of repair: Addressing faults becomes significantly more expensive as errors propagate through the codebase over time.

Iterative and Incremental Integration Testing

In contrast to the big bang approach, iterative and incremental testing involves integrating and testing modules incrementally. This method offers better fault isolation and allows for earlier feedback, improving the reliability of the system during development.

Key variations of incremental strategies include:

1. Hierarchical Integration Testing: Based on the system’s hierarchical structure, using either:

   • Top-down integration (starting from higher-level components).
   • Bottom-up integration (starting from lower-level components).

2. Thread-based Integration Testing: Focuses on testing portions of modules that provide user-visible functionality, regardless of hierarchy.

3. Critical Module Integration Testing: Prioritizes testing modules that are central to the system’s functionality or have the highest risk.

Comparison of Integration Testing Strategies

Aspect	Top-Down	Bottom-Up	Thread-Based
Starting Point	High-level modules	Low-level modules (leaves of hierarchy)	User-visible features (functional threads)
Use of Stubs/Drivers	Requires stubs for lower-level modules	Requires drivers for each module	Reduced (as threads minimize isolation)
Progress Visibility	Gradual, as higher-level functionality is tested first	Limited until higher-level integration	High, with features visible early
Integration Plan Complexity	Moderate (hierarchical structure)	Moderate (hierarchical structure)	High (features span multiple modules)
Testing Focus	Structural hierarchy	Structural hierarchy	User functionality

Both strategies are valuable in specific contexts. Bottom-up integration is well-suited for systems where lower-level modules form the foundation for functionality, while thread-based integration is ideal for projects where delivering user-visible features early is a priority. By understanding the strengths and limitations of each approach, developers can choose the strategy that aligns best with their project’s requirements.

Integration Testing: Top-Down

The top-down integration strategy starts with the highest-level modules, progressing toward the lower levels. This method aligns well with use-case-driven designs, where higher-level interfaces, such as command-line interfaces (CLI) or REST APIs, are tested first.

1. Initial Testing:

   • The top-level module is tested using test drivers that simulate user inputs.
   • Lower-level modules are represented by stubs, which mimic their behavior without full implementation.

2. Incremental Testing:

   • As lower-level modules become ready according to the build plan, they replace their corresponding stubs.
   • This process gradually increases the functionality that can be tested, reducing reliance on stubs over time.

3. Full System Testing: Once all modules are integrated, the entire system is tested to validate its complete functionality.

Benefits of Top-Down Integration	Challenges of Top-Down Integration
Early identification of design flaws at higher levels.	Requires stubs for unimplemented lower-level modules, increasing initial overhead.
Gradual increase in system functionality during testing.	Lower-level functionality remains untested until later stages.
Facilitates testing of high-priority features early in the development lifecycle.

Integration Testing: Bottom-Up Strategy

The bottom-up integration strategy begins with the testing of the lowest-level modules in the system hierarchy, often referred to as the “leaves” of the “uses” hierarchy. This method builds upwards, integrating and testing higher-level modules as lower-level modules are completed and verified.

1. No Need for Stubs: Unlike the top-down approach, stubs (which simulate the behavior of higher-level modules) are unnecessary in bottom-up testing because the process starts from the bottom-most modules.
2. Use of Drivers:
   • Drivers, which simulate the behavior of calling modules, are essential in bottom-up testing.
   • Each module may require its own driver, similar to unit testing, making this approach resource-intensive in terms of driver development.
3. Module Replacement: As new modules are developed, they can replace their corresponding drivers. This process ensures that testing becomes progressively more realistic.
4. Working Subsystems: Bottom-up integration often results in several working subsystems being created during the testing phase. These subsystems can operate independently before being integrated into the final system.
5. Final Integration: Once all modules are integrated, the complete system is tested to validate its overall functionality and behavior.

Advantages of Bottom-Up Integration	Challenges of Bottom-Up Integration
Testing can begin early with completed low-level modules, allowing for the verification of foundational functionalities.	Requires significant effort to develop and maintain drivers for low-level modules.
Working subsystems can provide incremental feedback and validation.	User-visible functionality may not be testable until higher-level modules are integrated.

Integration Testing: Thread-Based Strategy

The thread-based integration strategy focuses on integrating and testing modules in functional threads, where a thread represents a set of modules that collectively implement a user-visible feature or function.

1. Focus on Features: Each thread is designed to deliver a complete feature visible to end users or stakeholders, offering a tangible sense of progress.
2. Minimized Test Scaffolding: Compared to hierarchical strategies, thread-based integration often requires fewer drivers and stubs. Modules within the thread are integrated as soon as they are available, reducing the need for simulated components.
3. Complex Integration Plan: Planning the integration process in threads can be more complex, as it involves identifying which modules contribute to specific features and determining the order in which they should be developed and tested.
4. Incremental Progress: With each thread integrated, stakeholders can observe tangible functionality, fostering confidence in the system’s development.

Advantages of Thread-Based Integration	Challenges of Thread-Based Integration
Maximizes visible progress to users and stakeholders by delivering functional features early.	Requires careful planning to define threads and manage dependencies between them.
Reduces reliance on stubs and drivers, as modules within threads are often interdependent.	May necessitate additional coordination to ensure that threads align with the overall system design.

Integration Testing: Critical Modules Strategy

The critical modules integration strategy focuses on prioritizing the integration and testing of modules that carry the highest risk. These risks may stem from technical feasibility, complexity, or scheduling constraints. By addressing the most vulnerable or challenging parts of the system early, this strategy helps to mitigate potential delays and technical failures.

1. Risk Assessment:

   • Conduct a thorough evaluation of risks associated with the modules. These risks could be:
     • Technical Risks: Uncertainty about whether the module’s functionality can be implemented as designed.
     • Process Risks: Concerns about whether the development timeline for the module is realistic or achievable.

2. Risk-Oriented Testing:

   • Modules identified as high-risk are integrated and tested first, aligning with a broader risk-reduction approach.
   • Testing critical modules early ensures that potential challenges or “bad news” are uncovered as soon as possible, minimizing their impact on the overall project.

3. Thread-Like Approach: Although risk-oriented, this strategy often resembles the thread-based strategy, as it emphasizes delivering partial but functional results based on priority.

Advantages	Challenges
Proactive Risk Management: Addresses potential failure points early in the process.	Complex Planning: Requires a comprehensive risk assessment and prioritization process.
Informed Decision-Making: Early feedback on high-risk components allows for timely adjustments to the system design or development plan.	Dependencies: High-risk modules may depend on other parts of the system, necessitating careful planning to enable early testing.
Enhanced Confidence: By tackling the hardest parts of the system first, teams can reduce uncertainty and build confidence in the project’s feasibility.

Choosing the Right Integration Testing Strategy

The choice of an integration testing strategy depends on system complexity, project priorities, and stakeholder needs. Each strategy offers distinct advantages and trade-offs.

Simpler strategies like bottom-up and top-down work well for small systems or subsystems with clear hierarchies. Their simplicity makes them easier to plan and execute, but they may lack visibility into user-oriented progress. Advanced strategies such as thread-based and critical modules offer different advantages. The thread-based strategy focuses on delivering user-visible features, making it ideal for complex systems where stakeholder feedback is crucial. The critical modules strategy prioritizes testing modules with the highest risk, making it suitable for projects where certain components are more challenging or uncertain.

Combining strategies can be beneficial. For small subsystems, combining top-down and bottom-up approaches can streamline integration for small, hierarchical components. For larger subsystems, using a combination of thread-based and critical modules strategies is often more effective, as it balances risk reduction with visible progress. Threads can be further integrated hierarchically, employing top-down or bottom-up approaches within each thread.

A flexible approach is often necessary for projects to address their unique challenges. For example, a project might begin with a critical modules strategy to reduce risk early and then transition to a thread-based approach to deliver user-visible features. Hierarchical methods like top-down or bottom-up can then be applied to refine and integrate subsystems into a cohesive whole.

System (End-to-End) Testing

System testing, also known as end-to-end (e2e) testing, involves validating the functionality, performance, and reliability of a fully integrated system. It is conducted in a test environment that mirrors the production setting as closely as possible. This ensures the system behaves as expected under real-world conditions. The testing is usually performed by independent teams using a black-box approach, meaning they test the system without insight into its internal workings.

System testing can focus on functional aspects (verifying requirements) or non-functional ones (evaluating performance, reliability, scalability, and stress tolerance).

Common Types of System Testing

Functional Testing

Goal

The goal of functional testing is to ensure the system meets its functional requirements. It verifies that all features and functionalities described in the Requirements Analysis and Specification Document (RASD) are implemented and work as intended.

To achieve this, functional testing simulates real-world usage by following the scenarios outlined in the RASD. It compares the system’s behavior against the expected outcomes, focusing on user workflows, inputs, outputs, and system interactions. This approach ensures that the system performs as expected under typical usage conditions, providing confidence that it meets the specified requirements.

Performance Testing

Goal

The primary goal of performance testing is to identify and address bottlenecks and inefficiencies that could impact the system’s overall performance.

These issues may stem from various sources, including suboptimal algorithms, hardware limitations, or network constraints. By conducting performance testing, developers can ensure that the system operates efficiently under expected workloads.

To effectively carry out performance testing, it is essential to simulate workloads that closely match the anticipated operational demands. This involves creating realistic scenarios that the system is likely to encounter in a production environment. During these simulations, key performance indicators (KPIs) such as response time, throughput, and resource utilization are closely monitored. By analyzing these metrics, developers can identify areas where the system may be underperforming and determine potential optimizations.

Performance testing aims to achieve several critical objectives. First, it seeks to identify slow operations or inefficient processes that could degrade the user experience. By pinpointing these areas, developers can make targeted improvements to enhance performance. Additionally, performance testing helps to uncover hardware or network limitations that may necessitate scaling to meet demand. Finally, the insights gained from performance testing enable developers to optimize the system for expected usage scenarios, ensuring that it can handle the anticipated load effectively and efficiently.

Load Testing

Load testing evaluates how the system performs under varying levels of workload. It helps determine the system’s capacity and identify issues like memory leaks or resource mismanagement. The approach involves gradually increasing the workload until the system reaches its operational limits. By sustaining heavy workloads over an extended period, load testing can expose issues such as memory mismanagement or buffer overflows.

Example

Consider the code fragment:

static void ssl_io_filter_disable(ap_filter_t *f){
    bio_filter_in_ctx_t *inctx = f->ctx;
    inctx->ssl = NULL;
    inctx->filter_ctx->pssl = NULL;
}

In a load test, this piece of code could reveal memory mismanagement if repeatedly executed under heavy workloads.

Stress Testing

Stress testing ensures that the system can handle extreme conditions and recover gracefully from failures. It evaluates how the system behaves when pushed beyond its operational limits.

To conduct stress testing, the system is overloaded by exceeding its expected workload, such as doubling the number of concurrent users or HTTP connections. Additionally, resource failures are introduced, such as shutting down network ports or limiting system memory. The system’s response to these conditions and its recovery process are closely observed to ensure robustness and fault tolerance.

Real-World Example

Stress testing is closely related to chaos engineering, popularized by tools like Netflix’s Chaos Monkey. This technique involves deliberately introducing failures (e.g., shutting down random servers) to evaluate the system’s robustness and fault-tolerance.

Importance of System Testing

System testing is a critical step in the software development lifecycle. It ensures that the fully integrated system meets both functional and non-functional requirements and behaves as intended in real-world scenarios. By conducting various types of system tests, developers can identify and resolve issues early, reducing the risk of costly failures in production.