Testing strategies and quality assurance notes — Unit 4

Testing is the process of executing a program with the intent of finding errors. It is a critical element of software quality assurance and represents the ultimate review of specification, design, and code generation.

⇒1.1 The Goal of Testing

A common misconception is that testing proves the absence of errors. Testing can only show the presence of errors, not their absence. A successful test is one that uncovers an as-yet-undiscovered error.

⇒1.2 Verification vs Validation (Revisited)

• Verification: 'Are we building the product right?' (Does the software match the SRS and design documents?)
• Validation: 'Are we building the right product?' (Does the software meet the customer's actual business needs?)

⇒2.1 Key Principles

• Exhaustive testing is impossible: You cannot test every single possible input combination. Testing must be risk-driven and prioritized.
• Defect Clustering: A small number of modules usually contain the vast majority of the defects (Pareto Principle / 80-20 rule). Focus testing efforts heavily on these complex modules.
• The Pesticide Paradox: If you run the exact same tests over and over, eventually they will stop finding new bugs (because the code is fixed for those specific tests). Tests must be continually reviewed and updated.
• Testing is context-dependent: You test a safety-critical medical system very differently than an e-commerce website.
• Absence of errors is a fallacy: Finding and fixing defects doesn't help if the system built is unusable and doesn't fulfill the user's needs and expectations.

White-box testing (or glass-box testing) is a test case design philosophy that uses the control structure described as part of component-level design to derive test cases. The tester knows exactly how the code works internally.

⇒3.1 Objectives

Using white-box methods, the software engineer guarantees that:

• All independent paths within a module have been exercised at least once.
• All logical decisions have been exercised on their true and false sides.
• All loops have been executed at their boundaries and within their operational bounds.
• Internal data structures have been exercised to ensure their validity.

Basis path testing is a white-box technique first proposed by Tom McCabe. It enables the test case designer to derive a logical complexity measure of a procedural design and use this measure as a guide for defining a basis set of execution paths.

⇒4.1 Flow Graphs

The code is translated into a directed graph. Nodes represent statements, and edges represent control flow (loops, if-statements).

⇒4.2 Cyclomatic Complexity (V(G))

A software metric that provides a quantitative measure of the logical complexity of a program. It defines the number of independent paths through the code. That number is exactly the upper bound for the number of tests you must write to ensure all statements have been executed at least once.

• Formula 1: `V(G) = Edges - Nodes + 2`
• Formula 2: `V(G) = Number of enclosed regions + 1`
• Formula 3: `V(G) = Number of predicate (condition) nodes + 1`

Black-box testing focuses on the functional requirements of the software. It ignores the internal control structure. The tester treats the software as a black box, providing inputs and observing outputs.

⇒5.1 Objectives

Black-box testing attempts to find errors in the following categories:

• Incorrect or missing functions.
• Interface errors.
• Errors in data structures or external database access.
• Behavior or performance errors.
• Initialization and termination errors.

Unlike white-box testing, which is performed early in the testing process, black-box testing tends to be applied during later stages (Integration, System, Acceptance testing).

⇒6.1 Equivalence Partitioning

Divides the input domain into classes of data from which test cases can be derived. The assumption is that if one input in a partition causes an error, all inputs in that partition will cause the same error.

Example: A password must be 6 to 12 characters. Partitions: Less than 6 (Invalid), 6 to 12 (Valid), More than 12 (Invalid). You only need 3 tests, not thousands.

⇒6.2 Boundary Value Analysis

Experience shows that errors tend to occur at the boundaries of an input domain rather than in the center. BVA focuses on these edges.

Example: For the 6 to 12 character password, test exactly at the boundaries: length 5, length 6, length 12, length 13.

Unit testing focuses verification effort on the smallest unit of software design—the software component or module. It uses white-box testing techniques heavily.

⇒7.1 The Unit Test Environment

Because a component is not a stand-alone program, driver and/or stub software must be developed.

• Driver: A 'main program' that accepts test case data, passes such data to the component being tested, and prints the relevant results.
• Stub (Mock): A dummy subprogram used to replace modules that are subordinate to (called by) the component being tested. The stub provides hardcoded return values to simulate the missing module.

Unit testing is performed by the developer who wrote the code, often using automated frameworks like JUnit, pytest, or Jest.

If all modules work individually (Unit Testing passed), why test them together? Because data can be lost across an interface, one module can have an inadvertent adverse effect on another, and global data structures can present problems.

⇒8.1 Big Bang Integration

Wait until all modules are finished, throw them all together at once, and test. (Highly discouraged because isolating the cause of an error is almost impossible).

⇒8.2 Incremental Integration

Construct and test the program in small increments. Errors are easier to isolate.

• Top-Down: Start with the main UI/control module. Subordinate modules are replaced with Stubs. Gradually replace stubs with real modules.
• Bottom-Up: Start with the lowest-level utility modules. Use Drivers to test them. Gradually combine them into larger clusters, replacing drivers with the real parent modules.

⇒9.1 System Testing

Once the entire software is integrated, it must be tested against the external systems it interacts with (hardware, databases, other APIs). System testing validates the software against the original non-functional requirements.

• Recovery Testing: Forces the software to fail and verifies that recovery is properly performed.
• Security Testing: Verifies that protection mechanisms built into a system will protect it from improper penetration.
• Stress/Load Testing: Executes a system in a manner that demands resources in abnormal quantity or frequency to find the breaking point.

⇒9.2 Acceptance Testing

Conducted by the end-user (customer) to decide whether to accept the system.

• Alpha Testing: Conducted at the developer's site by a customer. The developer observes the user and records errors.
• Beta Testing: Conducted at the customer's site by end-users. The developer is not present. Users report issues back to the developers.

Every time a new module is added, or a bug is fixed, the software changes. This change can unintentionally break existing, previously working functionality.

⇒10.1 The Regression Process

Regression testing is the re-execution of some subset of tests that have already been conducted to ensure that changes have not propagated unintended side effects.

• Because regression testing can involve thousands of tests and must be run every time the code changes, it must be automated.
• A regression test suite contains three classes of tests: A representative sample of tests that exercise all software functions, tests focusing on the newly changed components, and tests focusing on components highly dependent on the changed components.

Manual testing is slow, expensive, and prone to human error. Modern software engineering relies heavily on automated testing.

⇒11.1 The Test Automation Pyramid

• Unit Tests (Base): Fast, cheap, and highly isolated. You should have thousands of these. They test individual functions.
• Integration/Service Tests (Middle): Slower and more complex. They test APIs, database connections, and component interactions. You should have hundreds of these.
• UI/End-to-End Tests (Top): Extremely slow, fragile, and expensive. They boot up a real browser and click through the application like a real user (e.g., using Selenium or Cypress). You should have relatively few of these.

An 'Ice Cream Cone' anti-pattern occurs when a team has massive amounts of UI tests but very few Unit tests. This leads to extremely slow build times and flaky test suites.

Testing and debugging are not the same thing.

⇒12.1 The Difference

• Testing: The systematic process of finding the symptom (the error). It proves that a bug exists.
• Debugging: The localized process of finding the cause of the symptom, and fixing the code. It is an art rather than a systemic process.

⇒12.2 Debugging Strategies

• Brute Force: The most common and least efficient. Adding `print` statements everywhere or stepping through every line in a debugger.
• Backtracking: Starting from the place where the symptom was observed, trace the source code backward manually until the site of the cause is found. (Only works for small programs).
• Cause Elimination (Scientific Method): Formulate a hypothesis about the cause. Design a specific test to prove or disprove the hypothesis. Repeat until isolated.

Quality is defined as 'conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software.'

⇒13.1 What is SQA?

SQA encompasses the entire software development process. It includes:

• An SQA process (defining standards and procedures).
• Specific quality assurance and quality control tasks (including formal technical reviews and multi-tiered testing).
• Effective software engineering practice (methods and tools).
• Control of all software documentation and changes.
• A procedure to ensure compliance with standards.

Testing is only one part of SQA (Quality Control). Quality Assurance is proactive; Quality Control is reactive.

To control quality, we must measure it. Software metrics provide quantitative data about the product and the process.

⇒14.1 Defect Metrics

• Defect Density: The number of defects found per KLOC (Kilo Lines of Code) or per Function Point. Identifies highly buggy modules.
• Defect Removal Efficiency (DRE): `DRE = E / (E + D)`, where `E` is errors found before delivery, and `D` is defects found after delivery. A high DRE (close to 1) means your testing process is excellent.

⇒14.2 Complexity Metrics

Tracking Cyclomatic Complexity. Modules with a complexity greater than 10 are significantly harder to test and maintain, and have a statistically higher chance of containing defects.

An FTR (often called an Inspection or Code Review) is the most effective quality control activity. Studies show that peer reviews find up to 75% of all defects before a single line of code is ever executed in a test.

⇒15.1 The Review Process

• A review involves 3 to 5 people (author, reviewers, and a moderator).
• Advance preparation is required (reviewers must read the code/document before the meeting).
• The meeting focuses on the product, not the producer. (Do not attack the author).
• The goal is to find errors, not to fix them during the meeting.

⇒15.2 Why Reviews are Superior to Testing

Testing only shows symptoms, forcing a developer to spend hours debugging to find the cause. A code review points directly to the cause (the bad line of code) instantly. Reviews can also check for maintainability, style, and efficiency—things automated tests cannot easily check.

ISO 9000 describes quality assurance elements in generic terms that can be applied to any business. ISO 9001 is the specific standard for quality systems in software development.

⇒16.1 ISO 9001 Certification

To become registered to ISO 9001, a company's quality system and operations are scrutinized by third-party auditors. The company must:

• Establish a documented quality management system.
• Document how they meet every requirement of the standard.
• Actually follow their own documented procedures.
• Provide mechanisms for continuous improvement.

ISO certification does not guarantee that the software is bug-free. It simply guarantees that the company has a standard, repeatable process for building software and handling defects when they occur.

Software reliability is defined as the probability of failure-free operation of a computer program in a specified environment for a specified time.

⇒17.1 Hardware vs Software Reliability

Hardware reliability decreases over time due to physical wear and tear (the bathtub curve). Software does not wear out. Software failures are due to design or implementation flaws. When a software bug is fixed, its reliability theoretically increases, though the fix itself may introduce new bugs.

⇒17.2 Metrics

• MTTF (Mean Time To Failure): Average time between failures.
• MTTR (Mean Time To Repair): Average time it takes to fix a bug once found.
• MTBF (Mean Time Between Failures): `MTBF = MTTF + MTTR`
• Availability: `MTTF / (MTTF + MTTR)`

Testing object-oriented software requires a different approach than traditional procedural software because of encapsulation, inheritance, and polymorphism.

⇒18.1 Unit Testing in OO

The 'unit' is no longer a single function; it is the entire Class. Because methods within a class interact and modify the shared internal state of the object, testing a single method in isolation is often meaningless. State-based testing is required.

⇒18.2 Impact of Inheritance

If a base class is thoroughly tested, does a derived class need to be retested? Yes. The derived class might inherit a method but use it in a new context, or it might override a method, breaking the assumptions of the base class. All inherited methods that interact with newly overridden methods must be retested.

Security testing verifies that information and systems are protected from unauthorized access, use, disclosure, disruption, modification, or destruction.

⇒19.1 Vulnerability Scanning and Penetration Testing

• Vulnerability Scanning: Automated tools scan the code and the live application for known security flaws (e.g., outdated libraries, open ports, SQL injection points).
• Penetration Testing (Ethical Hacking): Security experts manually attempt to break into the system using the same techniques malicious hackers use. This uncovers complex logical flaws that automated scanners miss.

⇒19.2 The OWASP Top 10

Security testing heavily focuses on preventing the most critical web application security risks, such as Injection (SQLi), Broken Authentication, Sensitive Data Exposure, and Cross-Site Scripting (XSS).

In modern software engineering, testing is not a phase that happens at the end; it happens continuously.

⇒20.1 Continuous Integration

Developers merge their code changes into a central repository several times a day. Every time code is pushed, a CI server (like Jenkins, GitHub Actions) automatically triggers the build process and runs the entire automated test suite (Unit and Integration tests).

• If a test fails, the build is marked 'broken', and the developer must fix it immediately.
• This ensures that the main codebase is always in a deployable, fully tested state.

⇒20.2 Shift-Left Testing

A DevOps philosophy meaning testing is moved to the 'left' (earlier) in the development lifecycle. Developers write tests before code (TDD), security scans run automatically during coding (DevSecOps), and quality becomes everyone's responsibility, not just the QA team's.