ISTQB Foundation Level Chapter 5: Managing the Test Activities

What this chapter covers

Test planning

A test plan documents the means and schedule for achieving test objectives, helps confirm that testing will meet established criteria, serves as a communication tool with stakeholders, and shows whether testing will follow the existing test policy and test strategy, or explains why it will not. You should be able to give examples of what goes into a test plan, which is a K2-level objective.

Typical test plan content includes the context of testing (scope, objectives, test basis), assumptions and constraints, stakeholders and their roles, communication arrangements, a risk register covering both product and project risks, the test approach (test levels, test types, techniques, deliverables, entry and exit criteria, independence of testing, metrics, test data and environment requirements), and budget and schedule.

In iterative development, two kinds of planning happen alongside this: release planning and iteration planning. Release planning looks ahead to a product release, refines the backlog, and sets the test approach across all iterations. Testers involved in release planning help write testable user stories and acceptance criteria, take part in risk analysis, estimate test effort, and plan testing for the release. Iteration planning looks ahead to the end of a single iteration. Testers involved here take part in detailed risk analysis of user stories, assess testability, break stories into testing tasks, estimate effort for those tasks, and refine functional and non-functional aspects of what is being tested. Recognizing how a tester adds value in these settings is a K1-level objective.

Entry criteria define the preconditions for starting an activity. If they are not met, the activity is likely to be harder, slower, costlier, and riskier. Exit criteria define what has to be true for an activity to be considered complete. Both should be defined for each test level and will vary depending on test objectives. Typical entry criteria include availability of resources, availability of testware, and an acceptable initial quality level of the test object, such as all smoke tests passing. Typical exit criteria include thoroughness measures (coverage achieved, number of unresolved defects, defect density, failed test case counts) and binary yes-or-no criteria (planned tests executed, static testing performed, all defects reported, all regression tests automated). Running out of time or budget can itself be a valid exit criterion, if stakeholders accept the risk of going live without further testing. In Agile development, exit criteria are often called the Definition of Done, and entry criteria for starting work on a story are called the Definition of Ready. Comparing and contrasting entry and exit criteria is a K2-level objective.

Estimation, prioritization, and the test pyramid

Test effort estimation predicts how much test-related work is needed to meet test objectives. Estimates are always based on assumptions and subject to error, and large tasks are usually broken into smaller ones for more accurate estimation. The syllabus describes four estimation techniques, and applying them is a K3-level objective.

Estimation based on ratios uses historical figures from past projects to derive standard ratios, for example a development-to-test effort ratio of 3 to 2, that can be applied to a new project's expected effort. Extrapolation gathers measurements early in the current project and projects the remaining effort forward, often by averaging recent iterations, which works well in iterative SDLCs. Wideband Delphi is an iterative expert-based technique where experts estimate in isolation, discuss any large deviations, and re-estimate until they converge on a consensus; Planning Poker is a common Agile variant of this. Three-point estimation uses an optimistic estimate (a), a most likely estimate (m), and a pessimistic estimate (b), combined as E = (a + 4m + b) / 6, with the measurement error calculated as SD = (b - a) / 6.

Once test cases are assembled into test suites, they need to be arranged into a test execution schedule, and applying test case prioritization is a K3-level objective. The three main strategies are risk-based prioritization (test cases covering the most important risks run first), coverage-based prioritization (test cases achieving the highest coverage run first, with a variant called additional coverage prioritization that always picks the test case adding the most new coverage), and requirements-based prioritization (test cases tied to the highest-priority requirements run first). Dependencies between test cases and resource availability can override pure priority ordering.

The test pyramid is a model showing that tests at different levels have different granularity, and recalling its concepts is a K1-level objective. Lower layers contain small, isolated, fast tests that check a small piece of functionality, so many are needed for good coverage. Higher layers contain complex, slower, end-to-end tests that check large pieces of functionality, so fewer are needed. The original model from Cohn (2009) uses three layers (unit tests, service tests, UI tests), though other models use component, component integration, and end-to-end tests, or other test levels from Chapter 2.

The testing quadrants, defined by Brian Marick, group test levels with appropriate test types, activities, techniques, and work products for Agile development. Summarizing this model and its relationship to test levels and types is a K2-level objective. Tests are classified along two dimensions: whether they are business facing or technology facing, and whether they support the team (guide development) or critique the product (measure it against expectations). This produces four quadrants. Q1 (technology facing, support the team) covers component and component integration tests, automated and part of CI. Q2 (business facing, support the team) covers functional tests, examples, user story tests, and API testing, checking acceptance criteria. Q3 (business facing, critique the product) covers exploratory testing, usability testing, and user acceptance testing, often manual. Q4 (technology facing, critique the product) covers smoke tests and non-functional tests other than usability, usually automated.

Risk management

Risk management helps organizations increase the likelihood of meeting their objectives, improve product quality, and build stakeholder confidence. The two main risk management activities are risk analysis (risk identification plus risk assessment) and risk control (risk mitigation plus risk monitoring). A test approach driven by risk analysis and risk control is called risk-based testing.

A risk is characterized by two factors: risk likelihood (the probability it occurs) and risk impact (the consequences if it does). Together these express the risk level, and identifying risk level from likelihood and impact is a K1-level objective. The higher the risk level, the more important it is to address.

Testing is generally concerned with two types of risk, and distinguishing between them is a K2-level objective. Project risks relate to managing and controlling the project itself: organizational issues (delivery delays, inaccurate estimates, cost cutting), people issues (skill gaps, conflicts, staffing shortages), technical issues (scope creep, poor tooling), and supplier issues (third-party failures). These can affect schedule, budget, or scope. Product risks relate to product quality characteristics, such as missing or wrong functionality, incorrect calculations, poor architecture, or security vulnerabilities. These can lead to user dissatisfaction, loss of revenue or trust, high maintenance costs, and in extreme cases physical harm.

Product risk analysis aims to provide awareness of product risk so test effort can be focused to minimize residual risk, ideally starting early in the SDLC. Risk identification generates a comprehensive list of risks using techniques like brainstorming, workshops, or cause-effect diagrams. Risk assessment categorizes those risks and determines their likelihood, impact, and overall level, using a quantitative approach (risk level as likelihood multiplied by impact), a qualitative approach (a risk matrix), or a mix of both. Explaining how this analysis influences thoroughness and test scope is a K2-level objective: its results help determine test scope, which test levels and types to use, which techniques and coverage to apply, how to estimate effort, how to prioritize testing, and whether non-testing activities could also reduce risk.

Product risk control covers the response to identified risks, through risk mitigation (implementing actions to reduce risk level) and risk monitoring (checking that mitigations are working and watching for emerging risks). Explaining what measures can be taken in response to analyzed risks is a K2-level objective. Beyond testing, response options include risk acceptance, risk transfer, or a contingency plan. Mitigation actions through testing include selecting testers with the right experience for a given risk type, applying an appropriate level of independence, performing reviews and static analysis, applying suitable techniques and coverage levels, applying test types that address the affected quality characteristics, and performing dynamic testing including regression testing.

Test monitoring, test control, and test completion

Test monitoring gathers information about testing to assess progress and check whether exit criteria are being met, such as coverage of product risks, requirements, or acceptance criteria. Test control uses that information to issue control directives, which are guidance and corrective actions to keep testing effective and efficient. Examples include reprioritizing tests when a risk becomes an issue, re-evaluating entry or exit criteria after rework, adjusting the schedule for an environment delay, or adding resources where needed. Test completion gathers data from finished test activities to consolidate experience, testware, and other useful information, and happens at milestones like the end of a test level, an iteration, a project, a release, or a maintenance release.

Recalling the metrics used in testing is a K1-level objective. Common categories include project progress metrics (task completion, resource usage, effort), test progress metrics (implementation progress, environment readiness, pass/fail counts, execution time), product quality metrics (availability, response time, mean time to failure), defect metrics (counts, priorities, defect density, detection percentage), risk metrics (residual risk level), coverage metrics (requirements and code coverage), and cost metrics (cost of testing, organizational cost of quality).

Summarizing the purpose, content, and audience of test reports is a K2-level objective. Test progress reports support ongoing test control and are usually produced regularly (daily, weekly) covering the testing period, progress against schedule, impediments and workarounds, metrics, new or changed risks, and plans for the next period. Test completion reports summarize a finished test activity and include a test summary, an evaluation against the original test plan's objectives and exit criteria, deviations from the plan, impediments and workarounds, metrics, unmitigated risks and unfixed defects, and lessons learned. Test progress reporting tends to be frequent and informal within a team, while test completion reporting follows a set template and happens once.

Exemplifying how to communicate test status is a K2-level objective. Options include verbal communication, dashboards (CI/CD dashboards, task boards, burn-down charts), electronic channels like email or chat, online documentation, and formal test reports. More formal communication tends to suit distributed teams, and different stakeholders typically want different kinds of information.

Configuration management

Configuration management (CM) gives testing a discipline for identifying, controlling, and tracking work products like test plans, test cases, test scripts, test results, and test reports as configuration items. Summarizing how CM supports testing is a K2-level objective. For a complex configuration item, such as a test environment, CM records what it consists of, the relationships between its parts, and its versions. Once a configuration item is approved for testing it becomes a baseline, and from then on can only change through a formal change control process, with a record kept of what changed so a previous baseline can be restored to reproduce earlier results. To properly support testing, CM ensures all configuration items, including individual test items, are uniquely identified, version controlled, tracked for changes, and linked to each other so traceability holds throughout the test process, and that documentation and software items are referenced unambiguously. CI/CD pipelines typically include automated configuration management as part of the DevOps pipeline.

Defect management

Since finding defects is a core test objective, an established defect management process matters. Reported anomalies might turn out to be genuine defects or something else, like a false-positive result or a change request, and this gets resolved as part of handling the report. At minimum, the process needs a workflow for handling anomalies from discovery to closure, plus rules for classifying them: logging, analyzing and classifying, deciding on a response (fix it, leave it, and so on), and closing the report. The same approach is recommended for defects found through static testing.

Defect reports serve three purposes: giving those resolving the issue enough information to do so, tracking the quality of the work product, and surfacing ideas for improving the development and test process. Preparing a defect report is a K3-level objective. A typical defect report logged during dynamic testing includes a unique identifier, a short summary title, the date observed and who reported it (including their role), identification of the test object and test environment, the context (which test case or activity was running, SDLC phase, technique or data used), a description of the failure detailed enough to reproduce and resolve it (including test steps, logs, screenshots, or recordings as relevant), expected versus actual results, severity (degree of impact), priority to fix, status (open, deferred, duplicate, awaiting fix, awaiting confirmation testing, re-opened, closed, rejected), and references such as the related test case. Some fields, like identifier, date, author, and initial status, are often filled in automatically by defect management tools.

Chapter 5 at a glance