Blog

What is software testing? Types, principles and process

Software testing finds defects and judges quality before users meet them. The main types, the seven principles, the test process, and what a pass cannot prove.

Software testing is the work of checking software to find its defects and to judge whether it is ready to release. Part of that work runs the program and compares what it does with what it should do. Part of it runs nothing at all: someone reads the requirements and spots two that contradict each other, or a tool scans the code and flags a variable that is never set. Both kinds count as testing, and a project that only does the first finds its problems later than it needs to.

This guide is written for people who commission or manage software rather than write tests. It covers what testing includes, how the many named types relate to each other, the seven principles testers work by, the steps of a test process, and the questions Google shows beside this search. Definitions follow ISTQB’s Certified Tester Foundation Level syllabus (version 4.0.1), the document behind its Foundation Level certification.

What software testing covers

ISTQB describes testing as a set of activities with two aims: discovering defects, and evaluating the quality of what the project has produced. The things being tested are called test objects, and code is only one of them. Requirements, user stories, designs and test scripts can all be tested, mostly by review.

Three distinctions from the syllabus clear up most of the confusion around the word:

  • Verification and validation. Verification asks whether the system meets its specified requirements. Validation asks whether it meets the needs of its users and other stakeholders in the setting where it will actually run. Testing does both.
  • Testing and debugging. A test exposes a failure, or a review finds a defect directly. Debugging is the separate job of reproducing the failure, locating its cause and fixing it. Afterward a confirmation test repeats the check that failed, to show the fix worked.
  • Testing and quality assurance. The terms get used as if they meant the same thing. ISTQB keeps them apart: testing looks at the product and is corrective, a form of quality control, while QA looks at the process the team follows and is preventive. QA is the responsibility of everyone on a project.

The vocabulary for what goes wrong is precise too. A person makes an error, which leaves a defect (a bug or fault) in the code or a document, and a defect that gets executed may produce a failure. Some defects fail every time they run, some only under particular conditions, and some never fail at all.

What a passing test tells you

The syllabus lists the usual objectives of testing. Among them: evaluating requirements, designs and code; causing failures and finding defects; checking that specified requirements, including contractual, legal and regulatory ones, have been met; and giving the people who make decisions, such as whether to release, the information they need. For a buyer, that last objective is the practical one. Test results are the evidence a release decision rests on.

That evidence has limits. The first of the seven principles below says testing can show that defects are present but cannot prove they are absent. A clean test run means the checks someone wrote found nothing. It says nothing about the checks nobody wrote. So the useful question about a test suite is what it checks. The number of tests, and the share that passed, say less.

Code coverage, the share of the code that tests execute, is often quoted as the answer. The authors of Software Engineering at Google (chapter 11) warn against reading too much into it: coverage records that a line was run, not whether anything checked what that line did. A high coverage figure with weak checks behind it proves little.

What is testing and its types?

Lists of “types of software testing” tend to mix things that are not alternatives to each other, because test types are named along different lines. Four questions sort them out, and every test has an answer to each.

QuestionThe answersAn example
Does the software run?Static testing examines work products without running them: reviews and static analysis. Dynamic testing runs the software.Reading the requirements for contradictions is static. Placing an order in a test copy of a store is dynamic.
What is being checked?Functional testing checks what the system does. Non-functional testing checks how well it does it: speed, security, usability and the rest.“Is the order total right?” is functional. “Does checkout stay quick with 500 shoppers at once?” is non-functional.
Where do the test cases come from?Black-box tests come from the specification, with no view of the code. White-box tests come from the code and its structure. Experience-based tests come from the tester’s knowledge.Testing a form against its written rules is black-box. Writing a test for each branch of an if statement is white-box.
How much of the product is under test?The test level: unit, integration, system or acceptance.A single price calculation is unit level. The whole store, end to end, is system level.

A fifth label describes why a test is run rather than what it is. After any change, confirmation testing shows that a fix worked and regression testing shows that nothing else broke. Both can happen at any level. The scale question has a guide of its own: levels of testing, with one feature followed through all four.

So a load test on a store’s checkout is dynamic, non-functional, usually black-box, and run at system level. Naming all four answers is a quick way to see what a test plan covers and what it leaves out.

Four questions that place any software test, answered for a load test on a store's checkout. Does the software run? Yes, so it is dynamic, not static. What is being checked? How well it does it, so non-functional. Where do the tests come from? The specification, not the code, so usually black-box. How much is under test? The whole store, end to end, so system level.

Static testing finds what running the code cannot

Static testing gets less attention than it deserves. ISTQB lists the defects it finds more easily or more cheaply than running the software: ambiguities, gaps and contradictions in requirements; inefficient database structures; undeclared variables and unreachable code; interfaces whose parameters do not match; some kinds of security weakness; and acceptance criteria that no test covers. It also reaches code paths that rarely run, which dynamic tests can easily miss. A contradiction caught in a specification costs an edit. The same contradiction caught after release costs a fix, a retest and a deployment.

Non-functional testing checks the qualities users notice

The syllabus sorts non-functional checks by the quality characteristics of the ISO/IEC 25010 standard: performance efficiency, compatibility, usability (also called interaction capability), reliability, security, maintainability, portability (also called flexibility) and safety. It warns that non-functional defects found late can seriously threaten a project, and that some of these checks need a special environment, such as a usability lab. Many of them can ride on functional tests: the same steps run, with a non-functional constraint added to the check, such as a time limit.

What is white-box testing?

White-box testing designs tests from the inside of the software: the code, its architecture, and the paths that work and data take through it. Its aim is to exercise enough of that structure. The two basic forms in the syllabus are statement testing, where the goal is for every executable statement to run at least once, and branch testing, where every outcome of every decision is taken: both sides of each if, every case of a switch, each loop’s choice to continue or stop. Reaching full branch coverage always gives full statement coverage as well. The reverse does not hold.

White-box testing finds defects even when the specification is vague or out of date, because it works from what was actually built. Its blind spot is the mirror image. If a requirement was never implemented, there is no code for a white-box test to find, and the omission goes unnoticed. That is why it sits beside black-box testing rather than replacing it.

Black-box techniques work from the specification instead. Take a rule that orders of $50 or more ship free. Boundary value analysis tests the values on each side of that limit, $49.99 and $50.00, because a limit coded one step too high or too low is a common mistake. Equivalence partitioning, decision tables and state transition testing are the other black-box techniques the syllabus teaches.

What is software performance testing?

Performance testing is non-functional testing of how quickly software responds and how much work it handles with the resources it has. Microsoft’s Azure Well-Architected Framework describes four kinds:

KindWhat it asksWhat it shows
Load testingDoes the system cope with expected traffic at normal and peak levels?A baseline, capacity limits, whether scaling works
Stress testingWhere does it break, and how?Maximum capacity, failure modes, how it recovers
Spike testingCan it absorb a sudden surge?How fast autoscaling reacts, whether queues hold, whether it degrades gracefully
Endurance or soak testingWhat goes wrong only after a long period of use?Memory leaks, exhausted resources, connection pool problems

Microsoft’s advice is to begin with basic load testing and add the other kinds as risks appear. For websites there is one more distinction. Lab tests run in controlled conditions before launch; field data comes from real visitors once traffic builds. A page can pass in the lab and still be slow on a cheap phone over a weak connection, which is why the three-second rule for websites is better judged from real visits than from one quick test.

Manual and automated testing

In manual testing a person follows the steps and judges the result. In automated testing a tool runs scripted tests and compares actual results with expected ones. The syllabus lists what automation buys: less repetitive work (regression runs above all), consistency, faster feedback, and more time for testers to design better tests. It is just as direct about the risks: expecting too much of a tool, underestimating the work of maintaining scripts, automating checks that a person would do better, and trusting the tool in place of critical thinking.

Some testing stays human by nature. Exploratory testing, where a tester designs and runs tests while learning how the product behaves, finds the problems nobody thought to script. Usability testing and acceptance testing by real users are often manual too.

Automated suites have a failure mode of their own: the flaky test, one that sometimes fails when nothing is wrong. Google’s book does the arithmetic. If each test has a 0.1 percent chance of failing for no reason, and a team runs 10,000 tests a day, someone investigates about ten false alarms a day. The authors note that as flakiness approaches 1 percent, tests begin to lose their value, because engineers stop reacting to failures. For the same reason Google sorts its tests by size rather than by name: small tests run in a single process, medium tests on a single machine, and large tests anywhere. Smaller tests are almost always faster and more predictable, whatever they cover.

Test automation tools fall into a few families. Unit test frameworks such as JUnit for Java, pytest for Python and Jest for JavaScript run tests written alongside the code. Browser automation tools such as Selenium, Playwright and Cypress drive a real browser through a user’s journey. Load testing tools such as Apache JMeter and k6 generate traffic. Static analysis tools read code without running it. These are examples of each kind, not a recommendation; the right choice depends on the stack and the team.

The seven principles of software testing

The syllabus sets out seven principles that apply to all testing. Each has a practical meaning for anyone paying for software.

PrincipleWhat it means when you commission software
1. Testing shows the presence of defects, not their absenceA clean run is evidence, not proof. Ask what the tests check.
2. Exhaustive testing is impossibleEvery input in every order cannot be tried. Effort goes where a failure would cost most, such as payments and personal data.
3. Early testing saves time and moneyReviewing requirements and designs before code is written is the cheapest testing a project buys.
4. Defects cluster togetherA few components usually hold most of the defects. A module that keeps failing deserves more testing, not less.
5. Tests wear outThe same tests repeated find fewer new defects over time, so tests and test data need updating. Automated regression runs are the exception, where repetition is the point.
6. Testing is context dependentA marketing site and a payment system need different testing. A single fixed recipe is a warning sign.
7. The absence-of-defects fallacySoftware can pass every test against its specification and still fail its users, which is why validation matters as much as verification.

The seventh is the one most often missed. A team can test every written requirement, fix every defect it finds, and still deliver a product nobody wants, because the specification was wrong. Early prototypes, reviews with real users and acceptance testing exist to catch that.

Software testing phases: the test process

ISTQB describes a test process as groups of activities. They look sequential, but the syllabus notes that they often run iteratively or in parallel and are tailored to each project.

  1. Planning. Define the objectives of testing and choose an approach that fits the constraints.
  2. Monitoring and control. Compare progress against the plan throughout, and act when it drifts.
  3. Analysis. Study the requirements and other test basis to decide what to test, ranked by risk.
  4. Design. Turn those conditions into test cases, and work out the test data and environment they need. Analysis decides what to test; design decides how.
  5. Implementation. Build the tests, the data and the environment, manual or automated, and arrange them into a run order.
  6. Execution. Run the tests, compare actual with expected results, log the outcomes and report anomalies.
  7. Completion. At a milestone such as a release, record unresolved defects as backlog items, archive what will be useful again, and report the results to stakeholders.

For a client, the visible outputs are the plan, the defect reports and the completion report. Asking to see a sample of each before work starts says a lot about how a team tests.

What is agile testing?

In agile development, testing happens inside every iteration rather than in a phase after the build. Three ideas from the syllabus describe how.

The first is the whole-team approach, which comes from Extreme Programming. Anyone with the right skills can pick up any task, and everyone is responsible for quality. Testers work with business representatives on acceptance tests and with developers on the test strategy and what to automate. The syllabus adds that this does not suit every setting: safety-critical work may need testers who are independent of the team.

The second is test-first development. In test-driven development (TDD), developers write a test, then the code that passes it, then tidy both. Acceptance test-driven development (ATDD) derives tests from acceptance criteria before the feature is built, and behavior-driven development (BDD) writes the expected behavior as test cases in simple natural language, usually in a Given/When/Then form, that stakeholders can read. All three move testing earlier, the idea known as shift left.

The third is the testing quadrants, a model from Brian Marick that sorts tests by two questions: do they face the business or the technology, and do they guide the team or critique the product?

Guides the teamCritiques the product
Business facingQ2: functional tests, examples, user story tests, user experience prototypes, API tests and simulations, all checking acceptance criteria, manual or automatedQ3: exploratory testing, usability testing, user acceptance testing, often manual
Technology facingQ1: component and component integration tests, automated and part of continuous integrationQ4: smoke tests and non-functional tests other than usability, often automated

A plan with nothing in one quadrant has a gap worth asking about.

What do software test engineers do?

The syllabus describes two roles. The test management role owns the test process: planning, monitoring, control and completion. The testing role owns the technical work: analysis, design, implementation and execution. One person can hold both, and in an agile team some management tasks fall to the team itself.

Day to day, a test engineer reads requirements and user stories looking for gaps, designs test cases, prepares data and environments, writes and maintains automated tests, runs exploratory sessions, and writes defect reports precise enough that a developer can reproduce the failure. When a fix lands, they run the check again. The syllabus names the skills that matter: thoroughness, curiosity, analytical thinking, domain knowledge and communication. It also notes that testers are often the bearers of bad news, so reporting defects without making them sound like personal criticism is part of the job.

How testing works on our builds

QA is the name of the testing role on our delivery teams, and it tests each release before it reaches users. Before any change ships, it is code reviewed and its dependencies are scanned for known vulnerabilities. Beyond that, the checks follow the build:

  • Websites. For sites built through our custom web development service, each page carries a performance budget, a ceiling on page weight and load time, and it must pass that budget in lab tests before launch.
  • SaaS products. Ahead of every release, automated tests act as one customer account (a tenant) and attempt to see, alter or export records belonging to a different one.
  • Education platforms. Load testing against the start-of-term peak happens before launch.
  • Live systems we maintain. When dependency scanning flags a security patch or an outdated library, the update is applied and tested before release, and restoring from backup is tested on a schedule.

If you are planning a build and want to know what it should be tested for, the first call is a good place to ask.

The full service behind this post: Custom Web Development Services.

Ready to scope your project?

Send us two sentences about what you need. We reply within one business day with the next step and any questions we need answered first.

Get a Quote