Here in the Central Architecture function for government, we often get asked how to approach various testing tasks and methodologies.
Key questions that come up are: what is exploratory testing and how should we do it?
Exploratory testing helps quality analysts and others involved in the testing field ensure systems and applications work for their users. Exploratory testing is often misunderstood as an approach but there a number of pointers you can follow to ensure you’re on the right track. Here are my top 10.
1) Focus on goals
Exploratory testing helps you exercise a system like a user while actively looking to identify bugs. Focus on these goals to maximise the value of your tests. Remember that exploratory testing can complement other testing methods that examine systems in different ways.
2) Plan your tests but don’t script them
You are not exploratory testing if you are following a script. However, exploratory testing doesn’t mean testing without control or good practice. You do need to plan your testing in advance.
Planning helps you clarify specific aspects of a system you want to examine including special data requirements or system needs.
3) Don’t aim to test too much
The aim of exploratory testing is not coverage - it’s to find the defects and issues in a system that you won’t find through other forms of testing. Typically these defects arise through edge cases of testing, but that doesn’t mean they are low impact.
Quite often edge case defects will have a high level of severity even if they are less likely to arise. This is due to the nature of exploratory testing – it focuses on the parts of a system that are away from the normal usage pattern and are less likely to be well tested.
4) Exploratory testing is a skilled activity
Typically, exploratory testing needs a greater level of testing skill and experience than other testing techniques. You are explicitly using and relying on the skill of the person doing the testing, so make sure they are the best you can get.
5) Keep a clear record of what you did
Don’t document something if there’s no value in doing so but do keep a clear record of what you did, how you did it and what you found out. This will give you an indication of how effective the session was, allowing you to optimise the process in the future.
When you need to formally document the outcome of a session, your report may include:
- a test charter outlining the scope and goals of your test
- a list of features tested
- notes on how the testing was conducted
- notes on any bugs found
- a list of issues (questions and concerns about the product or project that have arisen during testing)
- extra materials used to support testing
You can also report how much of the Exploratory Testing session you spent on each different activity, including:
- creating and executing tests
- investigating and reporting bugs
- setting up the session
6) Use exploratory testing alongside automated testing
Automated testing checks a system performs as the development team thinks it should, according to the identified needs. Exploratory testing checks the system performs as a user might expect.
It’s important to coordinate both testing types to ensure the values of both exploratory testing and automated testing are realised, e.g. when exploratory testing finds a defect, you can add an automated test to stop the issue from happening again.
7) Performance and non-functional testing can be exploratory
Performance and non-functional testing can also be exploratory, e.g. tracking an increasing load on a system, or measuring the time it takes a use case to fully complete from start to finish.
While performance and other non-functional testing hold a limited scope, they can still be exploratory. Whether you are tracking an increasing load on a system or measuring the time it takes an application to complete from start to finish, you can do so with the mindset of a user. You outline the scope of your testing in your test charter.
8) Choose the Exploratory Testing techniques that meet your needs
You’ll need to:
- research current exploratory testing techniques
- establish a common understanding of the techniques you’ll use and share this with all stakeholders
- decide on your scope (including timescales, process, and when exploratory testing will be used)
- ensure that the system is built to support exploratory testing, e.g. build functionality in slices that can be tested from start to finish as early as possible.
9) Don’t confuse exploratory testing with user acceptance testing
User acceptance is a testing activity that can (and probably should) be performed in an exploratory testing way. Don’t confuse the way of working with the type of test required.
Exploratory testing isn’t a phase of the development lifecycle; it’s an approach and technique that you should use throughout the life of the project. As soon as your bits of a system under development have a testable flow through them, you must test that flow. Exploratory techniques are a good way of doing that quickly and with high value output.
10) Don’t go mad on tools
There are many different tools you can use to do exploratory testing, from fully automated video capture and logging tools, to planning tools, to walls full of process diagrams and feature descriptions with accompanying timebox plans. However, the only tool you really need to do exploratory testing is a pen and some paper.
Tools are nice to have, but don’t put off starting exploratory testing if you don’t have access to them.
Do you have any tips around exploratory testing you would like to share? Add them in the comments box and let everyone know.
Don’t forget to sign up to the Government Technology blog.
1 comment
Comment by Mike Finn posted on
Great article.
Specially liked the item number 4.
Its actually not easy to do ET and way too often its considered as 'free for all' type of testing.
But as you pointed out in the article its far from it, and indeed requires structure and skills.
I often also find people needing support in form of common test objectives for generic elements.
There are usually lot more test objectives on common elements than testers might think, and that is where you need to have skills and experience. For example, considering simple email address collection field, there are some many different test objectives to test, far more than people may initially think. Search more with #TestObjectives or even simply check RFC5322.
Thanks again for great post!