This is an article I wrote for ReQtest over an year ago. To make it short: I wasn’t satisfied how they handled the publication of my article.
For a long time I’ve thought about putting it to my blog and now finally decided to actually do it. It’s quite old article, but I want it to be published as it was intended.
Few Words About Bugs
Bug reports are one of the main work products we produce as testers. As for definition of a bug, I like James Bach’s definition: “A bug is something that bugs somebody who matters.” Here I will though focus on the ones that are found by investigating the product. This is not the only aspect though. I recommend that you watch Rob Sabourin’s Let’s Test keynote from last year. It will reveal, how flexible the definition of a bug, can actually be.
It’s good to note, that finding bugs is not the only thing I do as a tester. I also provide information that will help other people made better choices. Recently, I wrote a blog post (Visibility of Our Work) that goes through the additional value we can provide as testers, besides finding bugs.
Let’s get back to finding bugs. When we find a bug and reach the point where it is filed to defect management system (or any other way of informing a programmer and possible other stakeholders about the bug), we have hopefully gone through many points that have a crucial effect on the probability of this specific bug being fixed. BBST Bug Advocacy course slides mentions a quote from the book Testing Computer Software (3.0) that highlights, how the amount of bugs we find, is not the main aspect, we should focus on.
The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the right bugs fixed.
In this article, I will try to outline some of the phases that increase the probability of bugs being fixed, with respect to paying attention to figuring out if something is a problem (this relates to TCS quote above). In the end I’ll also show an example of what my bug report would look like. I will write a bug report of a bug I found on XMind (mind mapping tool – www.xmind.net).
Is it a problem?
A crucial phase that we need to go through before we can start filing a bug is that we need to decide if we have actually found a bug. In other words we ask the question “Is it a problem?” One way to answer this question is using oracles. An oracle is something that will help us to possibly recognize if there is a problem. For example, a requirements document. When we notice that the product behaves in a way that is not consistent with how it is defined in requirements, we decide that there is a bug in the product. There are a lot of other oracles, like live oracles, which might be a product owner, or an experienced user that will confirm to you that some specific observation is a bug.
Michael Bolton wrote a great blog post where he went through the high-level list of different oracles that we might use to recognize if there is a problem. The list is as follows
a) A feeling like confusion or annoyance.
b) A desirable consistency between related things.
c) A person whose opinion matters.
d) An opinion held by a person who matters.
e) A disagreement among people who matter.
f) A reference document with useful information.
g) A known good example output.
h) A known bad example output.
i) A process or tool by which output is checked.
j) A process or tool that helps a tester identify patterns.
One important thing to remember about oracles is that they are fallible. Let’s say, for example, I feel strongly that there is a problem, which is a threat to the overall quality of the product. Though, according to the requirements, the product is working correctly. I might still consult another oracle (like the product owner or someone else whose opinion matters) and explain how I feel that the current behavior is a threat to the overall quality of the product. If I will not succeed in convincing the stakeholders about the threat to overall quality, I have at least given my opinion.
After we have recognized and concluded, that we have a bug in our hands, we move on to reporting it (there are probably different ways to handle this, but personally I like to solve the “Is it a problem?” issue, before I report the bug). It’s good to note though, that while some bugs can be reported instantly, others require more information gathering.
Isolate and amplify
I often want to understand (isolate) what the bug actually is in its most understandable form, before I move on to reporting it or discussing it more with stakeholders. My goal is that the programmer can clearly envision the bug in their head when reading my bug report. To achieve this, we can try to place the programmer to the seat of the end-user, and try to make her see the behavior with the eyes of the end-user.
In my current project I am monitoring real time server logging when I test our system (logs are located on a Unix server and logging is done with the help of the tail -f command). In the logs I can often see information, which will help the programmers to troubleshoot the problem. They can also access the logs by themselves but I will do my best to save their time by providing events that I’ve witnessed in logs while testing. Let’s remember though, that the logs you provide are not necessarily related to the problem you have reported. This is a risk, which needs to be acknowledged.
Besides isolating I also want to try to amplify the bug, if possible. As an example, I tested some time ago a message field that could be inserted text. It also had a functionality, which allowed you to attach files. I noticed that when you attach a PDF file to the message, the amount of characters, that the system allows you to insert to the message field, got smaller. Further investigations revealed, that if I increased the size of the file, the amount of characters available, decreased even more. Eventually, if I attached large enough file, I was not able to insert text to the message field. This information is good to mention, when you report the bug. It will help us understand the impact on the end-user, in greater detail.
We often understand more about the possible impact to the overall quality of our product when we have invested in increasing our knowledge about the bug. Also, the more we understand the impact on the overall quality, the more we can describe the possible consequences of this specific bug.
In order to get the bugs fixed though, we need to explain our findings. Great testers excel in their ability, to describe the behavior, which they have witnessed.
Testers need to be highly skillful at describing bugs
Bug reports are one of the main artifacts of our work and they also establish our credibility. Carelessly written bug reports (e.g. programmer can’t understand what the problem is, it’s hard to reproduce, there are a lot of grammar mistakes) give an impression that we are not professional testers. In some cases bug reports are our main way of being in contact with the programmers. Because of these reasons I think that we need to be extremely good at describing bugs and in the end writing bug reports.
Personally, when writing a bug report I focus on bug reports on information about configuration (e.g. browser, OS), providing steps for reproducing and attaching any files (e.g. screenshot, video, log files) that I think will help in reproducing and in the end fixing the bug. However, I can’t spend too much time on the report. Even if I would, there’s always a risk that programmer will not be able to understand my bug report and will fail to reproduce the bug. I try to minimize this risk, but also at the same time I realize that the time I spend on writing the report is time away from finding new information.
I’ve seen too many examples in my short career where there is a constant discussion between the tester and programmer regarding the additional information that the programmer needs in order to fix, reproduce or even understand the bug. Not always, but often these moments can be handled by writing the bug report well enough in the first place. That’s why I highly encourage people to do it professionally in the first place. Figure out as much (not too much) information as you can in reasonable time (requires judgment). Then tie the pieces together in a way that will maximize the quality of the information delivered. You can also ask for feedback from programmers, other testers or any of the stakeholders that handle your bug reports. Programmers could inform you, if they feel that some information is missing from the reports. You could also ask just general comments from other testers, regarding the overall quality of your bug reports.
Because I personally love examples and find them often the most convenient way of understanding something, I’ll show an example of a bug report I wrote regarding a bug I found while using XMind (mind mapping tool). It’s not a model of a flawless bug report but rather an example of how I currently write bug reports in my project.
One of the reasons I wrote this post is that I want to make my work public. By publishing my thoughts and my example bug report I will get feedback that will help improve myself. That’s also one of the ways of becoming a software testing expert. Surround yourself with bright and diverse people who are not afraid to criticize your ideas. That way you get feedback that truly helps you become better at your work.
Bug report example – XMind
Before I show the bug report, I’ll let you see the error I saw when using XMind:
Bug ID #1: User receives java.lang.NullPointerException when creating boundary for subtopic
Configuration
Windows Vista Enterprise – SP2 (32 bit)
XMind for Windows- v3.3.0
Summary
I received java.lang.NullPointerException –error when I created boundary two times in a row for subtopic, and then clicked left mouse button.
Steps to reproduce
- Select the [Central Topic]
- Create [Subtopic] with Tabulator or via menu
- Select the created [Main Topic 1]
- Press Ctrl+B+B
- Click the left mouse button when cursor is shaped as cross.
Expected
Selecting a subtopic, then clicking [Ctrl] down and two times [B] button, should not do anything when you press the [B] second time.
Actual
When I insert the boundary second time for subtopic and click left mouse button, when the cursor is shaped as cross, I get a java.lang.NullPointerException. I was not able to reveal any side effect, but user will feel confused, as s/he does not know if it has an effect on the behavior of XMind.
Attachments
I attached screenshot of the error message I received and also a video that shows how I reproduced the problem.
Conclusion
As you’ve seen, finding and reporting bugs, is not a trivial matter. It involves various challenging phases and those can easily vary between contexts. I’m constantly trying to find the balance between the effort I put to polishing the information I provide, and on the other hand, to finding new information. Knowledge about oracles and tools can help you to improve the information you provide. In the end it’s though you, who will make the final decision regarding the information, that is included or not included.
Seek ways of getting feedback for your bug reports, or ask if you could give feedback to others. Sharpen your skills of observing, recognizing and describing problems. If you don’t know where to start from, I strongly recommend BBST Bug Advocacy course (requires that BBST Foundations is passed) that is provided by Association for Software Testing. I will personally participate that (BBST Bug Advocacy) on November this year.
Also, feel free to comment (either here or on Twitter) my bug report or anything else that I’ve written on this article/post. Constructive feedback helps me move forward as a tester.
Flow of Testing 