Think About The Pain

Photo credit: wolfgangfoto (Creative Commons)

Photo credit: wolfgangfoto (Creative Commons)

I watched a while ago a great video from Michael Skok. There he mentioned interesting thing about new products. That’s Gain & Pain. Which were defined as:

Gain – The gain delivered to customer

Pain – The pain and cost for the customer to adopt

I thought about this from the perspective of testing and it felt like a good heuristic to consider while testing.

Think about the pain of adopting a product or feature.

It can be re-planning business process, providing training and training material related to new product, more work for people who are handling orders (because new product has helped in increasing the amount of orders), more work for people who are at help desk (to support people who are struggling with the new product), recruiting more people, investing on software licenses, users realizing that they’re not the subject-matter experts anymore for the new product, migrating existing production data into the new product…

And so on.

There’s probably often (if not always) pain when new products (or features) are being adopted. Try to think whose lives the product (or feature) will touch. Not just the end customers, but also the people (e.g. admins) who work behind the curtains.

Podcasts and Videos for Testers (and Others)

My friend asked me last week if I could share him any software development or testing related podcasts that he could listen while cycling. I said that I would reply to him when I find a good moment. I realized though that I should probably share those via blog post so others might benefit from it also. That’s the reason I’m writing this post now.

There’s just one catch. I haven’t listened that much podcasts. At least lately. On the other hand, I’ve listened a lot conference talks from Youtube & Vimeo, which might be useful also. Let’s see what I can come up with.

Podcasts

This Week in Software Testing

One of the first testing related podcasts that I found couple years ago, was This Week in Software Testing by Matthew Heusser & Michael Larsen. It would appear that some of the episodes can be downloaded without Software Test Pro registration, but at least my personal favorite (episode #4 with Jerry Weinberg) required registration. I know that people don’t like registrations, but I do think that Jerry’s episode alone is worth it on this case. There’s a small snippet of that episode on my past blog post.

AB Testing

Alan Page and Brent Jensen have had podcast episodes so far and I’ve listened few of them. Those have been interesting – especially if you want to have a peek inside Microsoft.

Testcast

This is a podcast that’s hosted by Trish Khoo and Bruce McLeod. I’ve known about this podcast for a long time, but haven’t yet started listening it.

Testing Podcast

I’ve only heard one podcast related to this – and it was one with Richard Bradshaw about Automation in Testing

Testing in the Pub

Based on the recommendation of Stephen Janaway, I added this here. Considering that it’s hosted by Stephen & Dan Ashby, I would expect it to be worth listening.

How To Think About Science

I’m not actually sure if this is a podcast or what, but nevertheless interesting series of episodes about science. I originally found this because of the recommendation by Michael Bolton.

Hanselminutes

This seemed like a good software development podcast based on one episode (#65 Scott Hanselman with Martin Fowler & David Heinemeier Hanson) I recently listened.

————————————–

These are the ones that I’ve had some experience with. There’s probably a lot more podcasts out there, especially if we expand the scope to software development in general.

Audio

Audio of several Let’s Test (2013) sessions

Me and my colleague recorded the audio of several sessions when we were at Let’s Test last year. Those can be found from my Dropbox shared folder. There’s also a blog post about that and it explains the background a bit more.

Audio of 4 Turku Agile Day (2014) sessions

I was giving a speak on Turku Agile Day this year and also recorded several sessions. After I had asked permission for sharing those, I put them on my Dropbox shared folder.

Videos

Besides audio & podcasts, I’ve personally focused more on Youtube and Vimeo videos from various conferences. I listen those while driving to work and back home. Below are many of the talks that I’ve enjoyed the most. I’ve listened all of them and can recommend those for that reason. I categorized them by the speaker as that’s probably the easiest way for me to do the categorization.

J. B. Rainsberger

Practical Tools for Playing Well with Others

7 minutes, 26 seconds

Extreme Personal Finance

Integrated Tests Are A Scam

The Economics of Software Design

Dave Snowden

Managing under conditions of uncertainty

How not to manage complexity

Tacit Knowledge

James Bach

Open Lecture by James Bach on Software Testing

Becoming a Software Testing Expert

Jon Bach

Telling Your Exploratory Story

Michael Bolton

Things Could Get Worse: Ideas About Regression Testing

How to get what you want from Testing

Cem Kaner

BBST Lectures

Zeger Van Hese

Artful Testing

Martin Fowler / Kent BeckDavid Heinemeier Hansson

Is TDD dead? [part I]

Is TDD dead? [part II]

Is TDD dead? [part III]

Is TDD dead? [part IV]

Is TDD dead? [part V & VI]

Neil Ford

Emergent Design

Abstraction Distractions

The Productive Programmer

James Lyndsay

There are Plenty of Ways to Manage Exploratory Testing

Alan Page

Where (Testing) Ideas Come From Testing

Keith Klain

How to Talk to a CIO About Software Testing (If You Really Have to…)

James Coplien

The New Defacto Scrum Standard

Julian Harty

Six Thinking Hats for Software Testers

Linda Rising

Fearless Change

The Agile Mindset — and Beyond

Fiona Charles

State of Software Testing – 2013 with Jerry Weinberg and Fiona Charles

Mind, Map and Strategy

Unburdening Testing. Finding the balance point for test documentation

Thinking Strategically About Testing

Jez Humble

Adopting Continuous Delivery

Jerry Weinberg

State of Software Testing – 2013 with Jerry Weinberg and Fiona Charles

Daniel Kahneman

@Google Presents: Daniel Kahneman

Daniel Kahneman on The Machinery of the Mind

Prof. Daniel Kahneman: “Thinking, Fast and Slow”

James Christie

Standards – promoting quality or restricting competition?

Ben Simo

There Was Not a Breach; There Was a Blog

Grady Booch

“I Think, Therefore I Am”

Dan North

Achieving Business Impact: Why Agile doesn’t scale

Interview with Dan North on Behavior-Driven Development

The Once and Future Kanban

Embracing Uncertainty

Allan Kelly

Continuous Delivery and Conway’s Law

Alan Richardson

Thinking Visually In Software Testing

The Evil Tester’s Guide to Technical Testing

Bret Victor

The Future of Programming

Inventing on Principle

Stop Drawing Dead Fish

Michael Skok

Startup Secrets Framework with Amazon Web Services

Kevlin Henney

Patterns for the People

Simon Sinek

Why Leaders Eat Last

If You Don’t Understand People, You Don’t Understand Business

W. Edwards Deming

A Theory of a System for Educators and Managers

W. Edward Deming Quality Guru

Russell Ackoff

If Russ Ackoff had given a TED Talk…

Systems-Based Improvement, Pt 1

Systems-Based Improvement, Pt 2

Systems-Based Improvement, Pt 3

Systems-Based Improvement, Pt 4

Scott Barber

“The ongoing evolution of testing in agile development”

Gojko Adzic

Reinventing software quality

Make Impacts Not Software

Visualizing quality

Jeff Patton

How hard could it be? What’s User Experience is and isn’t

Why common agile practice isn’t agile

Scott Hanselman

Information Overload and Managing the Flow

Amy Cuddy

Your body language shapes who you are

Gary Stager 

Seymour Papert — inventor of everything

Andrew Solomon

Depression, the secret we share

Chimamanda Ngozi Adichie

The danger of a single story

Udemy

Critical Thinker Academy

Few Words About Bugs

This is an article I wrote for ReQtest over an year ago. To make it short: I wasn’t satisfied how they handled the publication of my article.

For  a long time I’ve thought about putting it to my blog and now finally decided to actually do it. It’s quite old article, but I want it to be published as it was intended.

Few Words About Bugs

Bug reports are one of the main work products we produce as testers. As for definition of a bug, I like James Bach’s definition: “A bug is something that bugs somebody who matters.” Here I will though focus on the ones that are found by investigating the product. This is not the only aspect though. I recommend that you watch Rob Sabourin’s Let’s Test keynote from last year. It will reveal, how flexible the definition of a bug, can actually be.

It’s good to note, that finding bugs is not the only thing I do as a tester. I also provide information that will help other people made better choices. Recently, I wrote a blog post (Visibility of Our Work) that goes through the additional value we can provide as testers, besides finding bugs.

Let’s get back to finding bugs. When we find a bug and reach the point where it is filed to defect management system (or any other way of informing a programmer and possible other stakeholders about the bug), we have hopefully gone through many points that have a crucial effect on the probability of this specific bug being fixed. BBST Bug Advocacy course slides mentions a quote from the book Testing Computer Software (3.0) that highlights, how the amount of bugs we find, is not the main aspect, we should focus on.

The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the right bugs fixed.

In this article, I will try to outline some of the phases that increase the probability of bugs being fixed, with respect to paying attention to figuring out if something is a problem (this relates to TCS quote above). In the end I’ll also show an example of what my bug report would look like. I will write a bug report of a bug I found on XMind (mind mapping tool – www.xmind.net).

Is it a problem?

A crucial phase that we need to go through before we can start filing a bug is that we need to decide if we have actually found a bug. In other words we ask the question “Is it a problem?” One way to answer this question is using oracles. An oracle is something that will help us to possibly recognize if there is a problem. For example, a requirements document. When we notice that the product behaves in a way that is not consistent with how it is defined in requirements, we decide that there is a bug in the product. There are a lot of other oracles, like live oracles, which might be a product owner, or an experienced user that will confirm to you that some specific observation is a bug.

Michael Bolton wrote a great blog post where he went through the high-level list of different oracles that we might use to recognize if there is a problem. The list is as follows

a) A feeling like confusion or annoyance.

b) A desirable consistency between related things.

c) A person whose opinion matters.

d) An opinion held by a person who matters.

e) A disagreement among people who matter.

f) A reference document with useful information.

g) A known good example output.

h) A known bad example output.

i) A process or tool by which output is checked.

j) A process or tool that helps a tester identify patterns.

One important thing to remember about oracles is that they are fallible. Let’s say, for example, I feel strongly that there is a problem, which is a threat to the overall quality of the product. Though, according to the requirements, the product is working correctly. I might still consult another oracle (like the product owner or someone else whose opinion matters) and explain how I feel that the current behavior is a threat to the overall quality of the product. If I will not succeed in convincing the stakeholders about the threat to overall quality, I have at least given my opinion.

After we have recognized and concluded, that we have a bug in our hands, we move on to reporting it (there are probably different ways to handle this, but personally I like to solve the “Is it a problem?” issue, before I report the bug). It’s good to note though, that while some bugs can be reported instantly, others require more information gathering.

Isolate and amplify

I often want to understand (isolate) what the bug actually is in its most understandable form, before I move on to reporting it or discussing it more with stakeholders. My goal is that the programmer can clearly envision the bug in their head when reading my bug report. To achieve this, we can try to place the programmer to the seat of the end-user, and try to make her see the behavior with the eyes of the end-user.

In my current project I am monitoring real time server logging when I test our system (logs are located on a Unix server and logging is done with the help of the tail -f command). In the logs I can often see information, which will help the programmers to troubleshoot the problem. They can also access the logs by themselves but I will do my best to save their time by providing events that I’ve witnessed in logs while testing. Let’s remember though, that the logs you provide are not necessarily related to the problem you have reported. This is a risk, which needs to be acknowledged.

Besides isolating I also want to try to amplify the bug, if possible. As an example, I tested some time ago a message field that could be inserted text. It also had a functionality, which allowed you to attach files. I noticed that when you attach a PDF file to the message, the amount of characters, that the system allows you to insert to the message field, got smaller. Further investigations revealed, that if I increased the size of the file, the amount of characters available, decreased even more. Eventually, if I attached large enough file, I was not able to insert text to the message field. This information is good to mention, when you report the bug. It will help us understand the impact on the end-user, in greater detail.

We often understand more about the possible impact to the overall quality of our product when we have invested in increasing our knowledge about the bug. Also, the more we understand the impact on the overall quality, the more we can describe the possible consequences of this specific bug.

In order to get the bugs fixed though, we need to explain our findings. Great testers excel in their ability, to describe the behavior, which they have witnessed.

Testers need to be highly skillful at describing bugs

Bug reports are one of the main artifacts of our work and they also establish our credibility. Carelessly written bug reports (e.g. programmer can’t understand what the problem is, it’s hard to reproduce, there are a lot of grammar mistakes) give an impression that we are not professional testers. In some cases bug reports are our main way of being in contact with the programmers. Because of these reasons I think that we need to be extremely good at describing bugs and in the end writing bug reports.

Personally, when writing a bug report I focus on bug reports on information about configuration (e.g. browser, OS), providing steps for reproducing and attaching any files (e.g. screenshot, video, log files) that I think will help in reproducing and in the end fixing the bug. However, I can’t spend too much time on the report. Even if I would, there’s always a risk that programmer will not be able to understand my bug report and will fail to reproduce the bug. I try to minimize this risk, but also at the same time I realize that the time I spend on writing the report is time away from finding new information.

I’ve seen too many examples in my short career where there is a constant discussion between the tester and programmer regarding the additional information that the programmer needs in order to fix, reproduce or even understand the bug. Not always, but often these moments can be handled by writing the bug report well enough in the first place. That’s why I highly encourage people to do it professionally in the first place. Figure out as much (not too much) information as you can in reasonable time (requires judgment). Then tie the pieces together in a way that will maximize the quality of the information delivered. You can also ask for feedback from programmers, other testers or any of the stakeholders that handle your bug reports. Programmers could inform you, if they feel that some information is missing from the reports. You could also ask just general comments from other testers, regarding the overall quality of your bug reports.

Because I personally love examples and find them often the most convenient way of understanding something, I’ll show an example of a bug report I wrote regarding a bug I found while using XMind (mind mapping tool). It’s not a model of a flawless bug report but rather an example of how I currently write bug reports in my project.

One of the reasons I wrote this post is that I want to make my work public. By publishing my thoughts and my example bug report I will get feedback that will help improve myself. That’s also one of the ways of becoming a software testing expert. Surround yourself with bright and diverse people who are not afraid to criticize your ideas. That way you get feedback that truly helps you become better at your work.

Bug report example – XMind

Before I show the bug report, I’ll let you see the error I saw when using XMind:

xmind

Bug ID #1: User receives java.lang.NullPointerException when creating boundary for subtopic

Configuration

Windows Vista Enterprise – SP2 (32 bit)

XMind for Windows- v3.3.0

Summary

I received java.lang.NullPointerException –error when I created boundary two times in a row for subtopic, and then clicked left mouse button.

Steps to reproduce

  1. Select the [Central Topic]
  2. Create [Subtopic] with Tabulator or via menu
  3. Select the created [Main Topic 1]
  4. Press Ctrl+B+B
  5. Click the left mouse button when cursor is shaped as cross.

Expected

Selecting a subtopic, then clicking [Ctrl] down and two times [B] button, should not do anything when you press the [B] second time.

Actual

When I insert the boundary second time for subtopic and click left mouse button, when the cursor is shaped as cross, I get a java.lang.NullPointerException. I was not able to reveal any side effect, but user will feel confused, as s/he does not know if it has an effect on the behavior of XMind.

Attachments

I attached screenshot of the error message I received and also a video that shows how I reproduced the problem.

 

Conclusion

As you’ve seen, finding and reporting bugs, is not a trivial matter. It involves various challenging phases and those can easily vary between contexts. I’m constantly trying to find the balance between the effort I put to polishing the information I provide, and on the other hand, to finding new information. Knowledge about oracles and tools can help you to improve the information you provide. In the end it’s though you, who will make the final decision regarding the information, that is included or not included.

Seek ways of getting feedback for your bug reports, or ask if you could give feedback to others. Sharpen your skills of observing, recognizing and describing problems. If you don’t know where to start from, I strongly recommend BBST Bug Advocacy course (requires that BBST Foundations is passed) that is provided by Association for Software Testing. I will personally participate that (BBST Bug Advocacy) on November this year.

Also, feel free to comment (either here or on Twitter) my bug report or anything else that I’ve written on this article/post. Constructive feedback helps me move forward as a tester.

 

Start From The Past

After being in the computing business now for more than half a century, one thing worries me more than almost anything else: our lack of a sense of history. (Jerry Weinberg)

For a while now I’ve had a rule of thumb, heuristic, that I’ve followed when I’ve wanted to learn something new. I don’t actually remember if I learned it from somewhere or came up with it myself. Nevertheless, it’s been useful for me and I would like to share it with you.

Find Out Where It All Started From

If I want to learn something new, I will focus on the past and try to figure out where it all started from. This way I will lower the risk of being affected by Law of Raspberry Jam.

As for now, I think it’s most helpful if I share couple examples.

Case 1: Scrum

I wanted to know more about Scrum couple months ago. Considering the popularity of Scrum, its Wikipedia article is quite thorough. What’s most important, it has History section in it. Let me copy-paste the beginning of that section:

Scrum was first defined as “a flexible, holistic product development strategy where a development team works as a unit to reach a common goal” as opposed to a “traditional, sequential approach” in 1986 by Hirotaka Takeuchi and Ikujiro Nonaka in the “New New Product Development Game”.[1] Takeuchi and Nonaka later argued in “The Knowledge Creating Company”[2] that it is a form of “organizational knowledge creation, [...] especially good at bringing about innovation continuously, incrementally and spirally”.

After learning that, I went and read the paper. What’s interesting, paper didn’t mention e.g. sprints, daily stand-ups or backlog. It focused more on identifying characteristics. Characteristics that leading companies showed (in 1986). Later, while learning about origins of Scrum, I ended up watching James Coplien’s talk at Youtube: Patterns: The New Defacto Scrum Standard. I’ve seen it several times since then. Perhaps the thing that strike me the most, was that James said how Scrum had been affected by Toyota Production System (a.k.a. Lean).

Learning about the work of James Coplien also led me to Scrum Pattern Community & their published patterns. Based on all the previous, my current understanding is that Scrum is about identified patterns & in the end: Kaizen. Successful teams have had patterns that has been part of their success. Those have been identified and came part of Scrum. It also reminds me that every team has its own context that they operate on. This needs to be considered when trying to apply a pattern (e.g. daily stand-up) to yours.

Case 2: Agile

I’ve also activated with Agile lately (this year). Sure, I’ve been familiar with Agile Manifesto & its principles. That’s though quite simplistic & therefore it makes you wonder how to interpret it. That’s why I wanted to understand what people behind the Manifesto thought about it, when it was published. I found one good article about it where Martin Fowler & Jim Highsmith explained their thinking behind every point & principles. Good to notice how the article is from year 2001, when the Manifesto was published.

There’s a lot more, if you have time, energy & curiosity. Martin Fowler has an interesting article on his website from July, 2000, where he describes the New Methodology. Then there’s also an updated version of the same article from year 2005. I haven’t yet read those. But that’s where I will go next. These are the people who were behind Agile Manifesto, they are the ones whose thinking I want to listen, when learning about Agile.

I would be naive though, if I would assume that agile software development came to life 13 years ago (2001). There’s one great article that reminds us how people were doing things smartly already in mid-1950s. Here’s one quotation (by Jerry Weinberg) from that article (Iterative and Incremental Development: A Brief History):

We were doing incremental development as early
as 1957, in Los Angeles, under the direction of
Bernie Dimsdale [at IBM’s Service Bureau
Corporation]. He was a colleague of John von
Neumann, so perhaps he learned it there, or
assumed it as totally natural. I do remember Herb
Jacobs (primarily, though we all participated)
developing a large simulation for Motorola, where
the technique used was, as far as I can tell, indis-
tinguishable from XP.​​
​​​​
When much of the same team was reassembled
in Washington, DC in 1958 to develop Project
Mercury, we had our own machine and the new
Share Operating System, whose symbolic modifi-
cation and assembly allowed us to build the system
incrementally, which we did, with great success.
Project Mercury was the seed bed out of which
grew the IBM Federal Systems Division. Thus, that
division started with a history and tradition of
incremental development.
 ​
All of us, as far as I can remember, thought
waterfalling of a huge project was rather stupid,
or at least ignorant of the realities… I think what
the waterfall description did for us was make
us realize that we were doing something else,
something unnamed except for “software devel-
opment.”
 ​
Case 3: Testing
 ​
On December, 2013,  I wrote a blog post about history of software testing and especially how first software testing teams were formed. It’s a good description of how testers were starting to came into software development 50 years ago. Because they concluded that it would help if there were people who were focusing more on testing than others.
It’s good to note though that I’m talking about software testing. Testing, the mental activity of questioning in order to evaluate – that goes far back. I haven’t put that much effort on tracking the history of that yet.
 ​
While exploring the history of software testing I ran into different kinds of papers. One particular (Evaluation of the Functional Testing of Control Programs from 1967) hinted where the demand for excessively structured testing came from.
 ​
Other way to learn about the history is to ask from people who remember far enough. I asked for example from Jerry Weinberg about origin of test cases.
 ​
JerryWeinberg_TestCases
This is a good reminder of the context behind birth of test cases. While it may be beneficial to use them in your context 60 years later, do note that access to machines is not that abominable these days anymore. If that’s the case, other reasons for using them need to be considered. This same thinking should apply to anything that you’re leaning into & haven’t invented by yourself.
​​​
Try It Yourself
 ​
Next time you want to learn about something new (e.g. BDD, Specification by Example, Exploratory Testing, Lean, Unit Tests), start from its history. If there are few people who are behind it, try to find what they have to say about it. Or perhaps you can even ask from them via email, Twitter or some other way. This way you’re increasing your chances of understanding the subject the same way as the people who came up with it.
 ​
Good luck!
 ​
To show that I’m not alone, I’d like to end this to Martin Fowler’s words:
 ​
In order to understand something, I often find it useful to know how it came about.
 ​

Comparing BBST Foundations Courses

This is a post that should’ve been written long time ago, but here we are. Better late than never, I guess.

I participated on January, 2014 on my second BBST Foundations course and this was arranged by Altom. I had already been on the same course by AST on Autumn, 2012. Courses had similarities, but also differences. I will mention those in this post.

Do note though that it’s been almost two years since my AST BBST Foundations course and also about half year from the one by Altom. I don’t know how much AST course has evolved since then, so I can comment only the one I attended.

BBST Foundations Courses?

When I mentioned on Twitter that I was participating to BBST Foundations course, first responses assumed I meant the one by AST. I actually don’t know how many are aware that there are also commercial ones being arranged.

So, what are the BBST Foundations courses?

I could share that information here, but I rather recommend that you check their websites (where the information will be more up to date):

AST BBST Foundations Course

Altom’s BBST Foundations Course

In a higher level both courses had these four topics:

  • the mission of testing
  • the oracle problem
  • the measurement problem
  • the impossibility of complete testing

I think it’s good to also mention that my understanding has been that Altom doesn’t have privilege for arranging the commercial BBST Foundations course. I’ve understood that one can negotiate this with Cem Kaner and Rebecca Fiedler.

Why Did I Participate Again For The Course?

After I had been on the AST BBST Foundations course, I wrote a blog post about it. It was extremely hard for me then. I provided examples of that on the post:

In the end I was sleeping on the average 4-6 hours per night and it started to eat me inside. Actually now that I had first Monday since the course ended, I overslept 4,5 hours this morning and I thought if my body just decided to take the sleep debt I had kept building.

One might wonder why I decided to participate again for a course that was extremely hard? There were couple reasons:

  1. Cem Kaner as lead instructor
  2. Possibility to have interactive grading with Cem Kaner
  3. Updated course material (BBST Workbook)
  4. Challenging project was beginning at the same time as the course, so the course was a good way to refresh my thoughts on the subject

Besides those reasons I had also decided that I will not sacrifice my sleeping time this time. I wanted to sleep at least 7 hours a night and I think I managed to hold on to that fairly well.

What Were The Differences Between The Two Courses I Attended?

While the slides of the both (AST & Altom) courses seem the same, there was the BBST Workbook on Altom’s course that helped students to get more value from the course. Considering that it’s being sold in Amazon, I’d imagine you can benefit from it in future, whether the course is arranged by AST or Altom.

Other difference was that Cem Kaner was actively participating to the course by Altom and also provided interactive grading for first ten registered students. Others received also interactive grading, but with other instructors (Ru Cindrea, Maaret Pyhäjärvi & Alex Rotaru). In the course by AST students were reviewing each others exams & instructors weren’t that involved with students discussions as Cem were at Altom course.

The main feeling after Altom’s course was that Cem was the key difference when comparing the courses. He provided a ton of valuable feedback during the course for students who were answering to challenging questions. I still have many of those answers on my Evernote.

While it’s been almost two years from AST course, I felt that it was lacking a mentor or coach who would give feedback to majority of the students. I recall that they tried to help students but it was far from the level of feedback Cem provided on the Altom’s course.

I did felt though that there were more collaboration & assignments on AST’s course and also group assignments that involved communicating with others via e.g. Skype.

Which Should You Choose?

Considering the things I’ve said, I think you need to think which one suits you best. I’ll share few thoughts though.

If you’re on a strict budget, AST’s ($125 + $30 to $95 depending if your student or not) is significantly cheaper than Altom’s (~$811 + VAT).

I have to admit though that Cem’s presence on Altom’s course had a huge impact on its value for the students. While we are all humans, the vast amount of experience and knowledge behind every answer by Cem, provided valuable feedback to each student. When you added there the interactive grading session I had with him (almost 2 hours on Skype), I felt I had more guidance on the course, compared to one I had 2 years ago via AST. This proved to be valuable as I understood the purpose of one question in exam. I wouldn’t have understood it without the interactive grading session.

Also, as far as I’ve understood, Kaner, Fiedler & Associates, LLC, has access to latest materials. This means that these commercial courses will have in the future more up to date material.

In the end both of the courses are great, because they challenge a lot your thinking and let you see how diverse approaches people have for same problems. If you’re planning to participate to either of those, I recommend both of them. While keeping in mind what I said earlier.

What Is Quality?

For a long time I didn’t put too much thought into definition of quality. I just considered it to be an attribute. Something that is either good or not good. What’s so difficult about it?

Now when I look back – the very definition of quality is one of the main reasons that software development is so challenging.

It’s about value

I don’t know when I first heard Jerry Weinberg’s definition of quality. Maybe it was on BBST Foundations course. Or perhaps on Twitter. Or a blog post. Can’t remember.

Nevertheless, it changed my thinking. It changed how I view the world.

Jerry Weinberg defines quality this way:

Quality is value to some person

When I saw this – it made sense. I realized that we (people) like different things. Whether it’s food, drinks, movies, music, hobbies or books. We have our own preferences. One likes science fiction movies. Other one can’t stand historical or future related movies. Third wants to see only comedy. And the list goes on.

Why wouldn’t we also view software from our own perspective and value the things that are important for us? Sure, there are people who value same things, but there’s as much variety between what people value on software. Or products generally.

Who matters?

While quality is value to some person, that doesn’t mean that we’re trying meet everyone’s needs & requirements. There’s a lot of people on this planet. Only a small proportion is most likely the segment of people you want to impact on. Those are the people who matter (James Bach has added “who matter” to Jerry’s definition: “Quality is value to some person who matters“) And they’re not necessarily only customer or end-users. I think marketing, documentation or perhaps your CEO have their opinion about the quality of your product. If you’re interested of hearing their opinion and act upon it, then they matter also.

When we build software, we would like to meet the needs of the people who matter. Perhaps we have made thoughtful decisions about who we are concentrating on. Rational decisions you might say. But how rational they actually are?

It’s also about emotions & politics

Jerry Weinberg discussed on Agile Impressions (originally in Quality Software Management Volume 1: Systems Thinking – as far as I know) how our decisions about quality are political, but also emotional. 

While we would like to believe that our decisions are political decisions that are based on rational thinking – we can’t ignore our emotions and how they affect to our decisions about quality.

Considering that we all have our personal relationship with software. Meaning that we value things that vary from many people. How unlikely do you think it is that these personal preferences wouldn’t affect to our decisions about quality? That we wouldn’t make decisions that drive the product toward something that we personally would like to use? Or report bugs that are personally bugging us? Or criticize the usability based on what is good usability to ourselves?

I’m not saying that our personal preferences shouldn’t affect to our decisions. I’m saying though that we need to consider the possibility that this is happening. No matter how rational we want to see ourselves.

But – what is value?

I’ve seen Jerry’s definition of quality mentioned countless times. On the other hand, I haven’t seen his definition of value mentioned that often:

What are people willing to pay (do) to have their requirements met

There’s at least two perspectives that you can look this definition from.

  1. What are people (developers) willing to pay (do) to have their (e.g. end-users) requirements met
  2. What are people (end-users) willing to pay (do) to have their (e.g. end-users) requirements met

I guess you could add more variations to this, but let’s focus these two. In first case developers (i.e. programmers, testers, product owners, business analysts – anyone who is involved with making quality related decisions) put effort on meeting end-users requirements. If you’re building an app like Wedding Make Up on Google Play Store, you’re not probably putting too much effort on meeting my needs. That being because I’m not (probably) part of your people who matter.

On the other hand, as an end-user, app has basically no value for me. So, I’m not willing to pay or do anything to get that app.  Sometimes there’s a product that has value, but I might need to register myself to a webpage or pay X amount of money, and that will turn the value to negative. It depends.

If only our needs were static

While figuring out what people (whose lives our software touches) value is useful, there’s additional challenge. That’s the dynamic nature of our needs. They tend to change while time passes. Reasons vary. It might be that I get used to your product and will be expecting enchantments. It might be that I had one time need for your product (e.g. doing a video from the slides & audio of my talk). Or it might be that competitors add all kinds of features that I’m expecting from your product also. Needs are not static.

This leads us to Weinberg’s definition being extended (‘at some time’ being added by Michael Bolton – as far as I know) to:

Quality is value to some person, at some time, who matters.

What’s the impact of all this to my work as a tester?

This was not probably an easy to follow blog post. Largely because my thinking hasn’t settled related to this topic. I hoped that writing all this would help in it, but not sure it did that.

All this has though affected to my work as a tester. Perhaps the biggest impact is in realizing that I can’t put too much weight on my subjective opinion about the quality of the software. This being because it’s based on what I personally value on software. I’ve realized that it CAN be important, but it CAN also differ a lot from people who use our product. Own feelings & emotions act as a trigger for further investigation, but I need to also approach bugs & findings from other angle than my subjective angle.

 

Testing Quadrants As A Communication Tool

 

Screenshot at toukokuuta 15 23-13-23

 

Here’s a modification of Agile Testing Quadrants by Lisa Crispin & Janet Gregory (originally by Brian Marick). I’ve used this as a communication tool in our current project. As it might raise few eyebrows, let me explain couple things.

(Also want to mention that I’ve modified the Quadrants a bit from how they are actually on the project – i.e. what we are focusing on & what we are not focusing on)

Context & Background

I explained very briefly my current context on my previous blog post: Striving For Early Feedback

Besides that, I’ve had challenges with explaining what we (our test team of 2) is testing, with respect to all others on the overall system. Then I was listening Paul Gerrard’s talk (Agile Test Strategy) a while ago. There he mentioned Agile Testing Quadrants. Of course I knew them and had seen them several times before. I just couldn’t come up with the idea of them being useful in my current context.

Agile-Testing-Quadrants

I first used Crispin & Gregory Quadrants while describing our approach to one vendor. There was though many things that I didn’t personally see useful (e.g. Automated & Manual or ET being on Q3). Then I had a discussion with Jari Laakso on Twitter. After that discussion I realized – with the help of Jari – that I can modify the Quadrants in any way I want. And that I did.

Things I want to emphasize

With my modified Quadrants I want to emphasize basically 4 things:

  1. Our testing being a division between testing the expected vs. unexpected – Or in other words ‘Focusing on Expectations’ vs. ‘Focusing on Risks’
  2.  We are intentionally not focusing on testing certain things (e.g. unit testing) – but are aware of how that’s being done by certain people
  3. All testing is based on exploration – in other words: Learning. We don’t test for the sake of testing, but to learn more about our product with every test
  4. Testing mentioned on the top is about People and not only about Business. It’s about people whose lives our product touches (Marc McNeill – according to Dan North)

What was challenging

Perhaps the biggest challenge with Quadrants is that it’s quite challenging to fit different forms of testing there. For example in performance testing we might be testing the expected (e.g. explicit documented requirements), but also regardless of those. That would mean that performance related testing goes to many places.

In the end I found that that particular challenge was something I can live with. Considering I can emphasize those other things with the model.

What next?

I don’t know yet if I will continue to modify and improve the model to make it more useful while describing our testing approach. Or if I will use this model later on the following projects or companies. I might. Or I might not. What matters now, is that it makes it a bit easier (from my perspective) to explain what our testing looks like.