What Is Quality?

For a long time I didn’t put too much thought into definition of quality. I just considered it to be an attribute. Something that is either good or not good. What’s so difficult about it?

Now when I look back – the very definition of quality is one of the main reasons that software development is so challenging.

It’s about value

I don’t know when I first heard Jerry Weinberg’s definition of quality. Maybe it was on BBST Foundations course. Or perhaps on Twitter. Or a blog post. Can’t remember.

Nevertheless, it changed my thinking. It changed how I view the world.

Jerry Weinberg defines quality this way:

Quality is value to some person

When I saw this – it made sense. I realized that we (people) like different things. Whether it’s food, drinks, movies, music, hobbies or books. We have our own preferences. One likes science fiction movies. Other one can’t stand historical or future related movies. Third wants to see only comedy. And the list goes on.

Why wouldn’t we also view software from our own perspective and value the things that are important for us? Sure, there are people who value same things, but there’s as much variety between what people value on software. Or products generally.

Who matters?

While quality is value to some person, that doesn’t mean that we’re trying meet everyone’s needs & requirements. There’s a lot of people on this planet. Only a small proportion is most likely the segment of people you want to impact on. Those are the people who matter (James Bach has added “who matter” to Jerry’s definition: “Quality is value to some person who matters“) And they’re not necessarily only customer or end-users. I think marketing, documentation or perhaps your CEO have their opinion about the quality of your product. If you’re interested of hearing their opinion and act upon it, then they matter also.

When we build software, we would like to meet the needs of the people who matter. Perhaps we have made thoughtful decisions about who we are concentrating on. Rational decisions you might say. But how rational they actually are?

It’s also about emotions & politics

Jerry Weinberg discussed on Agile Impressions (originally in Quality Software Management Volume 1: Systems Thinking – as far as I know) how our decisions about quality are political, but also emotional. 

While we would like to believe that our decisions are political decisions that are based on rational thinking – we can’t ignore our emotions and how they affect to our decisions about quality.

Considering that we all have our personal relationship with software. Meaning that we value things that vary from many people. How unlikely do you think it is that these personal preferences wouldn’t affect to our decisions about quality? That we wouldn’t make decisions that drive the product toward something that we personally would like to use? Or report bugs that are personally bugging us? Or criticize the usability based on what is good usability to ourselves?

I’m not saying that our personal preferences shouldn’t affect to our decisions. I’m saying though that we need to consider the possibility that this is happening. No matter how rational we want to see ourselves.

But – what is value?

I’ve seen Jerry’s definition of quality mentioned countless times. On the other hand, I haven’t seen his definition of value mentioned that often:

What are people willing to pay (do) to have their requirements met

There’s at least two perspectives that you can look this definition from.

  1. What are people (developers) willing to pay (do) to have their (e.g. end-users) requirements met
  2. What are people (end-users) willing to pay (do) to have their (e.g. end-users) requirements met

I guess you could add more variations to this, but let’s focus these two. In first case developers (i.e. programmers, testers, product owners, business analysts – anyone who is involved with making quality related decisions) put effort on meeting end-users requirements. If you’re building an app like Wedding Make Up on Google Play Store, you’re not probably putting too much effort on meeting my needs. That being because I’m not (probably) part of your people who matter.

On the other hand, as an end-user, app has basically no value for me. So, I’m not willing to pay or do anything to get that app.  Sometimes there’s a product that has value, but I might need to register myself to a webpage or pay X amount of money, and that will turn the value to negative. It depends.

If only our needs were static

While figuring out what people (whose lives our software touches) value is useful, there’s additional challenge. That’s the dynamic nature of our needs. They tend to change while time passes. Reasons vary. It might be that I get used to your product and will be expecting enchantments. It might be that I had one time need for your product (e.g. doing a video from the slides & audio of my talk). Or it might be that competitors add all kinds of features that I’m expecting from your product also. Needs are not static.

This leads us to Weinberg’s definition being extended (‘at some time’ being added by Michael Bolton – as far as I know) to:

Quality is value to some person, at some time, who matters.

What’s the impact of all this to my work as a tester?

This was not probably an easy to follow blog post. Largely because my thinking hasn’t settled related to this topic. I hoped that writing all this would help in it, but not sure it did that.

All this has though affected to my work as a tester. Perhaps the biggest impact is in realizing that I can’t put too much weight on my subjective opinion about the quality of the software. This being because it’s based on what I personally value on software. I’ve realized that it CAN be important, but it CAN also differ a lot from people who use our product. Own feelings & emotions act as a trigger for further investigation, but I need to also approach bugs & findings from other angle than my subjective angle.



Testing Quadrants As A Communication Tool


Screenshot at toukokuuta 15 23-13-23


Here’s a modification of Agile Testing Quadrants by Lisa Crispin & Janet Gregory (originally by Brian Marick). I’ve used this as a communication tool in our current project. As it might raise few eyebrows, let me explain couple things.

(Also want to mention that I’ve modified the Quadrants a bit from how they are actually on the project – i.e. what we are focusing on & what we are not focusing on)

Context & Background

I explained very briefly my current context on my previous blog post: Striving For Early Feedback

Besides that, I’ve had challenges with explaining what we (our test team of 2) is testing, with respect to all others on the overall system. Then I was listening Paul Gerrard’s talk (Agile Test Strategy) a while ago. There he mentioned Agile Testing Quadrants. Of course I knew them and had seen them several times before. I just couldn’t come up with the idea of them being useful in my current context.


I first used Crispin & Gregory Quadrants while describing our approach to one vendor. There was though many things that I didn’t personally see useful (e.g. Automated & Manual or ET being on Q3). Then I had a discussion with Jari Laakso on Twitter. After that discussion I realized – with the help of Jari – that I can modify the Quadrants in any way I want. And that I did.

Things I want to emphasize

With my modified Quadrants I want to emphasize basically 4 things:

  1. Our testing being a division between testing the expected vs. unexpected – Or in other words ‘Focusing on Expectations’ vs. ‘Focusing on Risks’
  2.  We are intentionally not focusing on testing certain things (e.g. unit testing) – but are aware of how that’s being done by certain people
  3. All testing is based on exploration – in other words: Learning. We don’t test for the sake of testing, but to learn more about our product with every test
  4. Testing mentioned on the top is about People and not only about Business. It’s about people whose lives our product touches (Marc McNeill – according to Dan North)

What was challenging

Perhaps the biggest challenge with Quadrants is that it’s quite challenging to fit different forms of testing there. For example in performance testing we might be testing the expected (e.g. explicit documented requirements), but also regardless of those. That would mean that performance related testing goes to many places.

In the end I found that that particular challenge was something I can live with. Considering I can emphasize those other things with the model.

What next?

I don’t know yet if I will continue to modify and improve the model to make it more useful while describing our testing approach. Or if I will use this model later on the following projects or companies. I might. Or I might not. What matters now, is that it makes it a bit easier (from my perspective) to explain what our testing looks like.

Striving For Early Feedback

I’ve been for a while now at a new client. It’s been my second client while working as a consultant for my current employer (Comiq). When talking about challenges and the scale of testing problems, current project is on another level when comparing to previous ones. Luckily, there’s one thing that helps a lot. That’s the fact that we’ve jumped in couple months before the actual implementation (writing the code) starts. I could easily write a blog post of its own from being early in a project as a tester, but that’s not what I wanted to write today.

I wanted to write about early feedback.

Before we will go into that, let me open up the context a bit. Well, as much as I can while respecting the NDA.

Explaining the context

Wenjie Chang - Flickr

This project is one of those where one vendor is building a new solution that will have integrations to several existing systems. Those existing systems have their own vendors developing the integrations. The solution that we are interested of, is also only one piece of the puzzle for those other systems. They have all kinds of projects going on and many of them are related to things already in production. Production has usually higher priority when we are talking about urgent issues for our project.

You can probably figure out how the system architecture diagram looks like.


Lots of boxes and arrows. And as usual, system architecture diagrams are not the reality.

Motivation for early feedback

Considering the challenging nature of this project, there’s one thing we have been trying to emphasize a lot. That’s early feedback.

I’ve noticed how there have been examples on this client where problems are being found quite late in development process. Also the amounts of problems being found are high. This has at least two possible problems in itself.

Firstly, fixing a lot of problems in a specific point in time, will raise the probability of regression and also slowdown the ongoing development process.

Secondly, problems can be more challenging to fix when there’s a long time between introducing a problem (e.g. coding mistake) and actually finding it. One developer I’ve discussed lately, compared this to cancer that is spreading slowly. Spreading cancer being a metaphor for accumulating code that is being built over the one that is causing problems to someone who matters. Good to remember also that there are though factors (e.g. design of the code) that affect to how easily problems can be fixed later.

How we are trying to provide early feedback

Being early in a project has given us the privilege to sit down with vendors before the implementation (writing code) has started. Basically we have explained the challenges that I mentioned above and then we have discussed how vendors could help us achieving a state where problems are found earlier & how we also want to help them.

As for now, there are few practical things that we hope will help us achieving state where problems are being found rather sooner than later.

  • Reviewing specifications
  • Being part of discussions where decisions are made (e.g. workshops where functionalities are explored)
  • Testing in development environments (instead of waiting the software being put into “upper” environments later)
  • Having vendors physically in same place when e.g. new integration is implemented to development environment. Sort of Shake&Bake that James Bach has referred to in a video where he explains how to test in an agile software development team (https://www.youtube.com/watch?v=vqwyMaHcjQE)

There are few challenges though, that I’d like to discuss about.



Perhaps the greatest challenge is visibility. Well, lack of it.

If we want to test functionalities early, that would mean that we know when they’re testable in specific environments. Developer could always contact you directly when there’s something to test. But, what if there’s something new several times a week, or a day? And there’s, let’s say 7 vendors? Instead of testing you’re spending your time answering your phone or reading your emails.

When I realized the visibility challenge, I first wanted to start from checking how different vendors are providing visibility to their colleagues. After that we can think about providing visibility to us.

I’ve discussed with several vendors and discussions have been promising. Each of them have had their own way of visualizing their work. Our objective is to be able to determine, based on those, when specific functionalities / features are testable at certain environments.

Other challenge is scheduling. By scheduling I mean that pieces of software are being dropped in such a way that we have realistic change of testing them early. If there’s several new features being implemented on a same day, it can lead to feedback loop growing yet again, because we can’t test all of them at once. At least not while having good coverage. Whatever good coverage means in this case.

Not there yet

As I’m writing this, the overall approach is still shaping and near future will tell how those challenges will get tackled. There will also definitely be some kind of “Lessons Learned” type of blog post of this project when there’s enough meat over the bones. Before we get there I’m planning to share how I’ve used modified Agile Testing Quadrants as a communication tool in this project.


Prioritizing My Time

Prime-Time-Clock-by-zoutedrop-Creative-CommonsWhile procrastinating my blog post about BBST Foundations course (I’ve been on two of them and will compare them), I thought I could write a bit about how I prioritize my time.

Big Rocks

We all have 24 hours to spend. Every day. The way we use those hours, varies a lot though. A while ago I was listening an Udemy course about time management and it had a part where the presenter filled a glass jar with big rocks and sand. Moral of the story was that we should fill our life with big rocks (things that are important) first and only after that with sand (which represents all the little things). You can find similar video if you just google “big rocks time”.

Big rocks of my life are mainly:

  • Time with family
  • Work
  • Sleep

I think it’s good to open up those a bit.

Time with family

I have a fiancee and 3 years old son. When I get home from work (basically 4pm always), I spend time with them until our son is sleeping. When I say I spend time with them, I truly mean that. I don’t watch TV, use my laptop or play with my phone. When you have 3 years old son, being mentally engaged is important. Same applies to fiancee also.

Exceptions are really rare. I mentioned BBST Foundations courses. All the time I spent on those courses, happened when I was either at work, or when our son was sleeping.


With work I mean working 8 hours a day, 40 hours a week. That’s it. I’m quite strict about not doing overtime. Sometimes (like 2-3 times a month) I will be one hour longer at work, but those are rare.

We had a conversation about this last week on our Helsinki Software Development Lunches (we go have a lunch at Helsinki center and discuss about software development – anyone is free to join). There was a developer who said that he has also really strict about 40 hours per week rule. He said that he has been pushed on working overtime, but he has asked the underlying reason for something being really important and urgent. He continued that if there’s a life threatening situation or company is in bankrupt next day – then he can be flexible. How often this is the case though?


I need at least 7 hours of sleep per night. This is something I take also quite seriously and plan my activities so that it will be possible to sleep this much. Usually on weekends I will sleep longer, but I feel that 7 hours is enough during the week.

The Little Things

Those big rocks are something that are essential part of my life and will be affected very rarely. Sure, there are situations where I’m not entirely with my family after work. I might be making food, going grocery store or other similar things that needs to be done. Fortunately my son can be part of many of those. Depending on his mood.

As much as I would like to develop myself as a tester, I don’t want to do that by affecting big rocks. This is one of the reasons why majority of my learning happens while I’m at work, driving to work, putting my son to sleep (which requires just physical presence) or at the evening when I have a bit of my own time. I can’t even remember how many videos I’ve listened while driving to work. Videos from BBST to Oredev, Eurostar to TED Talks and Udemy courses to all kinds of Youtube talks. When I put my son to sleep I often read books. A while ago I read Jerry Weinberg’s ‘The Secrets of Consulting’ and ‘Are Your Lights On’.

It’s Not The Distance That Kills

I’ve seen several close people overworking and paying the price for it. Some are not here anymore and others, well there’s not much left of them. I’m also hearing these kind of stories while working and thinking that I’m not going to walk those paths. Because of these reasons, I’m strict about keeping the work hours on a level that will not affect negatively the whole. Work is only one part of the whole. Even though I love what I do and aim to become extremely good at it.

How about you? What are your big rocks?