Project Mercury and Death of Testing

After being in the computing business now for more than half a century, one thing worries me more than almost anything else: our lack of a sense of history. (Jerry Weinberg)

I’ve witnessed lately few great blog posts about the role of a tester and where we are going with what we understand as testing.

First Alan Page wrote his post “Death and Testing“. Then Trish Khoo delivered her thoughts in “Forget developers in test, we need testers in development“.

I’d like to share my thoughts about this subject also. That being, where should we be as testers?

Project Mercury

Project_Mercury_-_U.S.Man_in_space

When we’re talking about challenging software development projects, I think putting the first (hu)man on space is pretty high on that list. Project Mercury had this as a goal on the early 1960’s when US and Soviet Union had their space race going on. Jerry Weinberg, among others, had to develop software that would help in taking the astronauts to space and back, alive.

Jerry talked about Project Mercury on This Week in Software Testing podcast, 2010. I totally recommend listening it, as it’s one of the best podcasts I’ve heard (even though you need to register to softwaretestpro.com). I took the liberty and wrote a part of the podcast, especially the part where Jerry talks about Project Mercury:

Because of the complexity and worldwide nature of the system. I (Jerry) decided that we need to have a group of people dedicated to quality, including testing of the system. We created such a group, which is as far as I know, had never been done before.  They lived through the whole project, unlike a lot of testing groups today. They were involved right from the very beginning. So, for example they could comment on whether the software we were building, was in fact testable, by then. Which wasn’t sure at all to just automatically happen.

Because this was, not only first human life system, but it was the first worldwide online system, ever build, and presented many unique problems in testing. That’s how we got into this. It was after that that other people began to realize they needed separate testing groups. But it became different. Our testing group was composed of experienced and talented software developers.

Later on, managers began to see testing groups as a way to hire cheaper people and put them in testing, because they didn’t have to know how to develop software. I’m thinking a lot of managers thought all they needed to do is sit in there on terminal and bang on keys, like monkeys. Of course that’s simply not true, but it’s taken us a long time, for many people in industry to realize, that testing is as professional, a job, as we realized it back, literally half a century ago. And many managers today still don’t understand that.

In early 1960’s testing wasn’t yet defined, which meant that Jerry, and others, were not growing up with the same dogmas as we do these days. They didn’t have the mental (or concrete) walls that tend to separate testers from development teams these days. They just tried to come up with a good solution for the challenging problems they were facing. Therefore it was natural for them that few of the team would be putting more focus on testing than the others. Also, it was similarly natural that they were there since the beginning of the project and equally part of the team, as the others.

The fact that they were skillful developers is NOT the point. The fact that they were able to be valuable for their team, IS the point.

Boulding’s Backward Basis

I heard Jerry talking about Project Mercury in the podcast, roughly year ago. That moment was one of the first that led me to learning about Boulding’s Backward Basis (didn’t knew about the name then). When people talk about testers these days, many of them mean people who are working on a separate team or otherwise separated from the development team. Personally I think these people are growing up with dogma.

If you think of it, software development as it is, including testing, is not something that it’s MEANT to be. It’s though something that it’s TURNED OUT to be. Now would be actually a good time to mention Boulding’s Backward Basis (from ‘The Secrets of Consulting’ by Jerry Weinberg):

Things are the way they are because they got that way.

I’ll admit that I’ve not dig deep enough the history of our craft, so I could connect the dots and say what lead to where (besides what Jerry is explaining above). I do know though that the idea about testers being separated from the rest of the team wasn’t common long ago and it is these days. Some people talk about future and evolving, but I think we’re going back to our roots and that’s a good thing.

One dogma to another (and beyond)

I understand testing as a mental activity of questioning, observing, evaluating and in the end, exploring. This will never go away. I hope.

I don’t know if testers being integrated part of the software development team is how it should be. It does sound reasonable for me in the contexts I’ve been. Though, it would be naive to think that this is how it’s MEANT to be. We would just be jumping to a dogma after leaving another.

Keeping in mind that we don’t know what we’re doing (see Future of Programming by Bret Victor), and at the same time putting effort on exceeding our limits. Maybe that will move us toward something better.

Learning From Our Mistakes

Mistakes are often seen as a fruitful way of learning. If you do a Google search for “Learning from our mistakes” – you will get a lot of hits. Quotations, articles, TED talks, blog posts, books, references to Bible and of course… Oprah.

I also thought for a long time that there’s nothing special about learning from our mistakes. We make them, learn from them and don’t repeat them. So what?

Then I saw this talk by Kevlin Henney couple months ago. (The failure part starts around 17:30)

Kevlin referenced to Henry Petroski’s “To Engineer Is Human” book on his talk. Based on this book, he talked about overgeneralization and context of a failure.

Let’s say we make a mistake, learn from it and aim at not making it again. Now, this might lead to the overgeneralization problem.

We should be careful to get out of an experience only the wisdom that is in it – and stop there; lest we be like the cat that sits down on a hot stove lid. She will never sit on a hot stove lid again – and that is well; but also she will never sit down on a cold one anymore. (Mark Twain)

This was something I discussed briefly with Seriouspony (Kathy Sierra) on Twitter:

Kevlin also argued on his talk that:

When we learn from our mistakes, it’s highly contextual. We learn from our successes far better.

By mentioning the contextuality, he meant that our failures need to be happening in a context of successes. Then specific failures can be noticed. There are examples of this on the talk if you’re more interested.

Learning better from our successes?

Kevlin mentioned earlier that “We learn from our successes far better.” What did he mean?

First of all, he was referring to a study (by Earl Miller, Mark Histed & Anitha Pasupathy) which said (https://web.archive.org/web/20091222020319/http://www.asfct.org/documents/journal/2009-11/Vol1-2-9.pdf):

The assertion that we can learn something from every failure is often heard. This study by Earl Miller and his colleagues Mark Histed and Anitha Pasupathy of the Massa- chusetts Institute of Technology’s Picower Institute for Learning and Memory tests that notion by looking at the learning process at the level of neurons.

The study shows how brains learn more effectively from success than from failure. The researchers created a unique snapshot of the learning process that shows how single cells change their responses in real time as a result of information about what is the right action and what is the wrong one.

Brain cells keep track of whether recent behaviours were successful or not. When a certain behaviour was successful, cells became more finely tuned to what the animal was learning. After a failure, there was little or no change in the brain – nor was there any improvement in behaviour. This research seems to support SF’s assumption that analysing why something went wrong is unlikely to lead to ideas about how to create a better situation.

I found an article by MIT News (http://web.mit.edu/newsoffice/2009/successes-0729.html) that opened up the findings of the study much more. You might want to check that out if you’re more interested about this.

Despite the credibility of the study, it’s an interesting idea though. That we would learn better from our successes?

While ago I tweeted:

John Stevenson replied to my tweet:

Cognitive dissonance is a good point as it will help as in noticing when there’s a mismatch between what we have done and how it perhaps should have done, in our opinion.

Mistakes are though a matter of perspective. You’re applying some kind of oracle to the behavior or act that you call a mistake. Can you trust your oracle? Same of course applies to successes. Not being a mistake is also a matter of perspective.

Wrapping up my thoughts

This whole “learning from mistakes and successes” is still an ongoing process. I haven’t came up to any solutions, conclusions or deep realizations about it. But there are thoughts that I’m playing with at the moment:

  • In order to learn from failures, they need to be buried among successes. Being the odd ones that are spotted.
  • If we truly are learning better by doing something we consider successful, then making a mistake, should lead to changing the behavior to something we consider successful and will also prevent making the mistake again.
  • Are you sure that the mistake you made, isn’t something that is a mistake only in a specific and unique context?
  • While there needs to be thinking behind mistakes (why did it happen), there needs to be thinking also behind successes (why did it happen).

This is a subject that I’ve thought about here and there for a couple of months now. Hope it will give you something to think about.