I’ve been for a while now at a new client. It’s been my second client while working as a consultant for my current employer (Comiq). When talking about challenges and the scale of testing problems, current project is on another level when comparing to previous ones. Luckily, there’s one thing that helps a lot. That’s the fact that we’ve jumped in couple months before the actual implementation (writing the code) starts. I could easily write a blog post of its own from being early in a project as a tester, but that’s not what I wanted to write today.
I wanted to write about early feedback.
Before we will go into that, let me open up the context a bit. Well, as much as I can while respecting the NDA.
Explaining the context
This project is one of those where one vendor is building a new solution that will have integrations to several existing systems. Those existing systems have their own vendors developing the integrations. The solution that we are interested of, is also only one piece of the puzzle for those other systems. They have all kinds of projects going on and many of them are related to things already in production. Production has usually higher priority when we are talking about urgent issues for our project.
You can probably figure out how the system architecture diagram looks like.
Yes.
Lots of boxes and arrows. And as usual, system architecture diagrams are not the reality.
Motivation for early feedback
Considering the challenging nature of this project, there’s one thing we have been trying to emphasize a lot. That’s early feedback.
I’ve noticed how there have been examples on this client where problems are being found quite late in development process. Also the amounts of problems being found are high. This has at least two possible problems in itself.
Firstly, fixing a lot of problems in a specific point in time, will raise the probability of regression and also slowdown the ongoing development process.
Secondly, problems can be more challenging to fix when there’s a long time between introducing a problem (e.g. coding mistake) and actually finding it. One developer I’ve discussed lately, compared this to cancer that is spreading slowly. Spreading cancer being a metaphor for accumulating code that is being built over the one that is causing problems to someone who matters. Good to remember also that there are though factors (e.g. design of the code) that affect to how easily problems can be fixed later.
How we are trying to provide early feedback
Being early in a project has given us the privilege to sit down with vendors before the implementation (writing code) has started. Basically we have explained the challenges that I mentioned above and then we have discussed how vendors could help us achieving a state where problems are found earlier & how we also want to help them.
As for now, there are few practical things that we hope will help us achieving state where problems are being found rather sooner than later.
- Reviewing specifications
- Being part of discussions where decisions are made (e.g. workshops where functionalities are explored)
- Testing in development environments (instead of waiting the software being put into “upper” environments later)
- Having vendors physically in same place when e.g. new integration is implemented to development environment. Sort of Shake&Bake that James Bach has referred to in a video where he explains how to test in an agile software development team (https://www.youtube.com/watch?v=vqwyMaHcjQE)
There are few challenges though, that I’d like to discuss about.
Challenges
Perhaps the greatest challenge is visibility. Well, lack of it.
If we want to test functionalities early, that would mean that we know when they’re testable in specific environments. Developer could always contact you directly when there’s something to test. But, what if there’s something new several times a week, or a day? And there’s, let’s say 7 vendors? Instead of testing you’re spending your time answering your phone or reading your emails.
When I realized the visibility challenge, I first wanted to start from checking how different vendors are providing visibility to their colleagues. After that we can think about providing visibility to us.
I’ve discussed with several vendors and discussions have been promising. Each of them have had their own way of visualizing their work. Our objective is to be able to determine, based on those, when specific functionalities / features are testable at certain environments.
Other challenge is scheduling. By scheduling I mean that pieces of software are being dropped in such a way that we have realistic change of testing them early. If there’s several new features being implemented on a same day, it can lead to feedback loop growing yet again, because we can’t test all of them at once. At least not while having good coverage. Whatever good coverage means in this case.
Not there yet
As I’m writing this, the overall approach is still shaping and near future will tell how those challenges will get tackled. There will also definitely be some kind of “Lessons Learned” type of blog post of this project when there’s enough meat over the bones. Before we get there I’m planning to share how I’ve used modified Agile Testing Quadrants as a communication tool in this project.
Flow of Testing 