In my previous blog post, I pointed out that skills in testing matter. These skills are a bit like muscle memory; they can be learned by practice and attention to detail.
Now we come into the rub -- that you can’t practice test skills the way you can, say, hitting a ball or dunking a basket. Once you’ve tested the system once and know where the bugs are, a re-test exercise is trivial.
Or is it?
What we can do is look at the software from different perspectives:
This means we can look at the same software many times from different angles. In some cases, we can turn around and look at different software from that same angle.
It’s not quite putting the puck in the same place every time and making the same shot, over and over again, but it’s close.
The Second Problem
If you try this, the next problem you’ll have is the lack of examples. There just aren’t many of these in testing the way they exist in development.
My hope, in 2014, is to provide new examples worth referencing, right here on my Stickyminds blog.
Here’s one for today:
Many people are familiar with ParkCalc, the Gerald R Ford Parking simulator. It’s a great example of quick attack bugs; there are plenty and they are easy to find. At one conference, a "big name" in testing told me that ParkCalc was a bad example because it was "garbage in/garbage out" and “those bugs don’t matter if the system in general is useful."
Except, of course, he had no idea if the system was useful. Or at least, if he did, he didn’t do a very thorough job, because ParkCalc does have bugs even if you put in reasonable input.
Today’s challenge is to look at the real requirements, which you can find at the bottom of this page - then try to test for short-term parking fitness for use. You can limit your tests to one browser that does not have browser compatibility issues - you are looking for errors in the business logic.
Now let’s say that testing is expensive; you only get to run enter five values and click ‘calculate’ five times. Which values would you use? Why? Once you run them, what bugs do you find and what do you know about the software?
You can leave your comments here, or blog them and leave a link. I hope you see how this kind of exercise can be repeated, help develop skills. Also how sharing our experiences can help others develop skills - and also, maybe, just maybe, might be a little fun.
Once you’ve done that, here’s a question: What skill(s) does this exercise help develop?
More to come.
User Comments
Matt,
I went through your exercises and wrote about it on my blog:
http://justinrohrman.com/blog/practice-makes-progress/
Not to spoil it before reading, but I did find a bug that a real user could encounter while using ParkCalc. I'm interested to see what other people will find.
This exercise was fun, thanks for the thoughtful post.
Hey Justin!
How am I not surprised that you are the first one to step up to this, or that you would take the training wheels off by removing the request to limit tests to short-term?
Either way, that was nice. I especially like the "what I learned from each test idea" portion.
Toward the end, you said you'd be inclined to say, with 4.98 of the tests "passing", that the software would be fit for use. (Actually, you covered yourself and said you'd be /disinclined/ to call it /unfit/ for use, which is subtly different.)
Do you have a theory of verification on that? After 5 tests, what do you actually know? How do you know it?
Well, removal of the training wheels was sort of accidental :). I misread the mission. I'm pretty happy with the results either way though.
re: my verification theory
After running 5 tests, I was able to see that the application could work reasonably well under what I imaging to be common or likely to occur usage scenarios. My theory was based on personal experience using airport lot pay machines and also how I imagine people would use this calculator to anticipate cost.
I'm imaging a person that is just trying to get an idea of what it might cost to park in a lot. They probably aren't super concerned with the exact rate, just want a general idea. Maybe so they know how much cash to bring.
I think the slight tolerence between what the spec says should be charged and what was actually charged are small enough that this person might not care. I imagine that a person really concerned for small cash differences might catch a ride to the airport from a friend.
This was fun, I blogged my responses here http://www.metranterc.com/2014/01/practice-makes-progress-response.html
The biggest things for me was trying to focus and not just start playing with the app, and secondly, to react to what that app tells me to help define the next text....
That's great, Chris. I assume ParkCalc was new to you, and your slight changes were an attempt tp "make the test your own", which is fine. For those familiar with the challenge, I hoped my specifics would be a chance to dig deeper. Maybe try my specific request again in a week? I'd like to start conversations about the idea of test katas. :-)
One thing I didn't see in your blog post, and maybe I didn't make my ask clear enough in the first place:After those five tests, what do you know about the software? When the customer asks "can we ship it?" what do you say?
Hi, Yes it was the first time I'd seen ParkCalc - was it obvious 8-). In hindsite I could have stuck around the boundaries of the requirements (2 hours, 2 hours 1min, 2hours 30 min, 2 hours 31 min, 12 hours 1 min, etc). Partly this is becuase as the most you will pay a day in Long Term parking is $13 - then no one in their right mind would plan a stay of longer than 5 or 6 hours in a short term car park. Finding defects in calculations over multiple days is probably of little value.
However, playing in these areas showed me some rough spots in the application which would be worth conversations with the developers; 1) in the page source the code recognises invalid date/time formats and shows as red in code when the user updates the field, but the user isn't warned - which seems like an oversight/bug, etc. 2) If this is a short term calc then does the user really need to be able to select multiple days (any stay more than 6 hours and you may as well go in Long Term parking), 3) if the date/time format is important then why not force the user to use a calendar/date/time picker? 4) If you select a date from the calendar such as 01/09/2014 it autofills in the cell as 1/9/2014 - whereas the date format is meant to be mm/dd/yyyy - slight inconsistency.
If someone asked me if we could ship, my default answer is 'here's what I found you decide if it's useful/important/worthy of more investigation". I'd like to discuss these items with the developers, but balancing these findings against the fact that 'it's just a calculator for an airport carpark' would any of these 'issues lose business for the carpark, would there be any risk from these findings (only if the calc showed a lesser fee than someone was charged, and they subsequently sued ?)
Nice exercise. As a software tester, could I read the blog entries, consider the inputs used and then interact with the software? :)
Yes, Ajay, that'd be a fine way to make the test have both more depth/complexity and be more personal.
Thinking to practice and keep this topic as weekendtesting session. Thanks Matt
We had a weekend testing session 104 on this testing exercise and updated the test documents on WeekendTesting website.
http://weekendtesting.com/archives/3362
This exercise was fun and learning on different time zones and military times, thanks for the post.
Very interesting article.
Very useful.