Plenty of articles talk about testing schools of thought, testing theory, and testing tools. But how often do we talk about what we actually do when we test?
If you search the web for test exercises, you’ll likely find some programs that simulate a triangle, a word count exercise, or a parking calculator. Most of these exercises predate the Internet. Some are Windows programs designed for a monitor with one-tenth the pixels; they fit in a corner of today’s screens. Many are entirely pen-to-paper. The few that are web-based predate mobile devices and responsive design.
Two years ago, I decided to do something about it. I ran the WorksHop on Test Design, or “WHaTDa,” in Columbus, Ohio. My public promise was to develop some test training material at the workshop and give it away.
We took a word problem and put it online, and if I do say so myself, it is fantastic. So fantastic, in fact, that I was selfish: My company used it as an interview problem and a training exercise.
But it’s time to keep my promise and provide a test exercise to everyone. Here’s the setup.
The Palindrome Exercise
You are a recently hired tester. The company has software it wants to ship at the end of the day that tests palindromes—words that read the same backward as forward. (So “bob” is a palindrome, but “robert” is not.) You type in some text, click Submit, and the software tells you whether the text is a palindrome—pretty simple. The lead developer is out sick, but there is a junior developer who can fix bugs that you find. We need time to fix and retest, of course. The product owner is nontechnical, but he can answer questions.
That’s enough detail for you to test the software. Here’s the website where you can find it. (You can ignore the Anagram section; that’s not finished yet.) Check it out and leave comments about your favorite bugs.
But before you go and do that, think about it for a moment. The real power in this exercise is where you can take it when you have someone else playing the role of product owner. Let’s talk about a few places this could go:
- What browsers are you testing it in? What mobile devices? When do you stop?
- How long will it take you to test?
- What makes a bug, a bug? Which issues are bugs? Which are not?
- The product owner is worried about performance of the API—the call made on the submit button. Can you isolate the API? How would you performance test it?
- Can you find any potential security issues on the page? Accessibility? Internationalization?
- Can the software ship or not? (This often leads to a big stupid argument about the tester role, followed by “Can you at least make a recommendation? Can we have a conversation?”)
Testers who run these exercises typically don’t do well. They have a series of other skills. They’ll wiggle on the hook, trying to get the junior developer to do the research, or call the senior developer on her hospital bed. They’ll pull out dog-eared copies of How To Win Friends and Influence People or The 7 Habits of Highly Effective People. Or they won’t. Some testers explain that the questions above are for the developers—not their job. Some automators say, “Just hand me the test cases and I’ll automate them.”
And a few—just a few—testers have the technical skills to model the risks, do the technical investigation, and handle the uneducated customer. That sort of conversation requires a new set of skills: the kind we increasingly see in demand from our customers.
Skills for the Modern Software Tester
In the 1990s, software came on physical disks and ship decisions were expensive. Today, companies are trying to ship more often, improve the quality of code on the first release, and notice and fix problems more quickly. Programmers are writing unit tests, creating integration tests, and even driving the user interface, and it’s all hooked into continuous integration. Superficial tester skills—the skills needed to find obvious problems—are becoming increasingly marginalized.
That’s the nature of the modern web. Modern testers can specialize in a domain, such as databases, large text data sets, native mobile applications, API, or legacy systems, but to get the next job, testers will need to be able to learn new technologies.
And that’s just the hard tech skills. Of course, we also need to understand common failure modes, to have the people skills to get to know the customer, and to continue to learn to think.
For today, though, test the palindrome software. Pass it around. Let’s talk about how to make the exercise better—and about how we can all develop our testing skills for the future.
Great post Matthew!
I love the idea of providing a web-based application as a software test exercise! I found 2 bugs and 2 potential bugs in 5 min. No spoilers posted because I want to focus on your request (instead of giving away answers to your great interview exercise):
a) Host or link to a bug/discussion forum connected to "The Palindrome Exercise".
Level 1: Link to an open forum and let it run unmoderated. Easy to implement and unconstrained, but will make answers to your interview question discoverable and will provide limited productive discussion guidance. (GitHub project or ???)
Level 2: Link to a site that allows bugs to be reported publicly but not see other reports. This supports the interview exercise use case. But it blocks general discussion and collaborative improvements.
Level 3: Restrict the fully open discussion link. This enables separation of "test/interview use" and "discussion/collaboration use". COOL IDEA: Perhaps hide the open discussion URL in The Palindrome Exercise page HTML source). Someone who analyzes the source will be able to proceed to a deeper level.
COOL IDEA: Perhaps hide the open discussion URL in The Palindrome Exercise page HTML source). Someone who analyzes the source will be able to proceed to a deeper level.
Level 4: Report stats for # of bugs reported by triage state (untriaged, confirmed, duplicate, by design, rejected) and by priority. This feature requires some effort to create or find+configure. It also requires triaging bug submissions (ideally leveraging open discussion participants) but gives users some context and sense of contribution to the exercise.
Level 5: Require user to enter 5 bugs before they can get into the open discussion. Again this supports both the interview and the discussion use cases. Then require the new user to triage their own submissions once access is granted! This enables level 1-4 benefits plus the triaging and prioritization is in itself a great Interview/Testing exercise.
b) link to a survey (surveymonkey.com or whatever) that asks a series of questions for the Tester/interview candidate to answer regarding the Palindrome Exercise.
Example questions include:
I hope this is a good discussion starter. Many more ideas to cover. What do you think? Add to this discussion!
wow. There's so much here. Thanks man. I may be in touch; I assume you're on linkedin?
Dammit, I'm Mad!
great idea and article!!
I would add a link to specs or requirements document with bugs on them, to see the tester's involvement from the start of design.
also, to see how the tester creates a test plan design and estimates testing time, coverage and release gate.
Specs? Requirements? This is 2017, we are all agile now! There are no specs or requirements...at least I haven't seen any in years unless one counts the one and a half bullet points in a backlog item in TFS.
Recalling when you interviewed me for work at Excelon last year, we went through this exercise. I must say I was truly struck by the comparison with all the other tester interviews that I'd experienced, or rather, endured. These typically consisted of reciting the litany of test deliverables (i.e. test plan, test cases, test report) and maybe a logic puzzle or two. You, on the other hand, actually had me test something, live. Coming out of the interview, I felt with confidence that you had assessed my skills as a tester -- a genuinely novel experience for me. Thanks for your hard work on this!
Let's work on more things together in 2018, man!
That is about what I tested in two minutes. The basic function of checking and reversing appears to work with a handful of test phrases. That said, let's take care of the injection issue and fix the spelling errors, then ship. It will ship today no matter what because when management sets a ship date that is when stuff goes out the door. These days hitting delivery dates is way more important to businesses than removing all the issues. That is what unscheduled updates are for and those only come if enough customers complained.
Forgot one thing:
The performance concerns of the product owner are duly noted, but unless there are any hard metrics to confirm (x hits successfully processed per minute) I ignore this concern. I do not know if 100 successful hits per minute are good enough or not and what the response time needs to be.
Assuming a web API, Postman is your friend for basic tests, for performance testing an automated approach is needed unless we can get a bunch of people in the office to hammer the site at the same time. That brings me to another aspect for modern testing: crowd-sourcing. Get support and sales involved, they have the opportunity to learn the new features and they can deliver valuable test results at the same time.
Hey Tom -
Thank you for the serious and considered reply - but before we jump to postman is your friend, let's look at the question as-asked. Can you isolate the API? What is it? Once you have that we can talk strategy. :-)