Next-Generation Exercises for Software Testers

[article]
Summary:
How often do we talk about what we actually do when we test? Many of the test exercises out there predate the Internet and aren't as relevant anymore. This article contains some software you can play around with and use in mock test scenarios. These exercises will make you think about not only what you do when you test, but all the other skills that also go into being a tester.

Plenty of articles talk about testing schools of thought, testing theory, and testing tools. But how often do we talk about what we actually do when we test?

If you search the web for test exercises, you’ll likely find some programs that simulate a triangle, a word count exercise, or a parking calculator. Most of these exercises predate the Internet. Some are Windows programs designed for a monitor with one-tenth the pixels; they fit in a corner of today’s screens. Many are entirely pen-to-paper. The few that are web-based predate mobile devices and responsive design.

Two years ago, I decided to do something about it. I ran the WorksHop on Test Design, or “WHaTDa,” in Columbus, Ohio. My public promise was to develop some test training material at the workshop and give it away.

We took a word problem and put it online, and if I do say so myself, it is fantastic. So fantastic, in fact, that I was selfish: My company used it as an interview problem and a training exercise.

But it’s time to keep my promise and provide a test exercise to everyone. Here’s the setup.

The Palindrome Exercise

You are a recently hired tester. The company has software it wants to ship at the end of the day that tests palindromes—words that read the same backward as forward. (So “bob” is a palindrome, but “robert” is not.) You type in some text, click Submit, and the software tells you whether the text is a palindrome—pretty simple. The lead developer is out sick, but there is a junior developer who can fix bugs that you find. We need time to fix and retest, of course. The product owner is nontechnical, but he can answer questions.

That’s enough detail for you to test the software. Here’s the website where you can find it. (You can ignore the Anagram section; that’s not finished yet.) Check it out and leave comments about your favorite bugs.

But before you go and do that, think about it for a moment. The real power in this exercise is where you can take it when you have someone else playing the role of product owner. Let’s talk about a few places this could go:

  • What browsers are you testing it in? What mobile devices? When do you stop?
  • How long will it take you to test?
  • What makes a bug, a bug? Which issues are bugs? Which are not?
  • The product owner is worried about performance of the API—the call made on the submit button. Can you isolate the API? How would you performance test it?
  • Can you find any potential security issues on the page? Accessibility? Internationalization?
  • Can the software ship or not? (This often leads to a big stupid argument about the tester role, followed by “Can you at least make a recommendation? Can we have a conversation?”)

Testers who run these exercises typically don’t do well. They have a series of other skills. They’ll wiggle on the hook, trying to get the junior developer to do the research, or call the senior developer on her hospital bed. They’ll pull out dog-eared copies of How To Win Friends and Influence People or The 7 Habits of Highly Effective People. Or they won’t. Some testers explain that the questions above are for the developers—not their job. Some automators say, “Just hand me the test cases and I’ll automate them.”

And a few—just a few—testers have the technical skills to model the risks, do the technical investigation, and handle the uneducated customer. That sort of conversation requires a new set of skills: the kind we increasingly see in demand from our customers.

Skills for the Modern Software Tester

In the 1990s, software came on physical disks and ship decisions were expensive. Today, companies are trying to ship more often, improve the quality of code on the first release, and notice and fix problems more quickly. Programmers are writing unit tests, creating integration tests, and even driving the user interface, and it’s all hooked into continuous integration. Superficial tester skills—the skills needed to find obvious problems—are becoming increasingly marginalized.

Modern testers don’t need to write full-stack applications, but they likely know enough about accessibility, internationalization, platforms, simulating Wi-Fi speed, networking, HTML, CSS, TCP/IP, and JavaScript to be able to debug web applications. Many also probably know enough code to write a log analyzer, have the skills to analyze server logs, and understand enough about virtualization to create test machines and test servers, plus debugging, if not setting up and running continuous integration.

That’s the nature of the modern web. Modern testers can specialize in a domain, such as databases, large text data sets, native mobile applications, API, or legacy systems, but to get the next job, testers will need to be able to learn new technologies.

And that’s just the hard tech skills. Of course, we also need to understand common failure modes, to have the people skills to get to know the customer, and to continue to learn to think.

For today, though, test the palindrome software. Pass it around. Let’s talk about how to make the exercise better—and about how we can all develop our testing skills for the future.

User Comments

16 comments
Zephan Schroeder's picture

Great post Matthew!

I love the idea of providing a web-based application as a software test exercise! I found 2 bugs and 2 potential bugs in 5 min. No spoilers posted because I want to focus on your request (instead of giving away answers to your great interview exercise):

Let’s talk about how to make the exercise better—and about how we can all develop our testing skills for the future.

a) Host or link to a bug/discussion forum connected to "The Palindrome Exercise".

Level 1: Link to an open forum and let it run unmoderated. Easy to implement and unconstrained, but will make answers to your interview question discoverable and will provide limited productive discussion guidance. (GitHub project or ???)

Level 2: Link to a site that allows bugs to be reported publicly but not see other reports. This supports the interview exercise use case. But it blocks general discussion and collaborative improvements.

Level 3: Restrict the fully open discussion link. This enables separation of "test/interview use" and "discussion/collaboration use".  COOL IDEA:  Perhaps hide the open discussion URL in The Palindrome Exercise page HTML source). Someone who analyzes the source will be able to proceed to a deeper level. 

COOL IDEA:  Perhaps hide the open discussion URL in The Palindrome Exercise page HTML source). Someone who analyzes the source will be able to proceed to a deeper level. 

Level 4: Report stats for # of bugs reported by triage state (untriaged, confirmed, duplicate, by design,  rejected) and by priority. This feature requires some effort to create or find+configure. It also requires triaging bug submissions (ideally leveraging open discussion participants) but gives users some context and sense of contribution to the exercise.

Level 5: Require user to enter 5 bugs before they can get into the open discussion. Again this supports both the interview and the discussion use cases. Then require the new user to triage their own submissions once access is granted! This enables level 1-4 benefits plus the triaging and prioritization is in itself a great Interview/Testing exercise.

b) link to a survey (surveymonkey.com or whatever) that asks a series of questions for the Tester/interview candidate to answer regarding the Palindrome Exercise.

Example questions include:

  • Questions you posed in this article (how to test API for this app, ...)
  • How many bugs do you think you found?
  • List your candidate bug titles (Less than 60 char per bug title)
  • Describe the highest priority bug you found and why is it a priority? (open text - this also tests bug report/writing skill)
  • ...

I hope this is a good discussion starter. Many more ideas to cover. What do you think? Add to this discussion!

March 7, 2017 - 6:02pm
Matthew Heusser's picture

wow. There's so much here. Thanks man. I may be in touch; I assume you're on linkedin?

January 3, 2018 - 6:11pm
Omer Nir's picture

great idea and article!!

 I would add a link to specs or requirements document with bugs on them, to see the tester's involvement from the start of design.

also, to see how the tester creates a test plan design and estimates testing time, coverage and release gate.

March 9, 2017 - 3:30pm
Tim Thompson's picture

Specs? Requirements? This is 2017, we are all agile now! There are no specs or requirements...at least I haven't seen any in years unless one counts the one and a half bullet points in a backlog item in TFS.

March 18, 2017 - 11:06am
Thomas Sullivan's picture

Matt,

 

Recalling when you interviewed me for work at Excelon last year, we went through this exercise.  I must say I was truly struck by the comparison with all the other tester interviews that I'd experienced, or rather, endured.  These typically consisted of reciting the litany of test deliverables (i.e. test plan, test cases, test report) and maybe a logic puzzle or two.  You, on the other hand, actually had me test something, live.  Coming out of the interview, I felt with confidence that you had assessed my skills as a tester -- a genuinely novel experience for me.  Thanks for your hard work on this!

March 15, 2017 - 2:50pm
Matthew Heusser's picture

Let's work on more things together in 2018, man!

January 3, 2018 - 6:12pm
Tim Thompson's picture

Thank you for bringing training material into today's world....although, a lot things that were valid 20 years ago are still valid today, but a few items were added. Back then in a desktop app only world who cared about HTML or JavaScript injections?

Favorite bugs:

  • "Reversed" spelled wrong, but if the strings are in a resource file and I as a tester have access to source I can fix that myself, no need to waste time on a bug report and get a developer involved.
  • HTML/JS injection: the input needs to be sanitized, all input is evil! When entering <b>err</b> the reversal routine also generates the wrong result: >b/b<
  • Other text issues such as "This" being capitalized...although some UX designers insist on cringe-worthy formats, such as everything being all lower case
  • No maxlength set on input field, when entering a very long string the result runs off the page

That is about what I tested in two minutes. The basic function of checking and reversing appears to work with a handful of test phrases. That said, let's take care of the injection issue and fix the spelling errors, then ship. It will ship today no matter what because when management sets a ship date that is when stuff goes out the door. These days hitting delivery dates is way more important to businesses than removing all the issues. That is what unscheduled updates are for and those only come if enough customers complained.

 

March 18, 2017 - 11:02am
Tim Thompson's picture

Forgot one thing:

The performance concerns of the product owner are duly noted, but unless there are any hard metrics to confirm (x hits successfully processed per minute) I ignore this concern. I do not know if 100 successful hits per minute are good enough or not and what the response time needs to be.

Assuming a web API, Postman is your friend for basic tests, for performance testing an automated approach is needed unless we can get a bunch of people in the office to hammer the site at the same time. That brings me to another aspect for modern testing: crowd-sourcing. Get support and sales involved, they have the opportunity to learn the new features and they can deliver valuable test results at the same time.

March 18, 2017 - 11:13am
Matthew Heusser's picture

Hey Tom -

Thank you for the serious and considered reply - but before we jump to postman is your friend, let's look at the question as-asked. Can you isolate the API? What is it? Once you have that we can talk strategy. :-)

January 3, 2018 - 6:10pm

Pages

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.