Empirical research reveals that software testing tools profoundly impact practitioners' emotional well-being and professional identity. Beyond technical or usability hurdles, poorly implemented automation triggers negative emotional responses and demotivation. Organizations must consider the totality of "Testers' Lived Experience" (TX) to prevent project failure and role erosion.
"I don't really count what I did there as testing though, I ran scripts I didn't write, generated reports I didn't understand and quite honestly I don't think I could really tell you what the product I was testing did in any great detail. My only role was to make sure that the QC test run numbers were met and a spreadsheet was green."
It is not always easy being a tester! In my research, I heard from testers who were happy and fulfilled in cognitively challenging roles, yet there were others who were suffering from a loss of their professional identity and their autonomy, sometimes because of the tools they were forced to use.
In this article, the first in a series about my research findings, I focus on the human elements of tooling and automation and why this matters not just to testers, but also to team leads, managers and organisations. My research journey into tools to support testing started as just that: I perceived that teams faced initial implementation hurdles, followed by long-term maintenance difficulties. I started to wonder why tooling—which should be a benefit to individual testers and to teams—seemed to be challenging. Could I identify the root cause, and maybe suggest a solution? I was convinced this was a technical problem; how wrong I was.
Choosing to take an academic route to research this perceived problem was a wise decision. Academic processes are slow, exacting, and one is constantly challenged about one's methods and results, as well as how those are interpreted. My academic supervisors explained to me that before I started to pin down the problem and find a solution, I had to show evidence that there is a problem. My first question was not, "what are your problems?" but "what are your experiences?" This provided me with a rich collection of data that showed both the range of problems and successes that testers reported with their tools and automation. It was these first results that made me realise: this research is not just about tools and organisations, but fundamentally about people.
The key finding I want to share with you in this article is the intensely emotional language that testers use when talking about their toolset and automation; the tools and automation have a profound impact on the lived experiences of testers in their workplaces.
Industry experts and academic writers had already published three categories of blockers to success with test tools: technical, organisational, and usability challenges. My research confirmed all three (technical, organisational, and usability) are genuine barriers. Charts 1 and 2 show the results from the 2021 dataset, where usability issues were the most widely mentioned (109 of 111 participants; 511 references). Technical problems came second (102 participants; 466 references). Organisational issues were third (101 participants; 377 references), with 33 survey responses specifically highlighting management support—or the lack of it—as critical to success or failure.
Chart 1: Total number of survey references categorized by usability, technical, and organizational challenges.
Chart 2: The percentage of overall research participants who experienced each category of barrier.
The striking finding was the emotional content of testers' responses. Around 30% of survey participants used emotional language even though the questions were neutral and answered in writing, with no interviewer present to cue emotional responses. Face to face interviews, conversations and workshops also provided evidence of emotional content, despite not being elicited; I did not ask people how they felt, I asked them for a story about tool use, to share an experience.
The emotional range was broad: frustration, fear, pride, triumph, and anger were all present. Indicators of emotion included obvious emotional words ("frustrated," "scary," "proud"), emojis (including frowny faces), heavy punctuation, swear words, body language in interviews (one participant was bouncing on their chair; others were in tears), and the phenomenon of giving a restrained answer in a workshop and then a much more candid one privately, indicating fear: "I didn't like to say in front of the others...".
Sadly, negative emotions dominated the responses, and these in particular showed effect on lived experience, such as the despair and the desire to leave:
"I think I should leave my job and look for (a) company who actually values testing."
"...if we make this crap again, what's the point?"
Positive emotions, including pride and satisfaction, were also real and important:
"After being frustrated by it [the tool] (as it cost me a day and a billing run) I was quietly proud (of breaking it)."
"I have a beautiful framework..."
I had already analysed the responses and categorised problems into usability, technical and organisational. When I began to notice the emotion level, I reanalysed all the data. I compared the emotions with the number of problems reported. Notably, the number of problems a participant reported did not reliably predict the level of emotional language. Some people reported many problems without much emotion; others were highly emotional about a single issue.
Emotions were expressed about each of the types of challenge. Organisational failures, changes in direction from management and tools suppliers, and unreasonable expectations, all caused negative emotions:
"(The manager) didn't realise software is a bloody difficult thing to build... it took three years as opposed to three months."
"Frustrated [by] lack of internal support [...] even after discussions with the 'sponsor.'"
"Getting the hang of it [...] modifying processes [...] found it was no longer supported... Aargh!"
Tools not supporting the testing workflow, and even the choice of tools, all added to testers' cognitive load and frustration:
"(The best solution I have found) is a combo of (Tool 1) and (Tool 2), but neither does exactly what I want. Very frustrating."
"How to choose test tools in a 'jungle' out there."
Fear and powerlessness arose through encounters with new tools when testers were unsure if they had the skill set and capability and also when they wanted to learn new skills but were blocked from training. Organisational attitudes to training testers also raised frustration and anger.
"It is scary and I always get stuck. I am delaying the inevitable (frowny face)"
"It was my chance to learn and they ripped it away"
"The developers were given 3 months to learn the new framework, and the testers 3 weeks! What were they [management] thinking?"
These are just a few of the stories I was told, in conversations, interviews, surveys and workshops, where emotions ran high. But why does this matter? It matters because it has a potential or actual effect on motivation.
I identify three specific ways automation and tooling threatens testers' motivation:
1. Fear of redundancy. When automation aims to replace human testers entirely, those testers face a threat not just to their job but to their professional identity. Some responses reported organisations that had "sacked all the testers" in the words of one participant. The story at the top of the article supports Michael Bolton and James Bach's argument that "test automation" as a concept threatens to "dissociate people from their work."
2. Unbalanced job mix. People need a mix of routine and stimulating tasks to remain motivated. If automation removes the routine parts, only the most cognitively demanding and stressful work remains—and that imbalance is itself demotivating. If, on the other hand, people's jobs are reduced to just the routine, such as checking spreadsheets of results without understanding; that is a job for a tool—for AI—not for a person. A well-designed job is cognitively challenging and contains some lower key tasks. Testers use understanding, problem solving, lateral thinking, and exploration in a way that a tool cannot. A recent article by Dave Snowden describes types of reasoning and focuses on reasoning that a person can do, but a tool—even an AI tool—cannot do. This reasoning is both motivating to the testers and valuable to the organisation.
3. Frustration with flawed automation. When tools don't work reliably but are still deployed, testers must manage both the broken technology and the consequences of its failures. Roughly one third of participants in one of my studies reported problems of this kind.
Whether you are a tester, or have testing in your responsibilities, or lead testers, take the totality of testers' experience seriously. When you design a role, whatever you call it (and the fashion moves through SDET, quality engineer, tester, QA analyst…) think about their autonomy, professional pride, wellbeing, and their emotional responses to the technology they depend on. Acquiring the right tool to support testing now needs to include considering the emotional and lived dimensions of those practitioners' working lives. Just as you consider UX—the experience of your users, and some people are discussing DX, the experience of the developers with their tools, don’t forget this concept: TX: The Testers' Lived Experience of Tools and Automation.
When you are setting a tooling strategy, or acquiring a new tool, take time to understand who will be using it, and what they need, to support their work, and the lived experience of their tools. “Audit” means “to listen, to hear”: audit your testers, listen to their stories, and show them what you hear and will act on what they need from their tool set.
In part two of this series, we will dissect the illusion of usability and how it might adversely affect TX, and potentially increase shelfware… increasing waste. And, I will show how the research led to suggestions for how to better understand testers, TX, and testers’ requirements for their tools.
TX: The Testers' Lived Experience of Tools and Automation—the research concept developed across these papers.
LX (Lived Experience): The full impact of technology on a person's daily life, beyond immediate task use (after Porter, 2015).
UX (User Experience): All aspects of a user's interaction with a product and its provider (Norman and Nielsen, 2020).
Usability: Effectiveness, efficiency, and satisfaction with which a user achieves goals in a given context (ISO 9241-11:2018).
Shelfware: Tools acquired by organisations but not put into use.
HCI: Human-Computer Interaction—the research field studying how people interact with technology.
Mixed methods: The method used for this research, an approach combining quantitative (e.g. surveys, frequency counts) and qualitative (e.g. interviews, thematic coding) methods.
These papers were co-authored with Chris Porter, Mark Micallef, and Julian Harty, who contributed to the data analysis and writing of the paper:
- Evans, I., Porter, C., Micallef, M., and Harty, J. (2020). Stuck in Limbo with Magical Solutions: The Testers' Lived Experiences of Tools and Automation. *VISIGRAPP 2020*, 195–202. https://raw.githubusercontent.com/hci-lab-um/heuristics-for-test-tool-d…;
- Evans, I., Porter, C., and Micallef, M. (2021). Scared, Frustrated and Quietly Proud: Testers' Lived Experience of Tools and Automation. *ECCE 2021*, Siena, Italy. https://raw.githubusercontent.com/hci-lab-um/heuristics-for-test-tool-d…
- Bach, J. and Bolton, M. (2016). *A context-driven approach to automation in testing.* https://www.satisfice.com/download/a-context-driven-approach-to-automat…
- Graham, D. and Fewster, M. (2012). *Experiences of Test Automation: case studies of software test automation.* Addison-Wesley.
- Hackman, J. R. and Oldham, G. R. (1974). *The job diagnostic survey: An instrument for the diagnosis of jobs and the evaluation of job redesign projects.*
- Harty, J. (2011). Finding usability bugs with automated tests. *Communications of the ACM*, 54(2), 44–49.
- ISO (2018). *ISO 9241-11:2018 Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts.*
- Norman, D. and Nielsen, J. (2020). The Definition of User Experience (UX). https://www.nngroup.com/articles/definition-user-experience/
- Porter, C. (2015). *Designing for Experience — a requirements framework for enrolment based and public facing e-government services.* PhD thesis, University College London.
- Reid, S. (2015). *Motivated or Motivating? What sort of tester are you?* https://www.stureid.info/stuart-reid-software-testing/software-testing-… and https://www.slideshare.net/slideshow/motivation-and-behavior-presentati…
- Warden, R. and Nicholson, I. (1996). Motivational Survey of IT staff. *Software Futures.*
- Wiklund, K. (2015). *Impediments for Automated Software Test Execution.* PhD thesis, Mälardalen University.
- Snowden, D (2026) The Shape of Knowing https://thecynefin.co/the-shape-of-knowing/
Lets Hang!