Generative AI—Is It the Catalyst for Evolution in Test Automation?


Generative AI is changing testing practices by automating the creation of test cases, adapting to software changes and improving test efficiency. This highlights the growing importance of artificial intelligence in improving test coverage and accuracy, making test automation even more adaptive and intelligent. It has the potential to change the way software is tested, ultimately leading to higher-quality software products.

When I think about artificial intelligence (AI), I'm reminded of a time not too long ago when AI was just a futuristic concept, something we saw in sci-fi movies and read about in books. Little did I know that AI would soon become an integral part of my daily life, reshaping how I work and make decisions.

As a software test automation lead, I've seen the testing landscape evolve dramatically. But the introduction of AI genuinely revolutionized how I approached my work.

Before Generative AI

Automation testing has been an invaluable tool throughout the software development lifecycle, enabling rapid execution of test cases and reducing the time and effort required for regression testing. Automation frameworks and command sets have played a central role in ensuring software functionality, detecting defects and maintaining product quality. New AI tools, however, with neural networks and deep learning models continuously emerge, and most of them are very effective tools that definitely improve testing.

But these conventional testing methodologies possess their own inherent restrictions. Let us delve into a few of these limitations:

  • Maintenance costs: Automation scripts require constant maintenance as the software evolves, which increases the cost and effort of script maintenance.
  • Limited scope of testing: Automation testing focuses on predefined tests that often struggle to adapt to dynamic changes in the software environment.
  • Complex UI testing: Extensive automation of user interface (UI) testing can be complex, leading to gaps in test coverage.

Generative AI can help overcome some of these limitations, but it may also bring about new challenges.

Generative AI

Generative AI is a type of deep learning that can generate new data or content similar to the original data or content. For example, generative AI can create realistic images of faces, animals or landscapes that do not exist in reality, as well as create text that imitates the style and tone of a particular genre. Generative AI uses various techniques such as generative adversarial networks (GAN), variational autoencoders (VAE), and transformers to learn patterns and features of data and then generate new samples.

In recent years, the field of generative AI software testing has undergone significant change. Traditionally, automation testing has been the cornerstone of software quality assurance, ensuring the efficiency and accuracy of software functionality verification. However, Generative AI is poised to transform the automation testing landscape, introducing new approaches and capabilities that promise to change the way software is assured:

  • Test case generation: Generative AI can dynamically generate test cases by learning how the application works, identifying edge cases and creating test scenarios that humans might miss.
  • Self-correcting: Test automation tools that use AI algorithms can adapt to software changes and update test locators if they change, which reduces the maintenance of testers.
  • Improved UI testing: After learning UI, Generative AI can simulate human interaction with UI.
  • Larger data sets: Generative AI analyzes large data sets and generates test data with different combinations, discovering subtle flaws.
  • Shift-left testing: Generative AI facilitates error detection earlier in the development process, reducing the cost and effort of debugging later.

There are a few challenges and factors, however, that must be taken into account prior to the employing of any generative AI models:

  • Data Quality: Models need high-quality training data to perform effectively—garbage in and garbage out applies here.
  • Expensive: Machine learning & deep learning are an expensive technology to implement. It is estimated that the resources for developing a large AI model have roughly doubled since its inception.
  • Interpretability: Understanding AI-generated test cases and results can be challenging. Ensuring transparency and interpretability is crucial for trust and accountability.

After all this discussion on the drawbacks of traditional automation and the benefits and challenges of generative AI, the testing methodology and disciplines aren’t going anywhere. In fact, it creates new opportunities to build and learn new skills, tools, and roles. The requirements to optimize and go faster will never go away—and ChatGPT and generative AI help us automation testers do just that.

Below is an example which clearly states how ChatGPT or generative AI can help automation engineers to some extent, but to finish the work, the engineer needs to use their set of skills to fine-tune the solution provided by the generative AI tool.

Let’s say you want to test the UI for a web application and its browser-based app, and you decide to use chatGPT to help generate the automation script, sending the following message to ChatGPT:

“Generate selenium java script to visit ”” and choose the “Medical Care” option and click “See all health on Amazon.”

Here is ChatGPT’s response: 

 Image 1 

The script is close but there’s one major problem: the script is trying to click on an element which is actually not clickable, so when we run this script as it is, it will fail. An automation engineer can easily identify this issue. There are many more such examples which need human intervention and expertise. So, in this case, generative AI is capable of doing perhaps 80% of the work necessary to solve a problem, but the rest of the fine-tuned work needs domain-specific knowledge and expertise.


Generative AI is evolving at a rapid pace, but at this time it's not capable of replacing automation engineers. As generative AI evolves more, automation engineers need to also change their testing practices by using this technology to deliver bug-free products in a more efficient way. Will ChatGPT or generative AI eventually take automation engineers’ jobs? That we don’t know, but one thing we do know is AI will change the testing landscape forever.

User Comments

Bjorn van der Neut's picture

I find the example not as usefull with AI in mind. I would like to say something like:

"Generate Playwright test scripts to test page make sure you test input field, search filters, checkboxes and also mobile design etc.". 

And then It would generate multiple tests that I did not specified what to test. That would make the AI way more usefull.


November 29, 2023 - 5:03am
Arvind Kumar Bhardwaj's picture

Hi Bjorn,

Thank you very much for your insightful feedback. I appreciate your perspective on enhancing the usefulness of AI in the context of generating Playwright test scripts. Your suggestion to automate the generation of comprehensive tests, including testing various web elements like input fields, search filters, checkboxes, and mobile design on websites such as amazon, indeed reflects a more advanced and practical approach to leveraging AI in web development and testing.

The idea of AI automatically identifying and creating tests for elements not explicitly specified by the user holds great potential for improving efficiency and coverage in testing scenarios. It would not only streamline the testing process but also ensure a more thorough validation of web applications across different devices and user interactions.

I agree that the current example in the article might not fully capture the breadth of possibilities AI can offer in this domain. Incorporating AI to intelligently generate tests for a broader range of web elements and scenarios could significantly elevate its utility. This approach could lead to more robust web applications, as it allows developers and testers to catch potential issues early in the development cycle.

Your feedback is invaluable, and it opens up an exciting discussion about the future of AI in automating and enhancing web testing. I'll definitely explore this idea further and consider how AI can be applied more innovatively in test script generation to meet and exceed the evolving needs of web development.

Thank you once again for your contribution and for sparking an important conversation about advancing AI applications in our field.

Best regards,

Arvind Bhardwaj

February 11, 2024 - 2:02pm
ideya Labs's picture

Hey there! This article on AI revolutionizing test automation totally hits the spot. It's awesome seeing how generative AI is transforming the game, making test automation smarter and more efficient. The way it's shaping the future of QA testing is mind-blowing. Can't wait to see where this tech takes us next! Great read, thanks for sharing!

December 11, 2023 - 6:32am
Georgia Coleman's picture
I admire this article for its well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much.
January 15, 2024 - 5:06am
Arvind Kumar Bhardwaj's picture

Hi Georgia,

Thank you very much for your kind words and for taking the time to express your appreciation. Please feel free to share any thoughts or questions you might have about the topic; I would love to engage in further discussion.

Warm regards,

Arvind Bhardwaj

February 11, 2024 - 1:59pm

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.