Less Is More: Picking Your Test Automation Language


It's a classic dispute: Two test automation engineers can't agree on which programming language to use. In some contexts, the strong points of a certain language definitively make it the right choice, but what do you do when either language could work well for a project? That's when it becomes a managerial decision.

It was years ago, while working as a chip-level test engineer, that I first I witnessed an argument that has repeated itself a dozen times since. Two test automation engineers could not agree which programming language was better for their next-generation automation system: Pascal or C? (I did mention that it was years ago, right?)

Each side had weighty arguments: what’s easier to do in each language, how important strong string-processing is, what language has better libraries, etc. Additionally, both sides claimed that there isn’t really a great need to align on one language—it is possible to write modules in one language and use them as libraries for the other language.

The arguments continued for a while. If my memory serves, the automation was mostly written in C for a simple reason: It was the preferred language of the developer who wrote most of the code.

There are cases where the decision is clear-cut and easy, because in some contexts the strong points of a certain programming language definitively make it the right choice. But what do you do when this is not the case, and each language has a list of pros and cons?

Years later I was consulting for a team of testers writing a software test automation framework. Except for the fact that the languages discussed were now Visual Basic and C#, the rest was very familiar: the endless arguments; the teleconferences and face-to-face meetings, vowing to continue until we reach a decision; the presentation of proofs by each side about why their position is the correct one—all while employing both languages as each side tries to generate enough code to weigh the decision so that their language is chosen.

Who is right?

It seems to me that everyone is right … and no one is right.

The claims on each side are usually correct, which means you can’t settle the argument based purely on technology attributes. Comparing the pros of one language to those of the other is really an argument about the reasons for both languages to exist. Clearly each language has some aspects where it is better than the other. When you compare the positive and negative aspects of any language, it is hard to judge which parameter is more important. With no single strong argument that tilts the scales in favor of one or the other, the discussions drag on an on.

In these situations, I think it’s important to realize that the main consideration should be the number of languages, not which language.

Picking a Language Is a Managerial Decision

There is an “activation energy” for each new language used by a team. When context-switching from one language to the other, the engineers lose some time recalling the technical details of the language. You may need to duplicate some of your development tool (e.g., IDE, code control) and support systems, such as your nightly build and continuous integration. The engineers will need to learn how to work and be proficient with more tools. All this takes time, which costs money.

User Comments

Matt Griscom's picture

This phenomenon might be outside of your professional experience at Intel (at least, I hope so!) but scripting languages such as Ruby, Perl or Python are commonly popular.

I notice you explicitly sidestep this question in your article, but generally, the compiled languages have much more powerful capabilities that the scripted languages do not (OO exception handling, multithreading, synchronization, namespaces to manage complexity, XML support...)

I struggled with Ruby on a legacy project for months. No, Ruby doesn't even come close to doing what I needed it to do, but more on that below.

From extensive technical discussions on what is possible with Python, I think it's close, but not quite there, and it's definitely a poor choice for a large, complex project.

Perl? Doubt it.

Generally, scripting languages are widely used for "test automation" but their limited capabilities restrict them to an outmoded understanding of what that automation is all about and where it brings business value - really, the answer to the question "Why are we doing this?"

C or C++ can probably do it, but it's been a while since I've used those languages. Java can do it, too, if you don't need realtime, but my preference is C# which is more modern, better-designed, much better libraries, and more OO than Java.

Hope this helps!

Check out my work at MetaAutomation.net (and two code samples, to be updated later today). The 2nd edition of the book on MetaAutomation should be out in February.

December 29, 2015 - 5:42pm

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.