Column info : It’s Different in Test

Comment:
by vinayak shinde 12/20/2005

Hi Harry,i am not in accord with your statement.Tester must act some times like customer,some times like developer and some times like person with intention of braking of an application.When he will test application for positive test cases he must act like customer.when he will test application for negative testcases he must act like Person with braking intention.Most important is developers knows where will applcation will brake,thats why tester also needs to think like developer.I did all these things in my carrer.May be its conflict between your & my thinking.I will try to think like you.I know it will help me to imporove my work.

Comment:
by Matthew Heusser 9/7/2004

Interesting article, Harry, and you're right - the programming skills required for a tester writing perl scripts is going to be different than that for a coder writing a search algorithm. :-) In fact, I would submit that the tester should be using a higher level language with automated garbage collection or reference counting, so the linked list example is even more irrelevant.

Author's Response: 9/29/2004
Thanks, Matt. (And I'd like to point out that Matt caught me in an ambiguity in the problem description. The linked list has to be reversed in-place. See http://www.fidonet.m.nu/echomail/area/C_ECHO/48 for some thoughts on a solution.)

Comment:
by Tayssir John Gabbour 8/27/2004

The reason I agree is that I have a different "hat" on when I test my own software. It's a big difference mentally, very much like having played offense (US football) and now I'm on defense, looking for holes in the offense. I don't understand when I hear about these developers who are somehow shocked when testers do their job well. Then again, many don't understand the hacker spirit... hackers being something of inverted developers. http://c2.com/cgi/wiki?BlackTeam

Author's Response: 8/30/2004
I agree - the mind shift is important, and its hard to do it effectively on one's own software: you know it too well and you have a lot invested in believing that it works.

The offense/defense analogy to development/testing holds up well in several ways. For instance, it is important that the offense/development executes its pre-arranged plan, but defense/testing always has to adapt its strategy to whatever the offense/dev is doing. thanks!

Comment:
by Gene Fellner 8/26/2004

The correct response to Neha's quote, "The user would never do that," is, "Oh yes they will. Users make errors like everyone else." In my decades of software experience I have learned that some of the most catastrophic failures were caused by users doing something wrong that the developers didn't expect. Hitting "F4" instead of "5", searching on a null argument, clicking a radio button that wasn't supposed to be highlighted before realizing it was the wrong one to click. Testers are supposed to foresee ALL the things that end users will do, not just the things they are SUPPOSED to do.

Author's Response: 8/30/2004
True. And it gets worse since hacker/cracker types will be intentionally trying to do weird things. If it's so hard to predict what a well-intentioned user might do inadvertently, how can anyone guess what a malicious user would do? thanks

Comment:
by Neha Rastogi 8/26/2004

I completely agree with you Harry and this article of your's relates to so many of my experiences as a Software Tester. Often while discussing a bug with a developer I run in to a situation where the developer tells me "What you are saying is right but the user would never do it this way" and then what I say to this is "Yes that's right but I am not the user, I am the Tester".

Author's Response: 8/30/2004
Hi Neha - good point! And in the computer security world of today, your dev better not rely on the good will and predictable behavior of customers. thanks!

Comment:
by Sean Jorgensen-Day 8/26/2004

I also think a major difference between a tester and a devloper is that when accepting the product from the developer the tester is focused on finding as many bugs as they can. When selling the product off to the user the tester is showing that the product does what it says it will. Developers have a hard time understanding why we are always "breaking" their product. As a tester I have to put myself in the users shoes and distance myself from "owning" the product so that at the end of the day I don't take it personally, unfortunatly some developers do take it a personal attack when you point out problems with the product.

Author's Response: 8/30/2004
You can always try the old line "I didn't break it; it was already broken - I just pointed it out." It's unfortunate that some developers take bugs personally; more seasoned devs know that your job is actually to make them look good, so they welcome the help. Perhaps we need a line of T-shirts that say "I'm from Test... and I'm here to help!" :-) thanks

Comment:
by Stu Deevers 8/25/2004

Harry, I "somewhat agree" with the article, and it points out differences of opinions between a developer and a customer. I do agree with a couple of comments made, and that is "testers" bridge the gap between development and customer. A good customer ensures the product does what they want it to (hopefully based on the specification), and the developer programs to that specification. The tester then verifies the program written functions as specified, and then validates it is the correct program (notice the difference between verify and validate?). It is one thing to verify a program functions, but another to validate it does what the specification detailed. For example, is the "Save" dialog on the left, or on the right as specified? It doesn't matter if the "Save" dialog saves information if it is not in the specified location. A tester will verify the "Save" dialog functions as designed, and validate the "Save" dialog is as specified. As for testers being able to develop software, that is a whole other can of worms. I agree the test staff must be diverse, but to have every tester be able to write software is not always a good thing. A couple programmers sprinkled in, a couple tech writers and a couple people who "like to break things" that all have an analytical mind of what to verify and validate would make a good test team. This will return a great product, that hopefully, meets or exceeds all of the customer needs, and benefit the development staff with fewer defects because they each approach from a different angle, yet have a common goal. We all want to be recognized for our efforts and chosen profession, but until Managers, Customers and Development understand the benefit of a "real tester", we will be forever thought of as "those people who break things". Stu ...Collapse Comments

Author's Response: 8/30/2004
Hi Stu, thanks for your comments. I agree with you about tester diversity being a big plus to an organization.

However, regarding management's being able to "understand the benefit of a 'real tester'", I think getting people to appreciate testers will always be tough. When done well, the tester's contribution is almost invisible: everything just works. For instance, what would management think of a test team that was able to prevent the vast majority of defects so that they never showed up in the bug database? It takes pretty enlightened management to see what great testers are doing. It's like appreciating how well a platform diver can knife into the water with barely a ripple on the surface - a bellyflop is much easier to notice, but it isn't what you really want. regards,

Comment:
by Ashish . 8/25/2004

Partially agreed. A tester should know the development basics. This will help the tester to identify the root cause of the defect & present it to the development team correctly, instead of juggling around. However, when it comes to automation, a tester must be a good developer so that the script is developed quickly and is also reusable/maintainable.

Author's Response: 8/30/2004
I agree that it's good for a tester to know the basics of development; I don't agree that the tester needs to find the root cause of a defect - that's debugging and is best left to the developer (though the tester should provide a short, reproducible bug sequence).

As for test automation, I value design ability more than programming ability because the bigger problem is that most test automation is ill-conceived in the first place. (If you're interested, check out www.model-based-testing.org/intelligent.pdf for my thoughts on a good way to go about automating tests). thanks!

Comment:
by neill mccarthy 8/25/2004

Harry another good article. Once testers can define themselves clearly it will help other people that we need to influence to understand what we achieve. The 'schools of testing' much spoken about kind of help but for me miss the meat of what we do. Not sure I agree with the whole article, theres many grey areas that have been avoided, probably for the need of brevity. For example for Performance testing I actually do need the testers to be like good developers, bad test code in this on safety critical systems would be an issue... I do want them like a customer where they are involved in the acceptance testing of a system or assisting with the moderation of this testing. I do want them sort of like another tester, may be not in my team at the same time but certain key skills good testers have - thinking, acting and communicating I want to stand for all. I even like them to have the same idea as me occasionally, it helps me believe I am not totally off track in my thinking. Mind you I do like a good debate....Collapse Comments

Author's Response: 8/30/2004
Hi Neill, thanks for disagreeing! True, there are times that we want testers to be able to take on the guise of other specialties. But perhaps it's an issue of motivations: does your performance tester write slick code because she likes a good algorithm, or because she needs to do it to find a deep bug? I would think the latter...

As for testers defining themselves clearly - I wonder if this can ever happen since test is so closely tied to whatever development is doing. After all, in the tango of software development, test is the one dancing backwards most of the time... thanks!

Comment:
by David Coutts 8/25/2004

Harry, A big difference between testers and developers / analysts is that we don't produce anything that the customer can use. The customer knows that analysts (or whatever you prefer to call them) document requirements (hopefully). The customer knows that the developer produces code. If you're an Agile fan then working code is THE THING. If, like me, you work in a more traditional software project, then we all hope that the developer read the specification before he/she wrote the code. As testers, we then test the "theory" that the code matches the specification. It's a bit like those "Spot The Difference" pictures many of us did as kids. Each difference is a "defect". Hence, we are seen as non-productive (even destructive). Yet, and here is the key thing about testing, by detecting and eliminating error we help improve the software. So, software advances to its production status through the apparently non-productive and negative actions of testers. I've written a whole article on this which I've submitted to StickyMinds. In the meantime, read "The Unnatural Nature Of Science" by Lewis Wolpert. That should provide a clue as to the unnatural nature of testing... David Coutts dacoutts@optusnet.com.au...Collapse Comments

Author's Response: 8/30/2004
Hi David, If the developer provides the "code" of "working code", then it seems reasonable that the test team is responsible for the "working" part! :-) Truthfully, how would the customer feel if test was never involved? It would be like finding a warning label on the package: "This code compiled, but does not actually work". I think we produce the thing the customer can use the most: a decent night's sleep. thanks!

Comment:
by Danny Faught 8/24/2004

Hi, Harry, nice article. Funny, during the interview for my first testing job out of college, I was asked to write code to sort a linked list. A friend of mine who was a highly qualified tester but a bit rusty at coding flunked the interview. He went on to work for a competitor instead. By the way, note that a test toolsmith will indeed need to be a good programmer. I assume that when you say "tester" you mean someone who designs and/or executes test cases.

Author's Response: 8/30/2004
Thanks, Danny! You're right - by "tester" I mean someone who designs/executes tests. Someone who designs/implements a test tool is, in my mind, a developer who just happens to have testers as the customers. regards

Comment:
by David Schmelzer 8/24/2004

Kudos on your perspective here. It has been noted many times in our group that, as there are for other technical discplines, there is little formal (i.e. degree related) training on software testing. Testers evolve out of other areas. This makes the testing profession unique in ways, in that it blends many elements of other displines (developer, business analyst, product) together with certain personality traits (curiosity, attention to detail) in concert to face the challenges of our practice. While we as professionals are NOT exactly developers or customers, we do share some of these skills and attributes. It is a broadening understanding what these elements are, along with continued quality delivery of applications that will spotlight the unique contribution and value our profession puts forth. Good perspective and good read....Collapse Comments

Author's Response: 8/30/2004
Thanks for your comments, David. It's true that historically testers have wandered in from other disciplines. Yet, as test techniques get more advanced, we are starting to see courses tailored to testing (see www.testingeducation.org/general/othertestingcourses.html). It will be interesting to see where it all ends up. regards

Comment:
by Dave Lutzker 8/24/2004

Thanks for an enjoyable article. We certainly have a challenge in being recognized and appreciated. The first comment (by Bernard Homes) hit an excellent point - let's take our message to the developers and managers, and not just to ourselves! Distribute this link to your developer and manager friends and colleagues! What I find as an additional challenge is when working on a project team, I am always reminding the client that testing does not GIVE you quality, and that you can't "test quality into a product". Quality comes from the creators, and all we as testers can do is help you sleep at night - give you the appropriate level of confidence that the product will behave as expected. An "intangible benefit" when compared to developers. That's one reason why I scour the StickyMinds news bites that point out the fiascoes that occur due to poor software quality. If engineers and architects could use quality standards like software companies, we would truly be in trouble! OUR TIME WILL COME! ; ) ...Collapse Comments

Author's Response: 8/30/2004
Thanks for your thoughts, Dave. I agree with your assessment that our work is intangible: I once proposed a metric for my team as we headed into a release - "percentage of the team (system engineers, developers, testers, managers) sleeping well at night". It was admittedly a fuzzy metric, but I think it caught an interesting aspect of the test team's contribution.

As the industry continues to evolve, I think that test's contribution will play a more and more central role. Our time will indeed come! :-) Thanks -

Comment:
by Kamran Mirza 8/24/2004

Harry I am writing from Islamabad Pakistan, I am in SQA since 1998. Regarding your perspective I think what you suggest is quite right and agree with you that a good testing team should be diversified in “all aspects”. You are very right that testing team should openly discuss and at time should debate their perspectives. A tester can’t be like Customer or developer as customer (user) and developer will be having very limited knowledge (depending on job responsibilities in both cases) whereas tester should have the domain of the whole application and must try to think about the unthinkable combinations. Tester should always be doing something more that Business Process Assurancen (BPA) and basic path testing. Regards Kim...Collapse Comments

Author's Response: 8/30/2004
Thanks, Kim! You bring up the good point that the best testers make it a point to understand the whole application. Developers focus on getting their module working; customers focus on getting their work done; testers are often the only ones who have the opportunity and the charter to see the bigger picture. regards,

Comment:
by Darryl Hurmi 8/24/2004

I agree that testers should know how to program but not neccessarily to write test programs. If you know how to program you know were the common mistakes are and if you have a logical mind you can determine were the most likely places are for errors. My experience with customers and programmers doing testing is that customers do not try to break the code even when they are doing an acceptance test, Programmers can do good testing but don't ecpect them to get very far because they get caught up looking for the bug in the code instead of going on to find new bugs. The best skill that I believe that a tester should have is the ability to listen. Programmers wat to talk about there code and what it can do, if the tester has a GOOD understanding of what the code should do, the programmer will tell him were the program won't work as expected....Collapse Comments

Author's Response: 8/26/2004
Thanks for your comments. You bring up the good point that testers who can program usually have a good sense of where bugs are likely to reside. I often look at an application and think "well, I know where I would have put the bugs..."

Regarding listening to developers: I once heard of a tester who eavesdropped on developer conversations at lunch to discover which parts of the code the developers felt most anxious about. I prefer your idea of talking directly to the devs. :-)

Comment:
by Richard Whitehead 8/24/2004

I liked the article. I think several of the comments so far misunderstand what you said. I agree that customers may or may not be good testers, but the role of customer is very different from the role of tester. I also agree that testers use data as a means to manipulate the system, whereas a customer uses the system as a means to manipulate the data. I don't think the comments are quite giving you credit for saying that. Customers should be involved in all aspects of the development process, but I have never met a customer who could adequately perform any real testing process. The difference in viewpoint is too radical to be adopted overnight. Yes, customers define the results. But testing is more than the results....Collapse Comments

Author's Response: 8/26/2004
The fact that quality is something best defined by customers but best verified by testers makes me wish we had even better lines of communication between these two groups. thanks for your comment!

Comment:
by Gilles Girard 8/24/2004

Although I was sceptical after reading the title...you have now a new recruit ! Testing and Testers ARE a different beast nested between the developpers and the customers/business analysts. The best testing that I have done was when I had access to project allocated developpers and business analysts NOT when I was playing the developper or the business analyst while testing.

Author's Response: 8/26/2004
Hi Gilles, thanks for your comments! But why did you have mixed feelings about the title? regards,

Comment:
by Banurekha Narayanaswamy 8/24/2004

I agree with 'Testers are not like Customers'. Testers do more than just 'User like testing'. Customers may not perform negative testing, regression testing, boundary value testing. Also, their testing motive is to ensure that the application meet their requirements. However, an actual tester would go one step beyond to see if the application does anything unexpected apart from meeting the requirements.

Author's Response: 8/26/2004
Hi Banurekha, I agree absolutely - thanks!

Comment:
by Rodger Drabick 8/24/2004

Interesting article, Harry, and well-written, but I can't say I fully agree. Maybe I'm "picking nits", but I have a real problem with your statement "Jill Tester, on the other hand, has only a marginal interest in what happens to the data. In fact, she usually already has the answers in hand before she ever touches the keyboard. She is far more interested in what her data can do to the application." If Jill Tester doesn't know what the data needs to look like, she's missing the necessary set of "expected results" (unless she's doing pure exploratory testing, and doesn't have any idea what the data will look like...). Without expected results, she won't know if the test passed or failed. Also, on programs that deal with large databases, quite often data corruption becomes a problem, and if you "have only a marginal interest in what happens to the data", you as a test engineer could miss some significant, and hard to find, problems....Collapse Comments

Author's Response: 8/26/2004
Hi Rodger, thanks for commenting!

I don’t think I’m saying Jill doesn’t care at all about expected results. If Jill were testing a calculator program and entered “6 * 7 = “, she should certainly check that the answer comes back “42”. But she’s not really interested in the answer; she already knew the answer and this is mainstream functionality that the developer should have already been through. Jill wants to see whether those inputs cause something unusual to happen with the program. This is why “6 * 7 =” might not be a very interesting test case, but “6 / 0 = “ might be.

Your point about side effects like database corruption is a realy good one and underscores what Jill should be trying to do. If she inserts a record into the database, it is interesting to her (though not exciting) that the record successfully ends up in the database. However, if her action causes data corruption, that is both interesting and exciting! Regards,

Comment:
by Marc Duerr 8/24/2004

I'm sorry, but I cannot fully agree to what you say about the abillities of customers to test. We do here something called joint system tests, which involves the customer in a very early phase of the development. This includes test description writing, testing and problem reviews. From my point of view, the customers are the only ones, that can do a complete correct black box test, as they are the only ones, that define in the end the quality of a product (Quality is defined by the customers needs!). Of course you are right, that the normal end user of a browser, or other normal day PC software/hardware is not a good tester, but there are billions of dollars revenue made by software tailored to the customers needs in the industry. And they will do a ATP anyway before accepting a new software (and paying for it). The biggest advantage was, that we were able to deliver the new software with less overall bugs 1 month ahead of the schedule. You are a good tester, if you are able to read (at least partially) the code of a developer to understand the behavior in case of bug and you need to be able to write test descriptions in a way, that it is possible to get someone from the street and let him do the tests just using the test description!...Collapse Comments

Author's Response: 8/26/2004
Hi Marc – thanks for disagreeing - a different viewpoint is a good thing!

I agree that customers are the ones who should define what good quality is, but I will hold to my view that they are not the best people to determine whether that quality is actually being delivered. For instance, as a car buyer I might know what I think are the attributes of a good car, and I would probably insist on test driving the car, but I would still want real automotive professionals performing in-depth tests to ensure that the engine will not fall out of the car at 10,000 miles. Similarly, in the example you give where a tester writes the tests and hands them to a customer to run, would the customer be able to notice issues such as subtle performance degradation while running the test? I don’t think so, but that subtle issue might emerge as a big problem in the longer term.

Thanks again for your comments! regards,

Comment:
by J ’Brody’ Brodock 8/23/2004

I whole heartedly agree, testers don't make or fix things and a customer never want those things broken. As testers we have to be the bridge between. When people ask me what I do for a living, my canned response is, 'I break things'. The looks I get tell me much about the audience; owners are concerned, developers cringe or look relieved, and the non-technical get a puzzled expression on their faces and ask if I actually get paid for performing such destruction? Perhaps our task is to educate the masses on the value of a good bashing.

Author's Response: 8/26/2004
The only tweak I would add is that we don't actually "break" anything - the software was already broken. We just happened to be the ones who drew attention to that breakage. Thanks for your comments!

Comment:
by Andrew Raybould 8/23/2004

Harry, there is one trait that is common to both effective developers and effective testers, and that is being able to think analytically about systems. Without this skill, you are working by blind trial-and-error, which doesn’t work: for developers and testers alike, the problem space is almost entirely occupied by false trails.

For this reason, effective developers do spend a considerable amount of time thinking about how things could go wrong: this is inseparable from thinking about how to make them go right (a possible exception: management has created an environment that punishes efforts to get it right first time – but no developer can be effective in such a situation, and neither can testers, who will be swamped.)

Nevertheless, as there is no way to guarantee correctness (even formal methods depend on the correctness of specifications), independent testers are an essential part of high-quality development, and are appreciated by effective developers (it’s the ineffective ones who don’t like their bugs being found!)

One final word on skills: where concurrency is an issue, testers and developers alike really need an in-depth understanding of its dangers....Collapse Comments

Author's Response: 8/26/2004
Hi Andrew – Thanks for your comments. Yes, good developers and testers certainly will have a lot of traits in common. In fact, I think that the next few years will see a blurring of the distinction between the skills of these two groups. We are already seeing something like this in the model-based testing world where testers are designing and creating model programs that execute tests against the production program. [Maybe I should do a follow-up column called “It’s Not Really Different in Test” :-) ]
The distinction for me between developers and testers is where each group focuses its attention. I agree with James Bach’s assessment: “I see testers as the bodyguards of the project. We defend our developer's flank from failure, while they focus on creating success.” (www.stickyminds.com/se/S1934.asp).
Concurrency is a messy problem whose solution will underscore the need for cooperation across the whole organization, from writing better specs to making software more testable to running brutal test suites looking for the subtle bugs. Thanks!

Comment:
by Bernard Homes 8/23/2004

Hello Harry, I always say that developers and testers are not alike as one finds ONE solution to a problem, while the other aims to find ALL the way to break ... the poor unsuspecting program (is there a sadistic trait in all good testers ? ). If we wish to have testing recognized as a worthy profession by the development community, we need to have the profession recognized by managers (and developers), ... and that is difficult as you pointed out. I wonder if subcontracting testing abroad will not lead to the same hiccups as your example of left- vs right-hand users. This column is somewhat tester-centric (ie from testers to testers); it would be nice to see articles like this one in developers magazines (and have their feedback). A nice article, thank you....Collapse Comments

Author's Response: 8/26/2004
Hi Bernard, You are absolutely right that we have to educate managers and developers about what testers really do. We will need to think hard about how to reach them since there are few venues (conferences, magazines, websites) common to all the groups that need to be talking to each other. I think STQE's morphing to "Better Software" can be a good step in that direction since the focus is to improve software quality "regardless of your role in the software development lifecycle."

Regarding the issue of subcontracting software testing to a separate company: anytime the testers are separated either by space (as in outsourcing) or time (as in waterfall models) from the rest of the development team, there will be problems just because for the tester’s work to make a real difference it has to be delivered in a timely fashion, i.e., while the product is still forming. Yet more educational work for us to do. Thanks!