Category Archives: Testing

Testing the UI and API together with my new practice app!

It’s been a bit quiet lately, and that’s because I’ve been very busy with creating a workshop and new talk for TestBash Manchester! Now that it’s done, I can finally blog about some of the stuff I did to create the workshop, and release an app for folks to practice with!

Understanding The Problem

I have previously done a workshop on testing REST services, where I wrote an ASP.NET Core 2.1 API with a simple web front-end (using the Todo API example as a base to start from). For that workshop, participants would load up the code in Visual Studio and run the solution locally. This was a no-brainer, as part of the workshop was to write some automated tests in Visual Studio.

However, for the workshop at TestBash, I needed to account for an audience with a wider range of skills – I had done the above workshop at developer conferences and at work. Discussions with the folks that run the conference suggested a workshop without needing to code, to accommodate the most attendees.

So I realized I could use what I had written, and develop a workshop testing both the API and the UI that went along with it. We could test using Postman, Swagger, and dev tools in the browser. That could easily go 4 hours! Perfect!

But then I had a problem – how was I going to provide this application to the participants? I could host it somewhere – I tried putting it up on Azure, and it worked! But what about when one of the participants did something catastrophic to the data in the application? That could cause a whole host of problems, having shared data when asking participants to really try to break an application individually…

So what I really needed was to again have each participant running the application on their own machines – their own instances of the application.

Docker To The Rescue!

The obvious choice from there, for me, was Docker. I hadn’t worked with it too much on the “creating images” side, but had worked with it as a consumer of images.

I found some documentation on how to Dockerize my application but at one point, when right-clicking on the project in Visual Studio and selecting Add, I saw “Add Docker Support” so I clicked it! What could go wrong?


It asks you to target an OS – Windows or Linux. I chose Windows, because I’m on a Windows machine. And once you click OK it creates all of the Docker files for you! It’s pretty awesome! I compared the documentation to what Visual Studio automagically created for me, but it looked all good.

I followed the steps to build and run the image, and it was up and running! Awesome!

Then I had to get it to the participants somehow. I found this documentation on sharing your images, followed the steps and I was able to pull down my new image on another machine!


While I was prepping the prerequisite documentation for my workshop attendees, where they set up Docker and Postman and Chrome, I realized I needed to have people set their Docker configuration locally to run Windows images. After looking through the documentation, I found that folks on a Mac would not be able to run my image! Ack! Now what?!

Well thankfully, ASP.NET Core 2.1 applications can run on Linux equally as well as they can run on Windows! Yes! So I just had to set my local Docker config to run Linux machines, make a few modifications to the Dockerfile, build and publish it again. And now my application was accessible on Windows, Linux, and Mac machines without issue!

Now For The New App!

Ok, so now that we’ve got it all figured out, I can share the details for how you can use this Docker image to practice testing!

You can run the image locally, and have your own instance of a REST API and web UI to mess around with. This application has a Swagger for the API, and it can be reached via Postman as well as other tools.

It has a few known bugs (I’m not documenting them because that’s part of the fun of the workshop!) and quirks. If it ever gets in a bad state for you, simply stop the image, and start it up again! Remember to stop the image when you’re all done with it for the day – it does take up some resources on your computer.

Source code for the application can be found at

The information to get it running is in the Readme of the repo, here:


Once Docker is running, the application can be accessed at http://localhost:8080.

Swagger specification can be found at http://localhost:8080/swagger.

I hope you enjoy! Watch my website or twitter for where I’ll be giving this workshop next, as well!



A Lesson From My Testing Journey

On February 20th, I shared my career journey for the first time to a large audience. I gave a Ministry of Testing Masterclass called “Not All Who Wander Are Lost: A Career Experience Report” (a recording is available for Pro Dojo members here:

I’ve had kind of a wonky career path, and have at times felt some shame for “going backward” from a test lead to “just” an engineer. Now, though, I feel pretty good about my journey – I wouldn’t be who I am today, or where I am, if not for those experiences.

One lesson that I touched on, but I’d like to reiterate here, is to be honest with yourself about what you want to do.

I’ve had testers come to me and ask what their career path should look like, or what they should be aiming for. “What should my end goal be?”

Your career goal is for YOU to decide! Don’t let others make your life plans for you!

For sure, ask for advice about others experience. You don’t have to be a manager. You don’t have to be an architect. You DO have to be happy and fulfilled and grow and learn and get value and give value in your role. I can’t tell you what that looks like for you. Hell, I don’t know what it looks like for me! I’m taking my career a day at a time!

It’s great to have goals, but please, keep your goals flexible! If you realize on the way to becoming a manager that you really want to stay hands on, do that! It’s not the time to stick to your guns like “well, I wrote this down, I have to do it”. Your career should make you happy!

Every day, your experiences change you. You are a different person today than you were when you started your career.

Be honest with yourself about what you want from your career. You might wander a bit, but you won’t be lost as long as your true goal is happiness.


C# Advent Calendar – Combining Integration and UI Automation in C#

I signed up for the C# Advent Calendar with this post – be sure to check out the rest of the blog posts in the calendar! This post assumes you’ve got some experience with C# and Visual Studio, so I won’t be over-explaining some things. Comment if you get stuck or have questions!

If you’re here for the first time, welcome! I’m a “career QA”, been in the QA/testing field for just over 10 years now. I’ve been doing test automation with C# for several years, including CodedUI, Selenium, and integration tests (using NUnit or MSTest) for REST services, as well as unit testing. So I’m kicking off the C# Advent Calendar with a testing topic. Hope you enjoy!

My talk on Testing RESTful Web Services is all about evangelizing integration testing. Integration tests are sometimes missed between unit tests and GUI tests – developers may think it’s the QA’s job, and QA may think it’s the developer’s job, so they can fall through the cracks.


So why combine integration and UI tests? These are useful to see if the API and the UI are speaking the same language. They are often developed at the same time by different people, so it’s nice to have a safety net of tests that can be run whenever to verify things are still working properly.

I wrote a very simple web API for funsies (using this tutorial) and because it has an API and a web UI, it seemed like a great subject for this mixed tests treatment! My code can be found at my github, here:

Our test will:

  • do a GET via the API for an item
  • launch the UI and search for the same item
  • compare the responses

Creating The Class Library For The Tests

Because this is pretty small, I’m just going to use a single project for my tests. In a larger application I might split them out. I’ll create a Class Library for all of my testing stuff. The first thing I do in general is add all of the NuGet Packages I need.


I then set up my structure.

For the UI tests, I know I’ll be using the Page Object Model so I create a folder for those, and add my HomePage class. I also know I’ll need some utility methods, so I add a Utilities class. I’ll wait to add the test classes until I’m ready for them.


Setting Up The Page Objects

In essence the Page Object Model is decoupling the definitions of the elements on a page from the tests that use/manipulate them. This keeps the tests maintainable and not as brittle.

Our page is pretty simple. I’ve pointed out the elements we’ll want to put into the class. We’ll just be typing a number into the text field, clicking Search, and reading the response in the div below that.


Each of my elements has an ID – easily identifiable elements helps us QA folks use the elements in automation. If I have to use Xpath, for example, the tests can be more brittle than if I tell the automation “find this thing with this unique identifier”.

This is what my class looks like with the text field identified:

using OpenQA.Selenium;
using OpenQA.Selenium.Support.PageObjects;
namespace MontyPythonApi.Tests.PageObjects
	public class HomePage
		[FindsBy(How = How.Id, Using = "prodId")]
		public IWebElement ProductIdInput { get; private set; }

The FindsBy is how Selenium will be finding the element, and then the property is of type IWebElement (so Selenium can find it) and I can call it whatever I want. I like to use the type of element it is in the name to make it more clear, especially when there’s tons of elements in there.

I’ll add the rest of the page elements, and then initializing the elements via the PageFactory (part of Selenium.Support.PageObjects):

using OpenQA.Selenium;
using OpenQA.Selenium.Support.PageObjects;
namespace MontyPythonApi.Tests.PageObjects
	public class HomePage
		[FindsBy(How = How.Id, Using = "prodId")]
		public IWebElement ProductIdInput { get; private set; }
		[FindsBy(How = How.Id, Using = "searchButton")]
		public IWebElement SearchButton { get; private set; }
		[FindsBy(How = How.Id, Using = "product")]
		public IWebElement ProductDisplayOutput { get; private set; }
		public HomePage(IWebDriver browser)
			PageFactory.InitElements(browser, this);

Any methods specific to this page will also go here. We can add them as we need them.

Setting Up The Utility Methods

For our integration tests, we’ll need to make HTTP Web Requests, and then get the response back and deserialize it. I prefer to deserialize it in the test itself, but if you want to do it in the utilities be my guest 😀

Our utility class looks like this (the formatting sucks – check out the code instead here):

using System;
using System.Net.Http;
using System.Text;
namespace MontyPythonApi.Tests
	public class Utilities
		public static HttpResponseMessage SendHttpWebRequest(string url, string method, string content = null)
			using (var httpClient = new HttpClient())
				var httpMethod = new HttpMethod(method);
				using (var httpRequestMessage = new HttpRequestMessage { RequestUri = new Uri(url), Method = httpMethod })
					if (httpMethod != HttpMethod.Get && content != null)
						httpRequestMessage.Content = new StringContent(content, Encoding.UTF8, "application/json");
					return httpClient.SendAsync(httpRequestMessage).Result;
		public static string ReadWebResponse(HttpResponseMessage httpResponseMessage)
			using (httpResponseMessage)
				return httpResponseMessage.Content.ReadAsStringAsync().Result;

Now We Can Write Our Test!

Now to the good stuff.

I start with the SetUp method, which runs prior to every test. This is where I set the browser driver up – what Selenium uses to make the browser bend to its will.

using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
namespace MontyPythonApi.Tests
	public class IntegrationAndUiTests
                private IWebDriver browser;
		private static string webUrl;
		private static string apiUrl;
		private static string baseUrl;
		public void Setup()
			browser = new ChromeDriver();
			baseUrl = "http://localhost:20461";
			webUrl = baseUrl + "/index.html";
			apiUrl = baseUrl + "/api/products";

I instantiate the browser driver (using IWebDriver from Selenium) in the SetUp method because each test will need a new instance of it. Like all good tests, these need to be atomic.

We need a TearDown method as well, to clean up after ourselves.

public void TearDown()

I’m going to set up my test to first make the call to the API, then use an Assert to verify that I got a 200 OK response back. This isn’t necessary but I find it useful – my tests fail here rather then trying to deserialize later on, so I know where things went wrong more quickly.

So we know our test is going to make a call to the API about a product, then verify the data in the web UI. We have a requirement that if a product has a discount price, it’s the only price that will display in the web UI. We set up our test data so that the product with ID = 1 has a discount price, so that will be the subject of our test.

public void ProductsDiscountPriceDisplaysOnWebPage()
	//API call
	var uri = apiUrl + "/1"; //get the product with ID = 1
	var apiResponse = Utilities.SendHttpWebRequest(uri, "GET");
		"Did not get success status code; got " + 

Our test has started to take shape. We are making our call, and getting the response back. We’re verifying  it got a 200 OK back but nothing else with it yet.

Let’s deserialize the response, so we can get the discount price value. To do that, I need to add the reference to the API project first, so I can use the model.

Here’s my code now, after having deserialized the response, using the utility method to read the response content into a string:

public void ProductsDiscountPriceDisplaysOnWebPage()
        //API call
	var uri = apiUrl + "/1"; //get the product with ID = 1
	var apiResponse = Utilities.SendHttpWebRequest(uri, "GET");
		"Did not get success status code; got " + 
	Models.Product product = JsonConvert.DeserializeObject(Utilities.ReadWebResponse(apiResponse));

Now we have the data that is returned for the product. We can now launch the web browser with Selenium and get the data returned in the web UI.

I need to have the browser go to the url, and then instantiate my page object:

PageObjects.HomePage page = new PageObjects.HomePage(browser);

Now I can use the properties of page to interact with the elements on the page.

PageObjects.HomePage page = new PageObjects.HomePage(browser);

I’m essentially typing 1 into the search field, and clicking the search button.

Next I need to read the display of the data as it came back for that product. It comes back in a string with the format <Name> : $<Price> so I’ll need to parse it. Because this is something I’ll be doing on this page for more tests, I’ll add this utility method to the HomePage class.

public string GetPriceFromDisplayText(string displayText)
        decimal result;
	Regex r = new Regex("\\$(.*)");
	Match m = r.Match(displayText);
        decimal.TryParse(m.Groups[1].Value, out result);
	return result;

Definitely could hit some exceptions along the way, but for now we’ll leave it as-is.

Now our Web call looks like this:

PageObjects.HomePage page = new PageObjects.HomePage(browser);
var displayedPrice = page.GetPriceFromDisplayText(page.ProductDisplayOutput.Text);

And we can finally add that assert statement to check our values! Here’s our full test:

public void ProductsDiscountPriceDisplaysOnWebPage()
	//API call
	var uri = apiUrl + "/1"; //get the product with ID = 1
	var apiResponse = Utilities.SendHttpWebRequest(uri, "GET");
		"Did not get success status code; got " + 
	Models.Product product = JsonConvert.DeserializeObject(Utilities.ReadWebResponse(apiResponse));
	PageObjects.HomePage page = new PageObjects.HomePage(browser);
	var displayedPrice = page.GetPriceFromDisplayText(page.ProductDisplayOutput.Text);
	Assert.AreEqual(product.DiscountPrice, displayedPrice, "The prices don't match!");

Let’s run it and see what we get! Because the API and the tests live in the same solution, we’ll need to open a new instance of Visual Studio to run the tests locally.

Time For Some Results

And… our test fails!


There’s a bug in the UI code that should display the discount code. Would we have found this otherwise? Probably with a good unit test, but this is a nice way to combine our tests and see how the application really behaves.

I’m leaving the bug there in the repo, so you can follow along. All of the code demonstrated above is there.

Let me know what you think – is this something you could use? Is this too much overhead for your project? Is this useful?

Using Fiddler AutoResponder for quick service or response mocking

In my Testing  RESTful Web Services talk, I touch on service virtualization briefly, and I include a short demo of using Fiddler‘s AutoResponder. I’ve been meaning to look into other (free) tools but haven’t had the chance.

After a question on the Testers Slack, I realized I could do a short blog post on how I demo’d AutoResponder in my talk. It gives a quick idea of the tool and one way you could use it. I love Fiddler*, and there’s so much more you can do than what I’ve been able to use. Please don’t take this post as a “this is the only way you can do things” – I’ll be the first to admit that even though I’ve used Fiddler for years, I feel like there’s so much more to know about it!

*Note: I am not paid by Telerik/Progress for my endorsement; I just love the tool 😀

Read the rest of this entry

Using SQL Databases and REST together in Integration Automation with C# and NUnit

EDIT: One mindful reader noted that I should be using good practices to protect my code from SQL Injection. So there are some updates below, and the code is also updated! I used this as my guide:

One part of my talk and resources on Testing RESTful Web Services that I hinted at but didn’t quite get to was using SQL databases. A lot of us will be testing a REST service along with it’s backing SQL database, so it’s useful to know how to link them together in our C# automation!

I’ve added to my repository to include some checks against the AdventureWorks2012 SQL database, continuing to use C# and NUnit. Gitter’s database isn’t publicly available (nor should it be!) and I can’t find an AdventureWorks API that I could use for my checks – maybe that will be a later, more complete example. But for now, we’re doing some pretending!

If you’d like to follow along, grab the AdventureWorks2012 database backup from Microsoft by grabbing this zip AdventureWorks2012-Full Database Unzip and restore the database (google it if you need help :D).

Look through the tables, run some queries, and see what we’re working with. For the coding examples in the repo, I’m mainly looking at the Production.ProductInventory table, which depends on the Production.Product table for ProductIds. From the Product table, I’m going to pick a product to work with – I like #875: Racing Socks, L. Querying the ProductInventory table, I see there’s 288 (presumably pairs) of Large Racing Socks.

Let’s pretend that we have an inventory system that uses a REST service, which in turn grabs data from and updates our database. If we pull something off the shelf to send to a customer, we scan it into the system. The system identifies it, and sends a call like:

PUT http://ourwarehouse/api/products/875/inventory/-1

then the API goes into the database and essentially does:

UPDATE Production.ProductInventory 
SET Quantity = quantity - 1 where ProductID = 875

So with our checks, we would:

  1. Do a SELECT query on the database for the product and see what the quantity is
  2. Perform the PUT operation with the REST service
  3. Do the SELECT query again to see what the quantity is now
  4. Verify the new quantity is what we’re expecting

Utility Methods

First we can write some utility methods to execute our queries against the database. We start with a Utility class and add references to System.Data and System.Data.SqlClient.

Our first utility method is to execute a SQL query to get the quantity of a product. We’ll pass the product ID and connection string, and it will return a DataTable for us.

public static DataTable GetQuantityOfProduct(string productId, 
  string connectionString)
	string commandText = "SELECT Quantity FROM " +
                 "AdventureWorks2012.Production.ProductInventory WHERE " +
                 "ProductID = @ID;";
	using (SqlConnection connection = new SqlConnection(connectionString))
		SqlCommand command = new SqlCommand(commandText, connection);
		command.Parameters.Add("@ID", SqlDbType.NVarChar);
		command.Parameters["@ID"].Value = productId;
		using (var da = new SqlDataAdapter(command))
			var dt = new DataTable();
			return dt;

Another utility method we’ll want is to execute an Update SQL command, to update the quantity (since we don’t have an API).

Again this method takes in the product ID, as well as quantity and connection string, but this method returns a code, not a DataTable. If our command is successful, we’re expecting the return code to be the number of rows affected. If it’s -1, then something went wrong.

public static int UpdateQuantityOfProduct(string productId, int quantity,
 string connectionString)
	int code = 0;
	string commandText = 
            "UPDATE AdventureWorks2012.Production.ProductInventory " +
            "SET Quantity = @Quantity WHERE ProductID = @ID;";
	using (SqlConnection connection = new SqlConnection(connectionString))
	   using (SqlCommand command = new SqlCommand(commandText, connection))
		command.Parameters.Add("@Quantity", SqlDbType.Int);
		command.Parameters["@Quantity"].Value = quantity;
		command.Parameters.Add("@ID", SqlDbType.NVarChar);
		command.Parameters["@ID"].Value = productId;
		code = command.ExecuteNonQuery();
	return code;


Connection Strings

We’ll put our connection string for the database in our App.config file. We need to specify the server, which database, and the security we’re using. Mine is set up locally, with Integrated Security (Windows Authentication).

    <add key="dbConnectionString" value="Data Source=MSSQLSERVER12;
            Initial Catalog=AdventureWorks2012;Integrated Security=True"/> 

We’ll grab the connection string in the SetUp method in our test class:

private static string _connectionString;
public void SetUp()
     _connectionString = ConfigurationManager.AppSettings["dbConnectionString"];

The First Check

Let’s get started with our first check! Remember we’re going to:

  1. Do a SELECT query on the database for the product and see what the quantity is
  2. Perform the PUT operation with the REST service
  3. Do the SELECT query again to see what the quantity is now
  4. Verify the new quantity is what we’re expecting

0. Setup

First we need to specify a few things, including which product we’re going to use, how we’re going to modify the quantity, and create our query string.

int quantityModifier = -1;
string productId = "875";

So we’re going to take 1 away from our inventory, of product 875 or the amazing racing socks in large.


1. Do a SELECT query on the database to get initial quantity

First we execute the query, and get a DataTable in return. We could return just that individual field, but then our utility method wouldn’t be as useful or we’d have too many doing the same thing -sometimes we’ll want an entire row, sometimes we’ll want many rows. You can do it either way. Here, we’ll do the work in our test to get the field we want, which is the first row and first column.

DataTable initialInventoryDt = 
    Utilities.GetQuantityOfProduct(productId, _connectionString);
int initialQuantity = Int32.Parse(initialInventoryDt.Rows[0][0].ToString());

Now we can figure out what the expected quantity will be when we’re done with our check.

int expectedQuantity = initialQuantity + quantityModifier;

2. Peform PUT operation with the REST service

If we had this pretend REST service set up, our code now would look something like this:

string url = String.Format("http://ourwarehouse/api/products/{0}/inventory/{1}, 
    productId, quantityModifier");
HttpResponseMessage response = Utilities.SendHttpWebRequest(url, "PUT");
    "Response code to PUT was not successful");

However, we don’t. So we have to fake it by doing an UPDATE command to the database directly.


And we add this code:

int code = UpdateQuantityOfProduct(productId,expectedQuantity,_connectionString);
Assert.IsTrue(code == 1, "more than 1 row was affected, something went wrong");

REMEMBER this is ONLY because we don’t have that REST service and we’re faking what the service would do!

3. Do the SELECT query again to get the new quantity

Again we perform our select query, and grab the returned value

DataTable updatedInventoryDt = 
    Utilities.GetQuantityOfProduct(productId, _connectionString);
int updatedQuantity = Int32.Parse(updatedInventoryDt.Rows[0][0].ToString());

4. Verify the new quantity that we’re expecting

Now we just do our Assert!

Assert.AreEqual(expectedQuantity, updatedQuantity, 
    "Updated Quantity is not as expected; it is " + updatedQuantity + 
    " but should be " + expectedQuantity);

Other Checks

We could also do a check of the GET method, to make sure our service is pulling information from the right table. Our steps would be:

  1. Do a SELECT query on the database for the product to get the quantity
  2. Perform the GET operation with the REST service to get the quantity
  3. Verify the quantities returned from both match

This example is in the repo, so you can check it out there!

There are many other variations that we could perform, as well, but these are some basic building blocks.

Wrap Up

I hope that this walkthrough and the code help you to be able to automate checks against SQL databases as well as REST services! Let me know if you think something is missing, or if you need some clarity, or if you find a bug in my code!

Also apologies about the formatting here – I need to find a good code snippet plugin to use in WordPress! Let me know if you have any suggestions!

She used this one neat trick to model JSON as classes in C#, and her audience was stunned!

I recently presented my talk Testing RESTful Web Services for the first time at the QL Technology Conference in Detroit. In this talk, I cover why we should test web services, and how to test RESTful web services both with tools like Postman, and C# and NUnit.

I was not prepared for, mid-demo, resounding applause from the audience. I was demoing a cool trick I learned from a teammate, and apparently it was new magic to the audience. Awesome! So I decided to blog about it, to spread the word of this fancy magic and help everyone save time when testing RESTful web services within Visual Studio.

What was that magic? It’s the ability to take a JSON response and quickly turn it into a class – a super fast way to model JSON responses!



If you already know how to do it, awesome! Spread the word!

What you need:

  • a JSON response to copy/paste
  • Visual Studio, with a solution loaded (we’ll use C#, not sure how it works with other languages)

I’ll use the same examples from my talk – it uses the gitter API (

First, create a new class file in your project:

  • Right-click in Solution Explorer where you want the file to go
  • Select Add -> Class
  • Type a name and click Add

Then, paste the JSON as classes!

  • Copy the JSON you want to model as a class
  • Inside the class file in Visual Studio, put your cursor inside the namespace
  • Choose Edit -> Paste Special -> Paste JSON As Classes
  • Much rejoicing!

You’ll need to clean up the class a bit, but it will definitely save you time!ClassesAndJson.png

I hope you’re able to use this neat trick to help you in your testing of web services!

You will want to choose your JSON responses for modeling carefully – make sure it has all of the potential properties! Otherwise, down the road when you’re deserializing, you’ll hit a failure because the JSON will have an extra property that your class is missing.

Any other neat tricks you can share?

CodeMash 2016

CodeMash 2016 was awesome, as per usual. This year, I think we had a much larger turnout of testers than before. We even had some testing related open spaces!

I wasn’t able to go to all of the sessions I wanted to, but I live-tweeted the ones I did attend! I ended CodeMash with about 400 tweets, according to the rough count I got from Twitter (top is day 1 of CodeMash, before I started tweeting, bottom is after my final CodeMash tweet on Friday):



The big screens in the main hall where my face was far too often (tweet)

Here’s the Storify of the Precompilers I attended, which were

On Thursday, I attended the following sessions (which are in the Storify link below them):

  • Is your API leaking? Breaking APIs to increase security with Joe Kuemerle

  • A whole team approach to testing with Jon Kruger
  • Accessibility of the modern web with Brian Greene
  • Growing a Testing Whole Quality Team The Hard Way with Jim Holmes
  • Open Space – tester lean coffee
  • How do we solve for XX? with Kate Catlin

  • Jam Session!

On Friday, I attended the following sessions (which are in the Storify link below them):

  • Oppose Your Impostor! How to Be Freed From Feeling Like a Fraud with Marueen Zappala
  • How to Stop Hating your Test Suite with Justin Searls
  • Automation in Testing, Go Beyond Test Automation with Richard Bradshaw
  • My God, it’s Full of Tests with Kyle Lewis
  • Open Space – testing discussions
  • Secrets of Success in the Style of Glee with Jennifer Marsman

Coding the right thing: not just for developers

This tweet came across my feed recently and I emphatically agreed:


It occurs to me, however, that we as testers sometimes don’t see ourselves falling into the category of needing such advice. There’s the oft repeated “coding the right thing rather than coding the thing right”:


We throw together a script that walks through the application, that performs a particular use case, and we run it and we have this sense of security that if it passes, we are ensuring good code is going out the door.

But are we, as testers, doing due diligence with our code, the same way we expect the developers to? Are we doing design reviews to ensure we’re actually coding the right thing? Are we getting code reviews? Are we reviewing and refactoring our own code as the application code changes? Are we pair programming? From what I’ve seen, some of us are. Many are not.


We cannot advocate that “test code is as important as application code” if we don’t treat it as such. We can teach almost anyone how to write a script to automate use cases of an application. We should be teaching them how to think more – what will this code actually do? What does it accomplish? If it passes, what does it prove? Am I actually asserting anything, or am I just having it click buttons and fill in fields with no actual check in place? It is one thing to write a script to assist you in manual testing – perhaps to get you to a certain point so that you can get to the real testing quicker. It is another to write such a script and treat it as the test itself.

If you write test automation, I challenge you to reconsider what your code actually accomplishes, what a passing check means. Reconsider the design and architecture choices made. What did you mean to do? What did you miss?

I challenge you to treat your code as if it is application code. I challenge you to treat your code as you want the developers you work with to treat theirs. And remember that test automation is not testing (at least not until the machines take over) – you need a brain to test.

Should All Testers Be Automators? The Argument I’m Tired Of, But I Argue Anyway

It has come up recently, in several conversations – what is the ideal tester skillset on a team? Some folks are in the “make them all automators” camp, or “teach them all programming”, or even “get rid of manual testers”. I find that I have strong feelings in this argument, but perhaps my thoughts aren’t all together enough to make my case during these conversations. So here I’ll be spelling out my arguments, and the supposed counter-arguments, about the ideal tester skillset.

This is, of course, my opinion. You are certainly free to use the comments to pose yours 😀

What is the ideal project delivery team?

I feel like this is a good place to start, as it lays out my overall viewpoint on an entire team’s structure.

My ideal project delivery team (Agile of whatever flavor, because Waterfall is nowhere near ideal for me):

  • 3-5 developers
  • 1-2 business analysts
  • 1 tester focused on automation (test engineer, software developer in test, whatever you want to call it)
  • 1 tester focused on manual (“analyst”)
  • PM, PO, etc. etc.

Argument 1 – you don’t need two testers, the automator can do manual testing too

My problem with this argument is that when you have one tester on a team, there is hardly any time for automation. I, for instance, was the sole tester on a greenfield project. There was so much prototyping, trying things out, trying another newer technology out, etc. that there was no time to do automation. Nothing was automatable because it was going to have to change the next iteration anyway, so why even start?

If the project is in a state where the automator can actually write code, that’s where the majority of their time will be spent. Probably by choice. If I’m writing code, I feel like I’m getting more done – something much more tangible than a test case passed. And automation suites are in constant need of updating and maintenance, especially in a continuously changing codebase.

The fact is, if you want one tester to be on the team, they’re going to split their time between automation and manual testing. They’re going to do manual testing enough to get their automation test cases written. Which means that the amount of exploratory testing that should be done will not get done. Yes, automation helps to be able to do more exploratory testing. But if you’re working on that automation, you’re not doing as much exploratory testing either.

My Stand: two testers, each with their own focus, means you’re getting full attention on automation and on manual testing at the same time. One tester doesn’t have enough time to do both to a high enough degree of attention.

Argument 2 – well, can’t a BA pick up some of the manual testing?

Sure, a BA can do some testing. I hope they already are! However, a BA is someone that is closer to the business or to the client the software is for. They should be testing on a UAT kind of level. I wouldn’t expect a BA to know all of the various types of testing they can perform against a piece of software to the depth that a tester would.

Absolutely a BA’s experience is a benefit to the testing of the software being developed. But a BA also has other obligations – they have to research and write stories, collect acceptance criteria, and interface with the client/end-user. They also don’t have time to do two peoples jobs!

My Stand: a BA should do the job of a BA – yes they should do testing, on a UAT level. Testers are valued for their ability to think differently than other roles on project teams. Each role should be able to do its job.

Argument 3 – at least teach all of the testers how to code!

While I agree that anyone in a software testing role should understand how code is written, I feel that a “basics of programming” and “data structures and algorithms” classes would be enough. I don’t think that everyone has to know how to code. Not everyone wants to, and not everyone is adept at it and THAT IS OK! If a manual tester can’t code, that absolutely does NOT mean that they aren’t progressing in their career. If they are keeping up with technology, they understand the base architecture of the software project they’re working on, then they don’t need to know how to code to test!

What does any tester need to know, in my opinion?

  • basics of programming
  • data structures and algorithms
  • how web services/api’s work at a base level
  • how to use the browser debugger (if testing a web-based app)
  • how to troubleshoot various applications (debug logs, Fiddler, etc)
  • the technologies being used to create the application they’re testing
    • what they are at a basic level
    • how they’re being implemented
  • how to find out what code got built or pushed to their test environment – what changed, what to test
  • how to ask questions when they don’t know something
  • what questions to ask
  • how to just fucking google it sometimes
  • of course, standard testing stuff (exploratory, boundary, security, etc etc etc)

The important things for a tester to know are how to talk to the people around them, and various ways they should test the software they’re testing. Yes, understanding how a developer works, or what they need to do to go from requirement to finished software, is incredibly important for anyone working in software to understand. But they don’t need to know the difference between a List and an Array in C#, and when to use each.

Hey, understanding the limits of an int and how you can test a field that has an int datatype is great. Knowing how to see in the database that the field for a phone number has a float datatype and understanding that you can just put in a bug without testing the frontend because wtf were they thinking – that’s invaluable.

My Stand: So yeah, the more you know, the better. Knowledge is power, after all. But you don’t have to be able to write code to be a good tester. Period.

Argument 4 – we should just automate everything and get rid of manual testers

Lets go back to that “standard testing stuff” and “basics of programming”. Testers have a job because developers are human, and they make mistakes. Also, writing code is fucking complicated.

You can automate a lot of things. But you can’t automate a brain, curiosity, a set of eyes that see that weird thing happening on the taskbar that your automation didn’t catch because it’s not a HUMAN and you didn’t tell it to look there so IT DIDN’T!

A computer will do exactly what you tell it to. It can’t explore, it can’t see those subtle differences, or even the big honkin’ bugs unless you code it to do that. But then there’s always something else that your code is not specifically asked to do so it won’t do it.

I personally still feel squeamish about CI. I don’t like the thought of code going straight to production without a set of eyes on it. I will never be ok with CI.

Please, if you haven’t read it or you need a refresher, check out James Bach and Michael Bolton’s definition of testing and checking (automation = checking, btw, not actually testing):

My Final Stand: We need manual testers. We need automators for the regression checks, unit and integration checks, etc. But we will never NOT need people whose sole job is to manually test an application.

So there you go. We should absolutely not get rid of manual testers. Manual testers are not “at the end of their careers”. Manual testers, that are good at what they do and continue to learn, are indispensable.

Arguments? Further points? Let’s chat 😀