Approval Tests

Hello JRack

2009-09-12T14:27:00.001-07:00

Today Woody Zuill and myself put some of the finishing touches on a port of Rack to Java, and published it to the world. So let's do a quick show-and-tell to show you why we are so excited.

JRack is an extremely simple yet extremely useful and powerful way to create web apps.

Simple To Create
Let's look at how simple it is to create a "Hello World" web page.

public class HelloWorldRack implements JRack
{
  public RackResponse call(Map<String, Object> input)
  {
      return RackResponseUtils.standardHtml("HelloWorld");
  }
}

Wasn't that simple? Just 1 line to pass back the html we want displayed.

Simple To Run

Of course, this has just created it. What if we actually want to have this running on a Web Server, what would we have to do?

We can simply harness it through Jetty (you can also deploy on regular servers like tomcat). Jetty is a simple, standalone Web Servlet container. You can find out more about it here:

We'll add a main method to our HelloWorldRack:

public class HelloWorldRack implements JRack
{
   ...
  public static void main(String[] args)
  {
    JettyUtils.startJettyRack(80, new HelloWorldRack());  
  }
}

Now, if we run HelloWorldRack, we will have a fully running rack server serving the HelloWorld page.
Nice and simple. You are up and running with just two lines of code.

Simple To Test & Reuse
But what if you unit test this code? Don't you need a webserver to even run it? NO!
You can call the RackApp directly. Simply pass in what in used by the environment (in this case, nothing). Whats more we've added approval test support so you can simple approve the resulting html.

Put this code into a JUnit test class:

public void testHelloWorld() throws Exception
{
    Approvals.approve(new HelloWorldRack().call(null));
}

Okay - simple to write, simple to run, simply to test. There are many sites extolling the virtues of Rack. More and more, the Ruby community is migrating over to it. And now, you can have the same simplicity in the Java world. We'll write more about it later. You can download it here.
I also encourage you to look at the test examples here

Why Write Unit Tests?

2009-06-21T21:51:00.001-07:00

Why Write Unit Tests?

I've made a small video to go with this post

Approval Tests have allowed us to focus on the parts of testing at a much higher level of abstraction. Things that had been too hard to see have become visible. The benefits of Automated Tests are one of the areas that we can now more clearly appreciate. There are 4 main areas programmers can benefit from using Automated Tests (Unit and otherwise)

Specifications
Feedback
Regression
Granularity

Let's talk about each of these.

First, there is Specifications. Everyone knows that good requirements help programmers to produce better software. Well, you can't do better than a running, failing unit test. This gives a clearly defined goal, and provides a executable document of the Users Intent. Focusing on this particular area, has led to the emergence of Behavior Driven Development (BDD).

Fast Feedback. After you know what you want to write, it's very helpful to know that you are fulfilling that requirement. Feedback allows you to know when you have made a mistake, or a wrong turn. Imagine you are driving to Disneyland and you miss your turn. When would you like to know that? Immediately afterwards? 10 minutes later? 6 hours later? Or only when you reach the east coast? Feedback from your test can tell you within seconds if you didn't code what you thought you did. Compare that to sending it to QA and getting a bug report back next week. Focusing on this particular area has led to Test Driven Development (TDD), and is what they are referring to when they speak of bouncing between the red & green bars.

Regression Testing is what most people actually think of when they talk about testing. This is all about maintaining & assuring the quality of the code (by which I mean, is the code working). Focusing on this aspect of testing brings about Acceptance Testing and products like Selenium & Fitness.

And finally Granularity. Imagine the following bug report: "The software has a bug." While this is sufficient to tell you whether or not you have bug free software, it's not at all useful in helping you to remove that bug. (If you've ever done a google code jam, this is exactly what the bug reports look like by the way.) It's nice to have some granularity pointing you to the problem. Something like "if you select user #6 and run a expense report for July 7th, line 15 says $34 when it should say $65". Now that's helpful! Again, TDD is really the area most focused on small granular tests, partly because granularity and feedback are highly related.

Hopefully, you will start to see these 4 features of testing as you continue to unit test. I don't want to place any sort of hierarchy of value on them right now, just point out that they are all valuable. Of course, if you are using approval tests, you'll see how they can aid each and every one of these categories!

Approving the Weather

2009-04-25T18:45:00.001-07:00

How can I test the weather? We have a project where we need to know the weather report. This is possible thru a web API that will give us the weather report. However, we have no control over either the API or the weather. So how do we test this? The traditional approach is to encapsulate all calls to the weather API and then mock it out. The unfortunate part is this doesn't actually test the weather at all. It 'only' allows us to test the entire rest of the app in hopes that the encapsulated piece will work. Let's take a look at a new way to actually test the Weather API using approvals.

Feedback, Regression & Granularity :Unit Tests bring us a variety of things that are important in writing software: Feedback as we are writing the code, to know that we are writing what we wanted; Regression Testing so that once it works, it doesn't get bugs; and Granularity of Feedback, so when something goes wrong, we know exactly where and why it did.

A new approach. Let's look at a new approach that can get us all of these things, quickly and easily.

Feedback: First, we are going to need an encapsulated piece that loads the weather.
This is going to be a small class with two functions.

class WeatherLoader
{
getQuery(); // returns the query that's used to get the weather
execute(query); // executes the given query to get the weather
}

Now we're going to write a test that exercises this.

Approvals.approve(new WeatherLoader("San Diego"));

When we run this it will fail as all approvals do the first time and we will see the results of using this API.

Query:
GetCityWeather?city=San%20Diego

Result:
<WeatherReturn>

<Success>true</Success>
<ResponseText>City Found</ResponseText>
<State>CA</State>
<City>San Diego</City>
<Temperature>62</Temperature>
<Unit>Fahrenheit</Unit>

</WeatherReturn>

So now we have the feedback that our API is working.

Regression: Of course now that it's working we need it to work from this point on. Since we have no control over the weather, when we approve this test the only thing that will be approved will be the query, not the result. From now on as long as the current query matches the old query this test will pass. This test will pass without actually running the query against the web service.
This is important so we'll say it again, The test passes if the query hasn't changed.

This means the approval artifact would look like this :
GetCityWeather?city=San%20Diego

Granularity When Things Change: Of course all things change. Let's say in the future someone changes this call so that it returns the temperature in Celsius. Lets take a look at what would happen when you run that test.

Received	Approved
GetCityWeather?city=San%20Diego&unit=Celsius Result: <WeatherReturn> <Success>true</Success> <ResponseText>City Found</ResponseText> <State>CA</State> <City>San Diego</City> <Temperature>29.4</Temperature> <Unit>Celsius</Unit> </WeatherReturn>	GetCityWeather?city=San%20Diego Result: <WeatherReturn> <Success>true</Success> <ResponseText>City Found</ResponseText> <State>CA</State> <City>San Diego</City> <Temperature>85</Temperature> <Unit>Fahrenheit</Unit> </WeatherReturn>

Both queries are re-run: Notice that the temperature of the approved file is 85 not the 62 when it was originally approved. This is because the query is the only thing that is persisted. The result of the query is re-executed at the time of failure giving us the current temperature in the approved section.

Then notice that the query is re-executed for the received side. Through the beauty of diff tools not only do we get the large picture of what's going on but the details and granularity of everything that has now changed.

Testing the Indeterminate : This is a one line test that runs against an external Indeterminate system. It required zero setup and we have zero control over it, yet, in the end we have tests that give us Granularity, Regression and Feedback.

Other possible cases: Databases are another place where being deterministic can be a problem. Of course you can do a large amount of setup to force them to be deterministic. And, of course, this carries a high price in both writing the test and maintaining it. Let us suggest that you just accept that things aren't deterministic and use the above approach.

As a final thought. Agile was created by accepting that requirements change. This approach was created by accepting that not everything we test is deterministic.

Use Reporter Annotations

2009-03-29T10:21:00.001-07:00

Today I implemented a bit of dependency injection for the reporters in the Java version of approvaltests.

The idea is that you might want your own reporters for your project. Things to take advantage of something we didn’t think of. While you could have always done this by calling:


Approvals.approve(ApprovalWriter, 
                  ApprovalNamer, 
                  ApprovalFailureReporter)

I wanted to make it a wee bit easier.

Now you can do it by adding an annotation to any level of the stack calling. For example:


@UseReporter(FailedQuietApprovalReport.class)
public class MyQuietTest extends TestCase
{
    public void testSomethingQuiet()
    {
    }            
    @UseReporter(FailedDiffTextApprovalReport.class)
    public void testWithDiff()
    {
    }
}

This should make it easier in Java, and will probably appear soon in c#. You can download it now as part of

ApprovalTests Current 0.0.004

"Let it BDD" and a mention about approvals

2009-01-07T11:54:00.000-08:00

Here's a parody I wrote on testing code that mentions approvals.

Let it BDD

Talk at XPSD Thrusday Nov 6th

2008-10-31T19:03:00.000-07:00

we are giving a talk on approval tests at XPSD Thursday Nov 6th at 6pm, here's the teaser...

Approval Based Tests

Write your code, run it, see that it works. This is a familiar process to all programmers. The addition of 1 line of code can now make that a repeatable automated regression test!

Well tested code has been shown, over and over, to improve the quality and maintainability of software. Yet most projects are not well tested. Why is this? Too often programmers feel that testing imposes too high of an overhead on writing code.

In this talk, our goal is to make it easier and faster for you to write unit tests.

We will show how using approvals, in addition to asserts, will great increase not only the ease and speed of writing a test, but the completeness and maintainability of those tests; bridging the gap between people who don't write tests, and test driven developers.

"I feel like I have to write my code twice…" Right now, unit tests impose a difficult step upfront of knowing exactly what you want. Asserting this outcome then becomes cumbersome the more robust & detailed it is. After that, maintaining the test becomes equally cumbersome. Because of this, tests either do not get written, or get discarded, leaving untested code which is fragile, error prone and leads to the nightmare of maintenance so many experience.

After this talk, you should find yourself spending less time writing tests, yet having better tested code, a proven way to increase successfulness of your software.

"After I heard Dan & Llewellyn talk about approval tests, I tried it out… in the first 4 minutes I was testing my code" - C. Monkey

"I find myself writing more code just so I can write more approval tests!" - Script Kiddie

"All my life I'd sought approval from other developers, now I can approve on my own" – Noself E. Steam

"With all the time I saved, I actually learned to talk to women" - Mystery

Approval Artifact Files

2008-10-30T01:44:00.000-07:00

One key element of approvals is the approved artifacts that are created, and I wanted to take some time to go into these. First off, it is worth mentioning that for xUnit type tests, what is created is a file of the name

MyClass.myMethod.approved.extention
... and ...
MyClass.myMethod.received.extention

This file resides in the same directory as the source file for MyClass.

Many times this is simply a text file. The output being approved being taken from the toString() method, or a similar printer. Sometimes it is an html file, which is really just a txt file, sometimes it is a Pdf file or Png image file.

Q: Are these files kept in source control?

Yes, the approved files reside in your source control. This has 2 big benefits

1) The test are repeatable on different systems.

2) You can flip back through easy to view output, to pinpoint when and where a bug got introduced.

Q: What configuration is needed?

None.

Q: How can I write to a custom file format?

Simple, all Approvals.approve(object) calls simply construct an approver, and a reporter. The approver is normally a FileApprover, which takes a FileApprovalWriter and an ApprovalNamer.

To do a custom file output simply make your own ApprovalWriter and your own convenience approve() function

Downloads on source forge!

2008-10-21T08:26:00.000-07:00

This weekend we got the Java and C# versions packaged up on source forge.

Download them, try them out. Tell us what you think.

https://sourceforge.net/projects/approvaltests/

I know exactly what I want!

2008-10-16T14:59:00.000-07:00

Well...sometimes

When creating software, customers don't usually know exactly what they want at the beginning. Agile has checkpoints that allow for quick feedback; while the product is under development, customers can see features as soon as they are completed, giving them an opportunity to accept it or to suggest changes.

Approvals give developers the same opportunity. As a developer, you may not know exactly what you want. The more insight you can get from a result, the more you understand where you want the design to go. That tends to lead us to a more natural flow of coding. You start off with a design in your head, code up an example usage, and implement it. When the result is what you expect, you approve it. With your approved result, you also have a regression test that ensures your expectations don't get violated.

Here's an example test from a bowling game without approvals:

public void TestFourThrows()
{
game.Add(5);
game.Add(4);
game.Add(7);
game.Add(2);
Assert.AreEqual(18, game.Score);
Assert.AreEqual(9, game.ScoreForFrame(1));
Assert.AreEqual(18, game.ScoreForFrame(2));
Assert.AreEqual(3, game.CurrentFrame);
}

This is nice; we can see some of the usages for the game object. Now lets take a look at the approval version:

public void TestFourThrows()
{
game.Add(5);
game.Add(4);
game.Add(7);
game.Add(2);
Approvals.Approve(game);
}

The approval line doesn’t show us any usages of game like the first example. Glancing at the code we don’t even know what we expect either. When we look at the content of the approval we see:

Frame 1: 9
Frame 2: 18
Current Frame: 3
Total Score: 18

or

| 1 | 2 | 3* | Total |
| 9 | 18 | | 18 |

This output is much more expressive and gives us an understanding of the state of the game object. It allows us to change our internal design without having to change our test code. As we move forward we may find the need to add more information to the output. We make the change, fail the approval, then look it over; if we like the new result we approve it.

Those of us who push a design by writing tests first may have issues with this approach. Instead of creating assertions against primitive types for our expectation just trust the one in your head. While we code our expectations might change. Again, Approvals make it easier for us to change our code without changing the test code. As a discipline we still need to work in small units and design as we go. Our code can also skip those rare artifacts we sometimes need to create for testing. Because we've spent the time up front working out a visual for our code we can easily verify it.

When it comes to pairing, this allows both developers to come up with a common representation of the result they expect. When something fails, instead of double checking if our assertions were correct, we can look at the approval and discuss the expectation as a whole. We spend more time talking about the design instead of how to test it.

There are many ways to test code and they provide different solutions. Some methods push a design, some confirm them, some are used for regression testing. When you write tests, you need to decide if describing the interface is more important or the outcome. Approvals is another tool to help you write better code and get you closer to exactly what you want.

Approve is the new Assert

2008-10-14T21:38:00.000-07:00

assertEquals(5, person.age());
assertTrue(person.isFemale());
assertEquals("jane", person.getName());

Asserts are a great way of checking primitives. However, they get cumbersome when you want to assert an object. Even the above example really is asking for

assertPerson(person, "jane doe, female age 5");

Of course this quickly becomes a nightmare of trying to describe the person object. This is where approvals come in.

Approvals.approve(person.toString());

Now you can get a test with nicely formatted output

Name : jane doe
Age : 5
Sex : Female

The more complex the object, the greater the task of writing & maintaining the asserts becomes. While approvals always stay the same one line:

Approvals.approve(yourObject);

So whether it’s a single object, a tree of objects, a Gui Component, an Html page, a pdf document, an email, etc… if you can output it to a file, you can approve it. Then you can always view that output in whatever manner you wish. So there’s no need to find a fancy way to describe a pdf: just view it in acrobat, and if it’s good approve it.

Approval Tests (a pictures worth a 1000 tests) [repost]

2008-10-14T10:52:00.000-07:00

Scenario 1 [Testing a User Interface]

Core Concept :

Component gui = createGui();
Approvals.approve(gui);

This creates a snap shot of the gui: gui.received.png

This passes if gui.received.png == gui.approved.png

If you like the result, simply rename gui.received.png -> gui.approved.png and the test will pass.

Could you explain that in a diagram?

yes.

Why is this awesome?

“a picture is worth a 1000 tests”. Ever find that your unit tests aren’t giving you enough security? Have you started making lots & lots of asserts? Ever find your tests test too many things and the maintenance required to change anything is actually reducing your agility?

Approval Testing gives you 100% lock down on your gui, yet changes simply require a click of a button.

Where are the approved images stored?

Approved images are stored in your testing folder and in your source control. This makes the test repeatable on different systems & you can actually browse a visual of how your gui’s have changed over time!

How does it work for a web app?

String html = createWebPage();

Approvals.approve(html);

Instead of an image, now a text file (page.received.html) is created.

Scenario 2 [Behavior Driven Development]

Core Concept :

Object myObject = createObject();

/*temp*/ Approvals.approve(myObject.toString());
doSomething1();
/*temp*/ Approvals.approve(myObject.toString());
doSomething2();
/*temp*/ Approvals.approve(myObject.toString());
doSomething3();
/*temp*/ Approvals.approve(myObject.toString());
doSomething4();
/*temp*/ Approvals.approve(myObject.toString());
doSomething5();
Approvals.approve(myObject.toString());

As the code is created, the approval will move down, so That the final code is

Object myObject = createObject();
doSomething1();
doSomething2();
doSomething3();
doSomething4();
doSomething5();
Approvals.approve(myObject.toString());

Why is this awesome?

Many times a unit test is scattered with spot testing to assure things are done correctly. This creates a barrier to readability, and can lead to unneeded code.

Approval tests reduce the need to spot check, making the test cleaner and simpler.

What if I don’t want to change my toString?

No problem, create a toOtherString(), or a ObjectWritter, or Inspect the object. Sometimes we even create a ObjectVisualizer when an image is a better way to show the state.

What if there’s a bug in my code?

The first thing to do is to try to walk through your code and get an idea of the state of the object, so you can see where it went wrong... if only you had a nice way of visualizing the state of your object.... oh wait! that’s the first thing you created! I guess it will be pretty easy to debug.

Scenario 3 [The lock down]

Core Concept :

Say I have a web page that displays a user profile. I want to refactor it, but I have no tests. However, I do have 300 users in my current database.

String allPages = getAllUserProfilePages();
Approvals.approve(allPages);

Why is this awesome?

2 lines of code to completely lock down a legacy process. That is the definition of awesome!

Won’t this test be kinda slow?

Yes. But after you are done safely refactoring, you can/should remove the tests.

Is this test data independent? Will it be repeatable tomorrow?

No. But after you are done safely refactoring, you can/should remove the tests.

Should I keep this test after the refactoring?

No!

Approve(this.blog)

2008-10-14T09:14:00.000-07:00

We wanted a centralized place to put all of our thoughts on how to use approval tests.
Here it is.