Working Effectively with Legacy Code by Michael Feathers

I’ve been carrying around this book with me for the past few weeks and I find myself going back and reading parts of a chapter again and again, trying to soak in the concepts. This particular chapter, titled: I Don’t Have Much Time And I Have To Change It, struck me specifically when Feathers starts out talking about change and how often it happens in code bases and that steps should be taken to make every consequent change easier than the one before. When teams make it a point to only introduce changes to the code only if they have tests to cover that change, they find that their velocity slows down and people feel like they aren’t getting as much done as they need to. Feathers emphasizes that this is normal but if they persevere, they slowly realize that they are revisiting better code and changes get easier and easier to make.

I think that it’s important for teams to understand the benefits of testable code and how it eases the process of change so that they will invest the time and energy needed to get the code to that point. And I quote:

Ultimately, testing makes your work go faster, and that’s important in nearly every development organization.

Having code that is testable is only possible if we do one of the following (If I remember correctly, I think this is according to Uncle Bob):

1. Write the tests before writing the code
2. Design for testability

Doing the latter isn’t easy, and in fact it is so hard to achieve that people end up not doing it at all. So that leaves us with Option 1. But sometimes you come across code that doesn’t have any tests AND doesn’t look like it is testable, but you have to make some changes in a short amount of time. So what do you do? Michael Feathers suggests 4 approaches, and my colleague Mark documents it in a short and sweet way (basically he beat me to writing the post first).

1. Sprout Method
   Advantages: Separating new code from old code. New code is testable.
   Disadvantages: You’re giving up on the old code for now.
2. Sprout Class
   Advantages: Allows you to move forward with more confidence.
   Disadvantages: Conceptual complexity – moving new code into a separate class disrupts the flow of how key classes in the code base work together.
3. Wrap Method
   Advantages: A great way to introduce seams (this is a concept that Feathers talks about in a previous chapter, which we will visit in a later post) while adding new features.
   Disadvantages: This method requires you to rename the method that you want to change, and create a new method with the same name as the previous method before the rename was done. Sometimes this could introduce sloppy method names because we needed to do some renaming to make way for the new method.
4. Wrap Class
   This method is pretty much the same as Wrap Method except you’re extracting new behavior out into a new class. Feathers mentioned that 2 things push him towards choosing to use Wrap Classes and one of them is when a class has grown so large that he cannot stand to make it worse. Pushing behavior into a new class when the existing class is unmanageably large could also indicate that the original class might have too many responsibilities and thus needs to be gutted.

I like this chapter because it formalizes a lot of things that I try to do very often because of the large range of code bases that I work with, but find it difficult to nail it down to something that could be explained to someone else.

So how does the view model for the test in my previous post look like? Judging from the test cases written, we would have 3 public methods:

1. viewModel.valid
2. viewModel.hasDuplicates
3. viewModel.save which depends on the first two

public class SomeViewModel
    {
        public var dispatcher:IEventDispatcher;

        [Bindable]
        public var subscriptions:ArrayCollection;

        private var subscriptionsFromServer:ArrayCollection; 

        public function SomeViewModel() {}

        public function addNewSubscription():void
        {
            if(subscriptions == null)
                subscriptions = new ArrayCollection();

            subscriptions.addItem(SomeViewModel.createNewViewModel());
        }

        public function get valid():Boolean
        {
            if(subscriptions == null)
                return false;

            for each(var subscription:SomeViewModel in subscriptions)
            {
                if(!subscription.valid)
                    return false;
            }
            return true;
        }

        public function get hasDuplicates():Boolean
        {
            for each(var subscription:SomeViewModel
                                                 in subscriptions)
            {
                var remainingSubscriptions:ArrayCollection =
                                             removeFirstOccurenceOf(subscription);
                if(containsDuplicateValues(remainingSubscriptions, subscription))
                    return true;
            }
            return false;
        }

        public function save():void
        {
            if(hasChanges && valid)
            {
                var subscriptions:ArrayCollection =
                         SomeViewModel.convertToDomainModels(subscriptions);
                var event:SomeEvent =
                         SomeEvent.createSaveSubscriptionsEvent(subscriptions);
                dispatcher.dispatchEvent(event);
            }
        }

    }

Since Mate is event-based, the act of clicking on the ‘Save’ button causes the dispatching of an event, and the dispatcher is always the view, which is injected to the View Model via the Controller.

At the end of this, we have the functionality that pertains to the ‘Save’ button, but the view doesn’t need know about these complications. The controller will wire up the ‘Save’ button to invoke the save method on the view model, which in turn will only dispatch the event back to the controller if there is a need.

Continuing on my previous post, say we have the following view:

In this screen, a user is allowed to add a new subscription in the system that s/he cares about, view existing subscriptions, remove existing subscriptions and subsequently save those subscriptions. Let’s focus on a small portion of the functionality that is outlined in this view. When the user adds new subscriptions, saving is only allowed if:

1. All the required fields have been selected
2. There are no duplicates.

So how do we verify that behavior? Forgetting about the view for a moment, let’s think about the presentation model (I will call it view model) class, which holds the state and logic behind the view. There will be a ’save’ method on that class which will only perform its magic when the view model itself is valid, which means all required fields have been selected, and there aren’t any duplicate subscriptions. So let’s write the tests for that behavior:

    public class SomeViewModelTest extends TestCase
    {
        [Test]
        public function shouldNotBeValidIfThereAreNoSubscriptions():void
        {
            //given there are no subscriptions
            //then
            assertFalse(viewModel.valid);
        }

        [Test]
        public function shouldBeValidWhenAllSubscriptionsAreValid():void
        {
            //given
            viewModel.addNewSubscription();
            viewModel.addNewSubscription();

            //when
            for each(var subscription:SomeViewModel in viewModel.subscriptions)
            {
                subscription.type = TestHelper.SUBSCRIPTION_TYPE_A;
                subscription.group = TestHelper.GROUP_2;
                subscription.priority = TestHelper.ACTIONABLE_PRIORITY;
            }

            //then
            assertTrue(viewModel.valid);
        }

        [Test]
        public function shouldNotBeValidIfOneSubscriptionIsNotValid():void
        {
            //given
            viewModel.addNewSubscription();
            viewModel.addNewSubscription();

            //when
            viewModel.subscriptions[0].type = TestHelper.SUBSCRIPTION_TYPE_A;
            viewModel.subscriptions[0].group = TestHelper.GROUP_1;
            viewModel.subscriptions[0].priority = TestHelper.ACTIONABLE_PRIORITY;

            //then
            assertFalse(viewModel.valid);
        }

        [Test]
        public function hasDuplicatesShouldReturnTrueIfThereAreDuplicateTypeASubscriptions():void
        {
            //given
            var subscriptionsFromServer:ArrayCollection = new ArrayCollection();
            subscriptionsFromServer.addItem(helper.
                createSubscriptionTypeADomainModelWithInformationalPriority());
            subscriptionsFromServer.addItem(helper.
              createSubscriptionTypeBDomainModelWithActionablePriority(TestHelper.GROUP_1));
            viewModel.populateSubscriptionsFromServer(subscriptionsFromServer);

            //when
            viewModel.addNewSubscription();
            viewModel.subscriptions.setItemAt(helper.
                   createSubscriptionTypeADomainModelWithInformationalPriority(), 2);

            //then
            assertTrue(viewModel.hasDuplicates);
        }

        [Test]
        public function hasDuplicatesShouldReturnTrueIfThereAreDuplicateTypeBSubscriptions():void
        {
            //given
            var subscriptionsFromServer:ArrayCollection = new ArrayCollection();
            subscriptionsFromServer.addItem(helper.
                createSubscriptionTypeADomainModelWithInformationalPriority());
            subscriptionsFromServer.addItem(helper.
              createSubscriptionTypeBDomainModelWithActionablePriority(TestHelper.GROUP_1));
            viewModel.populateSubscriptionsFromServer(subscriptionsFromServer);

            //when
            viewModel.addNewSubscription();
            viewModel.subscriptions.setItemAt(helper.
              createSubscriptionTypeBDomainModelWithActionablePriority(TestHelper.GROUP_1), 2);

            //then
            assertTrue(viewModel.hasDuplicates);
        }

        [Test]
        public function hasDuplicatesShouldReturnFalseIfThereAreNoDuplicates():void
        {
            //given
            var subscriptionsFromServer:ArrayCollection = new ArrayCollection();
            subscriptionsFromServer.addItem(helper.
                createSubscriptionTypeADomainModelWithInformationalPriority());
            subscriptionsFromServer.addItem(helper.
              createSubscriptionTypeBDomainModelWithActionablePriority(TestHelper.GROUP_1));
            viewModel.populateSubscriptionsFromServer(subscriptionsFromServer);

            //then
            assertFalse(viewModel.hasDuplicates);
        }

    }
}

The view model has 2 attributes that are used to determine whether subscriptions can be saved or not: valid and hasDuplicates. Next we’ll take a look at the view model itself and how it ties back to the view.

I seem to be obsessed with testing lately and after watching J.B. Rainsberger’s video, I feel even more convinced that good developer testing plays a big part in producing code that is maintainable and not adverse to change in addition to providing the assurance to the developer that a functionality is done when they say it is.

Testing UI code is often a challenge and most projects I’ve been on defer that to acceptance tests that are driven from the browser. I’m not disregarding browser tests entirely but like I’ve mentioned in my previous post, that could be a suboptimal testing option.

Mate (pronounced Mah-tay), which is a tag-based Flex framework, provides the ability to define a separation between the state and behavior of the presentation of a view and the UI controls of the view itself, thus allowing for extensive unit testing of the UI. Before I delve deeper into how that works, let me talk a little bit about the Presentation Model pattern, advocated by Martin Fowler. Given a Model-View-Controller situation, the idea of the pattern is to remove all logic from the view into the presentation model, which is a model class that is part of the presentation (or view). This gives us a few advantages:

1. The view becomes dumb and simple. No logic resides here, just UI components.
2. The view could potentially be reused to display other data by just switching out the model.
3. And more importantly, we are now able to test the behavior of the view (i.e. what happens when a button is clicked, are multiple drop-down menus dependent on each other, and if so, how?)

We will take a look at how to test the behavior and state of the view in my subsequent posts.

On one of my previous projects, we started on a story (or functionality) by first writing the acceptance test for it. We had a custom-written BDD (Behavior-Driven-Development)-like tool that drove the test from the browser. I initially found it quite cool that it was set up in a way that it made it easy for developers to write acceptance tests. One of the downsides of writing these kinds of tests is the slow feedback loop. We of course also wrote unit tests but to verify a particular functionality, we ran the acceptance tests. Over time, the team built up a sizable suite of browser tests and the build started to become so brittle that it was red about 70% of the time. Another downside that we had with those tests was the fact that there were a lot of duplicates. Certain parts of the application were accessed with the same initial flow and each of those tests repeated them, hence slowing down the build even more.

I’ve recently learned that relying primarily on acceptance tests (or integration tests) isn’t such a good idea after all, in fact it’s actually quite a bad idea. In an application with many layers, like most applications are, we have to make sure that every single layer has its own suite of tests that verify behavior regardless of its external collaborators. J.B. Rainsberger has an excellent presentation called ‘Integration Tests Are A Scam‘ in which he explains why integration tests can cause a project more harm than good. He defines integration tests to be tests that pass or fail depending on the correctness of multiple objects with multiple unrelated methods that have multiple pieces of interesting behavior where interesting is complex enough to be tested on its own. Acceptance tests are a level of abstraction higher than integration tests, therefore increasing in complexity and number of points of possible failure.

Besides slow feedback, why else would you NOT want to write too many acceptance/integration tests? Say you have an application that is structured in the following way:

This is a typical set up of most projects that I’ve been on. Say for example that you have tests that run on the repository layer but depend heavily on data being in the database, and if the data changes or if the structure of the database changes, you will definitely have broken tests. But you say, “Each test that runs against the database sets up and tears down data that’s pertinent to that test only. And we recreate the database at every run of the build. So the database could be empty for all we care about.” Okay, that’s all good and fair, but are you going to write ALL your tests for the repository against the database? How many different permutations of those tests can you come up with that would test everything exhaustively? And if you are able to list all those permutations, how long do you think those tests will take to run?

Let’s take this scenario further. You have tests at the repository layer that depend on the database, and then tests at the service layer that depend on the repository layer. Your development team does not have complete control over the database and its data. It changes under your feet a lot and the build is broken almost all the time and the team does not have enough bandwidth and energy to keep fixing them. Then someone says, “We need to drive tests from the UI as well!”. Let’s take a look at how that would look:

As my colleague Prem has said many times, “How much hope do the tests driven from the UI will have of passing if everything underneath it is failing?“. Not only are you introducing tests that take longer to run, you’re also increasing the number of tests that will continue to fail thus making the lives of developers even harder.

I’ve learned that developers need to focus more on tests that are:

1. Fast, lightning fast.
2. Independent of external collaborators, and thus less brittle and prone to breakage because of something other than behavior change in the code immediately under test.
3. Exhaustive. Since they are fast to begin with, we can write as many tests for as many given scenarios as we can think of.

Change is inevitable. And to support change, developers need to be able to move nimbly. Writing tests in a focused manner at the unit level reinforces that nimbleness and allow developers to work more effectively.

Always leave your codebase in a better (or same, if you really can’t make it better) condition that it was in before, never worse. Disregard for this rule is how codebases disintegrate, rot and die.

I find it hard to imagine myself writing any sort of code without a test to back it up. But I’ve learned that not everyone shares that sentiment. So here’s an attempt to explain why developer tests are extremely vital to reliable and working software.

[Disclaimer]: These are not my thoughts alone. They have been collated from a number of books and blogs that I’ve read over the months. A few of which are Clean Code by Robert Martin and Working Effectively with Legacy Code by Michael Feathers.

[Yet another disclaimer]: My colleague Pat Kua reminded me that I should avoid blanket statements that say developers should write tests all the time. Thanks Pat! He writes about when tests should and should not be written here. So I should say that I’m talking specifically about developers writing production code in a focused mode with a good idea of how the resulting code should behave. And I’m not even bringing up Test Driven Development, but just the fact that having tests as an assurance that what you’ve written behaves in the way you expect and having that assurance continuously going forward.

1. Getting fast feedback
When you write a piece of functionality on a web application for example, arguably you could launch the application, drill down to where the code that you wrote is executed and test whether it behaves the way you expect it to. One of the problems with that approach is that the feedback loop is large. Depending on how your development environment is set up, it might take a whole 10 minutes (or more) to bring up the application on the browser, login (if you have to), drill down to the part of the application that you’re interested in and finally test its functionality. With a test, you could run it, and it would probably take 5% (or much less) of that 10 minutes to execute and tell you whether your code behaves the way you think it should. Different types of tests have different levels of feedback loops.

The lowest level of tests you can write are unit tests – these test the behavior of the lowest unit of code that you can have, which is usually a class. These tests should run extremely fast i.e. you could execute a few hundred of them in a matter of seconds. Then there are integration tests that test interaction between two or more components, these could involve database or file system access. These could potentially take longer to run since they interact with agents outside your application and should be written with care as to not increase the build time unnecessarily. Functional or acceptance tests come next. These are usually written against your application as a whole (i.e. through the browser) and should test features of the application on a higher level and mimic the interaction of a user with your application. These tests typically take the longest to run.

2. Tests document your code
This brings up yet another debate about how the functionality of an application should be documented for the users or future consumers of the code. As a developer, the best documentation of an application is always the tests. If I want to know what a class is supposed to do and how it’s supposed to behave, if there was a test, I could take a look at that and in theory, understand what I need to know in a relatively short time as opposed to trying to understand the code. Tests acting as documentation also beats external documentation because if a functionality changes, Word or PDF documents describing that change don’t always get updated and with time developers will lose visibility into how exactly the code should behave.

3. Gives you confidence to change your code
I often hear developers say things like, “I’m afraid to make that change there because I don’t know what will break after that”. That doesn’t mean that the particular developer is incompetent of understanding what the code is supposed to but it’s a symptom of a bigger problem which is the fact that there is no assurance whether a change we make in one portion of the application will not break a functionality in another portion. In his book, Working Effectively With Legacy Code, Michael Feathers points out that tests are “kind of like  a cloak we put over code we are working on to make sure that bad changes don’t leak out and infect the rest of our software. When we have a good set of tests around a piece of code, we can make changes and find out quickly whether the effects were good or bad”.

A lot of times there is a steep ramp up for us to be able to get to the point where we can write any kind of test. For example if our code base was in a condition where there is tight coupling everywhere, or there aren’t clear-cut components that are testable. These impediments would seem to indicate that writing tests would be a waste of time and that we should focus our energies on producing new functionality. Then what I would suggest you do is take a step back and evaluate your priorities.

Would you rather:

1. Spend a chunk of time now reorganizing your code and making it testable, so that you can reap the benefits of having the assurance that whatever you have built still works and will continue to work, and know that whatever that you are currently building or will build in the future will fail faster so that you can fix it and move on.
2. OR, having to go back and forth every time new functionality is built or when changes are made in existing functionality to make sure that they all still work as they should.

Ever since I was introduced to Mockito about a year ago, I’ve been a big fan. I wrote a couple of posts about it and promised a long, long time ago that I would post an example of a test written in Mockito and one in another mocking framework. The lucky chosen one was EasyMock. My biggest plugs for Mockito would be:

1. Test readability. Mockito tests are more concise and it discourages noise in test code.
2. The ability to clearly distinguish test expectations from test verifications so that you know what exactly you’re stubbing and what you’re testing.

Here’s an example of a class that I want to test: ProfileService.java. NOTE: Please ignore the design of the code itself. Let’s just say that all I wanted to do was to test it’s behavior without changing it. And I do know that it’s not the best, in fact far from the best.

There is one method in that class which retrieves a Profile from the database and finds all the contacts associated with that profile and goes through all of them and if there is an application in a contact that is already Closed, an exception is thrown and I’m not allowed to close the profile. Otherwise, the status of the profile is changed to Closed and this is updated in the database through the repository.

Let’s look at the EasyMock test first.

Notice that test verifications, i.e. that profileRepository.updateProfile() was called at the end, are interchanged with test expectations, thus blurring the line between what you’re setting up and what you’re testing.

Now let’s check out the Mockito version of the test.

First thing you should notice is that the test is much shorter, yes, only by a few lines, but it’s a few lines less confusing. The test is also easier to break into a logical given-when-then BDD-style scenario. There is also clear distinction between what you’re stubbing (expectations and setup) and what you’re actually testing (verifications).

I can’t see how EasyMock promotes cleaner and clearer tests, I really can’t.

Michael Feathers and Steve Freeman have a really good video talking about the phases TDD went through over the last few years. I just want to highlight the summary of their presentation.

1. Professionals test their code.
   And when they say ‘test’, I truly believe they mean writing automated tests. Nothing manual.
2. Separate what from how.
   What you’re testing should be set apart from how you’re testing it. For example, my colleague Mark recently spoke a little about how setting up a test correctly can sometimes turn into noise and reduce the readability of your tests. One way to avoid the noise is to use Builders.
3. Automatic tests confirm features.
   I’ve heard people say things like, “I had to go back and test some of the features from last iteration because something broke this iteration”. If we had a suite of automated tests (be it functional, integration or unit), we could avoid situations like that.
4. It’s a change in culture.
   And it still is, surprisingly. Even after test years, TDD is still a lesser known truth in some organizations. Heck, even the understanding of what a unit test is in shockingly lacking.
5. “Working” isn’t good enough.
   A lot of times we say things like “I tested the code, and it’s working.” So what exactly does ‘working’ mean? Does it conform to the business’ expectations? Does it perform well? Another colleague of mine, Phil, wrote about the 3 Things Agile Teams Should Care About – what does it mean to be done?. Let’s forget about being Agile for a moment and just focus on getting a particular functionality working the way as the business expected consistently.
6. Listen to the tests.
   What should you do if the tests are failing when you run the build? What should you do when the tests fail on the continuous integration build? We should listen to them. Tests are a feedback mechanism. They ensure the integrity of our system and we need to make sure that we pay attention to them at all times.
7. A working system provides feedback.
8. Focus on intent.
   What is the intent of the test that you’re writing? Is it expressed clearly in the method name? Are we just testing getters and setters or are we testing the behavior of the code?
9. When you’re lost, slow down.
10. It’s not only about testing.
11. Legacy code is code without tests.
12. Understand the principles behind the practices.
    Agile advocates individuals and interactions over processes and tools. Behind every practice is an intent. We should understand the intent first instead of blindly following the practice lest we fall into the trap of using a practice for a sake of using it.

So what exactly do I mean by ‘layer teams’? I mean structuring your software development team in such a way that you have a group of developers for each layer (or more), e.g. UI, services, database, etc. So a UI developer is only concerned about one specific part of the application, namely the UI, obviously, and the Services developer with the services, and so on and so forth.

Problems that I’ve seen surfacing from such a setup:

1. There’s an enormous amount of overhead time spent going back and forth between the different groups when defining interfaces. Maybe it’s the nature of the teams I’ve worked with in the past but IMHO this extra time could be spent writing tests that iron out the interactions between the layers.
2. Then there’s another chunk of time spent putting those pieces together and making sure they work correctly, i.e. integration.
3. This causes silos in the development team where only some people know about the UI, others about the back-end services, etc. This discourages developers to step outside their comfort zone to learn another person’s ‘trade‘.
4. Update >>> Introducing specialized teams also increases the amount of distrust between the teams. One example is the database management team holding on to the schema and data for dear life and believing that no developer should have access to change it. Maybe this is common situation across a lot of projects regardless of team setup, but separating the team does decrease their level of trust for each other.

What I think is better is to have every developer comfortable with every part of the application, even if it means taking a productivity hit – as I’ve heard it being said. Let’s say I have a story on the wall that says: “Add item to shopping cart”. Ideally, a pair of developers would start first by working with the QA to define acceptance criteria and writing the acceptance tests for that feature. (This is a side note to a session at Agile 2009 that I’m very interested in: Where Does Developer Testing End and Tester Testing Begin?. I wish I was able to go.) Then they would implement what is needed to be able to click through on the application to add an item to the shopping cart which eventually ends up in the database.