Get your project from zero to tested leveraging the power of Cucumber and Factory Girl

Testing can be a daunting task, especially for developers who are new to Rails. It sounds like a great idea, but where do you start? What do you test? What don’t you test? For many developers, the hardest part of testing is jumping in and getting your feet wet. Luckily, Cucumber makes getting started with testing easy, you’ll be testing your code form top to bottom in no time.

What is Cucumber?

Cucumber, in simple terms, is a framework for writing human-readable, meaningful tests that allow you to test the full functionality of your Rails project from database access to business logic to what’s displayed in your views.

From the Cucumber Wiki:

Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain-specific language and serves as documentation, automated tests and development-aid – all rolled into one format.

When you write your tests with Cucumber you get full-stack automated tests, documentation and a tool to help you communicate with your product owner.

Who’s the Factory Girl?

Factory Girl is a fixture replacement for Rails that allows you to create meaningful objects for use in your tests.

From the Factory Girl Github Page:

factory_girl is a fixtures replacement with a straightforward definition syntax, support for multiple build strategies (saved instances, unsaved instances, attribute hashes, and stubbed objects), and support for multiple factories for the same class (user, admin_user, and so on), including factory inheritance.

When you use Factory Girl in conjunction with Cucumber you can simplify complex test setup and use real objects from the database, not brittle mocks.

Getting Setup

Getting your project setup with Cucumber and Factory Girl is pretty straightforward. First, you’ll need a few gems installed:

    sudo gem install cucumber-rails
    sudo gem install webrat
    sudo gem install thoughtbot-factory_girl --source=http://gemcutter.org

Note: As of 0.5.0 the Rails specific functionality was extracted from the cucumber gem into the cucumber-rails gem

Cucumber comes pre-packaged with some generators for getting your project ready for testing. From the root of your Rails project, run the generator:

    ruby script/generate cucumber --webrat --rspec

You’ll notice that we’re passing a couple of flags to the generator, --webrat specifies that we want to use webrat as our automated browser and --rspec specifies that we want to use rspec as our test framework. If you don’t pass these options Cucumber will guess which options you want based on the gems which you have installed.

After running the generator, you will see some output about the files that were created:

    create  config/cucumber.yml
    create  config/environments/cucumber.rb
    create  script/cucumber
    create  features/step_definitions
    create  features/step_definitions/web_steps.rb
    create  features/support
    create  features/support/env.rb
    create  features/support/paths.rb
    create  lib/tasks
    create  lib/tasks/cucumber.rake

Here’s a brief description of what some of the important files do:

*config/cucumber.rb*: This is your cucumber environment, similar to production.rb or development.rb. Typically you’ll add cucumber specific config.gem directives and similar “cucumber only” items in here.

*features/step_definitions/web_steps.rb*: These are the webrat steps that you get for free with Cucumber and Webrat.

*features/support/env.rb*: This file has Cucumber specific setup and configuration options. This changes often, so it’s recommended to not alter this file directly, but to create your own custom env.rb (e.g. custom_env.rb).

*features/support/paths.rb*: Cucumber needs to be aware of the custom routes in your application that you plan on using when testing, this is where they are specified.

Cucumber from 10,000ft

Now that you’ve got your project configured to use Cucumber, let’s go over a high level view of the general concept of testing with Cucumber.

Cucumber uses “Features” to describe the functionality of your application. Features, stored in the features directory, are written in a simple human-readable format called Gherkin. The “Steps” in your feature files that describe the functionality of your application are backed up by step files in the features/step_definitions directory. The step files act like translations between the human-readable feature files and the underlying Ruby code of your Rails application.

Typically you will be using Cucumber in conjunction with Webrat to test your application. Webrat acts as the person behind the keyboard clicking around the site, filling out forms and viewing resulting pages. If you’ve ever tested your project by manually filling out forms and verifying output, that’s exactly what Webrat does for you, except it’s automated.

Let’s Get Testing Already

Rather than go into detail about TDD or BDD and why you should use them, I’m going to assume that you’re like most Rails developers looking to learn more about testing. You probably have an application or two out there in the wild, seemingly working just fine. However, you know that you should get some test coverage in there so that you can confidently make changes and debug problems as they come up. With that in mind, I am going to use a sample application that already has some functionality and we will add our Cucumber tests where needed.

A Veterinary Patient Management System

Our sample application is a simple patient management system for a Veterinarian that deals with Owners, Pets and Visitations. Let’s take a look at some of the models.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

 
    class Owner < ActiveRecord::Base
 
      has_many :pets
      validates_presence_of :name
      validates_presence_of :email
 
    end
 
    class Pet < ActiveRecord::Base
      belongs_to :owner
 
      validates_presence_of  :name
      validates_presence_of  :species
      validates_inclusion_of :species, :in => %w( dog cat bird snake ),
      :message => "Species: %s is not included in the list of accepted species"
 
    end
 
 
    class Visitation < ActiveRecord::Base
 
      has_one :pet
 
    end

The first thing for which we’ll be adding a Feature is the process for creating a new Owner record. When a new client calls the Veterinarian’s office, an employee needs to enter them into the system.

Writing a Feature

Cucumber features are very straight forward. The feature file explains a set of functionality by describing different cases through scenarios.

1
2
3
4
5
6
7
8
9
10

 
    Feature: Manage Owners
      In order to value
      As a role
      I want feature
 
      Scenario: title
        Given context
        When event
        Then outcome

Each Cucumber feature represents a desired software feature that a certain user of the system wants in order to achieve some end goal. The scenarios are formatted so that a context is setup, an event occurs and an outcome is evaluated.

Let’s examine our real Manage Owners feature.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

 
    Feature: Managing Owners
      In order to manage our client list
      As an employee
      I want to be able to CRUD owners
 
      Scenario: Creating a new Owner
        Given I am on the homepage
        And I follow "Owners"
        Then I should be on the owners index page
        Given I follow "New owner"
        And I fill in "Name" with "Clayton"
        And I fill in "E-Mail" with "clayton@example.org"
        And I fill in "Address" with "100 Cactus Rd"
        And I fill in "City" with "Scottsdale"
        And I fill in "State" with "Arizona"
        And I fill in "Postal Code" with "85000"
        And I fill in "Phone" with "480-555-1212"
        When I press "Create"
        Then I should see "Owner was successfully created."
        And I should be on the owners index page

Note: When you specify “And” in a Cucumber feature the step inherits the Given/Then/When from the previous step

Our scenario covers the act of creating a new Owner record. We first setup a context (I am on the homepage). Next we click a link in the navigation (I follow “Owners”). Once we’re on the index view for the Owners controller (I should be on) we can click another link to get to the new view for the Owners controller. From there we are simply instructing Webrat to fill out our form (I fill in) and click the submit button at the bottom (I press “Create”). Finally we are asking Webrat to read the resulting page and see if there is some text present (I should see).

Our feature is looking pretty good, but we still have a few more steps before we can get this scenario passing. Running the this feature from the command line will give us some output and more direction. Once you’re run rake db:migrate and rake db:test:prepare run the following from your application’s root directory:

  ruby cucumber features/manage_owners.feature

Looks like we’re failing, and Cucumber gave us some information about what’s wrong.

  Can't find mapping from "the owners index page" to a path.
  Now, go and add a mapping in patient-management/features/support/paths.rb

If we open up our features/support/paths.rb file we can add the correct path right below the default path for the “home page”. The paths file is really just a list of regular expressions that Cucumber uses to match named routes to words in scenarios.

1
2
3
4
5

 
  when /the home\s?page/
    '/'
  when /the owners index page/
    owners_path

With that little change we have our first passing Cucumber scenario! When running Cucumber features from the command line, Cucumber will print out a little summary message.

    1 scenario (1 passed)
    14 steps (14 passed)
    0m0.290s

Complex Scenario Setup

In our above example the setup, or Given steps, were pretty simple. However, sometimes you need more complex setups for your scenarios, like editing a record for example. Cucumber makes it easy to create objects in your scenarios using tables in multiline step arguments.

Here is a feature that goes through the process of editing an owner’s address. You’ll see the Cucumber table that is used to setup our existing owner record. The first row in the Cucumber table correspond to some of the attribute names for our Owner model.

1
2
3
4
5
6
7
8
9
10
11
12

    Scenario: Editing an existing Owner
      Given the following owners:
       | name    | email               | address       |
       | Clayton | clayton@example.org | 100 Cactus Rd |
      Given I am on the homepage
      And I follow "Owners"
      Then I should be on the owners index page
      When I follow "Edit"
      And I fill in "Address" with "567 N Scottsdale Rd"
      When I press "Update"
      Then I should see "Owner was successfully updated."
      And I should be on the owners index page

When we try to run this, Cucumber will give us the “shell” for our custom step definition “Given the following owners”.

    Given /^the following owners:$/ do |table|
      # table is a Cucumber::Ast::Table
      pending # express the regexp above with the code you wish you had
    end

Since we never created a step definition for this Cucumber feature, we can go ahead and add a new file, features/step_definitions/manage_owners_steps.rb. It is important to note that Cucumber will load all of the files in features/step_definitions and that steps can be used across scenarios and features. If you think you’re going to re-use a step definition, it might be a good idea to place it in something like features/step_definitions/shared_steps.rb.

Note: You should put some thought into your step organization as your project grows in size.

Cucumber takes the table argument from your scenario and turns it into an array of hashes, which is technically a Cucumber::Ast::Table object. By iterating through each hash of the array, we can build up an object that we can use later in our tests. Since we want this object to live in the database and we don’t want to describe every attribute of the owner in our scenario, we can use Factory Girl to simplify the process.

The factory for our owner is simple. I won’t go into the specifics of how to create factories as that’s a whole other article. You can read more about using factories on the github project page

Create a file to store your factories in features/support/factories.rb and copy the following.

1
2
3
4
5
6
7
8
9
10

    require 'factory_girl'
 
    Factory.define(:owner) do |o|
      o.name "John Doe"
      o.email "john@example.org"
      o.address "123 Elm Street"
      o.city "Phoenix"
      o.state "Arizona"
      o.zip "85000"
    end

When we use the factory to create our object, Factory Girl will take the attribute hash that we’ve passed in when creating the object and use those values for the object. Factory Girl will also fill in any blank attributes with the defaults we’ve specified in the factory definition. The record that is created will actually exist in the database, it is not a mock and it does not have any stubs, as far as our Rails app is concerned it is just a regular record in the database.

1
2
3
4
5
6
7

    Given /^the following owners:$/ do |table|
      table.hashes.each do |attributes|
        # {:name => "Clayton", :email => "clayton@example.org", \
        :address => "100 Cactus Rd"}
        Factory.create(:owner, attributes)
      end
    end

Another place we could use our Given the following owners step would be when creating a feature that deals with creating Pet records. An owner has_many pets while a pet belongs_to an owner. When we create a pet record, we need an existing owner record to which we’ll link the pet record. Start by creating a new feature file, features/manage_pets.feature.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

    Scenario: Creating a pet
      Given the following owners:
       | name    | email               |
       | Clayton | clayton@example.org |
      Given I am on the homepage
      And I follow "Pets"
      And I follow "New pet"
      When I fill in "Name" with "Bruno"
      And I select "Dog" from "Species"
      And I select "March 19th, 2005" as the "DOB" date
      And I select "Clayton (clayton@example.org)" from "Owner"
      When I press "Create"
      Then I should see "Bruno was successfully created."
      And I should be on the pets index page
      And a pet named "Bruno" should be owned by an owner named "Clayton"

In this example we have re-used our Given the following owners table. We have also made use of some new Webrat steps like I select for interacting with select lists and I select [DATE] as the [DATE LABEL] date for selecting a date from Rails’ date_selector helper. Finally, there is a custom step that we are going to use to make sure that the connection between our newly created pet and the existing owner is in place.

Let’s take a look at the step definition for our custom step above.

1
2
3
4

 
    Then /^a pet named "([^\"]*)" should be owned by an owner named "([^\"]*)"$/ do |pet_name, owner_name|
      Pet.find_by_name(pet_name).owner.name.should == owner_name
    end

When you place something in double quotes in your scenario steps, like “Bruno” and “Clayton” in our example, they are captured using regular expressions in the step definition. Cucumber then passes the matched values along so that then can be used in your assertion. We can find the pet based on the pet_name and make sure that the owner linked to this pet via the belongs_to association is the same as the owner_name we specified in our scenario. This is an example of how Cucumber can be used, with RSpec matchers, to make an assertion that has nothing to do with inspecting the DOM of a resulting webpage.

Note: Beware of the Conjunction Steps Anti-pattern

Scenario Outlines

Cucumber provides a very simple way to test multiple different situations with a single scenario. These might be edge cases or just repetitive examples that don’t require their own scenario.

We have some business logic in our application that determines when an appointment, or Visitation, can be made with the Veterinarian:

• Visitations less than three days from today require approval before being booked
• No visitations can be booked more than six months in advance
• No visitations can be booked in the past

We can easily represent this using a Cucumber Scenario Outline. Start by creating a new feature features/visitation_logic.feature. Also, because we’re going to be working with time and date specific business logic, this is a good time to point out that we can stub methods when using Cucumber, however, this practice is frowned upon and should only be used for things like dates and connecting to external APIs.

To add stubbing support add the following to your features/support/custom_env.rb file.

  require 'spec/stubs/cucumber'

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

 
    Scenario Outline: visitations
      Given the following owners:
        | name    |
        | Clayton |
      And the following pets:
        | name  |
        | Bruno |
      Given today is "<today>"
      Given I am on the home page
      And I follow "Visitations"
      And I follow "New visitation"
      And I select "Bruno (Owned by Clayton)" from "Pet"
      And I select "<date>" as the "Appointment Date" date
      When I press "Create"
      Then I should see "<message>"
 
    Examples:
      | today      | date               | message                                           |
      | 2010-01-01 | January 2nd, 2010  | Visitation requires short notice approval.        |
      | 2010-01-01 | January 31st, 2010 | Visitation succesfully created.                   |
      | 2010-01-01 | November 2nd, 2010 | Visitations cannot be booked that far in advance. |
      | 2010-04-01 | January 2nd, 2010  | Visitations cannot be booked in the past.         |

Cucumber will go through the Scenario Outline you have created and substitute the values in <>’s with their corresponding value in the Examples table at the bottom. Cucumber goes through each line in the Examples table and runs the scenario using your specified values.

First, lets implement the step for stubbing our today’s date.

1
2
3
4

 
    Given /^today is "([^\"]*)"$/ do |date|
      Date.stub(:today).and_return(Date.parse(date))
    end

In features/step_definitions/shared_steps.rb we can add the implementation for Given the following pets. We are cheating here because we’re not explicitly linking the Pet to the Owner in our scenario outline. This example isn’t so bad, but keep in mind that this association is not obvious when reading the scenario. This type of “behind the scenes” scenario setup is generally a bad idea as it obscures what’s really happening. Since we’ve already created the owner we can just find the first one in the database and create the pet records using the appropriate Rails’ association method.

    Given /^the following pets:$/ do |table|
      table.hashes.each do |attributes|
        Owner.first.pets.create(attributes)
      end
    end

The other steps in our scenario already exist, either because we created them, or because they come for free with the built in steps. In this one scenario we were able to test four different business logic outcomes. This test could be expanded to go beyond what is shown on the resulting page, perhaps to ensure that an Approval record was created for the short-notice appointment.

Advanced Cucumber Goodies

While you should now have an understanding of the basics of Cucumber, there are a number of other powerful features.

Tags

Cucumber allows you to “tag” your scenarios and features so that they can be run, not run or have special other tasks run before and after them. Out of the box you get the @wip tag (Work In Progress) which isn’t run as part of the normal rake cucumber:ok process.

Hooks

Using tags, you can run Before and After tasks that run some code before or after a given feature or scenario. You can even run these hooks before or after all of your features. You might need to do some cleanup of generated files in an After hook or some CPU expensive operation before a single scenario that you don’t want running before all of your scenarios.

Table Transformations

The tables that we have been creating use the exact model attribute names for their header rows. This is convenient for the developer, but makes the scenarios harder to understand. This is especially true if you have very strange or legacy attribute names like record_quote_ext_type__c. Using table transformations you can use “Record Extension Type” in your scenario table and map that to record_quote_ext_type__c in your step definition.

Calling Steps from Steps

It’s possible to take a multi-step process, expressed in Cucumber steps, and call them all at once from another step. You might have a few steps that describe logging in to a system (filling out login credentials, pressing a button etc.) You can easily put them all into one step, Given I am logged in, and then reference that elsewhere. This helps to reduce unnecessary repetition of steps and keep your scenarios to a manageable size.

Profiles

Cucumber allows you to specify a profile to use when running your features. By default Cucumber will not run any @wip tagged scenarios and the output will contain any failing files and the line number of the failure. However, if you wanted to run the @production tagged stories and output a nice HTML report, you could setup a new profile for that in config/cucumber.yml and use it when running your features.

Testing Javascript

You can even use Cucumber to test complex javascript. Tools like Celerity with Culerity or Selenium give you the ability go beyond the interactions provided by Webrat. A new addition to Cucumber is Capybara support which aims to unify the language used in steps so that you can use Webrat, Selenium and Celerity side-by-side.

Using Cucumber’s Table Transformations, you can easily build complex objects in a way that’s easy to read and understand for clients and developers alike.

My latest favorite feature of Cucumber are Table Transformations. I frequently use tables to build up complex objects and I’ve found that the regular old tables can be a little ugly, especially when your attribute names don’t make much sense on their own. I’ve also noticed that building up associations can be a little wonky, usually requiring more steps than seem necessary.

Conventional Table Usage

Let’s look at an example of how we could use a table, without transformations, to build up some objects for our scenario.

1
2
3
4
5

Scenario: Editing a Spirit
  Given I have a spirit with the following attributes:
    | spirit_type    | country_of_origin | age | brand        | lgcy_prod_sku | name       |
    | Scotch Whisky  | Scotland          | 12  | The Balvenie | SC38181       | DoubleWood |
    | Scotch Whiskey | Scotland          | 12  | The Macallan | SC38245       |            |

1
2
3
4
5

Given /^I have a spirit with the following attributes:$/ do |table|
  table.hashes.each do |attributes|
    Spirit.create!(attributes)
  end
end

Be sure to use create! in your tests to prevent false positives (Thanks Aslak!)

Now this is all fairly simple, and it looks pretty easy to implement, but I see some problems. First, what if you were actually friends with your DBA (bear with me) and you knew better than to have an attribute in your model like country_of_origin or spirit_type. Chances are those are going to be used by many other records and should be pulled out and made into their own Models, Country and SpiritType respectively.¹

So what does our scenario look like with those two new models?

1
2
3
4
5
6
7
8
9
10
11

Scenario: Editing a Spirit
  Given I have a country with the following attributes:
    | name     | continent |
    | Scotland | Europe    |
  And I have a spirit type with the following attributes:
    | name           |
    | Scotch Whiskey |
  And I have a spirit with the following attributes:
    | age | brand        | lgcy_prod_sku | name       |
    | 12  | The Balvenie | SC38181       | DoubleWood |
    | 12  | The Macallan | SC38245       |            |

It’s a little more complex, for sure, but it’s not totally unmanageable. However, the key part that’s missing is how to link the two spirits with their spirit types and countries of origin.

You could add some more steps, but then you’ve got a conjunction step which is inflexible and brittle.

1

  And the spirit named "The Balvenie" is from "Scotland" and is a "Scotch Whiskey"

You could go back to your original step, and try to do some behind-the-scenes stuff to map country_of_origin to the correct country_id, but that gets messy too.

Transform Your Tables

The first step to making good use of table transformations is to make your tables more readable. Start by change the header row of your table to use meaningful representations of the real attribute names.

1
2
3
4

Given I have a spirit with the following attributes:
  | Spirit Type    | Country  | Age | Brand        | Legacy Product Code | Name       |
  | Scotch Whisky  | Scotland | 12  | The Balvenie | SC38181             | DoubleWood |
  | Scotch Whiskey | Scotland | 12  | The Macallan | SC38245             |            |

We’ve turned that weird lgcy_prod_sku attribute into something that your Product Owner can make sense of and we’ve be able to add Spirit Type and Country of Origin back to the table. Now let’s look at the transformation that makes this all work.

1
2
3
4
5
6
7
8
9
10
11
12

Transform /^table:Spirit Type,Country,Age,Brand,Legacy Product Code,Name$/ do |table|
  table.hashes.map do |hash|
    spirit_type = SpiritType.create!({:name => hash["Spirit Type"]})
    country = Country.create!({:name => hash["Country"]})
    spirit = Spirit.create!({:age => hash["Age"],
                            :brand => hash["Brand"],
                            :lgcy_prod_sku => hash["Legacy Product Code"],
                            :name => hash["Name"]})
 
    {:spirit_type => spirit_type, :country => country, :spirit => spirit}
  end
end

The transformation step definition looks a lot like a regular table step definition. There is a regular expression, like anything else in Cucumber, that has the same values as the header row in the table from our scenario. Just like the table step definition we have a table object which is just an array of hashes. We can go through each hash, do the actual transformation, and then return something to our table step definition. We are using map (same as collect) to return an array of hashes, which is just what the table step definition is expecting.

You will also see that we’re creating three different records, which we are returning in the hash we create at the end. Let’s go through those step-by-step:

1. Create a spirit type object from the hash["Spirit Type"] value
2. Create a country object from the hash["Country"] value
3. Create a spirit object from the several related hash values
4. Put all of our created objects into a hash

While we’ve created the objects, we still need to create the associations. I like to leave this for the table step definition rather than the transformation since I think it’s more obvious what’s going on with the values when you’re viewing the table step.

1
2
3
4
5
6
7

Given /^I have a spirit with the following attributes:$/ do |table|
  table.each do |group|
    spirit = group[:sprit]
    associations = {:country => group[:country], :spirit_type => group[:spirit_type]}
    spirit.update_attributes(associations)
  end
end

There are at least a dozen ways to get the spirit associated with the country and spirit type, so don’t feel like you have to follow this pattern every time. Since we’ve sent our table an array of hashes we can iterate over each hash, group, and work with the individual rows. Here’s how:

1. Extract the spirit object from the hash
2. Create another hash with the country and spirit type that Rails can make sense of
3. Use update_attributes to update the spirit object with the new associations

Transformation Tradeoffs

We’ve been able to take our original multi-step scenario and simplify it to a single step. We are using the proper place, the step definitions, to do the associations and we have made our scenario much easier to read for non-developers working on the project. But what did we give up?

The biggest issue I’ve found with using table transformations is that they can be inflexible when you need to add more attributes to your dynamically created object. If you are writing features, using your table to setup objects and then realize that you need to add another attribute, you’re going to have to edit your table transformation step and how you create objects from the hash. When you take this a step further and try to have two different table definitions, you’ll be looking at having two nearly identical table transformations.²

If you’re not already using regular old Cucumber tables to create objects, use this guide to get started. If you are using Cucumber tables to create objects, try to re-factor one of your scenarios and use the table transformation strategy. Once you start using Cucumber tables and table transformations you’ll instantly improve the readability, portability and efficiency of your steps.

¹ Ignore for now the issues with spirit_type and Rails Single Table Inheritance

² I’m guessing that there is some way you can get around this with regular expressions and to have more flexible transformation table steps, but I haven’t tried it yet.

With a little hard work and my secret development ingredient, you can be a better Agile Developer

Recently my fellow developers at Integrum and I took a survey that helped us assess our team with regard to our Agile practices. When taking the survey, and now reviewing it later on, I was struck by how many of the questions were related to a single concept. Many of the problem areas that can be uncovered by the survey, along with examples of one’s successes, come back to this one theme.

Are programmers nearly always confident that the code they’ve written recently does what it’s intended to do?

• Is there more than one bug per month in the business logic of completed stories?
• Can any programmer on the team currently build and test the software, and get unambiguous success / fail result, using a single command?
• When a programmer gets the latest code, is he nearly always confident that it will build successfully and pass all tests?
• Are fewer than five bugs per month discovered in the teamʼs finished work?
• After a line item is marked “complete” do team members later perform unexpected additional work, such as bug fixes or release polish, to finish it?
• Are programmers nearly always confident that the code they’ve written recently does what it’s intended to do?
• Are all programmers comfortable making changes to the code?
• Do programmers have more than one debug session per week that exceeds 10 minutes?
• Do unexpected design changes require difficult or costly changes to existing code?
• Do any programmers optimize code without conducting performance tests first?
• Is there more than one bug per month in the business logic of completed stories?
• Are any team members unsure about the quality of the software the team is producing?

What’s the common theme among these stories, and the secret to better agile development? Testing, testing and more testing.

The negative outcomes implied by some of these questions can be solved by testing. Spending time fixing “completed” stories? Probably something you could have tested. Conversely, the positive benefits implied by other questions can be had via testing. Want to make your code more inviting and easier to deal with for new team members or people unfamiliar with the project? Give them robust and well-written tests.

Today I gave a presentation at Desert Code Camp about BDD, Cucumber, Webrat and User Stories. If you’d like to find review the slides or download the code I used during the presentation here they are.

Slides and Code

Desert Code Camp BDD Presentation Slides

http://github.com/clayton/desert-code-camp

If you’re using Cucumber and you’re not using Transformations you’re doing it wrong. I just started using these recently and ran into a problem with creating records using factory_girl factories that made use of sequences. While trying to create multiple Authlogic user records with a unique email and unique single_access_token using a Cucumber table, the functionality of Factory.next(:email) wasn’t working correctly, I would always get the same e-mail address. Turns out it was an easy fix, just had to use lazy attributes in my factory.

The Scenario, Step Definition and Factory

1
2
3
4
5
6
7
8
9
10
11
12

  Scenario: Presenter List
    Given the following presenters:
      | Name    | Bio                 | Website              |
      | Clayton | Rails dev @integrum | http://claytonlz.com |
      | Chris   | Scrum @integrum     |                      |
    And I am on the homepage
    When I follow "Presenters"
    Then I should see "Clayton"
    And I should see "Rails dev @integrum"
    And I should see "http://claytonlz.com"
    Then I should see "Chris"
    And I should see "Scrum @integrum"

My Step Definition
This uses the Transformation table below:

1
2
3
4
5

Given /^the following presenters:$/ do |table|
  table.each do |attrs|
    Factory.create(:user, attrs)
  end
end

This transformation takes a table like the one in my scenario above, and assigns the values to a hash using the actual model attribute names (Name isn’t an attribute on a user but name is). The regular cucumber step definition “consumes” this hash for each entry in the table and passes it to a Factory for creation.

1
2
3
4
5

Transform /^table:Name,Bio,Website$/ do |table|
  table.hashes.map do |hash|
    {:name => hash[:Name], :bio => hash[:Bio], :website => hash[:Website]}
  end
end

My Factory

This is a pretty basic factory for an authlogic user model, I’m using factory_girl sequences to give me a “unique” e-mail and single access token, which are required by Authlogic.

1
2
3
4
5
6
7
8
9

Factory.define :user do |user|
  user.email Factory.next(:email)
  user.name ""
  user.bio ""
  user.website ""
  user.password "password"
  user.password_confirmation "password"
  user.single_access_token Factory.next(:single_access_token)
end

The problem

The above scenario will fail when it tries to create the user records via the factories. You’ll see a validation error about how the user model requires a unique e-mail and single access token. You’ll be wondering, “hey why are my sequences working?”. When you inspect the log you’ll see that they are in fact NOT working.

The Quick Answer

The easy answer to this is that you need to use lazy attributes in your factory for the sequences so that they are loaded each time instead of once.

1
2
3
4
5
6
7
8
9

Factory.define :user do |user|
  user.email { Factory.next(:email) }
  user.name ""
  user.bio ""
  user.website ""
  user.password "password"
  user.password_confirmation "password"
  user.single_access_token { Factory.next(:single_access_token) }
end

Notice the curly braces around the sequences

The Longer Answer

The cucumber rdoc explains the Transform functionality, albeit somewhat hard to understand.

Registers a proc that will be called with a step definition argument if it matches the pattern passed as the first argument to Transform. Alternatively, if the pattern contains captures then they will be yielded as arguments to the provided proc. The return value of the proc is consequently yielded to the step definition.

I think the issue comes from something with the way these Procs are created, called and also their scope with regard to the step definition etc. I don’t think my ruby-fu is strong enough to give a good explanation but maybe I’m going in the right direction.

RSpec, Cucumber, Webrat, RCov and Autotest are a powerful combination of tools for testing your Rails app. Unfortunately getting them to all work nicely together can be a bit of challenge. I recently configured a development environment from scratch on OS X 10.5 Leopard and kept track of all of the little details.

Prerequisites

I’m assuming you’ve got the following installed:

• ruby
• ruby gems 1.3.1
• Apple development tools
• git
• rails >= 2.3.2
• You’ve added github to your gem sources (gem sources -a http://gems.github.com)
  
  

RSpec & RSpec-Rails

First let’s grab the rspec¹ and rspec-rails² gems.

sudo gem install rspec

sudo gem install rspec-rails

Cucumber

Next we’ll install the cucumber³ gem

sudo gem install cucumber

Webrat

Webrat⁴ is used by cucumber to simulate a browser for your integration tests. Webrat will also install nokogiri⁵.

sudo gem install webrat

RCov

I thought RCov⁶ would get installed with RSpec, but it wasn’t for me. You might not need to do this, but just to make sure…

sudo gem install rcov

Autotest

Autotest⁷ comes from ZenTest⁸ and allows you to have a kick ass workflow where you are constantly running relevant tests and less-constantly automatically running your entire test suite.

sudo gem install ZenTest

Optionally, Thoughtbot’s Factory Girl

Factory girl⁹ is a really helpful fixture replacement (and more) gem to use in conjunction with cucumber, checkout their much better explanation

sudo gem install thoughtbot-factory_girl --source http://gems.github.com

Optionally, Carlos Brando’s Autotest Notification

While autotest normally runs in a terminal window, it can be setup to hook into applications like growl or snarl. The Autotest Notification⁹ gem helps make this setup a lot easier.

You will need growl installed and configured for this step the installation instructions on this gems github page are very easy to follow.

sudo gem install carlosbrando-autotest-notification --source=http://gems.github.com

Next you need to turn autotest notifications “on”

an-install

A Sample Rails App

Let’s create a sample rails app for the rest of this guide.

rails sample-app

Configuring Environment Variables

Autotest relies on some environment variables to run all of your features and specs correctly. If autotest “hangs” after you try to run it, or it just never seems to be watching your specs or features, this will most likely solve your problem.

Open the test.rb environment definition file in sample-app/config/environments/test.rb and add the following.

1
2

ENV['AUTOFEATURE'] = "true"
ENV['RSPEC'] = "true"

These lines will test autotest to run, and look for changes to, your specs (rather than test unit tests) and your cucumber features.

Update

If you don’t want to add these environment variables to every rails project you’ve got on your machine, you can also choose to set them as environment variables in your .bash_profile or .bashrc (or whatever shell you’re using) files.

export AUTOFEATURE=true
export RSPEC=true

Unpacking Gems

Next let’s freeze (unpack) some gems that we’ll be using in our app. I’ve run into problems trying to use the system gems with cucumber, rspec and webrat, especially when I have multiple versions of any of them installed. Unpacking them into my rails app solves this problem for me.

mkdir sample-app/vendor/gems
cd sample-app/vendor/gems
gem unpack rails
gem unpack rspec
gem unpack rspec-rails
gem unpack cucumber

Because webrat (and nokogiri) are native gems, that is, they are built locally on your machine based on its architecture, we won’t unpack those.

config.gem support
The current accepted practice, when using rails 2.3, and as suggested by the rspec guy(s) is to use rails’ config.gem functionality.

Open sample-app/config/environments/test.rb and add the following lines:

config.gem "rspec", :lib => false, :version => ">= 1.2.0" 
config.gem "rspec-rails", :lib => false, :version => ">= 1.2.0" 
config.gem "cucumber", :lib => false, :version => ">= 0.2.3"
config.gem "thoughtbot-factory_girl", :lib    => "factory_girl", :source => "http://gems.github.com"
config.gem "webrat", :lib => false, :version => ">= 0.4.3"
config.gem "nokogiri", :lib => false, :version => ">= 1.2.3"

Your version numbers may be different, but these are all current at the time of writing.

Boot Strapping RSpec and Cucumber

Before you can get very far with rspec or cucumber you need to run the bootstrapping scripts to give yourself the default files and directories.

# From inside your rails app sample-app/
script/generate rspec
script/generate cucumber

Factories
Depending on where you’re going to use your factories the most, you might want to save your file in either spec/ or features/. I chose the latter. Only complete this step if you plan to use the FactoryGirl gem.

touch sample-app/features/factories.rb

Getting Accurate RCov Data

By default RCov is setup to only use your specs when calculating code coverage. If you’re using Cucumber and RSpec, you’ll obviously want to include both types of tests to calculate your project’s true code coverage.

I picked up this rcov rake task from my co-worker Jay McGavren it does all of the heavy lifting for you, we’ll just need to make a couple of changes.

Drop this file into sample-app/lib/tasks/rcov.rake and use it by calling rake rcov:all from your terminal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

require 'cucumber/rake/task' #I have to add this
require 'spec/rake/spectask'
 
namespace :rcov do
  Cucumber::Rake::Task.new(:cucumber) do |t|    
    t.rcov = true
    t.rcov_opts = %w{--rails --exclude osx\/objc,gems\/,spec\/,features\/ --aggregate coverage.data}
    t.rcov_opts << %[-o "coverage"]
  end
 
  Spec::Rake::SpecTask.new(:rspec) do |t|
    t.spec_opts = ['--options', "\"#{RAILS_ROOT}/spec/spec.opts\""]
    t.spec_files = FileList['spec/**/*_spec.rb']
    t.rcov = true
    t.rcov_opts = lambda do
      IO.readlines("#{RAILS_ROOT}/spec/rcov.opts").map {|l| l.chomp.split " "}.flatten
    end
  end
 
  desc "Run both specs and features to generate aggregated coverage"
  task :all do |t|
    rm "coverage.data" if File.exist?("coverage.data")
    Rake::Task["rcov:cucumber"].invoke
    Rake::Task["rcov:rspec"].invoke
  end
end

The important part here is on line 7, we want rcov to exclude our features directory. We obviously don’t need or want rcov telling us that our feature files are not “covered”. To solve this problem we’ve simply excluded the features directory from rcov’s processing.

We also need to slightly modify sample-app/spec/rcov.opts to get the full rspec + cucumber coverage data.

Your rcov.opts should look like this:

--exclude "spec/*,gems/*,features/*" 
--rails
--aggregate "coverage.data"

We again want to ignore our cucumber features and we also want to tell rcov to aggregate data in a file called coverage.data. This is used in the above rake task.

Write Some Specs and Features!

Act like you know what you’re doing and write some models, controllers whatever. Add some specs and features too.

Autotest Workflow

Open a terminal and make your way to your sample rails app and fire up autotest. You might see something like the following, depending on how many specs and features you’ve got.

$> autotest
loading autotest/cucumber_rails_rspec
opts 
...
 
Finished in 0.06276 seconds
 
3 examples, 0 failures
================================================================================
 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby /Library/Ruby/Gems/1.8/gems/cucumber-0.2.3/bin/cucumber --format progress --format rerun --out /var/folders/Aq/Aqp06i3dFnqse+tQgQA+1++++TI/-Tmp-/autotest-cucumber.75956.0 features
.................
 
4 scenarios
17 passed steps
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby /Library/Ruby/Gems/1.8/gems/rspec-1.2.2/bin/spec --autospec spec/models/intern_spec.rb -O spec/spec.opts 
...
 
Finished in 0.062995 seconds
 
3 examples, 0 failures
================================================================================
 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby /Library/Ruby/Gems/1.8/gems/cucumber-0.2.3/bin/cucumber --format progress --format rerun --out /var/folders/Aq/Aqp06i3dFnqse+tQgQA+1++++TI/-Tmp-/autotest-cucumber.75956.1 features
.................
 
4 scenarios
17 passed steps

The REALLY important stuff

1. make sure you’ve got “ENV['AUTOFEATURE'] = true” in your test.rb otherwise autotest won’t run your features automatically
2. make sure you’ve got “ENV['RSPEC'] = true” in your bash profile or else autotest won’t run your specs automatically
3. make sure you’ve got “–aggregate = ‘coverage.data’” in your spec/rcov.opts file if you’re going to use the above rake task and hope to get combined rcov coverage data between rspec and cucumber
4. make sure you’re excluding the features directory from rcov where required or else you’ll end up with misleading rcov data.

Gem Versions

Here’s a list of the current gems and their versions that I used in preparing this guide.

*** LOCAL GEMS ***
 
actionmailer (2.3.2, 1.3.6, 1.3.3)
actionpack (2.3.2, 1.13.6, 1.13.3)
actionwebservice (1.2.6, 1.2.3)
activerecord (2.3.2, 1.15.6, 1.15.3)
activeresource (2.3.2)
activesupport (2.3.2, 1.4.4, 1.4.2)
acts_as_ferret (0.4.1)
addressable (2.0.2)
builder (2.1.2)
capistrano (2.0.0)
carlosbrando-autotest-notification (1.9.1)
cgi_multipart_eof_fix (2.5.0, 2.2)
cucumber (0.2.3)
daemons (1.0.9, 1.0.7)
data_objects (0.9.11)
diff-lcs (1.1.2)
dnssd (0.6.0)
extlib (0.9.11)
fastthread (1.0.1, 1.0)
fcgi (0.8.7)
ferret (0.11.4)
gem_plugin (0.2.3, 0.2.2)
highline (1.2.9)
hpricot (0.6)
libxml-ruby (0.3.8.4)
mongrel (1.1.4, 1.0.1)
mysql (2.7)
needle (1.3.0)
net-sftp (1.1.0)
net-ssh (1.1.2)
nokogiri (1.2.3)
polyglot (0.2.5)
rack (0.9.1)
rails (2.3.2, 1.2.6, 1.2.3)
rake (0.8.4, 0.7.3)
rcov (0.8.1.2.0)
RedCloth (3.0.4)
rspec (1.2.2)
rspec-rails (1.2.2)
ruby-openid (1.1.4)
ruby-yadis (0.3.4)
rubynode (0.1.3)
sources (0.0.1)
sqlite3-ruby (1.2.1)
term-ansicolor (1.0.3)
termios (0.9.4)
textmate (0.9.2)
thor (0.9.9)
thoughtbot-factory_girl (1.2.0)
treetop (1.2.5)
webrat (0.4.3)
ZenTest (4.0.0)

El Fin

Hopefully this guide was useful or had that one little step that you needed to get everything working. I’m sure this will all be out of date in the coming weeks, but I’ll try to keep it as up-to-date as possible. If you see any errors, or can better explain some of the missing pieces, please post a comment. Thanks!

¹ http://github.com/dchelimsky/rspec/tree/master

² http://github.com/dchelimsky/rspec-rails/tree/master

³ http://github.com/aslakhellesoy/cucumber/tree/master

⁴ http://wiki.github.com/brynary/webrat

⁵ http://github.com/tenderlove/nokogiri/tree/master

⁶ http://rubyforge.org/projects/rcov/

⁷ http://www.zenspider.com/ZSS/Products/ZenTest/#rsn

⁸ http://www.zenspider.com/ZSS/Products/ZenTest/

⁹ http://github.com/thoughtbot/factory_girl/tree/master

¹⁰ http://github.com/carlosbrando/autotest-notification/tree/master

Updates
2009-12-08 – Removed “sudo” when describing how to unpack gems (h/t xdotcommer)

Shared example groups are a great feature of Rspec that help you simplify your tests and keep your code DRY. You setup shared example groups almost exactly like you would a regular set of specs, but these similarities can be slightly misleading.

Below we have an example model, spec and shared example group. Our Dog model has its own set of functionality, but as a mammal it should still have some aspects of being a mammal. We’ve got some specs in a shared example group that we use for testing all of our mammal models to make sure things don’t get too out of whack in the universe.

Our Example Model

1
2
3
4
5
6
7
8
9
10
11
12
13

class Dog
 
  attr_accessor :name, :mammal
 
  def initialize
    self.mammal = true
  end
 
  def greet
    "Hi, I'm #{self.name}, woof woof!"
  end
 
end

Our Example Spec

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

describe Dog do
 
  before(:each) do
    @animal = Dog.new
    @animal.name = "Bruno"
  end
 
  it_should_behave_like "a mammal"
 
  describe "Greet" do
    it "should respond with its name and a greeting" do
      @animal.greet.should == "Hi, I'm Bruno, woof woof!"
    end
  end
 
end

Our Shared Spec

1
2
3
4
5

describe "a mammal", :shared => true do
  it "should really be a mammal" do
    @animal.mammal.should be_true
  end
end

A Typical before(:each)

Typically, when you’ve got a describe block, you might use before(:each) to setup some scenario that is used for each spec in that describe block, pretty normal RSpec stuff. We’re using it above in our example spec to create a new Dog object and set that dog’s name.

Using before(:each) in a shared spec

What if you wanted to use a before(:each) in your shared spec? Expanding on our example above we can do something like this.

1
2
3
4
5
6
7
8

describe "a mammal", :shared => true do
  before(:each) do
    @animal.stub!(:has_body_hair?).and_return(true)
  end
  it "should really be a mammal" do
    @animal.mammal.should be_true
  end
end

Based on typical RSpec behavior, one would think that the stubbing of the has_by_hair? method on the instance of an animal, would only apply to the specs inside of the describe block of the shared example group. However, by specifying in the Dog spec that a dog “should behave like” a mammal, and thus using the shared spec, that stub will apply to all subsequent “it should” blocks in your model spec.

What if, for example, we had the following in our Dog spec.

  describe "Mutate into Lizard Dog" do
   # dog.mutate will remove body hair and make the dog cold blooded
    it "should mutate into a new species" do
      @animal.mutate
      @animal.has_body_hair?.should be_false
    end
  end

If we include this in our Dog spec, below the inclusion of the shared example spec, our test will fail. We’ve already stubbed out the has_body_hair? method as part of our shared example group, when we call it down here in this completely separate describe block, RSpec is just using the stub we setup previously.

It might be a design problem if…

Now while I’m considering this a gotcha, it may be that this is expected behavior, I couldn’t find anything specifically when researching this “bug” originally. It is also possible that stubbing behavior in shared example groups is frowned upon, and I’m just “doing it wrong”.

Ultimately, I tried using patterns that made sense to me and seemed to be in line with how RSpec works in general. A stubbed method inside the before(:each) of a describe block is usually only applicable to the specs and nested describes contained within. When I realized that this is not the case with shared example groups, it seemed like a gotcha.