Jenkins CI Build Monitor

Jenkins CI is a popular open source continuous integration server. I encourage teams to use continuous integration, however many teams setup the server, configure the automated e-mail blasts and think that’s all there is to it. Many teams don’t make the results of their builds part of an informative workspace. The team I’m currently working with fell into this trap and started having issues with failing builds going unnoticed. I looked around for some plugins and projects to help with this goal, but nothing quite met our needs.

Introducing Jenkins Radiator

I built Jenkins Radiator with a few things in mind:
  • We only care about passing jobs when all the jobs are passing
  • If a job is failing, it should be very obvious
  • If a job was failing and is being built again, we want to know
  • You shouldn’t be able to hide a job or otherwise ignore it, if it’s not important, get it off CI
  • A quick glance at the radiator from across the room should let you know if the build is green or not
Start Using Jenkins Radiator

Screenshots

Jenkins Radiator Failing Jobs

Jenkins Radiator Passing Jobs

Why Developers Avoid Testing

While the drive towards a culture of “always be testing” has existed for some time, the recent detour into software craftsmanship has helped to create a new style of getting the word out. What used to be, “Test driven development is a great way to approach solving a problem” has turned into “If you’re not testing you’re a shitty developer.” The testing evangelism going on in the community today is misguided and would convert more people if it addressed the real reason people aren’t testing.

It’s Like Losing Weight

To the ever testing software craftsman, it can be puzzling why people don’t write tests. The assumption much of the time is that developers are lazy, that they don’t care about the quality of their code or even that they think testing is an abject waste of time. In my experience this is almost never true.

People skip testing because, ultimately, it’s hard work. Think about what it takes to lose weight or to quit smoking:

  • Break familiar habits
  • Disrupt your personal relationships
  • Acknowledge that you’re deficient
  • Find help
  • Fail often before you succeed

As a developer working in an organization that does not maintain a culture of “always be testing”, it’s going to be difficult to:

  • Change the way you think about solving problems
  • Upset peers who are comfortable in the status quo
  • Admit that you don’t how how, what, when and why to test
  • Find a mentor willing to help you improve
  • Not give up every time you get stuck

End the FUD!

Think about what it means to start testing when you never have before. Not only do you have to come to terms with not knowing what you’re doing, you eventually realize that you’ve been “doing it wrong” until now. This is fearful self-doubt territory for a lot of people.

If you’re serious about getting more of your industry peers to start testing, stop:

  • Going on about craftsmanship; people don’t strive for mediocrity.
  • Playing the doomsday “If you don’t test…” game.
  • Talking about how testing makes you so much faster; their personal experience points to the contrary.
  • Overlooking their perception of testing.
  • Forgetting when you were in their shoes.

Instead, ask questions, get to the root of the problem, don’t settle for “testing makes me slow” or “that’s not my job”, dig deeper.

I believe that as an industry we should strive for increasing code quality, robust test suites and better problem solving. As the minority who are practicing TDD/BDD/test-first/TATFT etc., we need to change our tune. Let’s stop preaching the benefits and start addressing the fear, uncertainty and doubt that keeps smart, capable and dedicated developers from enjoying these benefits.

Let Conversations Write Your Tests

I consider myself an early adopter of Cucumber and have spent a lot of time using, thinking about, writing about and discussing cucumber and, at a higher level, behavioral driven development. Lately however, I’ve really been rethinking how I use Cucumber and why. Am I getting enough benefit from the overhead and abstraction of plain text features? Could I do the same thing with RSpec if I just approached RSpec differently? Am I cuking it wrong?

This shift in thinking is due in part to Liz Keogh’s Step Away from the Tools and Dan North’s Who’s domain is it anyway?. Both of these got me thinking about how I’m using the term BDD (rather loosely) and how much of an investment I’m making in a specific tool (Cucumber).

More Meaningful Planning

Having meaningful planning meetings with their customer/product owner is one thing with which many teams struggle. Too often, we go too fast, don’t uncover enough detail, use the wrong language, don’t understand “done” and leave too many loose ends.

To combat this we draw screens, discuss workflows, ask leading questions and a variety of other techniques. I felt that while I was doing these things, I was still frustrated with the other part of the planning process. I’ve never liked traditional tasking and I’ve never like the idea of having to translate all of the data gathered with the customer into some other form only to express it later in a test in yet another form.

What if I wrote my tests during the planning meeting?

I decided that changing the way I gathered conditions of satisfaction, defined “done” and discussed workflows with my product owner would give me the biggest boost in value delivered, so one day I did just that.

For the following examples, assume that we’re adding a feature to an e-commerce website.

As a user I should be able to manage addresses so that I don’t have to enter my information more than once

As I’m discussing the feature with my product owner, I will discuss with them the possible scenarios, including workflows, required to use the feature. Some scenarios for this feature might include:

  • When I am on my account page
  • When I am creating a new address
  • When I am deleting an address

These scenarios might have scenarios of their own:

  • When I am editing an address
    • When I successfully edit my address
    • When editing my address fails

So far I’ve been able to ask the product owner something like “So when I’m editing my address, and I miss some required fields, what happens? What do I see? Where do I go? What are the required fields?”. I can also draw pictures to explain the workflows and ask more questions “What’s on this page? Where is the error message displayed? Do I see error messages for the fields that are missing?”.

For each scenario, I can capture assertions that come from answers to the questions I’m asking:

  • When I am on my account page
    • I should see a link to “My Addresses”
  • When I click on the “My Addresses” link from my account page
    • I should be on my addresses page
    • I should see “My Addresses” in the page heading
  • When I am on my addresses page
    • When I have existing addresses
      • I should see each address listed in format xyz
      • I should see an “edit” link for each address listed
    • When I don’t have existing addresses
      • I should see help text explaining how to add an address

If you’re an RSpec user, you might be thinking, “Hey this looks like RSpec!” and it does. During the planning meeting I can capture these scenarios and outcomes and then use them nearly verbatim for my RSpec acceptance tests. Even better, when I run my tests, I can use the “documentation” format that RSpec gives you to output that’s almost identical to the scenarios above.

The conversation required to really define these scenarios and outcomes is challenging, but at the same time, very rewarding. I have also found that it’s pretty powerful to be able to sit with the product owner and view two nearly identical documents side-by-side knowing that one is automated test output.

Cucumber, Culerity and Bundler Errors

I had recently switched a rails 2.3.10 project, that makes heavy use of cucumber and culerity for javascript testing, to bundler for dependency management and immediately started receiving an error like this when I tried to run a javascript test which invoked jruby.

  @javascript
  Scenario: Ability to un-attend an event               # features/events.feature:22
JRuby limited openssl loaded. http://jruby.org/openssl
gem install jruby-openssl for full support.
    Given I am logged in as the admin                   # features/step_definitions/admin_steps.rb:12
      (eval):1:in `process_result': compile error
      (eval):1: Invalid char `33' in expression
      (eval):1: syntax error, unexpected tIDENTIFIER, expecting ']'
      Could not find rake-0.8.7 in any of the sources
                ^ (SyntaxError)

I searched and searched and couldn’t find a solution. I finally ended up using selenium to avoid this and that has been working okay for a while. Then today I decided to start looking again, figuring that something might have happened in the last few months, and I was lucky enough to stumble across this gist. The author was having a similar problem to mine, except that he was using rspec and steak. His solution was to add some code to his spec_helper.rb file. I added mine to my features/support/custom.rb file so that it would be loaded by cucumber. The fix is:

# Add this to features/support/custom.rb
if ENV['RUBYOPT']
  ENV['RUBYOPT'] = ENV['RUBYOPT'].gsub(%r{-rs*bundler/setup}, '')
end

I don’t know exactly what is happening, the only comment in the gist near the fix is suppress '-rbundler/setup' from RUBYOPT, but what’s important is that now my cucumber tests pass using culerity just like they did before.

Zero To Tested With Cucumber and Factory Girl

Get your project from zero to tested leveraging the power of Cucumber and Factory Girl

Testing can be a daunting task, especially for developers who are new to Rails. It sounds like a great idea, but where do you start? What do you test? What don’t you test? For many developers, the hardest part of testing is jumping in and getting your feet wet. Luckily, Cucumber makes getting started with testing easy, you’ll be testing your code form top to bottom in no time.

What is Cucumber?

Cucumber, in simple terms, is a framework for writing human-readable, meaningful tests that allow you to test the full functionality of your Rails project from database access to business logic to what’s displayed in your views.

From the Cucumber Wiki:

Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain-specific language and serves as documentation, automated tests and development-aid – all rolled into one format.

When you write your tests with Cucumber you get full-stack automated tests, documentation and a tool to help you communicate with your product owner.

Who’s the Factory Girl?

Factory Girl is a fixture replacement for Rails that allows you to create meaningful objects for use in your tests.

From the Factory Girl Github Page:

factory_girl is a fixtures replacement with a straightforward definition syntax, support for multiple build strategies (saved instances, unsaved instances, attribute hashes, and stubbed objects), and support for multiple factories for the same class (user, admin_user, and so on), including factory inheritance.

When you use Factory Girl in conjunction with Cucumber you can simplify complex test setup and use real objects from the database, not brittle mocks.

Getting Setup

Getting your project setup with Cucumber and Factory Girl is pretty straightforward. First, you’ll need a few gems installed:


    sudo gem install cucumber-rails
    sudo gem install webrat
    sudo gem install thoughtbot-factory_girl --source=http://gemcutter.org

Note: As of 0.5.0 the Rails specific functionality was extracted from the cucumber gem into the cucumber-rails gem

Cucumber comes pre-packaged with some generators for getting your project ready for testing. From the root of your Rails project, run the generator:


    ruby script/generate cucumber --webrat --rspec

You’ll notice that we’re passing a couple of flags to the generator, --webrat specifies that we want to use webrat as our automated browser and --rspec specifies that we want to use rspec as our test framework. If you don’t pass these options Cucumber will guess which options you want based on the gems which you have installed.

After running the generator, you will see some output about the files that were created:


    create  config/cucumber.yml
    create  config/environments/cucumber.rb
    create  script/cucumber
    create  features/step_definitions
    create  features/step_definitions/web_steps.rb
    create  features/support
    create  features/support/env.rb
    create  features/support/paths.rb
    create  lib/tasks
    create  lib/tasks/cucumber.rake

Here’s a brief description of what some of the important files do:

*config/cucumber.rb*: This is your cucumber environment, similar to production.rb or development.rb. Typically you’ll add cucumber specific config.gem directives and similar “cucumber only” items in here.

*features/step_definitions/web_steps.rb*: These are the webrat steps that you get for free with Cucumber and Webrat.

*features/support/env.rb*: This file has Cucumber specific setup and configuration options. This changes often, so it’s recommended to not alter this file directly, but to create your own custom env.rb (e.g. custom_env.rb).

*features/support/paths.rb*: Cucumber needs to be aware of the custom routes in your application that you plan on using when testing, this is where they are specified.

Cucumber from 10,000ft

Now that you’ve got your project configured to use Cucumber, let’s go over a high level view of the general concept of testing with Cucumber.

Cucumber uses “Features” to describe the functionality of your application. Features, stored in the features directory, are written in a simple human-readable format called Gherkin. The “Steps” in your feature files that describe the functionality of your application are backed up by step files in the features/step_definitions directory. The step files act like translations between the human-readable feature files and the underlying Ruby code of your Rails application.

Typically you will be using Cucumber in conjunction with Webrat to test your application. Webrat acts as the person behind the keyboard clicking around the site, filling out forms and viewing resulting pages. If you’ve ever tested your project by manually filling out forms and verifying output, that’s exactly what Webrat does for you, except it’s automated.

Let’s Get Testing Already

Rather than go into detail about TDD or BDD and why you should use them, I’m going to assume that you’re like most Rails developers looking to learn more about testing. You probably have an application or two out there in the wild, seemingly working just fine. However, you know that you should get some test coverage in there so that you can confidently make changes and debug problems as they come up. With that in mind, I am going to use a sample application that already has some functionality and we will add our Cucumber tests where needed.

A Veterinary Patient Management System

Our sample application is a simple patient management system for a Veterinarian that deals with Owners, Pets and Visitations. Let’s take a look at some of the models.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
 
    class Owner < ActiveRecord::Base
 
      has_many :pets
      validates_presence_of :name
      validates_presence_of :email
 
    end
 
    class Pet < ActiveRecord::Base
      belongs_to :owner
 
      validates_presence_of  :name
      validates_presence_of  :species
      validates_inclusion_of :species, :in => %w( dog cat bird snake ),
      :message => "Species: %s is not included in the list of accepted species"
 
    end
 
 
    class Visitation < ActiveRecord::Base
 
      has_one :pet
 
    end

The first thing for which we’ll be adding a Feature is the process for creating a new Owner record. When a new client calls the Veterinarian’s office, an employee needs to enter them into the system.

Writing a Feature

Cucumber features are very straight forward. The feature file explains a set of functionality by describing different cases through scenarios.

1
2
3
4
5
6
7
8
9
10
 
    Feature: Manage Owners
      In order to value
      As a role
      I want feature
 
      Scenario: title
        Given context
        When event
        Then outcome

Each Cucumber feature represents a desired software feature that a certain user of the system wants in order to achieve some end goal. The scenarios are formatted so that a context is setup, an event occurs and an outcome is evaluated.

Let’s examine our real Manage Owners feature.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 
    Feature: Managing Owners
      In order to manage our client list
      As an employee
      I want to be able to CRUD owners
 
      Scenario: Creating a new Owner
        Given I am on the homepage
        And I follow "Owners"
        Then I should be on the owners index page
        Given I follow "New owner"
        And I fill in "Name" with "Clayton"
        And I fill in "E-Mail" with "clayton@example.org"
        And I fill in "Address" with "100 Cactus Rd"
        And I fill in "City" with "Scottsdale"
        And I fill in "State" with "Arizona"
        And I fill in "Postal Code" with "85000"
        And I fill in "Phone" with "480-555-1212"
        When I press "Create"
        Then I should see "Owner was successfully created."
        And I should be on the owners index page

Note: When you specify “And” in a Cucumber feature the step inherits the Given/Then/When from the previous step

Our scenario covers the act of creating a new Owner record. We first setup a context (I am on the homepage). Next we click a link in the navigation (I follow “Owners”). Once we’re on the index view for the Owners controller (I should be on) we can click another link to get to the new view for the Owners controller. From there we are simply instructing Webrat to fill out our form (I fill in) and click the submit button at the bottom (I press “Create”). Finally we are asking Webrat to read the resulting page and see if there is some text present (I should see).

Our feature is looking pretty good, but we still have a few more steps before we can get this scenario passing. Running the this feature from the command line will give us some output and more direction. Once you’re run rake db:migrate and rake db:test:prepare run the following from your application’s root directory:


  ruby cucumber features/manage_owners.feature

Looks like we’re failing, and Cucumber gave us some information about what’s wrong.


  Can't find mapping from "the owners index page" to a path.
  Now, go and add a mapping in patient-management/features/support/paths.rb

If we open up our features/support/paths.rb file we can add the correct path right below the default path for the “home page”. The paths file is really just a list of regular expressions that Cucumber uses to match named routes to words in scenarios.

1
2
3
4
5
 
  when /the homes?page/
    '/'
  when /the owners index page/
    owners_path

With that little change we have our first passing Cucumber scenario! When running Cucumber features from the command line, Cucumber will print out a little summary message.


    1 scenario (1 passed)
    14 steps (14 passed)
    0m0.290s

Complex Scenario Setup

In our above example the setup, or Given steps, were pretty simple. However, sometimes you need more complex setups for your scenarios, like editing a record for example. Cucumber makes it easy to create objects in your scenarios using tables in multiline step arguments.

Here is a feature that goes through the process of editing an owner’s address. You’ll see the Cucumber table that is used to setup our existing owner record. The first row in the Cucumber table correspond to some of the attribute names for our Owner model.

1
2
3
4
5
6
7
8
9
10
11
12
    Scenario: Editing an existing Owner
      Given the following owners:
       | name    | email               | address       |
       | Clayton | clayton@example.org | 100 Cactus Rd |
      Given I am on the homepage
      And I follow "Owners"
      Then I should be on the owners index page
      When I follow "Edit"
      And I fill in "Address" with "567 N Scottsdale Rd"
      When I press "Update"
      Then I should see "Owner was successfully updated."
      And I should be on the owners index page

When we try to run this, Cucumber will give us the “shell” for our custom step definition “Given the following owners”.

    Given /^the following owners:$/ do |table|
      # table is a Cucumber::Ast::Table
      pending # express the regexp above with the code you wish you had
    end

Since we never created a step definition for this Cucumber feature, we can go ahead and add a new file, features/step_definitions/manage_owners_steps.rb. It is important to note that Cucumber will load all of the files in features/step_definitions and that steps can be used across scenarios and features. If you think you’re going to re-use a step definition, it might be a good idea to place it in something like features/step_definitions/shared_steps.rb.

Note: You should put some thought into your step organization as your project grows in size.

Cucumber takes the table argument from your scenario and turns it into an array of hashes, which is technically a Cucumber::Ast::Table object. By iterating through each hash of the array, we can build up an object that we can use later in our tests. Since we want this object to live in the database and we don’t want to describe every attribute of the owner in our scenario, we can use Factory Girl to simplify the process.

The factory for our owner is simple. I won’t go into the specifics of how to create factories as that’s a whole other article. You can read more about using factories on the github project page

Create a file to store your factories in features/support/factories.rb and copy the following.

1
2
3
4
5
6
7
8
9
10
    require 'factory_girl'
 
    Factory.define(:owner) do |o|
      o.name "John Doe"
      o.email "john@example.org"
      o.address "123 Elm Street"
      o.city "Phoenix"
      o.state "Arizona"
      o.zip "85000"
    end

When we use the factory to create our object, Factory Girl will take the attribute hash that we’ve passed in when creating the object and use those values for the object. Factory Girl will also fill in any blank attributes with the defaults we’ve specified in the factory definition. The record that is created will actually exist in the database, it is not a mock and it does not have any stubs, as far as our Rails app is concerned it is just a regular record in the database.

1
2
3
4
5
6
7
    Given /^the following owners:$/ do |table|
      table.hashes.each do |attributes|
        # {:name => "Clayton", :email => "clayton@example.org", 
        :address => "100 Cactus Rd"}
        Factory.create(:owner, attributes)
      end
    end

Another place we could use our Given the following owners step would be when creating a feature that deals with creating Pet records. An owner has_many pets while a pet belongs_to an owner. When we create a pet record, we need an existing owner record to which we’ll link the pet record. Start by creating a new feature file, features/manage_pets.feature.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
    Scenario: Creating a pet
      Given the following owners:
       | name    | email               |
       | Clayton | clayton@example.org |
      Given I am on the homepage
      And I follow "Pets"
      And I follow "New pet"
      When I fill in "Name" with "Bruno"
      And I select "Dog" from "Species"
      And I select "March 19th, 2005" as the "DOB" date
      And I select "Clayton (clayton@example.org)" from "Owner"
      When I press "Create"
      Then I should see "Bruno was successfully created."
      And I should be on the pets index page
      And a pet named "Bruno" should be owned by an owner named "Clayton"

In this example we have re-used our Given the following owners table. We have also made use of some new Webrat steps like I select for interacting with select lists and I select [DATE] as the [DATE LABEL] date for selecting a date from Rails’ date_selector helper. Finally, there is a custom step that we are going to use to make sure that the connection between our newly created pet and the existing owner is in place.

Let’s take a look at the step definition for our custom step above.

1
2
3
4
 
    Then /^a pet named "([^"]*)" should be owned by an owner named "([^"]*)"$/ do |pet_name, owner_name|
      Pet.find_by_name(pet_name).owner.name.should == owner_name
    end

When you place something in double quotes in your scenario steps, like “Bruno” and “Clayton” in our example, they are captured using regular expressions in the step definition. Cucumber then passes the matched values along so that then can be used in your assertion. We can find the pet based on the pet_name and make sure that the owner linked to this pet via the belongs_to association is the same as the owner_name we specified in our scenario. This is an example of how Cucumber can be used, with RSpec matchers, to make an assertion that has nothing to do with inspecting the DOM of a resulting webpage.

Note: Beware of the Conjunction Steps Anti-pattern

Scenario Outlines

Cucumber provides a very simple way to test multiple different situations with a single scenario. These might be edge cases or just repetitive examples that don’t require their own scenario.

We have some business logic in our application that determines when an appointment, or Visitation, can be made with the Veterinarian:

  • Visitations less than three days from today require approval before being booked
  • No visitations can be booked more than six months in advance
  • No visitations can be booked in the past

We can easily represent this using a Cucumber Scenario Outline. Start by creating a new feature features/visitation_logic.feature. Also, because we’re going to be working with time and date specific business logic, this is a good time to point out that we can stub methods when using Cucumber, however, this practice is frowned upon and should only be used for things like dates and connecting to external APIs.

To add stubbing support add the following to your features/support/custom_env.rb file.

  require 'spec/stubs/cucumber'
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 
    Scenario Outline: visitations
      Given the following owners:
        | name    |
        | Clayton |
      And the following pets:
        | name  |
        | Bruno |
      Given today is "<today>"
      Given I am on the home page
      And I follow "Visitations"
      And I follow "New visitation"
      And I select "Bruno (Owned by Clayton)" from "Pet"
      And I select "<date>" as the "Appointment Date" date
      When I press "Create"
      Then I should see "<message>"
 
    Examples:
      | today      | date               | message                                           |
      | 2010-01-01 | January 2nd, 2010  | Visitation requires short notice approval.        |
      | 2010-01-01 | January 31st, 2010 | Visitation succesfully created.                   |
      | 2010-01-01 | November 2nd, 2010 | Visitations cannot be booked that far in advance. |
      | 2010-04-01 | January 2nd, 2010  | Visitations cannot be booked in the past.         |

Cucumber will go through the Scenario Outline you have created and substitute the values in <>’s with their corresponding value in the Examples table at the bottom. Cucumber goes through each line in the Examples table and runs the scenario using your specified values.

First, lets implement the step for stubbing our today’s date.

1
2
3
4
 
    Given /^today is "([^"]*)"$/ do |date|
      Date.stub(:today).and_return(Date.parse(date))
    end

In features/step_definitions/shared_steps.rb we can add the implementation for Given the following pets. We are cheating here because we’re not explicitly linking the Pet to the Owner in our scenario outline. This example isn’t so bad, but keep in mind that this association is not obvious when reading the scenario. This type of “behind the scenes” scenario setup is generally a bad idea as it obscures what’s really happening. Since we’ve already created the owner we can just find the first one in the database and create the pet records using the appropriate Rails’ association method.


    Given /^the following pets:$/ do |table|
      table.hashes.each do |attributes|
        Owner.first.pets.create(attributes)
      end
    end

The other steps in our scenario already exist, either because we created them, or because they come for free with the built in steps. In this one scenario we were able to test four different business logic outcomes. This test could be expanded to go beyond what is shown on the resulting page, perhaps to ensure that an Approval record was created for the short-notice appointment.

Advanced Cucumber Goodies

While you should now have an understanding of the basics of Cucumber, there are a number of other powerful features.

Tags

Cucumber allows you to “tag” your scenarios and features so that they can be run, not run or have special other tasks run before and after them. Out of the box you get the @wip tag (Work In Progress) which isn’t run as part of the normal rake cucumber:ok process.

Hooks

Using tags, you can run Before and After tasks that run some code before or after a given feature or scenario. You can even run these hooks before or after all of your features. You might need to do some cleanup of generated files in an After hook or some CPU expensive operation before a single scenario that you don’t want running before all of your scenarios.

Table Transformations

The tables that we have been creating use the exact model attribute names for their header rows. This is convenient for the developer, but makes the scenarios harder to understand. This is especially true if you have very strange or legacy attribute names like record_quote_ext_type__c. Using table transformations you can use “Record Extension Type” in your scenario table and map that to record_quote_ext_type__c in your step definition.

Calling Steps from Steps

It’s possible to take a multi-step process, expressed in Cucumber steps, and call them all at once from another step. You might have a few steps that describe logging in to a system (filling out login credentials, pressing a button etc.) You can easily put them all into one step, Given I am logged in, and then reference that elsewhere. This helps to reduce unnecessary repetition of steps and keep your scenarios to a manageable size.

Profiles

Cucumber allows you to specify a profile to use when running your features. By default Cucumber will not run any @wip tagged scenarios and the output will contain any failing files and the line number of the failure. However, if you wanted to run the @production tagged stories and output a nice HTML report, you could setup a new profile for that in config/cucumber.yml and use it when running your features.

Testing Javascript

You can even use Cucumber to test complex javascript. Tools like Celerity with Culerity or Selenium give you the ability go beyond the interactions provided by Webrat. A new addition to Cucumber is Capybara support which aims to unify the language used in steps so that you can use Webrat, Selenium and Celerity side-by-side.

Use Cucumber Table Transformations To Build Objects

Using Cucumber’s Table Transformations, you can easily build complex objects in a way that’s easy to read and understand for clients and developers alike.

My latest favorite feature of Cucumber are Table Transformations. I frequently use tables to build up complex objects and I’ve found that the regular old tables can be a little ugly, especially when your attribute names don’t make much sense on their own. I’ve also noticed that building up associations can be a little wonky, usually requiring more steps than seem necessary.

Conventional Table Usage

Let’s look at an example of how we could use a table, without transformations, to build up some objects for our scenario.

1
2
3
4
5
Scenario: Editing a Spirit
  Given I have a spirit with the following attributes:
    | spirit_type    | country_of_origin | age | brand        | lgcy_prod_sku | name       |
    | Scotch Whisky  | Scotland          | 12  | The Balvenie | SC38181       | DoubleWood |
    | Scotch Whiskey | Scotland          | 12  | The Macallan | SC38245       |            |
1
2
3
4
5
Given /^I have a spirit with the following attributes:$/ do |table|
  table.hashes.each do |attributes|
    Spirit.create!(attributes)
  end
end

Be sure to use create! in your tests to prevent false positives (Thanks Aslak!)

Now this is all fairly simple, and it looks pretty easy to implement, but I see some problems. First, what if you were actually friends with your DBA (bear with me) and you knew better than to have an attribute in your model like country_of_origin or spirit_type. Chances are those are going to be used by many other records and should be pulled out and made into their own Models, Country and SpiritType respectively.1

So what does our scenario look like with those two new models?

1
2
3
4
5
6
7
8
9
10
11
Scenario: Editing a Spirit
  Given I have a country with the following attributes:
    | name     | continent |
    | Scotland | Europe    |
  And I have a spirit type with the following attributes:
    | name           |
    | Scotch Whiskey |
  And I have a spirit with the following attributes:
    | age | brand        | lgcy_prod_sku | name       |
    | 12  | The Balvenie | SC38181       | DoubleWood |
    | 12  | The Macallan | SC38245       |            |

It’s a little more complex, for sure, but it’s not totally unmanageable. However, the key part that’s missing is how to link the two spirits with their spirit types and countries of origin.

You could add some more steps, but then you’ve got a conjunction step which is inflexible and brittle.

1
  And the spirit named "The Balvenie" is from "Scotland" and is a "Scotch Whiskey"

You could go back to your original step, and try to do some behind-the-scenes stuff to map country_of_origin to the correct country_id, but that gets messy too.

Transform Your Tables

The first step to making good use of table transformations is to make your tables more readable. Start by change the header row of your table to use meaningful representations of the real attribute names.

1
2
3
4
Given I have a spirit with the following attributes:
  | Spirit Type    | Country  | Age | Brand        | Legacy Product Code | Name       |
  | Scotch Whisky  | Scotland | 12  | The Balvenie | SC38181             | DoubleWood |
  | Scotch Whiskey | Scotland | 12  | The Macallan | SC38245             |            |

We’ve turned that weird lgcy_prod_sku attribute into something that your Product Owner can make sense of and we’ve be able to add Spirit Type and Country of Origin back to the table. Now let’s look at the transformation that makes this all work.

1
2
3
4
5
6
7
8
9
10
11
12
Transform /^table:Spirit Type,Country,Age,Brand,Legacy Product Code,Name$/ do |table|
  table.hashes.map do |hash|
    spirit_type = SpiritType.create!({:name => hash["Spirit Type"]})
    country = Country.create!({:name => hash["Country"]})
    spirit = Spirit.create!({:age => hash["Age"],
                            :brand => hash["Brand"],
                            :lgcy_prod_sku => hash["Legacy Product Code"],
                            :name => hash["Name"]})
 
    {:spirit_type => spirit_type, :country => country, :spirit => spirit}
  end
end

The transformation step definition looks a lot like a regular table step definition. There is a regular expression, like anything else in Cucumber, that has the same values as the header row in the table from our scenario. Just like the table step definition we have a table object which is just an array of hashes. We can go through each hash, do the actual transformation, and then return something to our table step definition. We are using map (same as collect) to return an array of hashes, which is just what the table step definition is expecting.

You will also see that we’re creating three different records, which we are returning in the hash we create at the end. Let’s go through those step-by-step:

  1. Create a spirit type object from the hash["Spirit Type"] value
  2. Create a country object from the hash["Country"] value
  3. Create a spirit object from the several related hash values
  4. Put all of our created objects into a hash

While we’ve created the objects, we still need to create the associations. I like to leave this for the table step definition rather than the transformation since I think it’s more obvious what’s going on with the values when you’re viewing the table step.

1
2
3
4
5
6
7
Given /^I have a spirit with the following attributes:$/ do |table|
  table.each do |group|
    spirit = group[:sprit]
    associations = {:country => group[:country], :spirit_type => group[:spirit_type]}
    spirit.update_attributes(associations)
  end
end

There are at least a dozen ways to get the spirit associated with the country and spirit type, so don’t feel like you have to follow this pattern every time. Since we’ve sent our table an array of hashes we can iterate over each hash, group, and work with the individual rows. Here’s how:

  1. Extract the spirit object from the hash
  2. Create another hash with the country and spirit type that Rails can make sense of
  3. Use update_attributes to update the spirit object with the new associations

Transformation Tradeoffs

We’ve been able to take our original multi-step scenario and simplify it to a single step. We are using the proper place, the step definitions, to do the associations and we have made our scenario much easier to read for non-developers working on the project. But what did we give up?

The biggest issue I’ve found with using table transformations is that they can be inflexible when you need to add more attributes to your dynamically created object. If you are writing features, using your table to setup objects and then realize that you need to add another attribute, you’re going to have to edit your table transformation step and how you create objects from the hash. When you take this a step further and try to have two different table definitions, you’ll be looking at having two nearly identical table transformations.2

If you’re not already using regular old Cucumber tables to create objects, use this guide to get started. If you are using Cucumber tables to create objects, try to re-factor one of your scenarios and use the table transformation strategy. Once you start using Cucumber tables and table transformations you’ll instantly improve the readability, portability and efficiency of your steps.

1 Ignore for now the issues with spirit_type and Rails Single Table Inheritance

2 I’m guessing that there is some way you can get around this with regular expressions and to have more flexible transformation table steps, but I haven’t tried it yet.

The Secret to Awesome Agile Development

With a little hard work and my secret development ingredient, you can be a better Agile Developer

Recently my fellow developers at Integrum and I took a survey that helped us assess our team with regard to our Agile practices. When taking the survey, and now reviewing it later on, I was struck by how many of the questions were related to a single concept. Many of the problem areas that can be uncovered by the survey, along with examples of one’s successes, come back to this one theme.

Are programmers nearly always confident that the code they’ve written recently does what it’s intended to do?
Consider the following questions:
  • Is there more than one bug per month in the business logic of completed stories?
  • Can any programmer on the team currently build and test the software, and get unambiguous success / fail result, using a single command?
  • When a programmer gets the latest code, is he nearly always confident that it will build successfully and pass all tests?
  • Are fewer than five bugs per month discovered in the teamʼs finished work?
  • After a line item is marked “complete” do team members later perform unexpected additional work, such as bug fixes or release polish, to finish it?
  • Are programmers nearly always confident that the code they’ve written recently does what it’s intended to do?
  • Are all programmers comfortable making changes to the code?
  • Do programmers have more than one debug session per week that exceeds 10 minutes?
  • Do unexpected design changes require difficult or costly changes to existing code?
  • Do any programmers optimize code without conducting performance tests first?
  • Is there more than one bug per month in the business logic of completed stories?
  • Are any team members unsure about the quality of the software the team is producing?

What’s the common theme among these stories, and the secret to better agile development? Testing, testing and more testing.

The negative outcomes implied by some of these questions can be solved by testing. Spending time fixing “completed” stories? Probably something you could have tested. Conversely, the positive benefits implied by other questions can be had via testing. Want to make your code more inviting and easier to deal with for new team members or people unfamiliar with the project? Give them robust and well-written tests.

Cucumber Table Transformations with Factory Girl Sequences

If you’re using Cucumber and you’re not using Transformations you’re doing it wrong. I just started using these recently and ran into a problem with creating records using factory_girl factories that made use of sequences. While trying to create multiple Authlogic user records with a unique email and unique single_access_token using a Cucumber table, the functionality of Factory.next(:email) wasn’t working correctly, I would always get the same e-mail address. Turns out it was an easy fix, just had to use lazy attributes in my factory.

The Scenario, Step Definition and Factory

1
2
3
4
5
6
7
8
9
10
11
12
  Scenario: Presenter List
    Given the following presenters:
      | Name    | Bio                 | Website              |
      | Clayton | Rails dev @integrum | http://claytonlz.com |
      | Chris   | Scrum @integrum     |                      |
    And I am on the homepage
    When I follow "Presenters"
    Then I should see "Clayton"
    And I should see "Rails dev @integrum"
    And I should see "http://claytonlz.com"
    Then I should see "Chris"
    And I should see "Scrum @integrum"

My Step Definition
This uses the Transformation table below:

1
2
3
4
5
Given /^the following presenters:$/ do |table|
  table.each do |attrs|
    Factory.create(:user, attrs)
  end
end

This transformation takes a table like the one in my scenario above, and assigns the values to a hash using the actual model attribute names (Name isn’t an attribute on a user but name is). The regular cucumber step definition “consumes” this hash for each entry in the table and passes it to a Factory for creation.

1
2
3
4
5
Transform /^table:Name,Bio,Website$/ do |table|
  table.hashes.map do |hash|
    {:name => hash[:Name], :bio => hash[:Bio], :website => hash[:Website]}
  end
end

My Factory

This is a pretty basic factory for an authlogic user model, I’m using factory_girl sequences to give me a “unique” e-mail and single access token, which are required by Authlogic.

1
2
3
4
5
6
7
8
9
Factory.define :user do |user|
  user.email Factory.next(:email)
  user.name ""
  user.bio ""
  user.website ""
  user.password "password"
  user.password_confirmation "password"
  user.single_access_token Factory.next(:single_access_token)
end

The problem

The above scenario will fail when it tries to create the user records via the factories. You’ll see a validation error about how the user model requires a unique e-mail and single access token. You’ll be wondering, “hey why are my sequences working?”. When you inspect the log you’ll see that they are in fact NOT working.

The Quick Answer

The easy answer to this is that you need to use lazy attributes in your factory for the sequences so that they are loaded each time instead of once.

1
2
3
4
5
6
7
8
9
Factory.define :user do |user|
  user.email { Factory.next(:email) }
  user.name ""
  user.bio ""
  user.website ""
  user.password "password"
  user.password_confirmation "password"
  user.single_access_token { Factory.next(:single_access_token) }
end

Notice the curly braces around the sequences

The Longer Answer

The cucumber rdoc explains the Transform functionality, albeit somewhat hard to understand.

Registers a proc that will be called with a step definition argument if it matches the pattern passed as the first argument to Transform. Alternatively, if the pattern contains captures then they will be yielded as arguments to the provided proc. The return value of the proc is consequently yielded to the step definition.

I think the issue comes from something with the way these Procs are created, called and also their scope with regard to the step definition etc. I don’t think my ruby-fu is strong enough to give a good explanation but maybe I’m going in the right direction.

How To: Setup RSpec, Cucumber, Webrat, RCov and Autotest on Leopard

RSpec, Cucumber, Webrat, RCov and Autotest are a powerful combination of tools for testing your Rails app. Unfortunately getting them to all work nicely together can be a bit of challenge. I recently configured a development environment from scratch on OS X 10.5 Leopard and kept track of all of the little details.

Prerequisites

I’m assuming you’ve got the following installed:

  • ruby
  • ruby gems 1.3.1
  • Apple development tools
  • git
  • rails >= 2.3.2
  • You’ve added github to your gem sources (gem sources -a http://gems.github.com)

RSpec & RSpec-Rails

First let’s grab the rspec1 and rspec-rails2 gems.

sudo gem install rspec
sudo gem install rspec-rails

Cucumber

Next we’ll install the cucumber3 gem

sudo gem install cucumber

Webrat

Webrat4 is used by cucumber to simulate a browser for your integration tests. Webrat will also install nokogiri5.

sudo gem install webrat

RCov

I thought RCov6 would get installed with RSpec, but it wasn’t for me. You might not need to do this, but just to make sure…

sudo gem install rcov

Autotest

Autotest7 comes from ZenTest8 and allows you to have a kick ass workflow where you are constantly running relevant tests and less-constantly automatically running your entire test suite.

sudo gem install ZenTest

Optionally, Thoughtbot’s Factory Girl

Factory girl9 is a really helpful fixture replacement (and more) gem to use in conjunction with cucumber, checkout their much better explanation

sudo gem install thoughtbot-factory_girl --source http://gems.github.com

Optionally, Carlos Brando’s Autotest Notification

While autotest normally runs in a terminal window, it can be setup to hook into applications like growl or snarl. The Autotest Notification9 gem helps make this setup a lot easier.

You will need growl installed and configured for this step the installation instructions on this gems github page are very easy to follow.

sudo gem install carlosbrando-autotest-notification --source=http://gems.github.com

Next you need to turn autotest notifications “on”

an-install

A Sample Rails App

Let’s create a sample rails app for the rest of this guide.

rails sample-app

Configuring Environment Variables

Autotest relies on some environment variables to run all of your features and specs correctly. If autotest “hangs” after you try to run it, or it just never seems to be watching your specs or features, this will most likely solve your problem.

Open the test.rb environment definition file in sample-app/config/environments/test.rb and add the following.

1
2
ENV['AUTOFEATURE'] = "true"
ENV['RSPEC'] = "true"

These lines will test autotest to run, and look for changes to, your specs (rather than test unit tests) and your cucumber features.

Update

If you don’t want to add these environment variables to every rails project you’ve got on your machine, you can also choose to set them as environment variables in your .bash_profile or .bashrc (or whatever shell you’re using) files.

export AUTOFEATURE=true
export RSPEC=true

Unpacking Gems

Next let’s freeze (unpack) some gems that we’ll be using in our app. I’ve run into problems trying to use the system gems with cucumber, rspec and webrat, especially when I have multiple versions of any of them installed. Unpacking them into my rails app solves this problem for me.

mkdir sample-app/vendor/gems
cd sample-app/vendor/gems
gem unpack rails
gem unpack rspec
gem unpack rspec-rails
gem unpack cucumber

Because webrat (and nokogiri) are native gems, that is, they are built locally on your machine based on its architecture, we won’t unpack those.

config.gem support
The current accepted practice, when using rails 2.3, and as suggested by the rspec guy(s) is to use rails’ config.gem functionality.

Open sample-app/config/environments/test.rb and add the following lines:

config.gem "rspec", :lib => false, :version => ">= 1.2.0" 
config.gem "rspec-rails", :lib => false, :version => ">= 1.2.0" 
config.gem "cucumber", :lib => false, :version => ">= 0.2.3"
config.gem "thoughtbot-factory_girl", :lib    => "factory_girl", :source => "http://gems.github.com"
config.gem "webrat", :lib => false, :version => ">= 0.4.3"
config.gem "nokogiri", :lib => false, :version => ">= 1.2.3"

Your version numbers may be different, but these are all current at the time of writing.

Boot Strapping RSpec and Cucumber

Before you can get very far with rspec or cucumber you need to run the bootstrapping scripts to give yourself the default files and directories.

# From inside your rails app sample-app/
script/generate rspec
script/generate cucumber

Factories
Depending on where you’re going to use your factories the most, you might want to save your file in either spec/ or features/. I chose the latter. Only complete this step if you plan to use the FactoryGirl gem.

touch sample-app/features/factories.rb

Getting Accurate RCov Data

By default RCov is setup to only use your specs when calculating code coverage. If you’re using Cucumber and RSpec, you’ll obviously want to include both types of tests to calculate your project’s true code coverage.

I picked up this rcov rake task from my co-worker Jay McGavren it does all of the heavy lifting for you, we’ll just need to make a couple of changes.

Drop this file into sample-app/lib/tasks/rcov.rake and use it by calling rake rcov:all from your terminal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
require 'cucumber/rake/task' #I have to add this
require 'spec/rake/spectask'
 
namespace :rcov do
  Cucumber::Rake::Task.new(:cucumber) do |t|    
    t.rcov = true
    t.rcov_opts = %w{--rails --exclude osx/objc,gems/,spec/,features/ --aggregate coverage.data}
    t.rcov_opts << %[-o "coverage"]
  end
 
  Spec::Rake::SpecTask.new(:rspec) do |t|
    t.spec_opts = ['--options', ""#{RAILS_ROOT}/spec/spec.opts""]
    t.spec_files = FileList['spec/**/*_spec.rb']
    t.rcov = true
    t.rcov_opts = lambda do
      IO.readlines("#{RAILS_ROOT}/spec/rcov.opts").map {|l| l.chomp.split " "}.flatten
    end
  end
 
  desc "Run both specs and features to generate aggregated coverage"
  task :all do |t|
    rm "coverage.data" if File.exist?("coverage.data")
    Rake::Task["rcov:cucumber"].invoke
    Rake::Task["rcov:rspec"].invoke
  end
end

The important part here is on line 7, we want rcov to exclude our features directory. We obviously don’t need or want rcov telling us that our feature files are not “covered”. To solve this problem we’ve simply excluded the features directory from rcov’s processing.

We also need to slightly modify sample-app/spec/rcov.opts to get the full rspec + cucumber coverage data.

Your rcov.opts should look like this:

--exclude "spec/*,gems/*,features/*" 
--rails
--aggregate "coverage.data"

We again want to ignore our cucumber features and we also want to tell rcov to aggregate data in a file called coverage.data. This is used in the above rake task.

Write Some Specs and Features!

Act like you know what you’re doing and write some models, controllers whatever. Add some specs and features too.

Autotest Workflow

Open a terminal and make your way to your sample rails app and fire up autotest. You might see something like the following, depending on how many specs and features you’ve got.

$> autotest
loading autotest/cucumber_rails_rspec
opts 
...
 
Finished in 0.06276 seconds
 
3 examples, 0 failures
================================================================================
 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby /Library/Ruby/Gems/1.8/gems/cucumber-0.2.3/bin/cucumber --format progress --format rerun --out /var/folders/Aq/Aqp06i3dFnqse+tQgQA+1++++TI/-Tmp-/autotest-cucumber.75956.0 features
.................
 
4 scenarios
17 passed steps
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby /Library/Ruby/Gems/1.8/gems/rspec-1.2.2/bin/spec --autospec spec/models/intern_spec.rb -O spec/spec.opts 
...
 
Finished in 0.062995 seconds
 
3 examples, 0 failures
================================================================================
 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby /Library/Ruby/Gems/1.8/gems/cucumber-0.2.3/bin/cucumber --format progress --format rerun --out /var/folders/Aq/Aqp06i3dFnqse+tQgQA+1++++TI/-Tmp-/autotest-cucumber.75956.1 features
.................
 
4 scenarios
17 passed steps

The REALLY important stuff

  1. make sure you’ve got “ENV['AUTOFEATURE'] = true” in your test.rb otherwise autotest won’t run your features automatically
  2. make sure you’ve got “ENV['RSPEC'] = true” in your bash profile or else autotest won’t run your specs automatically
  3. make sure you’ve got “–aggregate = ‘coverage.data’” in your spec/rcov.opts file if you’re going to use the above rake task and hope to get combined rcov coverage data between rspec and cucumber
  4. make sure you’re excluding the features directory from rcov where required or else you’ll end up with misleading rcov data.

Gem Versions

Here’s a list of the current gems and their versions that I used in preparing this guide.

*** LOCAL GEMS ***
 
actionmailer (2.3.2, 1.3.6, 1.3.3)
actionpack (2.3.2, 1.13.6, 1.13.3)
actionwebservice (1.2.6, 1.2.3)
activerecord (2.3.2, 1.15.6, 1.15.3)
activeresource (2.3.2)
activesupport (2.3.2, 1.4.4, 1.4.2)
acts_as_ferret (0.4.1)
addressable (2.0.2)
builder (2.1.2)
capistrano (2.0.0)
carlosbrando-autotest-notification (1.9.1)
cgi_multipart_eof_fix (2.5.0, 2.2)
cucumber (0.2.3)
daemons (1.0.9, 1.0.7)
data_objects (0.9.11)
diff-lcs (1.1.2)
dnssd (0.6.0)
extlib (0.9.11)
fastthread (1.0.1, 1.0)
fcgi (0.8.7)
ferret (0.11.4)
gem_plugin (0.2.3, 0.2.2)
highline (1.2.9)
hpricot (0.6)
libxml-ruby (0.3.8.4)
mongrel (1.1.4, 1.0.1)
mysql (2.7)
needle (1.3.0)
net-sftp (1.1.0)
net-ssh (1.1.2)
nokogiri (1.2.3)
polyglot (0.2.5)
rack (0.9.1)
rails (2.3.2, 1.2.6, 1.2.3)
rake (0.8.4, 0.7.3)
rcov (0.8.1.2.0)
RedCloth (3.0.4)
rspec (1.2.2)
rspec-rails (1.2.2)
ruby-openid (1.1.4)
ruby-yadis (0.3.4)
rubynode (0.1.3)
sources (0.0.1)
sqlite3-ruby (1.2.1)
term-ansicolor (1.0.3)
termios (0.9.4)
textmate (0.9.2)
thor (0.9.9)
thoughtbot-factory_girl (1.2.0)
treetop (1.2.5)
webrat (0.4.3)
ZenTest (4.0.0)

El Fin

Hopefully this guide was useful or had that one little step that you needed to get everything working. I’m sure this will all be out of date in the coming weeks, but I’ll try to keep it as up-to-date as possible. If you see any errors, or can better explain some of the missing pieces, please post a comment. Thanks!

1 http://github.com/dchelimsky/rspec/tree/master

2 http://github.com/dchelimsky/rspec-rails/tree/master

3 http://github.com/aslakhellesoy/cucumber/tree/master

4 http://wiki.github.com/brynary/webrat

5 http://github.com/tenderlove/nokogiri/tree/master

6 http://rubyforge.org/projects/rcov/

7 http://www.zenspider.com/ZSS/Products/ZenTest/#rsn

8 http://www.zenspider.com/ZSS/Products/ZenTest/

9 http://github.com/thoughtbot/factory_girl/tree/master

10 http://github.com/carlosbrando/autotest-notification/tree/master

Updates
2009-12-08 – Removed “sudo” when describing how to unpack gems (h/t xdotcommer)