We have an internal application that does staffing, time entry, and now Paid Time Off (PTO) accrual, scheduling and management. It is quite nice, as it has replaced three existing systems, and replaced a number of manual, tedious tasks. I started it last year, as our current system was very inefficient. It was a simple Ruby on Rails app that I was able to get working in a few weeks. Over time the functionality grew.

We recently added in PTO accrual and functionality to debit PTO time. In doing so in a test driven manner was great, as we could put all the edge cases. What we found was that many of the requirements were plentiful with edge cases. For instance, we get paid on the 15th or end of the month, unless that is a weekend. Then we get paid on a Friday - Except if that Friday is a holiday, then the previous Thursday - Except - if that is also a holiday.... So it is really the previous non-weekend, non-holiday on or before the 15th or end of the month. The same holds true to determine the beginning of the billing period. If the Monday is the 2nd, than for some operations the effective billing period start is Tuesday the third.... Ugh...

Our initial thoughts were filled with visions of lots of very similar tests for all contexts that do things depending on the start or end of a billing period. We didn't want to have lots of duplicate test code to test all edge cases, so we decided to extend the Date object to have a few helper methods to do the complicated logic in one place. We get the full range of the billing period start and end, and then trim off any holidays or weekends. What this did, was allow us to test the complicated logic in unit tests with a full set of complicated edge cases. We created many crazy examples of billing periods starting and ending on weekends, holidays, and holidays before and after weekends. This created a very robust set of methods to be used anywhere in the system.

We then mocked a call to each Date helper method everywhere in the system where it cared about what day it was, and weather it was the start/end of a billing period. We had only a few edge cases now: is it the true effective billing period start/end, or not. Our tests could then focus on the guts of what it actually did, regardless of the effective date.

This demonstrates how powerful unit testing is, and how mocking can really keep your tests concise. It made not only our code cleaner, but our tests cleaner. It reduced duplicate code, and increased our assurance that the code will function as designed.

That being said, it still doesn't change the fact that implementing HR logic in any language is a pain in the ass. I have worked on a couple of systems for accounting departments in the past, and their business requirements are much worse than HR. Dealing with job codes, general ledger accounts, etc can make your head spin. All I know, is that without TDD, this system would be buggy.

Related posts:

1. The Testing Interviews
2. Ajax and Leaky Business Logic
3. The Hazards of Exposing Business Logic on the Client

It's time to play "Ask A Tester Person", where I answer questions that I've gotten via email or otherwise about Rails Testing topics.

If you have a question for Ask A Tester Person, send it to railsprescriptions at gmail.com.

I've got two questions today:

Question 1: I'm writing too many tests!

One subject I still have questions
about is how to test authorization in controller tests. In my
applications I've been testing three different user cases for every
controller action, but this leads to, for example:

test_edit_by_anonymous_user
test_edit_by_unauthorized_user
test_edit_by_authorized_user

So every action has at least three tests, which obviously means a
*ton* of tests for the entire application. This has always felt like
overdoing it but I haven't yet been able to convince myself that just
testing the authorization part of the code is sufficient. If I could
isolate the authorization mechanism for testing that might convince
me, but I'm not really able to do that (maybe my authorization process
needs to be redesigned?), and I'm not sure I'd be convinced
anyway...how the application responds in all three cases seems like
something that *should* be tested for every action...

I doubt that your authorization process needs to be redesigned, but then I've never seen your authentication process.

I'd recommend a couple of different things to clean this up a bit. On the assumption that unauthorized access has largely the same behavior across the application, you can create a boilerplate test like this example -- the example uses Shoulda, but the basic idea should work in any framework.

  self.def should_block_access_for_anonymous_user(*actions)
    actions.each do |action|
      should "block anonymous access for #{action}" do
        logout_current_user
        get action
        assert_redirected_to root_path
      end
    end
	end

Which you would then use as:

  should_block_access_for_anonymous_user :edit, :update, :delete

The implementation shown here is probably a little too simplistic for full use (you might need to pass more information than just the action for each request), but the simple version could easily cover the most basic authentication issues with very little fuss.

If your authentication system is more complicated, the other option is nested contexts -- again, this is in Shoulda, but can be adapted to RSpec or the Context gem:

context "GET edit" do

  setup do
    # generic edit setup here
  end

  context "with an anonymous user" do
    setup do
      logout
      get edit
    end

    should "not allow access" do
      assert_redirected_to root_path
    end
  end

  context "with an admin user" do
    setup do
      login_as_admin
      get edit
    end

    should "allow access" do
      assert_response :success
    end
  end

end

The upside of this compared with what you are probably doing is that it consolidate the setup between the different options. The downside is that it can be kind of verbose and hard to follow. But it's still potentially easier than writing three separate tests for each action.

Question two: I'm creating too many objects!

Okay, I actually can't find the email that this came from, which is driving me crazy, because how do you lose an email message these days? Anyway, the gist was that this person's tests were running very slowly because he or she was creating so many objects for each test -- I distinctly remember the number 50 to 100 being tossed around. So the question was how to avoid creating so many objects?

The bottom line is that there's no way that you should need to create 50 to 100 objects for every TDD test, or even for any TDD test -- it's pretty rare that a unit-level test really needs that much data to work. There are a lot of potential issues here. Three that spring to mind are:

• You are trying to test too much code at one time. It's possible (though still not very likely) that an integration test might need that many objects, but testing a single method almost never does
• You've transitioned from fixtures to factories, but you are still writing your tests as though you were using fixtures.
• You have perhaps an unnecessary fixation on providing "realistic" data in unit tests.

There's some overlap here. One cause of this is problem is that in fixtureland, there's very little marginal cost to creating new objects, so the tendency is to create fairly large aggregations of objects that cover all possibilities and have that be the universe for all the tests. When you transition to factories, though, keeping that big data blob around is not necessary. Since you can and should be custom-creating the data for each unit test, most model methods only need one model with specific attributes in order to specify the logic. Sometimes that model will need associated objects, but the factory tool can be set to create those automatically.

A special find method or named scope can be tested with as little as two objects -- one to be found and one to be skipped. Sort logic similarly can be tested with two or maybe three objects. If the logic is really complicated, it's better to do multiple tests with small amounts of data than one test with the whole shebang. The idea of a "unit" test is to verify one small piece of logic with as little data as possible, not to check program behavior under realistic data load -- that's what integration tests, performance tests, and actual user acceptance is for.

I feel like I'm answering the question "How do I create fewer objects in my tests?" with "By creating fewer objects in your tests", I hope this answer gives you some tools for minimizing the number of objects you need to create in your tests.

Related Services: Ruby on Rails Development, Custom Software Development, Testing and Quality Assurance

Related posts:

1. Elements of Testing Style
2. Rails Testing Frequently Asked Questions — The Non-Code Version
3. Rails Test Prescriptions is now on sale

Rails Test Prescriptions, the eBook put out by Noel Rappin, Director of Rails Development at Pathfinder, has been picked up by Pragmatic.

Congratulations to Noel - he's done a great job of furthering testing best practices in rails, and this is a great reward. As he said "I’m very excited by this. I’ve wanted to work with Pragmatic for as long as they’ve been publishing books, and I’m thrilled that this particular project will be able to get wider distribution and access to Pragmatic’s editorial expertise and skill."

* The current free “Getting Started with Rails Testing” ebook will continue to be available. If, at some time in the future, there’s a better Getting Started tutorial in the Pragmatic book, it may be offered as a replacement.

* The update site for current Rails Test Prescription owners will continue to be available for the foreseeable future.

* There will be one more official update to the current Rails Test Prescriptions, probably around the end of August. This will wrap up the chapter or two I’m working on, and tie up some other loose ends.

* After that, errata and information about changes to test tools will most likely be handled via this blog and an errata page on the rails test prescriptions site.

This is Noel's 4th book with a major publisher, following Professional Ruby on Rails, wxPython in Action and Jython Essentials. We're happy for Noel and happy to have him at Pathfinder.

Related Services: Ruby on Rails Development, Custom Software Development

Related posts:

1. More Kudos for Rails Prescriptions
2. Rails Test Prescriptions is now on sale
3. Announcing Rails Prescriptions

I'm writing a functional test:

should "do something functional"
  get :search,  rder_id => @order.id, :user_id => @user.id
  # and so on
end

The get call in that test simulates a browser request. Intuitively, you would (well, I would) expect this request to be identical to a request coming from the actual view, via a helper like link_to("search", :action => :search, rder_id => @order.id, :user_id => @user.id). At least, you'd expect that parameters hash in the controller to be the same between the

Makes sense, right? The testing call should set up the same environment as the actual call being tested.Well, if you've gotten this far, read the title that said "Annoyances", or have ever read any blogs, you know that isn't how Rails works. Specifically, the actual browser call has all its arguments converted to strings as part of going through the HTTP wire, but Rails, oddly, does not similarly convert the arguments in the test.

Most of the time it makes no difference -- if you pass the string or the integer directly to an ActiveRecord find method, everything works swimmingly. If you do a direct equality check on the parameter, though, you can get code that passes tests, but fails in the browser because the string value is no longer equal to the integer value.

For some time, I've had this on my list of annoyances that I don't have the energy to fix, right next to the slightly misaligned light over my desk, and the fact that they bought the wrong kind of Kleenex at the office that one time.

This week, Adam Milligan at Pivotal Labs tried to do something about it. Specifically, he created a Rails patch, then a plugin, that can raise a warning or an error when a non-string argument is passed to an HTTP method in a functional test.

Adam chose to fail or warn rather than silently convert the parameter because he feels that some arguments that might get placed in an HTTP method don't have clear correct conversions (his example is false), and he's trying to avoid setting up a whole different kind of difficult-to-diagnose test to app misalignments.

As much as I love that Adam did this, I kind of think that it's more consistent with Rails design to silently convert, to make the browser call and the test consistent. Still, having the code warn or fail (your choice) is a pretty good way to get into good habits and avoid having to track down weird test errors. So, thanks, Adam.

Related Services: Ruby on Rails Development, Custom Software Development

Related posts:

1. Rails, Ajax, RJS, and Testing
2. Rails Testing First Look: Blue Ridge
3. Elements of Testing Style

Dot, dot, dot, dot, dot -- tests are passing, looks like it's time for lunch -- dot, dot, dot, dot, F. F? F? But the code works. I know it does. I think it does. Why is my test failing?

One of the most frustrating times as a TDD developer is that moment when a test is failing and you don't know why, as opposed to the more normal case where the test fails as expected. Here's a grab bag of tips, tricks, hints, and thoughts to get us all through that difficult time.

Something Must Have Changed

This may be the most obvious piece of advice in the history of ever, but I find it's worth repeating, mantra-like, when confronted with a bad bug:

When a formerly-passing test fails, it means something changed.

It may be in the code, or the system, or the test. But it's probably not sunspots, and it's probably not evil spirits possessing your MacBook. (Unless you are either living in a Charles Stross novel or writing Perl, but I digress...)

Looking through recent changes can help figure out what the cause of the failure is. Git's bisect tool can do this automatically, or you can just look through recent changes in your source control viewer of choice. If the test was passing at one time, there's a good chance the answer is in there somewhere.

This is a great argument in favor of committing to your source control very, very frequently (especially when you are using git and can do local commits), so that your changes are very granular.

Isolate

When looking at a small number of failing tests, it's helpful to be able to run just those tests. Autotest is outstanding for this, since it will run the failing tests over and over until they pass. This is especially helpful if you have a number of failing tests that are not in the same test class.

This little code and terminal snippet is very helpful for quickly running one class at a time, which is almost like isolating a failing test, or at least close enough to be useful. Depending on your IDE and test framework of choice, you may also be able to run individual tests from the IDE.

Isolating tests makes the tests run faster when you are focused on just a few tests, and also makes any diagnostics you insert easier to interpret.

Two tips that I've stolen from listening to and reading Kent Beck:

• Back out your entire most recent change since your last passing test and start over. This works best if you work in very small increments, but it gets you out of the "I know I typed something wrong but I just can't see it" nightmare
• Replace all the expressions in the method under test with literals -- if that passes, then put the expressions back one by one until you find the culprit.

Diagnose

I have to say, I'm not a big fan of using stop-and-step debuggers. I've used them when I've been in an IDE, I've never really used the Rails command line debugger, but mostly I've found that not to be a great experience.

Normally, to diagnose what's going on in a test, I usually either add additional assertions in the test or have the code print information to the console. If I diagnose via assertions, generally I'm testing the values of variables in more detail.

For some reason, I see a lot of people using Ruby's puts method to write to the console -- I recommend p, which calls inspect on the object before printing, and generally results in more informative output. As a matter of course, I put require pp, which allows me to use pp to get pretty-printed output, which is nice for nested data structures. Also, y gives a YAML representation of the output -- very readable for ActiveRecord objects.

>> x = {1 => ['a', 'b'], 2 => 'c'}
>> puts x
1ab2c

>> p x
{1=>["a", "b"], 2=>"c"}

>> pp x
{1=>["a", "b"], 2=>"c"}

>> y x
---
1:
- a
- b
2: c

Especially if I have autotest running just the one test, I've been known to bury print statements all over the place -- controllers, Rails itself (often educational). Just remember to take them out when you are done.

Clear Your Head

Take a walk. Force your pair to solve the problem. Get a cup of coffee (actually, I hate coffee, get a Diet Coke). Take a nap. All those silly clear your head things really do work sometimes.

Band Aids

It's tempting sometimes to comment out the offending test and then your suite passes and all is well with the world again. That's generally a bad idea (although sometimes a major refactoring can genuinely make tests obsolete.

Hope this helps. What do you do to fix stubborn tests?

Related Services: Ruby on Rails Development, Custom Software Development

Related posts:

1. Rails Test Prescriptions is now on sale
2. Lesser Known Test Processes
3. Again With The Test Driven Development

What with upward of two people saying nice things about last week’s post, I’ve decided to keep going with part two of a look at some real testing code.

Most code-heavy tutorials show the code but not the tests — I’m doing the opposite here, and showing the tests, but not much of the code. Also, although I’m presenting these tests in chunks, you should realize that there was a lot of back-and-forth from Cucumber to tests to code and some backtracking, most of which I’ll spare you from having to wade through.

At the end of last week, I had run through the tests for spam-prevention code which worked by limiting the rate at which a user could send messages to other users of a particular social networking site. Cucumber was involved, and I think I went off on a tangent about writing lots of tests.

Once the basic rate limiting code was in place, the client and I came up with a couple of special cases. The application allows a user to explicitly reply to a specific message sent to them. We agreed that these replies should not count towards the rate limit on the grounds that a reply was, pretty much by definition, not spam. Similarly, we decided that any message between users with a friend relationship on the site also doesn’t count toward the rate limit. More subtly, we decided that if a user was inadvertently blocked, then reinstated by an administrator, that their rate count should drop back to zero, so the user doesn’t just get immediately re-blocked.

The Cucumber tests:

Scenario: Reply behavior
  Given I am a user who is not a new member
  When I send 4 messages in a day
  And I reply to a message
  And I send 1 message in a day
  Then 6 messages are sent
  And I am not blocked from sending further messages
  And the administrator does not get an email

Scenario: Friend Behavior
  Given I am a user who is not a new member
  When I send 4 messages in a day
  And I send a message to a friend
  And I send 1 message in a day
  Then 6 messages are sent
  And I am not blocked from sending further messages
  And the administrator does not get an email

Scenario: Unblocked behavior
  Given I am a user who is not a new member
  When I send 6 messages in a day
  Then I am blocked from sending further messages
  When I am unblocked by an administrator
  And I send 5 messages in a day
  Then 5 messages are sent
  And I am not blocked from sending further messages
  And the administrator does not get an email

These tests are quite similar in structure to the tests I started with last week. I might start thinking about combining some of the steps, but in Cucumber, I bias in favor of having the actual test be as clear and readable as I can get it. I don’t think these scenarios are so complicated that they require simplification.

Most of these steps have already been defined, here are a couple that aren’t. The first two steps are slight variants on the message sending step definition:

When /^I reply to a message$/ do
  original_message = Factory.create(:message, :sender => @recipient,
      :recipient => @sender)
  message = Factory.attributes_for(:message, :sender => @user)
  message[:in_reply_to] = original_message.id
  visit(messages_path,
      :post, {:recipient => @recipient.id, :message => message})
end

When /^I send a message to a friend$/ do
  @user.become_friends_with(@recipient)
  visit(messages_path,
      :post,
      {:recipient => @recipient.id,
          :message => Factory.attributes_for(:message, :sender => @user)})
end

One thing that you may have noticed about these step definitions. Despite all my jumping up and down about Cucumber being a black-box system, both of these steps bypass the UI to complete the setup. The first step creates a factory message and the second step creates a friend relationship. In both cases, the actual message send being tested passes through the UI.

There are basically two reasons for bypassing the UI in the setup. One is, of course, that it’s the setup and not the action being tested, the second is that being a purist all the time is a pain, and it’s much easier to get the background info out of the way directly.

The final missing step, however, is completely through the UI — the step logs the user out, logs in as an administrator, navigates to the page, unblocks the user, and then logs the administrator out and the original user back in.

When /^I am unblocked by an administrator$/ do
  @blocked = @user
  visit "/logout"
  Given "I am a logged in administrator"
  visit path_to("the admin messaging page")
  click_link("Unblock")
  @user = @blocked
  visit "/logout"
  Given "I am logged in"
  reset_mailer
end

A little on the Rube Goldberg side, perhaps, but it works.

With the Cucumber step definitions in place, the question becomes what tests and code need to be written to make these pass. For the first two scenarios that adjust the definition of what makes a message count toward the rate limit, the code change will be in the User and Message models. Since it’s a thin controller that defers to the models for data, there shouldn’t be any code change in the controller at all.

Now, that’s a much easier determination to make in hindsight than up front, and as it happens I did write controller tests for both of those scenarios, largely because they were so similar to the controller context chunks in last week’s post that it only took about five minutes to set them both up.

The real test action was in the user and message models. I added a database column for messages called counts_toward_spam. The idea is that this would be true for most messages, but false for replies, friend messages, or messages otherwise cleared by the admins. This requires some unit tests in the message class to support it. Again, I’m using Matchy here, and testing the boolean spam_message?, which is basically an alias for counts_toward_spam that I added for readability. In retrospect, adding the method was probably unnecessary.

context "spam count" do

  setup do
    @sender = Factory.create(:user)
    @recipient = Factory.create(:user)
  end

  should "give an ordinary message a spam count" do
    @message = Message.new_from_params(
        {:subject => "fred", :body => "body"}, @sender, @recipient)
    @message.sender_id.should == @sender.id
    @message.recipient_id.should == @recipient.id
    @message.should be_spam_message
  end

  should "not give an ordinary message a spam count if it is a reply" do
    @message = @message = Message.new_from_params(
        {:subject => "fred", :body => "body", :in_reply_to => 11},
        @sender, @recipient)
    @message.should be_reply
    @message.should_not be_spam_message
  end

  should "not make a message between mutual friends a spam count" do
    @sender.become_friends_with(@recipient)
    @message = Message.new_from_params(
        {:subject => "fred", :body => "body"}, @sender, @recipient)
    @message.should be_friend_message
    @message.should_not be_spam_message
  end
end

The message tests simply set up the different classes of message and verify that they are classified appropriately, they are pretty straightforward.

I also added a test to user to verify that the spam count was being used appropriately — this is part of the same context as last week’s user tests. The user has already sent four messages in the setup.

should "not count a reply message" do
  message = Factory.create(:message, :sender => @user,
      :created_at => 10.minutes.ago)
  message.in_reply_to = 3
  message.update_count_toward_spam
  message.save!
  @user.spam_message_count.should == 4
  @user.update_message_block_status.should be_nil
  @user.should be_able_to_send_messages
end

It’s interesting what you see when you go back over a chunk of code to explain it to other people. In this case what I notice is the update_count_toward_spam method, which is part of Message that unsets the count_toward_spam_ field if the message is a reply and is automatically called as part of the controller create message code. However, strictly speaking, it should be tested in the message class rather than here (although, the message controller tests would also exercise it).

That’s the bulk of the testing for those features — at the time, I didn’t think there were any other special cases.

The part about unblocking a user requires controller tests, because the unblock action needs to be written in the controller. I’m making it part of the message controller, although I know that RESTfully speaking it probably should be a separate resource — the legacy app is such a REST tangle that it’s not worth it.

The controller test context looks like this — the setup creates a blocked user and calls the unblock method.

context "GET unblock" do

  setup do
    ActionMailer::Base.deliveries.clear
    @recipient = Factory.create(:user)
    @user = Factory.create(:user,
      :message_sending_status => User::BLOCKED_STATUS)
    @user.should_not be_able_to_send_messages
    admin!
    get :unblock, :user_id => @user.id.to_s
    @user.reload
  end

  expect { @user.should be_able_to_send_messages }
  expect { assert_redirected_to :action => :admin }

  should "send email to unblocked user" do
    assert_sent_email do |email|
      email.to.first == @user.email &&
      email.from.first == "do_not_reply@singlestravelintl.com"
    end
  end
end

The assertion tests verify that the user can send messages again, and that the user gets an email from the system to that effect. The associated user model test validates that the user’s spam message count is reset back to zero, which was the specific point of this round of tests (in the actual application, there were other Cucumber scenarios supporting basic administrative behavior).

context "unblocking a user" do

  setup do
    Message.new_from_params(Factory.attributes_for(:message,
        :created_at => 10.minutes.ago), @user,
         Factory.create(:user)).save!
    @user.update_message_block_status.should == "blocked"
    @user.should_not be_able_to_send_messages
    @user.unblock_message_status
  end

  expect { @user.should be_able_to_send_messages }
  expect { @user.spam_message_count.should == 0 }

  should "reblock" do
    @user.block_message_status
    @user.should_not be_able_to_send_messages
  end

end

Anyway, the code that I wrote along side these tests to make them pass seemed to work fine, and we deployed to production.

Shortly thereafter, we got a bug report. I haven’t stressed it here, but new users were limited to a single message in their earliest time on the system. We heard from a new user who sent a message, received a reply, and then replied back to that message, only to find that he was blocked, even though the reply message shouldn’t count toward the rate limit.

Interesting. It’s one of those bugs that almost turns philosophical — exactly when in this process does a user become blocked? In the code that I originally wrote, a user was blocked as soon as he or she hit the rate limit, and before sending another message. In fact, the code should wait until the user tries to send that next message to consider that user blocked, because a user who is at the rate limit border should still be able to send replies and friend messages without being blocked.

So, bug. Since the bug involves multiple interactions, I started in Cucumber:

Scenario: New User Behavior on reply
  Given I am a new user
  When I send 1 message in a day
  And I reply to a message
  Then 2 messages are sent
  And I am not blocked from sending further messages

All those step definitions exist, but the last two steps fail without further work. I wasn’t initially sure whether the code change for this would wind up in the controller or not — as it happened, it turned out to be a minor controller change and minor model change but it took me a few tries to get it right.

Since I already had a decent controller harness for similar scenarios, it took almost no time to adapt to the new condition:

context "with a borderline new user and a reply" do

  setup do
    reply = Factory.create(:message, :recipient => @user,
        :sender => Factory.create(:user))
    flexmock(User).should_receive(:new_user?).and_return(true)
    flexmock(Message).should_receive(
        :toward_spam_in_last_24_hours).and_return(1)
    post :create, :recipient => @recipient.id,
        :message => Factory.attributes_for(:message, :sender => @user,
            :in_reply_to => reply.id.to_s)
    @user.reload
  end

  expect { @user.messages_sent.size.should == 1 }
  expect { @user.spam_message_count.should == 1 }
  expect { assigns(:message).should_not be_new_record }
  expect { @user.should_not be_message_sending_blocked }
  expect { assert_no_email_to_administrator }
end

I have to say, in retrospect, that I’m not 100% sold on the use of the mock package here — it’s a clear user of mocking to limit the test’s exposure to the model layer, but I’m not convinced it makes the test more clear or readable. This setup and tests is very similar to the other bundles of controller test and makes most of the same assertions.

The main change was in the user method, triggered by the following tests — one of which tests the positive sequence, one the negative — and yes, the setups probably should have been combined.

should "move a new user to blocked mode after first message" do
  Timecop.freeze(Date.today)
  new_user = Factory.create(:user, :created_at => 1.day.ago)
  m = Message.new_from_params(Factory.attributes_for(:message),
      new_user, Factory.create(:user))
  m.created_at = 2.hours.ago
  m.save!
  assert_equal(2.hours.ago, new_user.last_message_sent_time)
  assert_equal("blocked", new_user.update_message_block_status(true))
  assert new_user.message_sending_blocked?
end

should "allow a new user a second message that is a reply" do
  Timecop.freeze(Date.today)
  new_user = Factory.create(:user, :created_at => 1.day.ago)
  m = Message.new_from_params(Factory.attributes_for(:message),
      new_user, Factory.create(:user))
  m.created_at = 2.hours.ago
  m.save!
  assert_equal(2.hours.ago, new_user.last_message_sent_time)
  assert_nil new_user.update_message_block_status(false)
  assert !new_user.message_sending_blocked?
end

The key here — which would be easier to see if the setups were combined — is the next to last line where update_message_block_status is called. That’s the method called by the controller when a new message is sent, and the false argument means that the new message does not count toward the rate limit. So in the second test, the user can still send messages, while in the first test, the user does move to a blocked status.

Having the existing suite of tests around the controller and model behaviors was a big relief when fixing this bug — it’d be easy to make a fix for this issue that affected one of the other scenarios (I know because I made fixes that broke a lot of tests…). This was a case where the tests clearly made me more confident in the fix that I made.

Related Services: Ruby on Rails Development, Full Life Cycle Software Development

Related posts:

1. A Real Testing Example
2. Elements of Testing Style
3. The Testing Interviews

As sort-of promised in last week’s post, I’m going to work through a real-world test example, with an eye toward explaining how and why I tested the way I did. Hopefully, I’ll be able to do this at blog-post length. If not, well, there’s always next week.

This site, which was a legacy rescue, allows users to send messages to each other within the site without having to give away their other contact information. The problem is that nefarious spammer types were creating logins and immediately sending messages to large numbers of the user population, irritating them. After some deliberation, the client decided on a rate-limiting strategy, where a member could only send a certain number of messages in a day, and a new member could send even fewer messages a day. Messages above that point would require administrative action to unblock the user’s privileges.

I started with Cucumber tests — the first two are representative of the initial batch. Everything you see here is actual code, only slightly tweaked to anonymize details of the site. But I’m trying not to present simplified “example” code. That said, this is the final state of the code, there were some intermediate steps and false trails that I’m sparing you from having to read about. I’m also focusing on the tests here, rather than the resulting application code.

Background:
  Given the user database is cleared
  Given a group of recipients.

Scenario: Normal User behavior
  Given I am a user who is not a new member
  When I try to send 5 messages in a day
  Then 5 messages are sent
  And the administrator does not get an email

Scenario: Excessive User behavior
  Given I am a user who is not a new member
  When I try to send 6 messages in a day
  Then 5 messages are sent
  And I am blocked from sending further messages
  And the administrator gets an email

This test uses an implicit style, so a lot of the details are in the step definitions. I start by writing the step definitions one by one until I get to one that I can’t make work without writing new code. As it happens, since not sending email is the current state of the system, the entire first scenario should work without any new code. It’s very important to include that scenario, though, to ensure that the new changes don’t break the basic behavior.

Here’s what the steps look like — the user who is not a new member defers to the boilerplate RESTful Authentication step for logging in:

Given /^I am a user who is not a new member$/ do
  @user = Factory.create(:user, :created_at => 5.months.ago)
    Given "I am logged in"
end

Given /^I am logged in$/ do
  visit "/login"
  fill_in("email", :with => @user.email)
  fill_in("password", :with => @user.password)
  click_button("Sign in")
end

Next up, sending messages. Since the point of Cucumber is to treat the application as a black box, the step definition uses Webrat to simulate number of posts to the create message RESTful action. (The @recipient is created in the background action, which I didn’t show here because it’s not very interesting.)

When /^I try to send (.*) message(.*) in a day$/ do |count, plural|
  count.to_i.times do
    visit(messages_path, :post, {:recipient => @recipient.id,
             :message => Factory.attributes_for(:message,
                                :sender => @user)})
  end
end

So this step definition allows you to match “send 3 messages” and “send 1 message” by adding that little group at the end of the message, that group needs to have an associated variable in the block argument list, but it’s just ignored. Somebody with better offhand regular expression skills could easily make it so the end of the expression only matches “message” or “messages”, but I can live with it being a little overly matchy for now.

Closing out the normal case, the step definition to test emails sent uses the excellent
email_spec plugin, which creates RSpec matchers and Cucumber step definitions for email testing. This one checks an all_emails method, filters out any methods sent to the administrator, and makes sure the right number exist.

Then /^(.*) message(.*) (is|are) sent$/ do |count, plural, verb|
  emails_out = all_emails.select do |e|
    !e.to.include?("admin@admin.com")
  end
  assert_equal(count.to_i, emails_out.size)
end

The administrator email step definition uses the email_spec steps directly — and yes, I could have included those steps explicitly in the Cucumber scenario. I chose not to, on the perhaps dubious grounds that I wanted to keep explicit string literals out of the scenario. But it doesn’t make that much difference either way.

Then /^the administrator (gets|does not get) an email$/ do |status|
  if status == "gets"
    Then '"admin@admin.com" should receive an email'
  else
    Then '"admin@admin.com" should not receive an email'
  end
end

Everything here passes for the normal case. So far, my main note is that all the step definitions are really simple — it’s almost impossible to misinterpret them.

At this point, I move on to the blocking case. There’s one more step definition to write. In actuality, the “then 5 messages are sent” step will fail, since the blocking isn’t in the code. So, I would normally jump to writing regular tests at that point, but since I’m here, I’ll present the last step definition.

Cucumber is a black box, therefore in order to detect if the user is blocked, we need to find some place in the application that will display it. After their initial notification, the user doesn’t see anything confirming their block status. But the administrator does, via an admin screen that actually hasn’t been written yet. So, the cucumber tests logs the user out, logs an admin in, and checks the admin page for a row associated with the user.

Then /^I am blocked from sending further messages$/ do
  @sender = @user
  visit "/logout"
  Given "I am a logged in administrator"
  @admin = @user
  visit path_to("the admin messaging page")
  assert_select("tr#?", dom_id(@sender, :blocked_row), :count => 1)
end

This step definition is a bit more complicated than the others, and it’s also taking it slightly on faith that the row with the correct DOM id will actually have the information the admin needs. (The tradeoff in view testing, as always is faith an flexibility vs. certainty and brittleness…)

Okay, there are genuinely failing Cucumber steps, it’s time to make them pass.

A couple of things to note:

• After going back and forth on this some, I’ve decided that Cucumber tests don’t really replace controller tests, except in maybe the simplest cases. My rationale is that Cucumber and controller tests look at different error conditions, and that Cucumber tests are really (partially) replacing the hand-testing that I would be doing to verify that the feature is working. Cucumber is the 30,000 foot view of the application, I still feel the need to test from the ground.
• This app was a legacy with effectively no tests, leading me to have to write some extra tests to cover the normal, non-blocking case. If the whole app was TDD, then that functionality would already have tests, but since I’m working on the new features, I need to go back and cover the base functionality.

The basic ideas is write the tests in the class where the code is going to go. I can guess that code will need to go in the MessageController, the Message model, and the User model. In fact, I’m going to need a new field in the user model, a string representing message sending status. (It’s a string and not a boolean, because I know from a later requirement that there will be more than two states). I actually will wait to write the migration, though, until a test compels it.

My preference is to start with the controller tests — it’s the easiest way into the system for me.

Here’s the batch of tests I wrote to cover the normal sending a message and not getting blocked functionality. These tests will probably look weird to just about everybody since I wrote them using a combination of Shoulda, Zebra, and Matchy. Zebra gives nice one-line tests, and Matchy gives a sort-of RSpec syntax that I sometimes like.

context "POST CREATE" do
  setup do
    ActionMailer::Base.deliveries.clear
    @recipient = Factory.create(:user)
    @user = login!
  end

  context "with a clean user" do
    setup do
      post :create, :recipient => @recipient.id,
          :message => Factory.attributes_for(:message,
                :sender_id => @user.id)
    end

    expect { @user.messages_sent.size.should == 1 }
    expect { @user.spam_message_count.should == 1 }
    expect { assigns(:message).should_not be_new_record }
    expect { @user.should_not be_message_sending_blocked }
    expect { assert_no_email_to_administrator }

    should "create and send" do
      assert_sent_email do |email|
        email.to.first == @recipient.email &&
        email.from.first == "do_not_reply@singlestravelintl.com"
      end
    end

  end

### Outer context continues

What to say about these tests…

• In controller tests, I organize contexts by action. The outer context setup clears the email and creates a couple of users — the login! method creates a factory user and simulates a login.
• The internal context posts the message
• The various expect blocks each resolve into a test that passes if the block returns true. The first one checks that a message has actually gone into the database (messages_sent is an association on user). The spam_message_count is what is used to determine if a user is blocked — that’s not going to pass yet, because the concept of a spam count is new to the app. The third tests that the message assigned in the controller is actually saved. The fourth checks the message_sending_blocked? method of a user, which is also going to need to be written. The last calls a helper method to determine if an email is sent to the administrator, which should only happen if the user is blocked.

Many of these tests pass as is. The ones that don’t are dependent on the new user features. Which means we need user tests.

I realize that some of you are starting to think that having all these tests is ridiculous overkill — I mean, you’ve got your Cucumber tests, you’ve got your controller tests, and now model tests. That’s, like, triple the work, isn’t it?

No, I don’t think it is. My rationale for doing all this testing goes something like this:

• In my head — which is, admittedly a strange place to be — these are all different tests covering different aspects of the same feature. They could all fail independently of each other, at least in theory. Each of them is exercising a different part of the code, and crucially, each of them is responsible for potentially testing error conditions at in their own space.
• It’s not that much extra time — I’m presenting the code in larger chunks than I actually wrote it (can you imagine how long this would be otherwise?). In the rhythm of testing, it’s a very short piece of test, followed by a very short bit of code, and the amount of tests grows over time.

Anyway, the user tests cover the case where a user has four messages, and then the transition to five messages. The setup is pretty similar (another reason why the extra tests don’t take as long to write as you might think)

context "rate limiting" do

  setup do
    ActionMailer::Base.deliveries.clear
    Timecop.freeze(Date.today)
    @user = Factory.create(:user, :created_at => 1.month.ago)
    4.times do |i|
      Message.new_from_params(
          Factory.attributes_for(:message,
              :created_at => i.hours.ago, :body => "Message #{i + 1}"),
          @user, Factory.create(:user)).save!
    end
  end

  should "correctly count messages" do
    @user.spam_message_count.should == 4
    @user.update_message_block_status.should be_nil
    @user.should be_able_to_send_messages
    assert_number_of_emails_to_administrator(0)
  end

  should "move user to blocked mode after fifth message" do
    Message.new_from_params(Factory.attributes_for(:message,
        :created_at => 10.minutes.ago), @user,
        Factory.create(:user)).save!
    @user.update_message_block_status.should be("blocked")
    @user.should_not be_able_to_send_messages
    assert_number_of_emails_to_administrator(1)
  end

## Outer context continues

Most of this should be clear given all my blabbering so far (and I have no real clear reason for abandoning Zebra in this section of tests). One thing is that the method update_message_block_status is actually responsible for changing the user status after a message is sent.

This is getting way long, so back next with with how this played out with new requirements, and a kind of interesting bug.

Quibbles with my testing style or process should go in the comments.

Related Services: Ruby on Rails Development, Custom Software Development

Related posts:

1. Real Testing Example, Part Two
2. The Return of the Cucumber
3. Using Cucumber for Acceptance Testing

Cover image from Amazon.com

It's been way too long since I blathered on about style issues. Today I'd like to talk about testing style. This article assumes you are already writing tests and already using something approaching a Test-Driven Development process -- I'm not here to argue about process, at least not today.

Today the topic is the actual construction of individual tests, how to name them, how to group them, where to get data from and the like.

I suspect I'll think of five more things right after I post this, so look for an update sometime in the future. The update will also address all the places where everybody tells me that I'm totally wrong.

Name tests after their expected behavior

The first part of each test is its name. By now, there's no excuse for still using the def test_underscore_instead_of_spaces naming convention. Rails 2.3.x, Shoulda, and RSpec all support a style where the test name can be an actual string with actual readable spaces and everything.

A test should be named after the expected behavior it verifies. Being specific is helpful here -- the name is potentially important documentation.

GOOD: test "should use last_name, first_name as display"

LESS GOOD: test "should correctly display names"

BAD: test "name display"

If you can't think of an easy name to describe the behavior being tested, this is a strong hint that the test is either a) richly deserving of being split into multiple parts or b) unneeded. Most of the time, it's (a).

Use contexts to group related tests

I'm also at the point of feeling like there's no good reason not to use contexts. If you don't want the weight of using Shoulda or RSpec, then just use the Context gem to get simple contexts.

So, what gets grouped in a context?

• A set of tests that all share a common data setup
• A set of tests that exercise a common method, as in controller tests that all work on the same action

Obviously, those two options tend to go together.

The context name should reflect the commonality -- either the name of the method being exercised, or something denoting the common data setup, like "as an administrative user".

Nesting

Contexts can also be nested, allowing for setups to be extended. I don't have very strong ideas on when nesting contexts is advisable, my current guidelines look like this:

• Only nest a context if the inner context uses all of the outer context setup and then adds on (similar to the guideline of only creating an Object subclass if the subclass uses 100% of the parent class data)
• Try to keep the distance between the different setup methods as small as possible -- the problem is that having to trace setup code up multiple pages and across several generations of contexts is a real pain.

Actually, the problem I'd really like a clean solution for is what to do if I want the inner context setup to be run first -- for example, a controller action that I want tested for multiple user types. The controller action is logically in the parent context, but the login as the various user type, logically in the inner context, needs to happen first. Putting a large context for each user type separates the tests for each action, which I'd rather avoid. Anybody smart got this one solved yet?

Setting Up Data

The guideline for creating data for tests is simple: create the minimum amount of unique data that you can while still having a meaningful test. In many cases, a model unit test will only require one object, and in many of those cases only a small number of attributes are actually relevant to the test. (One of the advantages of factory tools is the ability to highlight just those relevant attributes).

Even search and report methods can often be unit-tested with only two models in the database -- one to be found, and one to be ignored. (You'll often need more than one test, but each individual test is small and focused on one unit).

Setup should be as close to the actual test as possible, granted that it often makes sense to group repeated patterns into setup methods.

Single Assertion Style

Shoulda has popularized a testing style where most of the test is in the setup and each individual testing method has a single assertion. I have mixed feelings about this style, though I use it sometimes.

Guidelines:

• If you are going to do single-line tests, definitely use a tool that optimizes single-line tests like Zebra or Testbed.
• I think single-assertion tests work best when setting up relatively simple model tests that don't require a lot of nested contexts.
• Especially in controller tests, single assertion tests can lead to a lot of extra calls to the setup and a slower test suite, so that's something to keep an eye on.

When tests should be DRY and other fascinating questions?

The key thing about tests is that they are also code, so pretty much every style issue that applies to regular code also applies to tests.

That said, tests do have a special place in your application in that they are the part of your code that doesn't actually have tests of their own. So there are some special guidelines for tests.

Don't get cute. There's an old programing truism that if debugging is harder than coding, then by definition any code that is as clever as you can write it is impossible to debug. Similarly, any test written at the edge of your ability is going to be impossible to verify. It's not that metaprogramming and other fancy stuff has no place in testing -- Shoula macros depend on metaprogramming, for example. It's that you probably don't want an individual test to depend on the subtleties of Ruby's object system.

You can repeat yourself a little. I place a higher value in tests on being able to read the whole test in one glance than I do in regular code (I expect regular code to be spread across various methods, I expect a test to be self-contained). As such, I tend to allow somewhat more code duplication in tests than I do in application code, especially in setups. That said, keeping setups short has a value as well, and when the setup is creating enough objects that it's obscuring the point of the test, I'll refactor the setup to a helper method and reuse that. But in application code, I'll generally blast duplication immediately, but in test code, I'll often wait until I have some pain. Is that a good thing? Not sure.

Your turn. Where am I wrong?

Related Services: Ruby on Rails Development, Custom Software Development

Related posts:

1. The Testing Interviews
2. Ask A Rails Tester Person
3. Elements of Ruby Style

Recently I have taken a bit of a detour into the world of telephony, working with Asterisk-Java, which by itself is a very valuable tool, and worth knowing a bit about if you are integrating a system with Asterisk. While it is a Java-based library, I am integrating it into a Grails application.

We have a fairly comprehensive suite of unit tests asserting that desired behaviors are triggered upon certain events initiated through the Event API. This is made even easier, as usual, with Groovy-- specifically on the testing front. Here I show two hypothetical test cases, followed by supporting code that works specifically with the Asterisk-Java classes...

 
import org.gmock.GMockTestCase
import org.asteriskjava.manager.event.JoinEvent
import org.asteriskjava.manager.event.LeaveEvent
 
def queueObject = // .... some object representing an internal queue
 
/*
 * Verifies caller entering a call queue will trigger behavior
 */
void testCallIAddedToQueue() {
  mock(queueObject).add('SIP/123', '3125551234').once()
  dispatchJoinEvent([channel: "SIP/123", callerId: "3125551234"])
}
 
/*
 * Verifies caller leaving call queue will trigger behavior
 */
void testCallRemovedFromQueue() {
    mock(queueObject).remove("SIP/123").once()
    dispatchLeaveEvent([channel: "SIP/123", dateReceived: new Date()])
}
// ...
 

The testcases above are fairly straightforward, hiding much of the details involved when working with Asterisk-Java. The 'queueObject' can be anything, really, as long as it contains behavior which we wired into the event-handling system Asterisk-Java provides. The important part here is how we can easily bind properties to the underlying Event types, and dispatch them so that our behavior is invoked.

 
private def dispatchJoinEvent(properties) {
  dispatchEvent(JoinEvent, properties)
}
 
private def dispatchLeaveEvent(properties) {
  dispatchEvent(LeaveEvent, properties)
}
 
private def dispatchEvent(eventType, eventProperties) {
  def event = eventType.newInstance([:])
  eventProperties.each { name, value -> event."${name}" = value }
  play {
    asteriskService.connection.dispatchEvent(event)
  }
}
 

Here I have a 'service' class defined in Grails which I call 'asteriskService'. It encapsulates access to the Asterisk-Java ManagerConnection instance, which in turn can dispatch specific events.

Of interest to those new with Groovy is the binding of properties within 'dispatchEvent()'. It looks perhaps a bit more complex than one would hope, but unfortunately JoinEvent and LeaveEvent do not have default constructors, requiring the code to 'new' them up with a placeholder argument, and then set any properties on the fly.

Asterisk-Java is worth checking out, but particularly nice is using a bit of Groovy to tie it in nicely. While this can benefit the application, you can see how it helps ease testing-- particularly as you start using more and more of what the library offers.

Related posts:

1. GWT Testing With Groovy? (Heck yeah!)
2. Grails: Delegating to GORM Persistence in Java
3. Groovy 1.6 and Per-Instance Metaclass

I mentioned last week that the RubyMine post was replacing what I had meant to write about. Well, this week we finally get to it...

It's been about ten weeks since I wrote about Cucumber the first time and the second time. Since then, I've continued to use Cucumber and now seemed like a good time to update some thoughts on how and why it seems to be working for us.

The big headline, of course, is that I'm still using it after ten weeks. I'm pretty quick to abandon tools that aren't pulling their weight, so just the fact that Cucumber is still in the toolbox means that despite the time that it takes to write Cucumber tests and step definitions, I'm finding the process of writing the tests and the tests themselves to be valuable.

Where I'm using it

Right now, I've been using Cucumber primarily on two different projects.

• An internal project that has a well-defined testing process, and 100% coverage for unit tests on models and for functional tests on controllers.
• A client project that we took over from the original coders, who wrote no tests at all. All new features come in with tests, but we haven't

Cucumber plays out a little differently in each case.

How I'm Using it

I struggled a bit with how to integrate Cucumber into the testing process, and eventually came to the following rule of thumb:

Just as all lines of code should be in response to a failing unit or functional test, all new unit or functional tests should be in response to a failing line of Cucumber.

The process seems to go something like this:

1. In conjunction with a stakeholder of some kind, write the cucumber tests for the feature, usually this will be multiple scenarios. At this stage, don't worry about setting up data, and I think you're better off focusing on making the test as human-readible as possible, but see below for some thoughts in that direction.




2. Start writing step definitions for the Cucumber tests, step by step. In many cases, you'll find yourself adding some steps to create data (especially to the background step of the feature). Especially on a mature project, you'll often be able to write several steps without needing new code in the app itself -- "Given I am a logged in user", "When I visit the project page"... in essence, the setup.




3. In the fullness of time, you'll come to the meat of the feature, and you won't be able to make the Cucumber line pass without writing application code. At this point, you drop into a normal TDD process, with the goal of moving the application forward enough to make this line of Cucumber succeed.




4. What happens here depends a little bit on personal taste and the project team. For the internal project, which has a goal of 100% functional test coverage, I do write functional tests even if they overlap the Cucumber test strongly. On the legacy project, I do sometimes drop do the model/unit tests from the Cucumber test if there doesn't seem to be a substantial difference between the acceptance and functional test.




5. When I think the regular TDD process has gotten me enough code to pass the Cucumber line, then I run Cucumber again to verify, and move on to the next line.

This process has a lot going for it. The TDD part is always focused on a specific, definable goal -- the next line of Cucumber test. The cycles between acceptance and regular testing are short enough that the two feed into each other (sometimes the Cucumber tests change in response to the shape of the code, and sometimes vice versa), and they work together to keep the code simple, complete, and covered.

Cucumber vs. Functional testing

The big question is to what extent the Cucumber tests duplicate or replace functional tests. At the moment, my answer is somewhat, but not completely. I see the Cucumber tests as being the way to go for testing a work flow of more than one one action, but I still find it a little easier to set up data in the functional test framework. That said, I don't think that adding the functional tests has cost me that much more time, and there are still enough gaps in Cucumber/Webrat (most notably Ajax) that I'm reluctant to give up controller tests entirely. I expect my thinking on this will change as we have projects that had Cucumber from the very beginning.

General vs. Specific

I'm having a stylistic issue with the Cucumber tests between putting the specifics in the Cucumber or in the step definition. This came up most sharply in the legacy project. We were working on a feature that made a distinction between new users and old users. My first pass at the Cucumber test started off like this:

Given a user named "Noel Rappin" who has been a member for 5 months

With a step definition that was something like this:

Given /^A user named "(.*)" who has been a member for (.*) months$/ do |name, mo|
  @user = Factory.create(:user, :name => name,
      :created_at => mo.months.ago)
end

The idea was that "5 months" was just a stand in for "an old user", but the client, quite understandably, wanted to know what the deal was with five months -- the specificity of the rule seemed to imply that it was important functionally. Eventually, I changed the rule to be more like this:

Given /^A user who is not a new member$/ do
  @user = Factory.create(:user, :created_at => 5.months.ago)
end

Once I saw this issue, I started seeing it everywhere. The second form has the advantage of being more general and more requirements like, and the disadvantage of hiding a lot of magic detail in the step definition. The original version has the advantage of putting specific data front and center and the disadvantage of not making it clear which details are important.

I'm not aware that there's a consensus on the style issue. My tendency, right now, is to favor the general form on the theory that it's a good thing to make the Cucumber test as readable as possible and let the step definitions sort it out. On the other hand, this seems like a good way to get really messed up by having the step definition not quite match what you expect, and have that be hidden.

Related Services: Ruby on Rails Development, Custom Software Development, Agile Development

Related posts:

1. Cucumber Rocks – But it’s not a replacement for unit tests
2. Using Cucumber for Acceptance Testing
3. Again With The Cucumbers

Cucumber is also an excellent choice for getting coverage on a legacy code base. Many of our clients have existing applications that were developed by cheap firms. They finally realize that they should have gone with a development company that writes automated tests, and works in a true agile manner. When initially looking at the code, we often get chills. There are usually zero tests. Most of the code sits in huge controller methods. We generally will use cucumber to start to get coverage for the features that the client wants changes to. We do write unit and functional tests, but we generally only assert the things we are changing. Over time, the unit and functional test coverage increases, and when we make changes we generally will refactor lots of the existing code base.

With that said, I have seen the use of cucumber on some of our projects change the way some developers code. They go from true TDD to crappy non-TDD. Many methods on the model don't have unit tests! How can we call this TDD?

I do admit that the functional tests can be a lot lighter, as you are getting coverage from the cucumber suite of tests. However, It doesn't mean that you can skip writing unit and functional tests.

This is how I incorporate cucumber into my development, while still maintaining 100% code coverage.
1. Write cucumber tests from the acceptance tests from the requirement
2. See all of the tests fail
3. Write unit tests, see them fail, write the code until the unit tests pass.
4. Write functional tests, see them fail, write code until the functional tests pass
5. Continue until I think the feature is fully implemented
6. Run the cucumber tests and see if anything still fails. - Add unit/functional tests to address failing cucumber tests. Once all tests pass, run the test coverage to ensure there are no holes (this generally doesn't happen, as I am practicing TDD).

Related Services: Ruby on Rails Development, Agile Development, Custom Software Development

Related posts:

1. The Return of the Cucumber
2. Using Cucumber for Acceptance Testing
3. Again With The Cucumbers

Same Photo by Nick Atkins Photography

My cucumber obsession continues unabated. I've spent a lot of the last week adding Cucumber to projects, writing about it in a very long section for the Rails Test Prescriptions book, and generally trying to figure out how to use this tool, not to mention trying to figure out why this tool has been so interesting to me this week.

Some follow-up thoughts since last week's post.

• I'm really excited about the idea of using Cucumber as a way to bootstrap testing on a legacy project that has no tests. The normal problem is that you need to refactor in order to write good tests, but since you don't have good tests, you have no way to prove that your refactor didn't break anything. Cucumber's advantage here is that it can treat the Rails application as a black box, so you can throw in some Cucumber tests without touching the original code. This will give you some trip wires that will let you know if something gets really broken in your refactor. Plus, it's a good way to learn your way around an unfamiliar project.
• For what it's worth, I seem to be getting a fair amount of reuse out of step definitions within the same feature. That is, after I've set up most of the baseline scenarios, if I think of a new oddball one, most of the steps that make it up are already defined.
• After pointing out last week that my first real Cucumber example actually passed in Cucumber after failing in the browser, I need to mention that this week I had a big Cucumber success, where my unit tests passed, but the browser and Cucumber didn't behave correctly. In this case, the unit test was performing operations in a different order then the browser code, and created a false pass.
• I'm not completely sure why, or even whether I'll feel this way in a few days, but Cucumber view tests feel much more readable than pure Test::Unit view tests, even with the indirection of the step definition. Even if Cucumber was just a DSL for creating readable view tests, I still think it'd be useful.
• I don't think there's a definitive sense of what great Cucumber style is, but, at the moment, I'm skeptical of the advantages of bundling Cucumber steps into the step definitions. I appreciate the advantages of reducing duplication, but I think that Cucumber is most valuable if the actual feature page is as open and accessible as possible. I'd rather have more steps in Cucumber with short definitions than fewer steps and more complex definitions.

Related posts:

1. The Return of the Cucumber
2. Cucumber Rocks – But it’s not a replacement for unit tests
3. Using Cucumber for Acceptance Testing

Photo by Nick Atkins Photography

An consistent nuisance problem when testing Rails applications is the "unit test gap". This happens when the model test passes and the controller test passes, but the application as a whole fails because there's a mismatch between the output produced by the model and the input given to the controller test. In theory, Rails integration tests can solve this problem, but they aren't really designed for it, and nobody uses them much anyway.

An easy-to-use tool that solves that end-to-end testing would be great. Somebody should really write a blog post about that.

So, as I was saying. We continually run into an issue with our clients over defining requirements at the level that developers need to keep going, without getting bogged down in long stretches of design. There's a big gap between "users should be able to upload photos to their pages" and all the different details of permissions, validation and the like than need to be answered at some point. The Agile process suggests that those requirements be created as closely as possible to the code, which leads to the question of how best to keep the customer in the loop while the developers need these decisions to be made.

An easy-to-use tool that lets users read or create requirements that developers can build from and run as acceptance tests would be great. Somebody should definitely write a blog post about that.

Which brings us -- finally -- to Cucumber, a tool for creating automated acceptance tests. It's flexible enough to solve both problems. It can be used as a developer tool to drive regular TDD testing, and as a client tool for managing requirements.

It's got loads of potential. And some pitfalls and tricks. The Cucumber web site has great documentation on installing and using. I want to focus on how Cucumber can work within a development process.

Here's a Cucumber test:

  Background:
    Given I am logged in
    Given I am in a project

  Scenario: Make High Priority
    Given I have a task with medium priority
    And I am on a page that displays tasks
    When I click on "priority_up" for that task
    Then the task should display high priority

The structure of a Cucumber test is pretty simple. You've got your "Given" lines, which specify prerequisites, then you've got your "When" lines which are user actions, finally the "Then" lines list results. The Background items are loaded before each scenario in the feature, and therefore consist primarily of "Given" lines.

This test is actually runnable as is, although Cucumber will just tell you that it doesn't know what any of the lines actually mean yet. The Cucumber test runner output, by the way, is outstanding. Color coded results of each step, tied to the line where each step is defined, a nice summary, and snippets of code to insert for undefined steps. The output goes a long way toward making Cucumber easy to use.

In order to make this test actually work, you have to define it. Cucumber gives you a handy file to place the definitions in. Cucumber task definitions are Ruby code. Here is the definition for the first step in the scenario above:

Given /^I have a task with (.*) priority$/ do |priority|
  @task = Task.make(:priority => priority, :project => @project,
      :feature => @feature)
end

Let's break this down a little bit. The starting point of a Cucumber step definition is a definition line containing a regular expression and taking a block. Any group specified in the regular expression is passed as an argument to the block (and matching text is very helpfully presented in bold from the command line output).

In this particular case, the text "I have a task with high priority" matches this step definition, and passes "high" to the block, where it's used to create the new task (using the Machinist fixture replacement plugin -- Cucumber plays very nicely with all the fixture replacements).

Here are the definitions for the other three steps.

Given /^I am on a page that displays tasks$/ do
  visit("/tasks")
end

This one is pretty straightforward, and uses Webrat to simulate a browser hit to that specific URL.

When /^I click on "(.*)" for that task$/ do |button|
  #click_link(dom_id(@task, button))
  if button == "priority_up"
    visit("/tasks/#{@task.id}/upgrade")
  elsif button == "priority_down"
    visit("/tasks/#{@task.id}/downgrade")
  end
end

The commented line here is what I would do if it was a normal link -- again using Webrat to simulate the link. However, Webrat doesn't follow rails link_to_remote links (at least not yet), which is what I actually have in this project. So I'm faking it with a truly ugly hack. I'm assuming there's a better way to do this.

Finally, we get to the actual assertion.

Then /^the task should display (.*) priority$/ do |priority|
  response.body.should =~ /priority_task_#{@task.id}/
  response.body.should =~ /#{priority}/
end

Since the response here is an Ajax response, all I can do is a match on the output. If it was ordinary HTML, I could use the full weight of assert_select and the like.

With these definitions in place, the tests will run and fail. You can then spin off into your regular TDD process to make everything work. In practice, I actually do this one step at a time. Write the step definition, then make it work, then on to the next step. Over time, this takes the place of some controller and (especially) view testing. Also, you should fine some ability to reuse step definitions.

I have several points to make about this.

• It seems like the Cucumber steps should be written from the user perspective in plain language as much as possible. So "I should see a high priority icon" rather than, "the page should contain high_priority.jpg". Along that line, thought the Given statements probably do need to touch the database, the When and Then statements should work entirely at the user action and HTML response level. Complete end-to-end testing is, to me, the place where Cucumber has the most to offer.
• It's also clear that the amount of efficiency Cucumber adds or subtracts from your development process is largely dependent on how well you write the step definitions. The Given and When lines need to be as simple as possible, and everything relies on the Then lines giving an accurate representation of what the application is doing.
• Cucumber tests have the same problems as view tests in general -- if they are detailed enough to be fully exact, then they are probably too brittle, but a more robust test (based on DOM ID, for example), could easily be fooled by a sloppily written view layer.
• The exercise of creating the Cucumber tests is, all by itself, quite useful in specifying exactly what the application is going to do. Unlike a lot of other requirements definition languages, Cucumber is really low-ceremony and easy to generate. So far, I've mostly used Cucumber in projects where the developer, not a client is generating the tests. Even then, the process of thinking these scenarios through was very helpful in driving development. Having the tests be runnable was a bonus.
• That said, it's not hard for the tests to basically lie to you. The batch of tests referenced here is from a feature I actually built using Cucumber. First time I tried it in a browser, with all the Cucumber tests passing, the browser failed. For an interesting reason. The priority column was a new addition, and the code assumed that the tasks would have one of three values. Existing tasks however had a nil value, which broke the code. A rake task to preset the existing tasks fixed the problem, though. So -- this is not a substitute for actual human testing.
• Which leads to another question about Cucumber -- where to draw the line between Cucumber and existing tests. It seems to be that Cucumber is not the place to test all your error conditions (although if an error condition has a user-visible component than a generic error test is probably appropriate). I'm not sure I've completely found the groove as to when a test goes in Cucumber and when it goes into a controller or view test -- model tests still do seem to be completely separate, I had no trouble writing model tests off of my Cucumber scenarios. (I know that some people are using Cucumber to replace controller tests -- I'm not convinced yet that's the way to go.
• I'd be very interested to see how this played out on a larger project. Would the number of feature files become unwieldy? Would multiple developers be continually re-inventing the step definition wheel? Would that matter?

One thing that kept me from trying out Cucumber in the past was a variant on the same thinking that keeps people from testing in general - the fear that you're just going to be writing more code, and taking more time, to little or no benefit.

My experience so far, as I explore what Cucumber can do, has been largely positive. Where I was starting with Cucumber and only a vague idea of how the user interaction would play out, writing the scenarios at the Cucumber level felt very valuable and gave the development a clear path that I wouldn't have otherwise had. That said, there is extra code being written, and it's clearly possible to get really tangled in getting the step definitions right.

As developers, we're kind of conditioned to believe that if we're solving a complicated problem, we're doing something valuable. At it's worst, writing the step definitions felt like a complicated problem that wasn't adding much. Luckily, a lot of that was due to my own unfamiliarity with Cucumber. As I got better at it, most of the step definition part came quickly. Ultimately, I think I wrote better code as a result of starting with the Cucumber definitions -- it definitely kept me from writing unneeded code, since most everything flowed from a Cucumber step.

I definitely recommend giving Cucumber a try -- the potential for really cleaning up what is often the messy process of determining what a feature should do is very high. I'm really looking forward to reading (and writing) more details of what successful strategies with Cucumber look like.

Related Services: Ruby on Rails Development, Custom Software Development

Related posts:

1. The Return of the Cucumber
2. Again With The Cucumbers
3. Cucumber Rocks – But it’s not a replacement for unit tests

This is the cover

Rails Test Prescriptions: Keeping your Application healthy is now on sale. This is a beta, partial release.

Rails Test Prescriptions is a comprehensive guide to testing your Rails application, covering both the mechanics of writing tests and the style for writing good and useful tests.

I'm excited to be publishing this electronically in a way that allows me to keep the book up to date as Rails changes over the upcoming months and years. Your $9 purchase entitles you to all updated versions for the life of the book.
The current table of contents looks like this:

A basics section:

• Basics—creating test files, running tests
• Model tests
• Controller tests
• Integration tests
• View tests
• Fixtures
• Helper tests
• Mocks and stubs

The more detailed section currently has:

• Refactoring setups
• Testing style guide (incomplete)
• Fixture replacements
• Testing with Date/Time
• Faker

Planned for the future:

• Third-party view testing
• Security testing
• route testing
• Ajax testing
• Mailer testing
• File Upload Testing
• Refactoring assertions
• Shoulda
• RSpec
• Cucumber
• Webrat
• Dealing with legacy projects
• Autotest
• Tarantula
• Heckle
• RCov

Related posts:

1. More Kudos for Rails Prescriptions
2. Rails Test Prescriptions to be Published by Pragmatic
3. Announcing Rails Prescriptions

Most of the developers at Pathfinder Development are old hands. We've been around the block a few times. We've worked on quite a number of software projects over the years. We've all gravitated toward agile development not because it's the latest buzzword, or because it feels so good (though it does when it's done right), but because everything else feels so bad.

Take testing, for example. Testing girds you, the product manager, for the moment of truth -- deploying or shipping a new release. In the monolithic or waterfal processes, we would often wait until the end, after all of the development was done to "QA" the application. Often you'd throw in some performance testing to see if this monstrosity you'd just built would actually handle the load you were going to throw at it.

After weeks or months of testing, fixing, hoping, praying and integrating, you'd finally deploy. Odds are you'd ship with some pretty serious bugs. But the quality of the software, in terms of bugs and errors, was pretty poor and the expense and effort of the developers fixing the bugs was akin to the contractor applying spackle to a drywall that was more hole than whole.

Let me go through some of the testing we do today, what the previous alternative was, and what the benefit for your product is.

Continuous Integration Rocks

Every time you check in some code, the entire system builds, is tested and deployed to a development environment, then tested some more. This is the heartbeat, the core of the Agile feedback loop. The alternative was building part of the application on you local environment, occasionally checking out code from other developers, and doing a painful build and integration at the time of deployment that could take weeks or months to work out the kinks. You would not discover basic compile time problems for days or even weeks.

For the product manager the equation is simple: bugs are more expensive to fix the farther they get from the time of original development. Automate your tests and you find things out right away.

TDD and Unit Testing Rocks

We work in OO languages for the most part, so Unit tests are a natural fit for the classes we develop. We practice test driven development -- writing the tests first -- since we know that writing tests afterwards is like making tea with a used tea bag. We strive for 100% line and branch coverage. Every time we identify a bug, we write a test for it.

Agile testing is cumulative, each iteration produces automated tests that protect your code from unintended side effects in the next iteration. Since most software has more development effort expended on it after it has been released than before, the cumulative tests act as an early warning system against those side-effects. I think of all those times that a change broke a test and therefore broke the build during an iteration. In the old way of doing things that was a bug that no one would have noticed and that likely would nevery have been detected or fixed.

For the product manager, automated, cummulative tests mean you can quickly and confidently modify your product to meet changing market demands without having to worry about and brittle and buggy code base.

Releasing Working Code Every 2 Weeks Rocks

In the old days we would wait until close to the end of the project or phase before bringing all of the code together in a working, useful system. That pretty much made testing something you could only do at the end. Now we release a system every 2 weeks that is useful in the business sense and can be tested and critiqued. Can this be done? Well, if you are designing an air traffic control system and you need all of the features at once -- you want both the take off safely and land safely user stories at the same time -- then Agile may not be for you. But for the other 99.9% of software applications, releasing every 2 weeks is very much doable.

For the product manager this means you can test and measure whether you application performs up to spec from the beginning and as the application grows and evolves. You can do usability testing throughout. You can do security testing throughout. Knowing whether you application so far meets the spec allows you to predict when you will be done with development to a much greater precision as you approach the end of development.

Releasing Working Code Every 2 Weeks Rocks Again

Traceability. How I love that word. In the old days, keeping track of whether you tested something was a major headache. On a project with 12 developers and several months of development you sometimes lost track whether something had been tested, even with spreadsheets and project plans. Today we automate and never bite of more than we can chew in 2 weeks. That means the tester and the developer work or even pair together to ensure that our little collection of features has all been tested.

For the product manager this means less headache verifying for compliance or auditing that everything that the team said it had tested has in fact been tested. More peace of mind. Fewer ulcers for the product manager.

Is Sliced Bread Really so Wonderful?

For those practicing Agile already, the benefits listed above may seem rather short (there are many other benefits beyond testing) and mundane. After all, this is all stuff you take for granted. But for those product managers still wallowing in big, monolithic processes or halfheartedly adopting aspects of Agile: jump in, the water's fine. You'll notice when the pain stops and then, hopefully like childbirth, you'll forget all about it.

I've only scratched the surface here on Agile testing. To my readers: any notes or observations on testing you'd like to share?

Related posts:

1. Agile Development and Testing: Unit or Montecarlo?
2. Cucumber Rocks – But it’s not a replacement for unit tests
3. 800 on Your Math SAT, Software Development and Bugs

my economics final by "diabetesisfun"

So, if you're lucky, you've mostly missed another Internet pile-up about unit tests and so on, this one started by Joel Spolsky and some comments he made on his podcast.

I've already written a little bit about what Joel said, and I don't want to rehash that. But I have seen a recurring theme in arguments that testing is overrated -- namely that the testing that those people are writing about doesn't seem to be related to the testing that I actually do.

I'm going to put the summary up front:

• "Unit Testing" and "Test Driven Development" are not identical concepts
• It would seem, on the available evidence, that those of use who are TDD or BDD backers haven't done a great job of explaining this.
• TDD has code quality benefits that are completely separate from whatever function it has as a code verification engine.
• It would seem, on the available evidence, that those of use who are TDD or BDD backers have done a really poor job of explaining this.

In my head, Test Driven Development is a process where you write a short little test, write a short little bit of code to pass the test, refactor and move on. There's constant feedback between code and tests, and you move between the two freely.

This is probably an over-generalization, but it seems like "Unit Testing" often refers to a process where you write a lot of code -- days or weeks worth -- then go back and try to write tests against it. That does sound painful. You can never quite remember what the code was supposed to do, and retrofitting tests to existing code is never fun. If that was the only kind of testing I was familiar with, and some yahoo came around shouting about how I needed to test 100% of the time, I'd probably think that guy was crazy.

Luckily, that's not the way I test or the reason I test.

Put plainly, I write code test first because I write better code. And I write it faster. And when there are bugs, I find them quicker. And I enjoy coding more. Your milage may, of course, vary, but my milage from TDD is consistently good.

I write better code: Code written using TDD has a very distinctive style and structure. Smaller methods, smaller classes, very strongly decomposed, not a lot of duplication, not a lot of side effects. Mostly, this is due to the way in which small methods that do one thing are easy to test -- the small structure evolves organically from the co-existence of the tests.

I write code faster: Here's an idealized thought experiment. Let's say I'm writing a browser feature, and it takes, say, 30 seconds for me to exercise the feature in the app. Let's say it takes a full minute to write the initial test, but only a few seconds to run the test once it's written. How many times do I have to check what I've written before the test pays for itself? Plus, the test is always there when I need to come back to the feature.

I find bugs faster: Even granted that TDD doesn't catch all possible bugs, my experience in TDD-developed systems is that bugs are easier to find and fix, largely as a result of the smaller, less duplicated structure. Generally, the bugs are more localized

I enjoy coding more: Because it feels like a game. But hey, that's just me, right?

I leave you with this Rails testing Haiku, courtesy of Pathfinder's own Alice Toth:

flowing dots on screen
unbroken but for an F
my work is not done

Feel free to leave your own testing Haiku, or anything else, in the comments.

Related posts:

1. Test-Driven Development As Part of the Process
2. Web 2.0 Expo: Behavior-Driven Development with Rails and RSpec
3. Rails Test Prescriptions is now on sale

Geoffrey Grosenbach of PeepCode and Topfunky.

Gregg Pollack of Rails Envy, the Rails Envy Podcast and Rails Activist fame.

Enjoy!

Related posts:

1. The Testing Interviews
2. Getting Started With Rails Testing Guide Now Available
3. Rails Test Prescriptions to be Published by Pragmatic

These interviews will be just as important as Frost/Nixon

Over at the Rails Prescriptions blog, I'm going to be posting a series of interviews with various and sundry Rails folks -- I'm cross-posting the first one here. To get the series started off, I've decided to use myself as a guinea pig and see if I can hold up to the grilling.

A couple of quick notes before I get to it.

• The free e-book Getting Started With Rails Testing is now available. Check it out.
• Those of you lucky enough to be in the sub-zero Chicago area can come to theChicago Ruby.org meetup Saturday, January 17th to hear lots of interesting stuff, including me talking about testing and Rails some more.

And on with the interview.

How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?

I started with automated testing very shortly after reading Kent Beck's original XP book. The XP book came out at a time when I was very receptive to the ideas -- I had just come off a project that had suffered from a lot of regressions and all the other kinds of pain that XP promised.

It didn't take long for me to see how much better my code was when I did a lot of testing... what was more surprising was how much more fun writing code test-first turned out to be. The quick feedback and the ability to clean up code with confidence turned out to be really satisfying.

What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?

When I'm adding a new feature, I tend to start with a skeleton controller test that validates that the controller sets the right variables. I don't put much code in the controller, so then I move to testing the model for the new functionality. If the view logic seems to require it, then I'll add view tests after I get the view in place. I go back and forth between test and code pretty quickly, but sometimes the code will get ahead of the tests, especially when doing view layer stuff. I don't use integration tests very much at the moment.

I seem to have moved back into the core Rails test features recently, although I still use Shoulda for contexts and for the additional assertions. My most recent project used core Rails plus Matchy. I've also been using the various factory replacements for fixtures, which I like quite a bit.

What's the most interesting thing you've discovered about testing recently?

The biggest change I've made recently is using factories for generating test data, which makes the tests much more readable and stable by keeping the setup closer to the actual test.

Is there a tool you wish you had for testing that you don't think currently exists?

I wish I had a really good way of validating view logic, none of the ones I've tried have been completely satisfying. It'd also be nice to have more sophisticated coverage reports. Of course, these things might actually be impossible...

What advice would you give somebody looking to write more effective tests?

Automated testing is much easier and more valuable if you keep a tight feedback loop between your tests and your code.

Related posts:

1. More Testing Interviews
2. Rails Test Prescriptions is now on sale
3. Getting Started With Rails Testing Guide Now Available

I'm pleased to announce that the getting started guide, conveniently titled "Getting Started With Rails Testing" is now available for download at the Rails Prescriptions site.

It grew to be a bit longer than I originally intended, but I'm very happy with how it turned out. It starts with an empty Rails application and walks through the first couple of feature additions, adding tests for controllers, models, views, and basic security.

Download, read, enjoy, tell all your friends, let me know what you think.

Related posts:

1. Rails Test Prescriptions is now on sale
2. More Kudos for Rails Prescriptions
3. More Testing Interviews

It's at 3pm, location and other information is available at their meetup.com site.

Looking forward to it -- see you there!

Related posts:

1. Why Chicago is Rails-town, USA
2. The Testing Interviews
3. Testing various roles in ruby on rails