Live Ruby: Testbed, part two

Noel Rappin, Tuesday, December 4, 2007 @ 4:30 pm

In Part 1 of this post, I started a to demonstrate using Ruby Metaprogramming to implement a potentially useful testing API:

testbed "call my test function" do |input|
subject = MyObject.new(input)
subject.method_to_test
end
verify_that(1).returns("fred")
verify_that(2).returns(3)
verify_that(0).raises(Exception)

Last time we covered the testbed call, now, let's move on to verify_that and returns.

The verify_that method is a class method of TestCase, just like testbed. In order for the method chain to work, it needs to return some object that can accept a returns call. There are three options:

• verify_that can return the TestCase class, and returns could be another class method. I rejected this out of hand, on the grounds that it's kind of ugly, and also feels like it could lead to some tangles if the code gets more complicated.

• verify_that can return a the testbed instance, and returns can be an instance method of Testbed. I actually walked down this road for a block or two, but it implies that Testbed needs to have a current state for the most recent input and output values. I didn't like the code that produced, and I think it might have some problems later on.

• verify_that could return a brand-new class, Verifier, which stores a specific state to be verified. This is the design I chose -- I never met a design with extra objects that I didn't like...

The next test covers the creation of the Verifier object:

def test_creating_a_verification
FakeTestA.testbed("this is a fake testbed"){ |a| a * 2 }
verifier = FakeTestA.verify_that(1)
assert_equal(FakeTestA, verifier.test_class)
assert_equal([1], verifier.arguments)
assert_equal(:test_this_is_a_fake_testbed_1, verifier.testbed.method_name(1))
end

The test creates a testbed as before, then simulates another class call, this one to verify_that, then validates that the verifier object knows about the test class, the arguments, and also can work with the testbed to generate the method name for the eventual test.

To make this work, we need to define verify_that, the Verifier class, and Testbed#method_name. Individually, all of these methods are straightforward.

First, define verify_that inside the TestCase class singleton block where testbed is already defined:

def verify_that(*arguments)
Verifier.new(current_testbed, self, *arguments)
end

Right now, the Verifier class is just the initializer -- this needs to be defined inside the Test::Unit module along with the existing Testbed class.

class Verifier

attr_accessor :testbed, :arguments, :test_class, :value

def initialize(testbed, test_class, *arguments)
@testbed = testbed
@test_class = test_class
@arguments = arguments
end
end

Finally, the method name, an instance method of testbed. I'm building the method name up from the name of the testbed and the arguments being verified.

def method_name(*arguments)
"test_#{name.gsub(" ", "_")}_#{arguments.join('_')}".to_sym
end

Okay, the next step is to implement the returns method, which generates an actual test method and verify that the test actually runs. This leads to the philosophical question of how you test a test case. After some banging around with the Test::Unit docs, this is what I came up with:

def assert_test_run(method, error_count, failure_count, run_count, assertion_count)
test = FakeTestA.new(method)
runner = Test::Unit::UI::Console::TestRunner.new(test, 0)
result = runner.start
assert_equal(error_count, result.error_count)
assert_equal(failure_count, result.failure_count)
assert_equal(run_count, result.run_count)
assert_equal(assertion_count, result.assertion_count)
passed = error_count == 0 && failure_count == 0
assert_equal(passed, result.passed?)
end

I'm not prepared to defend this as the best way to test a test case, but it does seem to work. It's creating a TestRunner with quiet output (the 0 argument -- otherwise, you get spurious test running output, which is confusing). It runs the test runner, and validates the various result counts against the passed in values.

The first test is to prove that a returns call creates the method and then allows the method to run correctly.

def test_makes_return_successful
FakeTestA.testbed("this is a fake testbed"){ |a| a * 2 }
verifier = FakeTestA.verify_that(1)
verifier.returns(2)
assert FakeTestA.public_method_defined?(:test_this_is_a_fake_testbed_1)
assert_test_run(:test_this_is_a_fake_testbed_1, 0, 0, 1, 1)
end

The first step in implementing this is Verifier#returns, which simply stores the value, then defers to the test class to create the actual method.

def returns(value)
@value = value
test_class.define_test(self, testbed)
end

This gets a tiny bit hairy. The method is getting created within the test class, but all the data is in the verifier and testbed objects. Ideally, I'd like to actually define the method within returns, but define_method is private, and so has to be called in the test class singleton block, right below verify_that:

def define_test(verifier, testbed)
define_method(testbed.method_name(*verifier.arguments)) do
assert_equal(verifier.value, testbed.actual(self, *verifier.arguments))
end
end

Using define_method makes metaprogramming so pretty... The method being defined takes it's name from the method_name method we just wrote. Then it asserts the equality of the value passed to returns, comparing it to the actual value of the testbed with the verifier arguments, which is a simple block call -- the method is an instance method of Testbed:

def actual(inst, *arguments)
block.bind(inst).call(*arguments)
end

The tricky thing here is that the actual test case instance -- self in the testbed.actual call above, and inst in the arguments to the actual method -- has to be bound to the block. The statement block.bind(inst) sets the value of self within the block being called to the instance. This enables any instance variables created in the test class (in a setup method, for example) to be accessible in the block. Without the bind call, then self inside the original testbed block is the test class, and not the test instance, because when the testbed block is defined, it's defined within a class method.

And that's it -- I'm always amazed at how much you can do in Ruby with just a few lines of code. Some follow up tests prove this works in the case where there's a failing method, and in the case where we want to have multiple verifications against the same testbed:

def test_makes_return_unsuccessful
FakeTestA.testbed("this is a fake testbed"){ |a| a * 2 }
verifier = FakeTestA.verify_that(1)
verifier.returns(3)
assert FakeTestA.public_method_defined?(:test_this_is_a_fake_testbed_1)
assert_test_run(:test_this_is_a_fake_testbed_1, 0, 1, 1, 1)
end

def test_two_methods
FakeTestA.testbed("this is a fake testbed"){ |a| a * 2 }
verifier = FakeTestA.verify_that(1)
verifier.returns(2)
verifier = FakeTestA.verify_that(2)
verifier.returns(2)
assert_test_run(:test_this_is_a_fake_testbed_1, 0, 0, 1, 1)
assert_test_run(:test_this_is_a_fake_testbed_2, 0, 1, 1, 1)
end

And I thought it was worth it to have a separate file test that should work given the syntax as it would actually be used in the wild. In a separate file, try:

require 'test/unit'
require '../lib/testbed'
class TestbedTest < Test::Unit::TestCase

testbed "this should work" do |input|
input * 2
end
verify_that(1).returns(2)
verify_that(0).returns(0)

end

It should all work.

Next time: adding raises and any other functionality that strikes my eye, plus packaging as a Rails plugin.

agile ajax

Comments

• Leave a comment
• Comment feed for this post
• TrackBack URL

None so far.

Leave a comment