Factory tools for fixture replacement: a comparison

Noel Rappin, Friday, May 22, 2009 @ 10:49 am







Assembly line picture by harry_nl

Continuing on the general theme of blog posts that people actually asked me to do... this week, I'll be taking a comparative look at fixture replacement tools. The general request was to have one place that compared the features of the various libraries so that a prospective fixture replacer might make a more informed choice.

So here goes, I'll try to keep the editorializing to a minimum, but no promises.

Fixture replacement or factory tools replace the global nature of Rails fixture data with the ability to easily create new objects based on sample valid data previously defined by the developer. Factory tools have several advantages, including keeping each test independent of other tests, since you are no longer using global fixtures, and allowing the important data for each test to be defined in that test.

Updated to correct some errors caught in the first few comments.

Self-promotion alert: A full description of why to use fixture replacements and an in-depth look at one of these tools is available in the full version of Rails Test Prescriptions available wherever fine PDF books are sold as long as you think the only place fine PDF books are sold is http://www.railsrx.com.

The competitors

As I write this, there seem to be four factory tools that have some user base:

Each of these tools has subtle or more advanced features that are outside of the scope of this quick guide -- be sure and check out the tools that look appealing to you.

Defining Factories

The basic structure of each of these tools is similar. You create a file made up of factories, each of which defines the basic data for a model. First, let's discuss that file. Each tool places it in a different location.

Location of factory data
Factory Girl Autoloaded if in: test/factories.rb or spec/factories.rb or test/factories/*.rb or spec/factories/*.rb, otherwise manually loaded.
Fixture Replacement db/example_data.rb, automatically loaded by module. Also, code in that file must be declared inside module FixtureReplacement. Commentator says that factories can be defined inline in your tests as well, but I don't see docs for that feature.
Machinist test/blueprints.rb, by convention, but could probably go anywhere.
Object Daddy Typically, in /spec/examplars/model_exemplar.rb. Object Daddy puts the factories inside the ActiveRecord model class definition, so this is a place that monkey patches the model and will be auto loaded during testing, however the data can be placed in the actual definition in app/models.



Just as a note, my personal preference here is to keep the tools in the test directory -- the placement for FixtureReplacement in the db directory bugs me. Object Daddy is the only tool that doesn't expect the factories to be defined outside the model.

Within the factory file, the general idea is that multiple blocks are defined, one to each model, more or less. This table contains the syntax for creating a block -- we'll cover what goes inside that block in a moment. Let's assume here a model called Project


Creating an individual factory
Factory Girl
To define a factory matching a model name:

Factory.define :project do |p| ... end

Or, to define a factory and assign it to an
arbitrary class:


Factory.define :active_project, :class => Project do |p| ... end

Fixture Replacement
attributes_for :project do |p| ... end
Machinist
Project.blueprint do end


For a blueprint with a label:


Project.blueprint(:active) do end

Object Daddy N/A. Since Object Daddy puts the generators in the actual ActiveRecord model class, it doesn't need an explicit block.



There's a slight difference in structure here. Since Factory Girl and Fixture Replacement are essentially defined in their own namespaces, the factory blocks take a single argument, which is the model being created. Machinist adds a class method to ActiveRecord base, so it doesn't need that argument -- the block will be evaluated inside the class that the blueprint method is called on.

Defining Attribute Defaults

Within each factory block, individual attributes can have default values specified. Typically these are either static values, or random values (the Faker gem is very helpful here). Each tool has a slightly different syntax for defining basic attribute defaults.

Unless otherwise specified, these lines defining the defaults would go inside the factory blocks listed above. Also, unless otherwise specified, you can do other arbitrary code inside the factory definition to calculate any intermediate values you want.


Default attribute data
Factory Girl
For static defaults:

p.name 'Test Project'

For dynamic defaults

p.name { some_dynamic_code }

Attributes can depend on earlier defined ones, if the block takes an argument


p.long_name { Faker::Company.name }

p.short_name { |a| a.long_name[0, 2] }

Fixture Replacement p.name = "Test Data"

The right hand side can be an arbitrary expression, and can, I think, reference previously defined attributes.
Machinist Static data is declared simply:
name "Test Data"

Dynamic data can take a block

name { "Test Data" }


Previously defined attributes can be referenced. All dynamic random data should be described in a Sham, see next section.

Object Daddy
Inside the active record model, four forms.

generator_for :name, "Test Name"


generator_for :name do "Test Name" end


generator_for :name, :method => :fake_name


generator_for :name, :class => NameGenerator


See next section for how to use these in sequence.



Some slight difference here between Fixture Replacement's use of an assignment statement, Machinist using a method that takes a block, and FactoryGirl using a method that takes a value or a block. Object Data does everything a little differently

Defining Sequences of data

One nice feature of these factory tools is the ability to create sequences of related non-repeating data, whether that's just calling projects "Test 1", "Test 2" and so on, or a more complex sequence. Here is how to define a data sequence in each tool. Note that even though these examples take an index and are sequential, they can also be used with just random data.


Defining sequences
Factory Girl
Outside the define block


Factory.sequence :name do |n| "Test Project #{n}" end

Inside the define block:

p.name { Factory.next(:name) }

Fixture Replacement No explicit support, but does define a 'random_string' method
Machinist
Outside the blueprint block

Sham.name { |index| "Test Project #{index}" }

Inside the blueprint index

name { Sham.name }

But, if the Sham and the attribute have the same name, this can
be written as just

name


And Machinist will make the right connection

Object Daddy The generator_for method takes a :start option to define the initial value of the sequence. The block and method versions take an optional argument, prev, which is the previous value of the sequence which can be used to generate the next value. The class version can maintain it's own sequence, as long the class provides a 'next' method.


Defining Associations

A factory also allows you to define default associations with dependent objects. This isn't always what you want -- my experience is that you are frequently better off specifying the associations in the actual test. For the purposes of the example, let's assume that the Project belongs to a Client, and again, all this code is in the block defining the example, unless otherwise specified. These also assume that there is a separate factory definition for Client.


Support for associations
Factory Girl p.client { |a| a.association(:client) }

Which can be abbreviated:

p.association :client


In each case, extra arguments are treated as overrides to the default
data of the other object.

Fixture Replacement p.client = default_client


Additional arguments to the method are treated as overrides.
Machinist client { Client.make }

Again, additional arguments can be specified as overrides. If the attribute and the blueprint have the same name, this can be shortened to:

client

Object Daddy A belongs_to association is automatically created (by calling generate on the associated class) when needed.


Also note that has_many style relationships can not be created directly in any of these tools, because ActiveRecord needs the object to be saved first. These relationships need to be created in the actual test.

Creating Instances

Having spent all this time defining attributes, you are probably going to want to actually declare some in your tests.

First, you actually need to include or require some things to allow your tests to be aware of the factory replacement tool.


Steps needed to load factory data
Factory Girl No further steps once installed, unless you want to include a factory file from a non-standard location
Fixture Replacement In Test::Unit, Test::Unit::TestCase must include FixtureReplcement. In RSpec, the Spec::Runner.configure block must have the line config.include FixtureReplacement
Machinist The blueprint file must be required in test_helper.rb or spec_helper.rb. You must add a setup or before block called before each test containing the line Sham.reset
Object Daddy No further steps once installed, unless you include generator files in a non-standard location.


Then you need to actually write the lines in your test that will create your data.

In general, all these methods allow you to override the defined default values by passing them as key/value attributes to the method creating the object. In general, this works for attributes as well as associations. The examples below will show creating the object with the name field overridden, just to give a sense of the syntax. Some of the tools have different options depending on whether you want the object saved to the database or not.


Steps needed to load factory data
Factory Girl Create and save:
Factory.build(:project, :name => "New Project")

Create, don't save: Factory.create(:project, :name => "New Project")

Create as as series of stubs: Factory.stub(:project, :name => "New Project")

Return the attributes as a hash:
Factory.attributes_for(:project, :name => "New Project")
Fixture Replacement Create and save: create_project(:name => "New Project")

Create and don't save: new_project(:name => "New Project")

Already saw "default_project" for creation as a dependent object.
Machinist Create and save: Project.make(:name => "New Project")

Create and don't save: Project.make_unsaved(:name => "New Project")

Return the attributes as a hash: Project.plan(:name => "New Project")

Named blueprints can be invoked by having the name be the first argument to any of the above methods. Project.make(:active)
Object Daddy Create and save: Project.generate(:name => "New Project")

Create and save, but raise exception if invalid: Project.generate!(:name => "New Project")

Create and don't save: Project.spawn(:name => "New Project")


To me the biggest difference here is that Object Daddy and Machinist call new class methods of the actual class, while Fixture Replacement creates methods of Test::Unit, and Factory Girl uses its own object.

And In Conclusion, I'm Done

That's a quick look at the available tools for using factories to replace fixtures. I've had personal success using Fixture Replacement, Factory Girl, and Machinist -- I haven't had a chance to use Object Daddy yet. I slightly prefer Machinist, but not by enough that you should take that seriously if one of the others looks good to you. (I should mention that Factory girl will allow you to use some of the syntax from the other tools if you've taken a liking to it).

The biggest thing to remember when using these tools is to avoid setting up large amounts of data for each test -- that'll slow things down a bit. Instead, you want focused unit tests that create a minimal amount of data for each test.

Related posts:

  1. Review of fixture_replacement2 plugin – Update
  2. ChicagoRuby meeting ‘Test Prescriptions’ recap
  3. Live Ruby: Testbed, part 1
  4. Ask A Rails Tester Person
  5. Live Ruby: Testbed, part two

Topics:

Software Development Software Development

Comments: 14 so far

  1. Awesome comprehensive play-by-play comparison. I also prefer Machinist.

    I definitely gotta echo the point about creating the minimal data necessary for a good test. One colleague of mine once created a test which ran the same assertion against 1,000 different permutations of the data, basically testing that every input X between 0 and 1,000 would output the expected Y; I gotta hand it to him for being thorough, but not only was it unnecessary, but it made the test run so slowly that I had to comment it out just so I could be productive with autotest.

    Comment by Chris Vincent, Friday, May 22, 2009 @ 1:37 pm

  2. Thanks for posting this. Right now it’s useful to me in suppressing the shiny-object urge to try every new tool I see. Laid out this way, they don’t seem different enough to be worth the effort of switching from factory_girl, which I’m already using.

    I think you got build and create reversed for factory_girl, by the way: Create saves, build doesn’t, just like ActiveRecord.

    Comment by Erik Ostrom, Friday, May 22, 2009 @ 4:54 pm

  3. Great in-depth analysis of the various popular alternatives.

    A few corrections about FixtureReplacement:

    As of FixtureReplacement v 2.1, db/example_data.rb is no longer necessary.

    You can also call your factories inline, if you so please:

    http://gist.github.com/116456

    And define them that way, too:

    http://gist.github.com/116457

    You could also create your own wrappers around it:

    http://gist.github.com/116459

    Comment by Scott Taylor (smtlaissezfaire), Friday, May 22, 2009 @ 7:25 pm

  4. Great stuff.

    A small correction for Machinist. You need to use a block to assign a simple value in a blueprint.

    name ‘Test Data’

    works fine

    Machinist FTW.

    Comment by Adam Meehan, Friday, May 22, 2009 @ 7:55 pm

  5. woops. should read *don’t need*

    Commenting from iPhone can be sloppy

    Comment by Adam Meehan, Friday, May 22, 2009 @ 8:00 pm

  6. Dataset deserves a mention http://aiwilliams.github.com/dataset/

    Comment by Jim Gay, Saturday, May 23, 2009 @ 7:49 pm

  7. [...] Agile Ajax » Factory tools for fixture replacement: a comparison » Pathfinder Development [...]

    Pingback by Ennuyer.net » Blog Archive » 2009-05-25- Today’s Ruby/Rails Reading, Monday, May 25, 2009 @ 1:30 pm

  8. I’ve been using factory girl for a while and like it a lot. However, the issue I keep finding with these tools is they tend to slow down your test suite. Unsurprisingly, creating the data for each test directly within the test is slower than preloading fixture data and then just reusing it in each test.

    I tried to address this (with some success) with a gem:

    http://github.com/myronmarston/factory_data_preloader/tree/master

    It should be compatible with any fixture replacement, though I’ve only personally used it with factory girl.

    How have others solved the speed issue?

    Comment by Myron Marston, Wednesday, May 27, 2009 @ 6:25 pm

  9. @Myron — I’d argue that in general if you are creating common data that is being reloaded in multiple tests that you aren’t using the factory tool optimally. (That’s from my experience, but I’ll also grant that’s kind of a purist argument rather than a pragmatic one)

    The advantages of a factory tool are best realized if you create the fewest number of objects you need to expose each test, and if you create them as close to the actual test as possible. Creating more objects slows the tests down a lot. Creating the data away from the test has the same problem as fixtures — test values that seem magical because you can’t see the data that caused them.

    Mocks are another alternative for speeding up tests… More on that tomorrow.

    Comment by Noel Rappin, Thursday, May 28, 2009 @ 8:22 am

  10. [...] on the theme of comparing similar things that I started last week, this week I’ll be taking on Mock Object [...]

    Pingback by Ruby on Rails » Comparing Ruby Mock Object Libraries » Pathfinder Development, Friday, May 29, 2009 @ 12:24 pm

  11. I do like machinist a lot, and use extensively in my projects. Factory girls supports that machinist syntax, but I didn’t feel the need to change. machinist works pretty well.

    @myron Usually a avoid creating unnecessary data, I just call “make” when that record will be used. some auxiliary data, I may also use make_unsaved, and it will not hit the database.

    Comment by Marcus Derencius, Saturday, May 30, 2009 @ 2:29 pm

  12. @Noel-

    In principle, I agree with you: you get the most benefits of a factory tool by creating the test data as close to the test as possible. And if your primary reason to switch to a factory tool from fixtures is to be abel to see the data creation right next to the test, so that you don’t have to switch to another file to see the contents of a test object…then preloading a bunch of data using my gem (or a similar solution) would be a bad idea.

    In my case, my primary reason for hating fixtures and moving to a different tool is because I want all my test data to be created through the models themselves, rather than just being inserted directly into the database. This means my test data has passed validations, and any associated records created by before/after save callbacks will also exist.

    The test suite on my current project is at 2500+ tests and counting. I wish it was faster than it is. My factory_data_preloader gem has been a big help in speeding it up, but it’s still slower than I’d like it (about 7-8 minutes total right now, I think).

    I use mocks quite a bit too. It’s a technique I find myself using more and more. Maybe we wouldn’t have the speed issues if I had use mocking early on.

    Comment by Myron Marston, Saturday, May 30, 2009 @ 7:42 pm

  13. [...] http://www.pathf.com/blogs/2009/05/factory-tools-for-fixture-replacement-a-comparison/ : une comparaison des outils pour remplacer les fixtures dans Rails [...]

    Pingback by Dev Blog AF83 » Blog Archive » Veille technologique : Conditions d’utilisation, Lecture, HTML5, Mozilla, Vim, Git, Ruby, Rails, Javascript, Microframeworks, PHP, Python, Bases de données, Tuesday, June 9, 2009 @ 6:09 am

  14. [...] solid review of different factory tools that go beyond standard fixtures (Factory Girl, Fixture Replacement, Machinist, and Object [...]

    Pingback by Agile Ajax » ChicagoRuby meeting ‘Test Prescriptions’ recap » Pathfinder Development, Wednesday, June 24, 2009 @ 1:27 pm

Leave a comment

Powered by WP Hashcash

Launch: Pathfinder Newsletter

    Get a monthly update on best practices for delivering successful software.

    Subscribe via email


    Subscribe via RSS      RSS icon

Categories

Topics

Blogroll

Search

WordPress

Comments about this site: info@pathf.com