I need to write a data change migration (splitting data from one table into another table, and removing redundant columns). Given that the product is in production now, I must have a test that proves my migration works. So, how do I test my migration? I need to pull the database up to the point prior to the migration, setup some data, run the migration, then see that the results are correct. This requires that all previous migrations run fine, which is currently not the case (some old pure-data migrations are of course failing hard).

I’ve been searching, talking with co-workers, I just don’t know what the solution to this is. I’ve personally always had the mindset that all migrations should be up- and down-able for this reason. Any one have suggestions here?

One thing I will NEVER do is re-run a migration. A migration is a run-once and once only script (per database) and therefore I have no problem with it updating any of my database state whether schema or data.

Sometimes changing a schema requires setting data to a value such as adding a column and setting a ‘default’ value based on existing data. A migration allows that to be done inline so that migrations can run one after each other without having to stop to run a rake task.

To clarify my run-once statement, once a database is in production, there is no way I’m dropping and re-creating with migrations. It’s live and must stay live so backups become the order of the day there.

I put migrations into three categories. Structural changes (add/remove tables/columns, etc.), static data (a zip code to lat/long conversion table), and data (static or dynamic) changes (populate a new join table between users and topics).

Structural changes are what migrations are great at. Use it and prosper. (schema.rb is unfortunately usually shite in a big team in my experience, mainly due to conflicts, and its ever-changing nature.)

For static data, I have not yet found a good solution for bulk loading large amounts of static data into a newly created database, especially a test database (fixtures don’t cut it for this stuff). Rails just isn’t built for that, and it’s frustrating sometimes. You end up in twisty little maze of implementations, all different.

Data changes are trouble, and I recommend against them as well. You have two basic choices, either use the model (slow and harms refactorability) or direct SQL which bypasses your validations, so can lead to subtle data errors.

*I* feel #14 should be NEVER use models in your migrations. If you find yourself tempted to use models in your migrations, you have clear evidence that you need a rake task. A one-off rake task, perhaps, but a rake task.

I’ve probably had a dozen models that I’ve refactored away over the years that broke somebody’s migration, and the only option was to go back and change their migration, which feels dirty for a reason.

That said, I also believe that a fresh development database should always be creatable from the migrations and that you should set up CC.rb to do a migrate to create its databases.

A great back and forth related to this is on this very site at: http://www.pathf.com/blogs/2008/07/resolved-should-schemarb-be-included-in-your-source-control/

– Morgan

In the environment I work with I couldn’t do that anyway – I develop on MySQL but deploy to Oracle. The primary keys that Rails sets up on Oracle start at 1001 whereas on MySQL they start at 1. So right off the bat you couldn’t even have migrations which used primary keys/object IDs.

So I’d always be doing :

Faculty.find_by_code(’SBS’).update_attribute(:name, ‘Southampton Business School’)

…using a column (in this instance called code) which I know will never change, rather than…

Faculty.find(1001).update_attribute(:name, ‘Southampton Business School’)

…which is decidedly more risky and I wholeheartedly agree is a disaster waiting to happen.

Overall I think my view on what is acceptable/workable and what isn’t is influenced by the fact I’m usually the sole developer working on projects and I rule the database with an iron fist so (fingers crossed) I don’t get any surprises.

If I was part of a large team where a lot of people could be interfering then perhaps my outlook on migrations would be different and my use of them would change. Still, it’s useful to be aware of the potential pitfalls.

“If your application is going to be doing something non-standard, just don’t bother with REST. Save yourself (and everyone else) a little bit of time.”

I think you’re a little misguided as to what REST actually is. If you’re going to be doing something non-standard, just what exactly should you be bothering with? Wait.. a minute, I guess if you’re doing something non-standard you’re probably not doing REST. Yes, of course!

Scott, I think that you and I are actually agreeing on a lot of these points… at least, that’s the impression I get from your comments. You say you don’t agree with using migrations to load default data: that’s pretty well what I’m saying here. Put them in a separate file. Load them from migrations with rake tasks.

Your point that migrations should represent deltas is completely correct. You should never go back and change existing migrations. But that’s one of the two options available to you when you have your data in your migrations: and opposed to the other one, it’s the only way you can be 100% sure that the data is inserted correctly without failure. But I think you can see why this is a really bad idea.

The other option is having a migration only to change data existing in the database. This makes two huge assumptions that may prove unfounded: that the data exists and that it hasn’t been changed already. If you’re on a project with a lot of people, inserting such a dubious migration into the application is a sure way to make lots of exciting new enemies.

I’m actually surprised I’m getting a lot of pushback about #13, and you guys do have some really good points about it. Consider this, then: let’s say you have a model Faculty that has and belongs to many Teachers (to shamelessly steal part of your example, Anton). You decide to insert a join model between them called Tenures, and you drop the old faculties_teachers table. You suddenly lose all your associations between Faculty and Teachers. So, before you do this, you have to prepopulate the Tenures table with the appropriate faculty_id and teacher_id. Now the static data in your migration no longer represents the actual state of your data.

If you had a rake task to generate Faculties, Teachers, and the relationship between them, though, it’s unlikely anything at all would change in that task: it would still be able to do faculty.teachers << teacher so nothing would change.

I guess my point is that static data tends not to be as static as one thinks. If you have data that is literally totally immutable, then go ahead and put it in a migration. But I’ve come across very little that is literally totally immutable, and by putting data in a migration you risk the data integrity of your project. Rake tasks that run after migrations are a lot safer, more repeatable, and require less work on your part.

So that’s the reasoning behind #13.

I think rule 13 is acceptable as long as you are using it on what I would term “static data”. That is data which is never changed by the end user (and certainly not data input by the end user) but is used by your application to perhaps populate a drop down box or something similar.

I work in a University which is divided into faculties. Now these faculties are pretty constant but once in a while they change name slightly and I don’t want to be unloading and reloading the faculties table with the new names because I’ve got live data in the production system so instead I use migrations.

Now scenarios like the above I personally think are OK. However you raise a good point about using YML files and rake to populate the defaults and that’s something I think I’ll also do now so that anyone setting up a dev server from scratch can have the correct values in there without having to trawl through every migration.

In summary I think, if used correctly on static data, rule 13 is acceptable as long as you have the YML files and rake tasks to build the data from scratch without performing every migration. So the migrations are used purely for migrating existing databases where the data is referenced and cannot be deleted.