Comments on: SQLAlchemy Migrate Process Hiccups

By: Lucas

Lucas — Wed, 04 Feb 2009 16:33:29 +0000

Seems like Hans and Doug solution combined should be new migrate plan.

Add a
set_table_version(tablename, version)

Version could be a combination of numbers (production.testing.development)(1.3.503)

then when doing upgrades use the solution that goes through each upgrade version from 0 but first check if this version of the upgrade was applied? (isEligible(CurrentTableVersion,upgradeTableVersion)

If upgrade was not applied, apply it,
if upgrade was applied move on to the next item.

This would make sure you only upgrade from what you are on forward.

There should also be a command to reset it all. For example after few years you are at production 40.1.0, and you don’t want to check all the upgrades from day1. There should be a reset that takes the current 40.1.0 and makes it a first upgrade.

Now we just needs some code samples to fill these parts in and connect it to migrate.

Thanks,
Lucas

By: Jorge Vargas

Jorge Vargas — Wed, 10 Dec 2008 08:19:03 +0000

I got very little experience with migrate, but I think there is a 3th, from migrate docs:
“A freshly versioned database begins at version 0 by default. This assumes the database is empty. (If this is a bad assumption, you can specify the version at the time the database is declared under version control, with the version_control command.) We’ll see that creating and applying change scripts changes the database’s version number. ”

This implies you can alter the initial version number. Although I’m not sure if you can do that with a migrate command, so back to your example: Say it is your project and I’m the one hoping in from time to time, you will give the the current model and the current db number, which I’ll insert as my first migration, and with that we should be able to fool the db.

Now that’s all theory as I haven’t been able to get migrations going for this project

By: Hans

Hans — Sat, 06 Dec 2008 15:31:19 +0000

I should also clarify that the end solution for us was a little more involved than simply storing version number in the table comment. Because we are using an ORM that generates the SQL based on a schema.xml file (Propel, to be specific), we modified our build system to split out all of the DDL related to indexes, foreign keys, views, etc. Before we actually run the migration code for each table, we do things like drop all FKEY constraints on the table. In some cases we have to drop other dependent objects too (e.g. views). At the end of our migration code we actually re-apply all of that dependent object creation DDL; much of it does not get applied (because it was never dropped), but that has proven a clean way of upgrading non-table objects.

So, the point I wanted to clarify is that while the table-version-in-model method only addresses versioning for tables, we found that this was all that we really *needed* to version since we could drop and recreate all the other (dependent) objects in the system.

We also moved all of the comment-setting code into database functions — we have a set_table_version(tablename, version) method, for example — which does some fancy regex to replace or prepend the version information to a comment. And a get_table_version(tablename) to get the current version. End result just looks like “v1.0.2 – Table to store user data.”

In any event, the approach shouldn’t scare you like PHPNuke There was actually some thought put into how this works — and keeping the abstraction points in place to avoid coupling it too closely to any one particular database or even any one particular versioning mechanism. (We could always re-implement the Upgrade->isEligible(Table) method to use something besides table comments to determine eligibility.)

-H

By: Doug Hellmann

Doug Hellmann — Sat, 06 Dec 2008 13:21:13 +0000

I haven’t used sqlalchemy-migrate, but the way we deal with this in the alter scripts we wrote at work is to have each migration test the database to see if it needs to be applied. So if a script wants to add a column, first the test() method is invoked and it looks for the column. If it isn’t found, it returns True, and the alter() method is run. If the column is found by test(), it returns False and the alter() method is not run.

We found this to be much more workable than depending on recording which alterations have been made to a given database — that’s just redundant information if you have the ability to look at the schema.

By: percious

percious — Sat, 06 Dec 2008 04:52:22 +0000

Thanks for the reply hans. This was exactly the kind of discussion I was hoping to get. You have a unique solution to the problem, and I must admit it has the benefit of being flexible in that you could use other migrate solutions since the migration data is stored within the database, but it scares me a little in the way that PHPNuke does… But seriously, I am interested in all solutions people have for this problem. The more I think about it, the more I like solution 2.

By: Hans

Hans — Sat, 06 Dec 2008 00:47:49 +0000

Well, I haven’t used Migrate yet; I look forward to being able to do that on an upcoming project. I had to solve a similar problem with our PHP ORM solution not that long ago. What we ended up doing was actually storing versions of our model in the database — in the comments on table names, to be specific. We also had a lightweight set of DB utilities and baseclasses that we could extend with our actual migration scripts. Yes, this solution does require that you use an RDBMS that supports comments (we use postgresql), but in practice it actually has worked very well and it provides some additional assurance that our migration scripts won’t try to migrate an already-upgraded table. This is certainly not the best solution, but I wanted to offer another lightweight option to this migration issue.

-H