Archive for December, 2008

DBSprockets is back, Baby!

Friday, December 12th, 2008

So I have had a pretty long hiatus from working on dbsprockets, but I’m back… with a vengeance.  So, I worked hard to get Rum working in TG2, but struggled my ass off and got nowhere.  Left tangled in a web of peak-rules that I did not want to decipher, I began to think about dbsprockets again.  I mean, were we really that far off from what RUM is offering?

The answer turned out to be no.  All I needed to do was to replace the dbsprockets primitives with a class structure.  This turned out to be about an 8 hour job.  Not bad for a day’s work.  And now you can do things like:

from dbsprockets.declaratives import FormBase
from myWebapp.model import User
...
class RegistrationForm(FormBase):
    __model__ = User
    __limit_fields__ = 'user_name', 'email_address', 'display_name', 'password', 'verify'
    __required_fields__ = 'user_name', 'email_address', 'display_name', 'password', 'verify'
    email_address = TextField
    verify = PasswordField('verify')
    __base_validator__ =  Schema(chained_validators=(FieldsMatch('password',
                                                        'verify',
                                                        messages={'invalidNoMatch':
                                                                  "Passwords do not match"}),))
registration_form = RegistrationForm()
class ATurboGearsController(BaseController):
    @validate(form=registration_form.__widget__)
    @expose('genshi:sproxtest.templates.register')
    def register(self, **kw):
        pylons.c.widget = registration_form
        return dict(value={})

This turns out to be a much simpler way of handling forms, because now you can subclass to your heart’s content (for instance, make one base user form which you subclass for admin, registration, profile and login), or even come up with your own wacky WidgetSelector that chooses widgets for you and subclass as you desire. Here is initial documentation, which I will express in more detail at a later date when I have more time to do it the right way.

The simple fact is that you can customize form widgets with ease, limit fields to a set of fields, drop a few fields, basically anything you can think of to change how the database schema actually displays on the page.  The same is true of field validators.  Simply define an attribute of the class that has a validator, widget instance, or widget type, and dbsprockets will do the right thing.  If you want to override both the field and the validator, all you must do is create a widget wit the validator attribute populated.  The greatest thing is that your forms (and tables!) will change with you as your database schema migrates.

Right now I am keeping the primitives way of getting the values from the provider to populate the tables/forms.   This is likely to change in the future.  The primitive way of doing forms is now deprecated.  I will be adding deprecation warnings to the code.  Hooray!

Look for ajax support in the near future on both the view and data side of things.  I have a clear understanding of ExtJS (2.0.1), JSON, and LGPL now and I am not affraid to use it.  In the mean time, a dev version of dbsprockets 0.5 is up at pypi.

CTS solution, a new keyboard?

Monday, December 8th, 2008

http://www.flickr.com/photos/finpix/2804562484/

Any of you who know me know that I have been struggling with carpal tunnel syndrome for the last year or so.  A combination of working with a laptop in an ergonomically undesirable position, frequent days longer than 12 hours coding, and my personal winter vocation (ice climbing) put me in pretty consistent pain.   The problem with ice climbing is as you reach to place your tool into the ice, your wrist bends at an acute angle.  Then, to move your feet up, you must weight the bent wrists.  The pain got to the point last February where I had to completely stop climbing ice, despite having a trip planned for Ouray.  Even rock climbing had to stop, relegating me to short day trips to the Flat Irons (mostly footwork climbing) until things improved.

The pain lessened long after the ice was melted, and I was able to return to  a pretty regular climbing schedule, but often pressures from my clients would cause the wrist pain to flare up.  I used wrist braces to help keep my wrists in correct position, but I was simply unable to maintain my usual pace without going to bed at night with aching wrists.

West Gully of Black Lake, RMNPWith ice season approaching, I had to find a solution to this problem, as I really love climbing ice, it brings you to some pretty spectacular places.  At Plonecon this year, I saw two people with the same unusual keyboard, a Kinesis Advantage.  I asked them about it, and everyone gave it good reviews, saying that it really helped them.  When I got back from Plonecon I plunked down $250 for a refurbished keyboard at home, and started carting it back and forth to work.

This keyboard is not one who’s transition you can take lightly.  As you can see, it has an unusual physical layout, but if you have been using a Microsoft Natural Keyboard, the transition may not be too bad.   One think that took a while to get used to is that the keys are aligned vertically.  This creates a lot less strain, but is quite unusal at first. After a month or so of use, I can say that my typing skills have returned to what they once were, even in a programming environment, where you have to use weird symbols, and the arrow keys a lot.

The keyboard has helped me reduce the amount of pain in my wrists enough that I can resume ice climbing.  However, the change comes about not only because the keyboard offers better ergonomics, but it has changed the way I type, equalizing the work that both hands do.  If your keyboard is more than 6, months old, pick it up, and hold it at an angle so you can see which keys have a glossy shine, and which ones are still opaque.  You can learn a lot about a way a person types by examining their keyboard.  Do you use both shift keys?  Which  thumb do you use to hit the spacebar?

The biggest struggle for me with the new keyboard whttp://farm1.static.flickr.com/127/321996427_ae5fb8fa54.jpg?v=0as the fact that the “b” key is on the left hand.  I’ve been typing that one with the right hand for as long as I can remember.  When I look at how I used to hold my hands over the keyboard, I can see that the left hand is bent a little, which makes hitting the “b” key a little more difficult, which is why I probably did that with the right hand.  Also, I used the left shift key exclusively.  I still struggle with that, but I am getting a better about using the right one as well, you just have to take a little more time when typing capitals to be able to use both.  (This would really suck if I was a JavaProgrammer)

Besides helping improving my typing prowess, the keyboard offloads a lot of work to the thumbs, eliminating the wrist-snapping alt-tab combination that caused the majority of my problems… In fact, ctrl-c, ctrl-v, ctrl-x, where I spend a lot of my time is now a two-handed affair.  This takes a lot of the stress off my left hand, which had much worse pain than the right.  Lastly, the arrow keys are moved to below the left and right hands, and while this is a little weird to get used to, it is nice not having to move my right hand to get to those keys.  It also forces me to use both shift keys more, since you must use them to highlight text the way you need for cutting and pasting.

While the Kinesis keyboard is pretty unusual, and a bit hard to get used to, the keyboard comes with a set of exercises which really helped me get used to it.  The first day, the exercises probably took > 2 hours, but after a week of doing them, I had that down to < 25 minutes.  In between I used my old keyboard, because I just had to get work done, but I used the Kinesis more and more over time, and after a month, it is now full time, and I really don’t like using standard keyboards, but the transition going between the two is not too bad.

This is kind of a long post, but if you are having RSI, and aren’t sure that spending $250+ dollars is going to be worth it, perhaps this blog post will help you decide.   My refurbished keyboard at home is not discernibly different from the brand new one my work purchased for me, and both have worked for a month rock-solid.  The one thing I noticed was that the Kinesis had a somewhat higher energy profile, requiring me to replace my 4 port usb hub with a powered one.  I might add that I had the mouse and old keyboard plugged into the Kinesis, so it was in fact drawing the power of 2 keyboards, if you have the ability to simply replace your old keyboard, you may not see this problem.  Lastly, you are able to re-map the keys on the keyboard, which I did with the up/down keys.  This opens up the posibility for a dvorak layout, but the home keys are different from the other keys, so it might be worth it to invest in the more expensive version that comes with special keys designed specifically for this layout if that is what you prefer.

SQLAlchemy Migrate Process Hiccups

Friday, December 5th, 2008

So, I’ve done migration processes for two medium-large database schemas (50-100 tables) and I have found what I believe to be a disconnect in the process of migrating and a database and developing a database application.


The Problem
——————-
Here is the problem in general.  I am sorry it is so long winded, but it’s hard to see what is going on without this full expression.

You create a schema, and use that schema to create a database.  Maybe
you are using Pylons, and using setup-app to create the schema. Everyone is happy in schema land, and then someone realizes you need to make a change, and maybe even more changes in the future.  You decide to use migrate, because hell, migrate will make things easier. So, you write the migration, modify your model code and everything is honky dory.  Clients are happy, your developer feels like he has a maintainable codebase.  Your project begins to grow.  Awesome.

Not Awesome.  Your second developer needs a development environment. You give them your model, they setup-app it and they are ready to go. While they are plugging along, you realize you need a new migrate, so you create it, and upgrade your development version of the database, which is now at migration 2.  The production also goes off without a hitch, it’s now on 2.

What about your new development buddy?  When he ran migrate_version, he set his database up to version 0.  So, migrate will try to do 0->1 and then 1->2, but 0->1 will fail because his database is already at version 1 even though his version table says its 0.  Now he is in pain, and re-creates his entire database to get moving, and the whole cycle starts over again, although he can work… until you do the next migration.

This is not ideal, and I have been struggling with it at one of my clients.  In that situation, I am the developer buddy, bopping in from time to time to lend a hand.  I spend probably 5% of my time getting up and running, where if their migration system was somehow linked to their database model, I could be synchronized, and actually use it for a development tool, rather than a dusty old production tool.

Solution 1
—————
So, I came up with my own solution, and have talked to a close friend about an alternative solution.  I’d like to discuss both of them as they pertain to migrate, in the hopes that we can improve migrate, and  re-connect the development environment with the production one.

My solution is to connect the model base code to the migration version.  The way I accomplish this is to add a version.py file at the root of my model code, which has, incredibly, the version of the migration changeset that matches this model definition.  I also have a custom database creation script which creates a migrate_version table and fills it with a record of the correct version number.  Now I can svn up, and migrate up and everyone is happy. The problem with this method is that I have to maintain a damn file with the version number, and not go insane doing it.  Since my model contains 3 different database schemas in it, that means I have to maintain 3 variables in my version.py .  Personally, I don’t mind this, but I had to write more documentation than the amount in this post to make certain that my predecessors (or me in 6 months) can figure out what the heck is going on.

Solution 2
————–
The second solution, which Mark came up with, is to preserve the initial model.  Then, you do all of the migrations to bring you up to the current codebase.  This does not require any version file tracking, and if your migrations add boilerplate entries, you don’t have to record them in a second place. I think solution 2 is reasonable, but could be more time consuming to execute (which is probably not really a problem with everyone’s 2ghz machines these days).  The problems I see are that the boilerplate entries you make for production may a not be the same as you make for development.  This makes testing a bit harder if they differ. I also feel that solution 2 is more prone to problems, simply because there are a lot more cranks to turn to get a development database.

How can migrate help
—————————–
The way to provide solution 1 to the problem in migrate would be to allow the developer to connect his migrations to his model code, and allow migrate some level of control over the “versions” module in the model code.  So, when you do a migrate commit, you are also saying that the model code supports this version of migration.  It will probably only take me  a few hours to make this happen.  We then create an easy way to create the migrate_version table by importing something.  This way you can have a custom createdb script that also creates the migrate_version table.  Perhaps we could even provide a reasonable createdb template.

Solution 2 only requires us to provide a custom creation script.  For this we will have to preserve the initial schema (schema 0) and then upgrade.  We could probably also include a template instead of a script so that you can manage boilerplate on your own.

My own conclusions
—————————-
I lean more towards solution 1, mostly because I feel that it will provide for easier boilerplating, because you will have access to your whole model at once, instead of bits and pieces of your model as it moves from one version the next.  I also think the boilerplate for dev. is likely to be a lot different than that of prod for most folks, and I feel like solution 1 offers an easier way of handling it.  It also seems like solution 1 is much more efficient, because you don’t have to go and re-do all of that migration work, you have already arrived at the correct-for-now solution.

If you made it this far congratulations!  Now, go grab a cup of coffee, stretch your arms, come back and tell me what you think.