Posts Tagged ‘Turbogears’

Pycon 2010 Dump… Sorta

Monday, March 1st, 2010

I decided this year I am going to forego my Pycon dump, and focus on the news that matters most to my readership.  There are plenty of reviews of talks, and I went to a bunch of good ones, had a great time at wsgi house, blah, blah, blah.  What _you_ probably care about is what’s cutting edge in TurboGears, and the sprints are where it’s at.

We had 12 sprinters on day 1, I think 10 on day 2 and then 8, 4.  So yeah, there’s some manpower behind the next-gen TG stuff.  And that was our focus.  I don’t think anyone lifted a 2.0 finger, all the effort has gone into 2.1.  With Luke Macken’s help, we closed 21 tickets.  I think we have all the road-blocks out of the way for the forthcoming b2 release, which will likely be the last beta before release candidates.  The api is pretty solid now, I don’t expect to expand it an more, so it’s going to be feature frozen in b2.  There are only 2 blocker tickets before 2.1rc1.

Mark Ramm and Jenny Steele overhauled the visual components of the 2.1 docs and came up with a better way of updating our public face.  Jenny has also put forth an effort in c5t (a new cms based on mongo and tg2) to improve our image library capabilities.  She also spent some time updating the TG admin so it looks way better now.  I’ll post some screenshots when I do the 2.1b2 release.

C5t got quite a bit of love actually.  Rick Copeland added Ming 0.2 support since the api changed a bit from 0.1. Christopher Brown fixed a bunch of the styling.  Kevin Mitchell added some tests.  Jason Galyon worked on per-page authz, and Jorge Vargas worked on the editor implementation.  I created a new c5t.website project so that we can work on the core and the public image independently.  This is valuable especially since c5t does not ship with TW1, but we wanted to add mongodb support to the admin on the website, and TW1 is required for Sprox to do it’s magic in the admin presently.

Kai Groner and Eric J (sorry I don’t know your last name) both made some contributions to Sprox.  Eric added Ming support after I refactored the database orm later organization in Sprox.  Ming is a layer on top of pymongo that allows you to enforce schemata in your collections.  This is perfect for selecting validators (the validator in the schema becomes Sprox’s schema), and fairly simple to select widgets with.  Eric had every provider test that works with SAORM working, except for one that looks like a pymongo bug on the limit operator.  Kai worked on handling inline-forms in Sprox.  Looks like most of the provider code is working, we just need to add in tw.dynforms in an intelligent way to hook the rest of the application up.  While this is not a trivial task, it looks as if we are about half-way there for supporting inline forms. All of this work will go into sprox 0.7.

I also spent some time with Sprox, and added jquery support, but it’s not ready for release.  At the end of the sprints I had a working jquery table with pagination in the TG Admin, but some polish is going to be required to get that fully working.  When I’m done, you will be able to easily swap between nothing, dojo, and jquery, and mootools support, or mix and match at any level.  You can also switch between mongo, or any rdbms SA supports, and hopefully couchdb soon.  It’s kinda scary that’s true.

TurboGears 2.1a3 Released!

Sunday, November 29th, 2009

In keeping with our monthly releases, the TurboGears team presents 2.1a3. This release is primarily a bugfix release. There are no compatibility issues with this release. Here are the major items fixed.
* Fixed problems with Beaker Secret Key functionality. (Thanks
Sanjiv)
* Fixed infinite loop problem associated with lookup.
* Fixed RestController so delete() works with nested RCs.
* Refactored configuration support to handle non-SA persistance.
* Deprecated lookup in favor of _lookup.

This is likely to be the last alpha version before the beta lock down. b1 may have some minor api changes from this point, but it does not look like there is anything major. We are this means that we will be in bugfix-only mode starting mid-january, as we march to a final release for Pycon in 2010. The documentation team spent a considerable amount of time working the docs, they are posted at: http://www.turbogears.org/2.1/docs/ . Our todo list is down to 114, from the 140 we ballooned to since the last sprint. It would be great to have the todo list under 50 before formal release, but we have been working on those items which are priority, so the docs are really not in that bad of shape despite the high number of todo items.

TurboGears Black Friday Sprint

Friday, November 13th, 2009

If you are from the US, you may be familiar with the custom of racing to your local commercialization hub before the sun rises and beating your fellow man over the head to get that cute little fifi doll your daughter wants for christmas.  This endeavor is often fueled by the previous day’s binge on tryptophan-laced poultry.

black_friday

TurboGears wants you to know there _is_ an alternative to this ruckus behavior.  While it may not offer a method of transference of the latest flu virus strain, you may come hang out with the leaders of our community on irc://#turbogears@irc.freenode.net . and ask questions and help your fellow man.  In the spirit of thanksgiving, you may decide it would be good to return the 1000s of lines of code contributed by helping us document that mess*!

November 27, 2009 starting around 9am MST, ending when Chris passes out, we will convene and work on the 134 todo items left in our documentation.  If some of you are near Denver/Boulder that day, I would be happy to organize an on-site meetup, please let me know.  So, the day after thanksgiving, why not act gluttonous for one more day (with your writing) and lets see if we cant close another 60 todo Items as we did on the last doc push.  Be you an advanced super 37173 user, or someone completely new to TG, we have got something for you to do!  Lets work together to make the formal 2.1 release of TG the best yet, with the highest level of completed documentation evar!

cheers.

-chris

* the term “mess” is not used literally in this sense.

A TurboGears Weekend in review

Sunday, October 11th, 2009

A few weeks ago the Front Range Pythoneers decided to organize an “Uncon” where people show up to discuss various topics on-the-spot.  This is the sort of event I really enjoy participating in, so I of course agreed to attend.  At the same time, I was approached by the guys from Developer Day to do a talk on TurboGears.  You can imagine the conundrum I faced, but thanks to the willingness of the DD organizers to be flexible, and some creative planning, I was able to participate in both.

Speaking at Developer Day was a new experience for me because I was talking to folks that were not necessarily versed in Python,never-mind TurboGears.  The conference appeared to be somewhat Rails heavy, but it was refreshing to see organizers reaching out to the greater web community to provide a well rounded conference.  The nice thing about speaking to a wider audience was that I was able to expound some of the history of the Python web, as well as describe TurboGears at a high level without worrying about boring the audience.  I was quite nervous speaking at first, because I have not done so in a few months, but seemed to settle into a groove by the time I showed an example of how easy it is to inject repoze.profile into a TG application and provide a cachegrind display to find any slowdowns in your app.  I hope that this example was able to express how versatile WSGI is.

I stayed the morning at DevDay and I am glad that I did.  Chad Fowler gave an address on what it means to remain passionate as a developer over the life of your career.  I think his idea that providing structure to your life definitely allows you to achieve amazing things.  His real-life examples were poignant and well received.  I’ll be checking out his book soon.  The other talk that I found interesting was Jeremy Hinegardner’s talk which basically discussed the numerous non-relational persistence methods available.  I thought his method for showing examples of the different methods was great. For each one he had a simple succinct example that showed the  benefits for the given persistence framework.  He allowed the audience to choose from the frameworks he discussed in his talk.  Jeremy was an engaging speaker, and I would not hesitate to sit in on one of his talks in the future.

Google in Boulder, CO by krossbow.After a bit of DD-provided BeauJos, I headed over to the UnCon.  They too were having pizza provided by Google.  Google Boulder was a great sized venue for the 40 people that attended.  It was exciting to see so many new faces in attendance.  It seemed to me that the “regulars” were doing a lot of demoing, while the new folks watched on, but there was also a lot of discussion that happened.  I showed how to use repoze.profile and runsnakerun to

analyze the results.   Zooko immediately installed runsnakerun and tried it on his app.  It is always nice to have immediate gratification for having taught someone something, even more so when the person voluntarily tries what you think is “so cool.”  I got to show off some of the work I am doing for www.getmvp.com, since much of it is prototypical of the Extension Solution that I hope to provide with a combination of Pylons and TG.  Also on display was TW2.  It was great to show how simply one could express all MVC elements of a widget in one complete package.

Sunday I ran the first TurboGears WorkShop.  If you follow my blog, you may have read a few posts about how I think we can improve sprinting, but I’ve come to the realization that our less-than-stellar sprint performance is really due to a need for improvement in the organization at large.  I have decided to add a WorkShop Series to our tireless effort for improvement of TurboGears, both from a technical aspect, and one of the community.  I was up late on Wednesday creating a basic tutorial-type plan for Sunday, and I finished up with about 80 pages of documentation to provide workshop goers, basically by selecting items from the TurboGears documentation.  My goal for the sprint was to provide sprinters with a working example of TG at the end of the day, with a little bit of work accomplished customizing the Admin.  I asked sprinters to bring their own databases, to utilize sqlautocode as an example database for their new application, and while no one provided the class with one, we were still able to succeed with one that I provided as a backup.  5/6 people succeeded in this, and while there were some rough edges, I think I have an idea that is workable for a 3-6 hour WorkShop that will succeed with a little bit of polish.

I am still formulating the ideas for TurboGears workshops.  I have started to contact folks I know throughout the country, in order to provide venues for these workshops.  So far I have Boston, Dallas, San Fancisco, Atlanta, and Boulder (Denver) lined up.  I think with little effort, I could also add Ann Arbor, and probably Washington D.C.  The idea behind a workshop is that you arrive with a varying amount of knowledge in TG, and you leave with a greater knowledge than you arrived with.  You are encouraged to bring an existing project to hack upon, or to create a new one that we can play with.  I will provide a rough outline of what we might do in the tutorial, but if the group decides to go off in a different direction, that’s okay too.  If you are interested in participating in one of these WorkShops as a mentor, or providing venue space, accommodations, etc. I would love to hear from you.  Right now I am in the organization phase, expect a blog post announcing the official plan in the near future.

Thanks!  Without the efforts of a number of individuals this weekend would have been much less successful than it actually was.  I want to thank Ben Scofield for inviting me to talk at developer day, and for shuffling the schedule so I could participate in both conferences.  Greg Holling put in a great effort to organize the Uncon, and Google provided an awesome venue for us to use.  Three volunteers from Google Boulder provided their time, and even gave a tour of the facility to conference goers.  They weren’t even Python developers…  Jim Baker and Matt Boersma both showed up to provide access to Bivio so that we could have our first-ever TG WorkShop.  Lastly, I’d like to thank Bruce Eckel for making the trip down from the mountains to provide his unique perspective.

TG’s Killer Features: Object Dispatch

Wednesday, September 23rd, 2009

When people ask me what really sets TurboGears apart from the rest of the frameworks out there, I throw away everything we have built upon our foundation and find one thing remaining.  Object Dispatch.  Dispatch is probably a heinously boring topic for some folks.  For me, it’s a fascinating topic, one full of complexity, twists, and turns.  This is my main contribution to TurboGears, creating a dispatch system which is flexible enough to handle the most complex url mapping you can come up with, simple enough for the average Joe to use.  Like most of the technologies TG builds upon, the dispatch system is designed to be easy to start with, and easy to mould when you find that the defaults don’t meet your needs.

Let’s look at TG’s humble roots.  CherryPy was probably not the first framework to realize that nested classes are a fantastic way to organize urls.  Afterall, web addresses are nested, right?  (More on that later)  As the web server of choice for our founder, Kevin Dangoor, CherryPy offered a solid foundation in what remains in my mind to be the killer feature of TG.  Without OD, you pretty much don’t have TurboGears.  (You have Pylons).  Speaking of our brethren, lets look at a few frameworks to see how you dispatch a simple url.

Let’s take this url for instance:

rules/section/3

Pylons:

in routes.py you would add the line:

map.connect(’/rules/section/:id’, controller=‘section’ action=‘show’)

create the controller:

class SectionController:

def show(self, id):

return ‘i am a section, hear me roar’

Django:

edit urls.py to add:

(r’^rules/section/(?P<id>\d+)$’, ‘myapp.views.section’),

create the controller:

def section(request, id):

return HttpResponse(‘i am a section, hear me roar’)

Both are remarkably similar here, but I would give the advantage to Django for two reasons.  First, it uses regular expressions.  That means I don’t have to learn yet another dialect for matching strings.  Second, it specifies the controller directly, instead of the framework determining the location of the controller code with some magic lookup method.  Lastly, it provides a serialized method for routing, which means that you can provide a set of lookups in a file which can be digitized, modified, and re-serialized.  This is a huge advantage if you are planning on making a large system of swappable parts.   However, I don’t think the routes system is bad, and I have built my own system around it to manage the above problems, for a very plugin-required system I am building.  By the way, I’m no Django expert, so please feel free to comment on my code samples and I will fix them if need be.

Okay, so let’s look at the TG version of the same thing.

routing match code:

huh?  what’s routing?

controller code:

class SectionController:

@expose()

def index(self, id):

return ‘i am a section, hear me roar’

class RulesController:

section = SectionController()

class RootController:

rules = RulesController()

Hmm.  Notice the lack of weird symbols in the router code?  Oh, waitaminute, there _is_ no routing code!  That’s one less thing for someone to learn, one less thing to have to code.  Except there’s a catch.  That little @expose; what’s that all about?  Well, because the routing is done with introspection, you have to tell the dispatcher that something is _allowed_ to be dispatched upon or not.  So, already you have addressed some security concerns, where otherwise you would have to test that your routes don’t end up in a method that should not be exposed to the outside world.  And in complicated routing, this can be a definite issue.

So, Object Dispatch provides a provably simpler path to nirvana for the web programmer, but it’s just not cutting it for you.  Well, first off, with TG2 you can easily drop back and use Pylon’s routing mechanism.  Secondly, we build a RestController, which routes based on HTTP Verbs.  Lastly, in TG 2.1 you have the cream-de-la-creme of dispatch power.  You may write your own dispatch mechanism.  Theoritically, you could write a DjangoDispatchedController, that uses regular expression to determine the enclosed members’ dispatch mechanism.

Wait, huh?  Okay, so dispatch works in a similar manner for every framework, but not every framework can so easily switch dispatch mechanisms on the fly.  Here’s basically how it works in TG:

First, we go through the normal Pylons routing, this usually deposits us at the RootController of your app, unless you have overridden the pylons routes for your app to do something else.  Next, the routing mech takes your url, and splits it by ‘/’ into a list of strings.  It now looks for attributes of the root controller that match the first string in the list.  If that attribute is a method, and the method’s number of arguments jives with the remaining items in the list, the dispatcher says it’s done, and fires that method.

If the attribute found in the initial dispatch is an instance of a class, and the class has no _dispatch method, it traverses that class with the remaining elements of the list, repeating down the list until it finds a method that matches.

Now, if the attribute is an instance, and it has a _dispatch method, the dispatcher will then _become_ the _dispatch method, and dispatch will continue in whatever method was identified by the new _dispatch method.

So, if you read that whole description, I commend you, but if you only looked at the picture that works too.  Basically what I am saying here is that TG2 provides you with the ability to create your own method for dispatching your controller code based on the remaining URL in the path (that’s the “black” box).  And it works too, this is how we implemented TGController and RestController, and soon we will have an AmfController that does a Remote Procedure Call method of dispatch, to help simplify controller code for our Adobe Flex friends.

TurboGears is in my mind designed correctly, with very little tradeoffs made for the sake of pragmatics.  This no-sacrifices approach to the framework is yielding results.  As a developer, choosing TurboGears means that you can get started pretty easily, and are not stuck once you hit the limitations of TG’s developer’s imaginations.

I have decided to make this blog entry into a series on what makes TG 2.x a compelling choice as a web framework.  The next blog entry will be:  TG’s Killer Features: SQLAlchemy (obvious, no?)

Pycon 2009 Recap

Friday, April 3rd, 2009

It felt like this year Pycon was executed to near perfection. Many struggles I had with last years Pycon were addressed both by the organizers, and some creative thinking.
In this post I will recap everything that happened from my perspective.

WSGI House

I gathered a few close friends from the TG team and a couple of wildcards for perspective to share a house for the continuum of the conference.  Having a house gave us a place to go home to at night and meet with friends, often staying up late talking about issues surrounding our favorite software.  Having a focused group I feel is important because you spend less time off on wild tangents.  The first (and pretty much only) rule of the house was that you pay the same amount whether you stay one night or nine.  At least one of our members was encouraged by this rule to stay for the sprints which he hadn’t done before. Success!

Tutorials

For me, tutorials got off to a shaky start, but we seemed to recover nicely.  TurboGears has a lot of momentum right now, and it makes it hard to come up with a succinct tutorial when there is so much functionality to cover.  I think we were able to recover and that our students managed to soak in enough knowledge from our proverbial fire hose to create some useful applications.  I think we have a good start on a new book.

I was extremely impressed with the quality of students who were attending my ToscaWidgets tutorial.  Every single student finished every example.  I chose Pylons to give the tutorial, and although it is a little harder to integrate TW in the stream than does TurboGears2, it installed quickly and flawlessly.  Overall, I think the tutorial was a success.

Talks

This year I did not focus on attending the talks, but instead chose wisely based on speaker and topic and allowed my feet to do the walking if the talk became uninteresting.  I definitely missed some talks, but the AV team has done an incredible job putting the talks up on blip.tv so that I can review them later.

This year I did not miss Raymond Hettinger’s talk on AI in python and was enthralled by a speaker who could successfully put a page of code on the screen and keep my interest.  I showed up to support Philip Jenvey in his talk on Pylons on Jython but was impressed by his ability to provide a succinct example on where Jython really shines.  I am hoping that more people take a second look at this really well done presentation.

Now, I am a SQLAlchemy supporter through and through, but find the domain of database mapping an interesting echosystem.  While the ORM panel was littered by advertising chatter from one of the panelists who did not even write an ORM, an obvious dis-inclusion was Robert Brewer who wrote Dejavu, a very nice way to map persistent resources of different types for use in an “objecty” way.  Bob’s talk was especially interesting and makes me wonder if SQLAlchemy could leverage some of the work with AST that Bob beautifully displayed with some of the most amazing one-handed keyboarding I have ever seen.

Open Space

Well, I said I was going to give a talk at the Open Space, and ended up not doing so.  Part of the problem was the utter lack of projectors in the OS rooms, and part of it was a reluctance to break up the collaborative/discussive vibe that was going on in these sessions.  WSGIers hammered out a 2.0 spec, which involved a discussion I only monitored in passing.  I was disappointed by the lack of people who showed up for the GSoC BOF, but I think the economy held back a lot of students from attending Pycon.  It was also nice to allow my feet to walk around and see what was up in different projects.  I met one guy who took REST way to far and got to express some of my dissatisfaction with one of the available tools.  On a more positive note, the TG BOF was well-attended  and it was nice to see so many users wondering what was up in TG land.

Sprints

This year I refused to let the noobs get me down and actually wrote some code.  I am sorry if I did not act as a good host of the TG project, but we have some important milestones coming up and I just wanted to get work done on that.  Sprinting remains a cornerstone of our development process and I will see if we can’t get our monthly
sprints happening again in 2009.  I was however able to completely re-engineer our dispatch system, and while it is not currently 100% complete, it should be finished in a matter of days.  RestController now supports variable arguments for get_one, delete, and put, as well as supporting lookup and default.  Anyone can actually now create their own dispatch mechanism, since this functionality has been generalized.  Simply subclass Dispatcher, override _dispatch() and go to town.  I look forward
to seeing what kind of crazy code this brings to TG land.  A lot of discussion has been had on how to make “plugins” or “extensions” for TG, and you can rest assured that we will have this functionality soon.

Thanks

Thanks to all of my house mates who put up with my “mothering”.  Thanks to all of you who tolerated my “um”s at my talk on Sphinx, and especially to Georg Brandl who answered some questions.  Thanks to the organizers, volunteers and staff that came together to create what has been my best Pycon to date.

Pycon 2009

Wednesday, February 4th, 2009

So, Pycon registration has been up for a few days, I will be speaking both on and off-podium (read: open space) and providing assistance to and presenting tutorials.  Here is a run down of what I am planning in case you wanted a little bit more in-depth information.

Tutorials:

Turbogears2 Beginner and Intermediate:

I will be assisting Mark Ramm by giving individuals help installing and using the new TurboGears2 framework.  Mark is an experienced tutorial presenter, an expert in the technology, and in general a fun character to spend a few hours with.  When you leave his tutorials you should expect to have a working version of TG2 on your machine, along with an understanding of Model, View, and Controller paradigms.  Middleware, Forms, and REST will also be covered.  One note, if you are getting started with TG2, it’s best to have it installed and running if you plan to attend only the Intermediate Section.  We will not be going over installation in the second-half.

 Toscawidgets: Test Driven Modular Ajax:

I am presenting this tutorial which will describe how to use the valuable Toscawidgets package to create web content.  If you are currently use WSGI technology, and are interested in creating reusable, modular web content, this is a perfect way to get started.  I will show you how to configure TW middleware to work with pylons (which is applicable to other frameworks like repoze.bfg, paste, or even plone/Grok).  I will then describe how you might use this middleware to generate web forms.  The last few hours of class will be devoted to using the JavaScript utilities of TW to create an Ajaxified website, and test it using YUITest.

The Big F’ing Tutorial: Development Using the repoze.bfg Web Framework

I will assist/present with Chris McDonough about this up-and-coming framework who’s goals are to utilize bits of the zope 3 framework, wsgi, and new technologies to make a lighting-fast web server.  Those of you who are familiar with Zope technologies may be interested to find how nicely some of the familiar bits of zope are integrated with wsgi with repoze.bfg.

 Presentations:

Using Sphinx and Doctests to provide Robust Documentation

This is a 1/2 hour slot which describes how you can integrate tested documentation with your source code… with sanity!  I go over a quick install of Sphinx, and use some screencasts to demenstrate how to add, run, and display doctests using it.

Open Space:Agile Development with SQLAlchemy and Python Testing Tools

I really enjoy giving this talk, and even though it was not accepted as a formal talk, I will find a venue by way of Open Space to express my knowledge of Testing, SA, and Nose.  I have given this talk a few times now, and it’s fairly polished.  My presentation, while on some dry topics, won’t put you to sleep.  Carefully prepared screencasts and photograph-punctuated slides makes the 45 minutes breeze by.  Questioneers/Hecklers welcome!

 Sprint Topics

I want to spend some time with the Dispatch of TG2, and probably push Sprox further a bit.  If you are just starting with TG, please feel free to contribute.  Sprinting is a great way to learn a lot from the experts in the domain.  We usually do a meet-greet-install the night before the sprints.  Oh, and I’ve been known to provide refreshments to all of our sprinting hordes (read: FREE BEER).

So, I hope to see all of you there!  If you see me in the hall, feel free to introduce yourself and tell me what you are using Python for!

Coding Binge

Tuesday, January 27th, 2009

I haven’t written to the blog in a while.  Quite frankly, I’ve been busy.  In the last 30 days, I have released 3 software new packages, updated 1, deprecated 1, participated in a sprint that lasted a virtual 2 weeks, closed countless tickets, and pushed forward TG2 functionality.

TG2b4 was released last Saturday.  This was mostly a bug-fix release, but b3 is where the new functionality really came into the scene.  TG2b3 is the first build to include Sprox, a new library for schema-generated widget generation.  Sprox is the offspring of DBSprockets.  I decided I liked the declarative part of DBSprockets so much I wanted to spin it off as it’s own entity.  Sprox looses DBSprocket’s table-based dependency, utilizing the mapping provided by SQLAlchemy.  I realized that much of DBSprocket’s code was doing precisely what SQLSoup was designed to do, and decided to focus my efforts on making and extremely configurable widget base.  The result was a considerable removal of the cruft that was associated with DBSprockets.  Sprox releases with an excellent documentation base provided by Sphinx.

There has been a bit of resistance to Sprox, people were/are confused/upset about my providing yet more options for schema based widget generation.  The fact is I have yet to find anything that performs as well as Sprox from a developer/speed standpoint, and I needed to provide our TurboGears user base with a better way to administrate their site, and also allow them to use that tool component-wise in their system.  I think this method for developing widgets is well done in other frameworks, and we need a solid answer to this problem.  Sprox is just that.

The next step was to re-work Catwalk to use Sprox.  This took a little effort, and I put in RESTful URLs while I was at it, but struggled with making the URLs work within TG2’s dispatch system.  The result was as close to REST as you can get without conforming to a set standard.

The result of hacking REST into Catwalk got me thinking, and I decided to implement content-type dispatch as well as RESTful dipatch in TG2.  I went back for another round on Catwalk, and converted it over to the standard.

I’ve also been toying around with Dojo at NREL.  I’m pretty much done with ExtJS due to licensing issues, a not-so-hot codebase, and weak support from IRC.  It’s bad when you go on to ask a question on the channel as a 6 month-user of a software project and end up spending all your time answering everyone else’s questions (as the most experienced person in the room).  Something must be said for an organization that does not push paid consulting as a primary focus on their website…  #dojo has been an exceptional resource for getting my work done.  Those guys know their software, and lend a great hand to help you with it.

Back to the topic at hand… I was able to shoe-horn Dojo into Sprox with little effort, and implemented DojoCatwalk, which worked, but was ultimately not what I wanted.  What I really wanted was configurability.  I started work on tgext.admin, which was supposed to provide enough functionality to replace tgcrud, a command to auto-create crud in your own TG application.  To support tgext.admin, I created a new package called tgext.crud, which provided a CrudRestController, which is a simple way of providing crud for any object in your model.  AdminController combines this functionality with that of Mark’s lookup code to provide a fast, configurable set of tables/forms/etc for all objects in your model.  AdminController takes a declarative AdminConfig as input which provides a consistent way to create your administrative toolset.  Did I mention it does Dojo tables with ajax loading?  Yeah.

I’m not done with this binge yet.  Catwalk is going to mutate one more time before I’m through with it.  It is going to become a default-configured AdminController specifically designed to work within the context of a quickstarted TG2 application.  I had one blocker ticket which was solved last weekend, so it’s time to get Catwalk good and finished.

Google Summer of Code Mentor Summit Recap

Wednesday, October 29th, 2008

I spent last weekend in Sunnyvale, CA at the GSoC mentor summit.  This was a great opportunity to rub elbows with a number of experts in various OSS projects.  It was great to make connections with people in and out of the Python community.  First off, I just want to thank Google, and Leslie Hawthorne for having us. Google was an excellent host, and I would jump at the chance to attend a Google-hosted event again.  Google is using GSoC to promote the open source community.  By providing funding for students to work for a summer on software instead of flipping burgers, they hope to save the world, one open source project at a time.  I have my own take on what Google’s true intentions are, but I don’t really want to discuss that in this blog entry.  That is to say, I don’t think Google’s intentions are in any way evil, but they are definitely self-serving.  In any event, it is remarkable the amount of time, energy, and money that goes into this project, which is at this time one of a kind.  Here’s hoping that other companies will consider the not-so-small buy in for sponsorship which could lead to even more OSS development.   

As for the summit itself, there were lots of talks.  Since it was run like an open-space conference, it took a little while to work up some speed the first day, but I never found myself want of something to do.  I was not surprised that the talk on monetizing your OSS project was wildly successful.  I was surprised at how easy it was for sessions to get off on tangents, but it was interesting to see how people in similar fields were solving projects.  I attended a session on Scientific Computing (since I work for NREL) and it seemed that the biggest problem the room faced was packaging/deployment of software.  I appears this problem cross-cuts platforms, language, and even hardware barriers.  It also seems that you either have a choice to make: stick with old software that is stable, or upgrade to new stuff that almost always breaks other things.  Sound familiar?  I wonder how long it will be before everyone runs isolated virtual machine images for each of their projects… I bet some already are.  

By day two I was pretty exhausted of participating in sessions, and I just wanted to hack with some like-minded folks.  It turned out that at least 8 people at the summit were hacking on making their application run on Python 2.6, so we all got together and shared the good vibe while we solved our individual problems.  The main barrier for TurboGears 2 running on Python 2.6 was ToscaWidgets and TurboJson.  First off, peak-rules needed some minor tweaking to get a few of the bugs fixed.  Thanks to Philip Eby for implementing my patch (in a slightly different way) and making it all work.  ToscaWidgets was also breaking with the SimpleJson trunk, but this turned out to be an implementation error, and I think I have it mostly sorted out.  

Yesterday evening Mark and I finished up packaging the !first beta! of Turbogears2.  I put a few custom eggs in the index from my GSoC hacking, and suddenly we have a Python 2.6 compatible release.  Also, of interesting note Gustavo Narea cranked out tgext.authorization which handles most of the security for TurboGears2.  Now I just need to get cranking and fix DBSprockets Primitives to work with Rum, and then fix silverplate to support Rum also.  (Perhaps this is my way of telling people to move in the Rum direction).  

Mark and I will be attending PyWorks in November, and doing a sprint then.  So, if you are in town, or interested in sprinting remotely, please don’t hesitate to sign up.  I will be speaking at PyWorks on using PasteScript, Virtualenv, SQLAlchemy, and Nosetests to create robust database applications. Do you think there is anything else I could cram in a 45 minute talk?  I will also be helping Mark out at his TG2 tutorial, so if you aren’t signed up for PyWorks, get crackin’ !

Open Source Software and Production Engineering

Saturday, September 27th, 2008

In the past I have struggled with the fact that some of the OSS code that I write wraps other’s work, creating an evil dependency which is not easily rectified.  I am then bound by the wrapee’s software release schedule, as are those who choose to use my software.  Yes, I’m talking about ToscaWidgets.I wanted to break free from this paradigm, and I have taken initial steps in my ToscaWidget wrapper for Yahoo YUI.  Here comes a huge block of code, but don’t let it scare you, I will explain the problem it intends to solve:


class YUILinkMixin(Link):
    params = {'basename': '(string) basename for the given file.  if you want yuitest-min.js, the base is yuitest/yuitest',
              'suffix': '(string) "", min, beta-min, or beta.  Default is "min"',
              'version': '(string) select the yui version you would like to use. Default version is: '+__YUI_VERSION__,
              'external': '(boolean) default:False True if you would like to grab the file from YAHOO! instead of locally',
              'yui_url_base':'The base url for fetching the YUI library externally',
    }

    version = __YUI_VERSION__
    external = __DEFAULT_LINK_IS_EXTERNAL__
    yui_url_base = __YUI_URL_BASE__
    modname = "tw.yui"
    extension = 'js'
    default_suffix = __DEFAULT_SUFFIX__
    _suffix = ''

    def __init__(self, *args, **kw):
        super(Link, self).__init__(*args, **kw)
        if not self.is_external:
            modname = self.modname or self.__module__
            self.webdir, self.dirname, self.link = registry.register(
                modname, self.filename
                )
        if 'suffix' in kw:
            self.suffix = kw['suffix']
        else:
            self.suffix = self.default_suffix

    def _get_suffix(self):
        if self._suffix == '':
            return ''
        return '-'+self._suffix

    def _set_suffix(self, value):
        self._suffix = value

    suffix = property(_get_suffix, _set_suffix)

    @property
    def external_link(self):
        link = '/'.join((self.yui_url_base, self.version, 'build', self.basename+self.suffix+'.'+self.extension))
        #xxx:check for existance of this resource and choose -min -min-beta -debug or no suffix as alternatives.
        return link

    def _get_link(self):
        if self.is_external:
            return self.external_link
        return tw.framework.url(self._link or '')

    def _set_link(self, link):
        self._link = link

    link = property(_get_link, _set_link)

    def abspath(self, filename):
        return os.sep.join((os.path.dirname(__file__), filename))

    def try_filename(self, filename):
        abspath = self.abspath(filename)
        if os.path.exists(abspath):
            return filename
        return False

    @property
    def filename(self):
        #make basename windows/qnix compat
        basename = self.basename.replace('/', os.sep)
        basename = self.basename.replace('\\', os.sep)

        basename = os.sep.join(('static', self.version, 'build', basename))
        #try the default
        filename =  basename+self.suffix+'.'+self.extension
        if self.try_filename(filename):
            return filename

        #try '' if the default suffix is min
        if self.default_suffix == '':
            filename =  basename+self.suffix+'.'+self.extension
            if self.try_filename(filename):
                return filename

        #try min if the default suffix is ''
        if self.default_suffix == 'min':
            filename =  basename+'.'+self.extension
            if self.try_filename(filename):
                return filename

        #try debug
        filename =  basename+'-debug'+'.'+self.extension
        if self.try_filename(filename):
            return filename

        #try beta-min
        filename =  basename+'-beta-min'+'.'+self.extension
        if self.try_filename(filename):
            return filename

        #try beta
        filename =  basename+'-beta'+'.'+self.extension
        if self.try_filename(filename):
            return filename

        #try beta-debug
        filename =  basename+'-beta-debug'+'.'+self.extension
        if self.try_filename(filename):
            return filename
        return None

    @property
    def is_external(self):
        return self.external

class YUIJSLink(JSLink, YUILinkMixin):
    pass

class YUICSSLink(YUILinkMixin, CSSLink):
    extension = 'css'
 
So, the question is, how does using this code differ from using direct links to the YUI library, or using the ubiquitous JSLink Widget provided by ToscaWidgets, which is found in just about every other tw.js_wrapper available?
1) Testabliltiy - First and foremost, I wanted a way to test a new version of the library with my JS code to see if it would work, without having to change the wrapper at all.  A bit of a monkey-patch, but if you change the version of YUI that is fetched by altering one variable in the tw.yui library before doing subsequent imports.  If the current version is not available in the widget library, you can point it directly at yui's online library.  Run your tests, see what breaks, fix it, and you are now compatible with the new library.
2) Debugability - Synchronous with Testability, you can also set a flag for tw.yui to include all of the debug versions of the yui modules so you can dig down into the code using firebug if need be. (The default is minified)
3) Compatibility - Since you now can control what version of YUI you want to use, you can retro-fit an older project with Toscawidgets by downgrading the version of YUI that tw.yui is using.  You can also try out the newest version of YUI that may tw.yui is not yet using, and with the exception of new modules that have not been integrated into tw.yui, it should still work.
4) Maintainability - Since we now have the ability to swap versions, the maintainers of tw.yui can keep multiple versions of yui in the library, and deprecate them as they see fit.  This provides maintainability, as you would expect to see deprecation warnings about your WSGI Application's use of an out-of-date JS library.  This helps the whole process of knowing what libraries are current to what version, which can be a real bear, given all of the things we all must keep track of.
5) Producibility - The bottom line always comes down to whether or not you can pipe your development software up to production level efficiently.  (Is the code producible).  With the ability to test compatibility, and drop down to different versions of dependent libraries when compatibility is a concern, you will be able to keep all that works on your production server working, while bringing up new or updated functionality incrementally.  Keep in mind that the above code demonstrates the ability for system-level defaults of the javascript link widgets to be overridden for any particular widget instantiation.
These 5 items I call Production Engineering.  Basically, Production Engineering is developing a process which allows you to integrate working development code into your stead-fast production site, and allow you to back out software which may have mistakes.  A good software engineer realizes that people make mistakes, and there needs to be a way to buy some time so those mistakes can be fixed.  I believe that YUILinkMixin is a step in this direction, and I hope to extend it in the future to provide an even more robust toolset in line with Production Engineering.The complete source code for tw.yui is available at www.toscawidgets.org/hg .