percious.com – Python

Archive for the ‘Python’ Category

Sprint Organization: But can we do more?

Tuesday, October 13th, 2009

The ultimate sprint incentive would be for companies to put up a bounty to fix bugs, or otherwise provide support for an open source project. This is the toughest thing to sell I think, but if Google can do it, why can’t other organizations? I have seen this attempted in the past with mixed results. I think it was a half baked idea, and that we can do better than that. The challenge is metrics, and making sure that everyone gets their fair share. I mean, at that point, sprinting becomes payment for work done, the sprinters are just hired guns for a weekend.

Autograph by kugelfish.

I think the challenge has been, and will always be metrics. How do you measure the work and who did it, and how much is each thing done worth? I think the sprint organizer is the greatest asset in this situation. He usually has the best idea about what needs to be done, what has been done, and ultimately who did what quantity/quality of work. So you can use him as your metric definer.

My idea is this: A sponsoring company provides the OSS organization with a certain amount of money, and what they would like to see achieved in an organized sprint. This money is given to the organization as a donation regardless of the outcome of the sprint. The donation may also be given to the organization with no stipulation of task, but the sprint organizer must choose a topic of interest in order to guide the sprint.

The sprinters all agree to a set method for dissemination of funding provided by the sponsoring organization. My suggestion would be to split all of the funding equally, but in a capitalist nation, it is hard to justify giving the same funding for a person who did a little work on a piece of documentation vs. someone who spent 40 hours putting together a full tutorial. Another method would be to allow the sprint organizer to disseminate the funding as he sees fit, but the sprinters would have to agree to this before work is started. The work would have to be completed by a certain deadline to obtain the funding, because project organizers do not have time to track down who did what 4 months after the fact, it’s just not practical. Finishing a week after a weekend sprint seems reasonable to me.

I am sure to see lots of comments about these ideas because when money gets involved, everyone gets uppity. (one of the great ideas behind OSS is that there is no money paid for the actual software)

So, there exists possibilities for setting up a sort of Round-up style sprint. The goal is to provide sprinters incentive for providing bug fixes, or even to document a part of the code that is otherwise hard to get anyone to document. I would love to open a discussion about this topic, as I see it as a completely different business model when it comes to software development.

But you have to ask yourself, “What’s in it for the company?” Well, first I think you have to ask yourself why Google has been running programs where they pay for OSS development, no questions asked, for 5 years now. But here are some insights while you ponder that. The easiest reason to understand is that the sponsoring company has some bugs in OSS software they use that they need fixed, and a team of experts can fix them in a weekend, or their staff can submit bug fixes, go through the rigamarole of OSS contribution, and things get done in months.

The next less obvious reason for a company to provide funding for a sprint is that they are using OSS software that could use a little help in the documentation department. By paying for documentation, they are getting a cut-rate deal on the experts that usually created the software providing documentation for those things that may be crucial to their business’s success. This also reduces their dependency on any one employee who may have in-depth knowledge on a particular software package. By ensuring the OSS software that the company uses is well documented, the company ensures that the intellectual know-how for that piece of software remains with the project, not with the employee that may leave at some later date.

The last simple reason I can think for a company to sponsor OSS is for recruiting reasons. Typically the people involved in the sprinting process are the ones that know the most about it. It is also be a chance to evaluate an employees enthusiasm for work in general. Those who are sprinters are more likely to work well with others and stay with projects for the long haul, in my experience.

If software development is described as herding cats, gathering and directing sprinters is like herding stray cats. I once had a stray cat visit me when I was in my potato-tuna days, and I gave it some of my tuna, because it looked in worse shape than was I. And you know what, that cat always returned for the tuna. Now, I’m not much of a cat person (current cat count: zero) but I think that if software developers are truly analogous to cats, they might show up at your door once, but it’s much easier to get them to come and visit regularly if you give them some tuna once in a while.

Posted in Agile, Software, Sprint, Turbogears | 48 Comments »

Sprint Organization: Is our Current Model Broken?

Tuesday, October 13th, 2009

How does one entice people to thrive by sitting and writing documentation? Rather, how do you reach out to a greater audience of developers that you could enlist for the betterment of your project? The classic model has been, organize a theme, have the “elders” show up, and get some work done. This sort of works.

The problem with this model is that I find a ton of people who basically show up to the sprint for a free tutorial. You get them up and running with your source code, hoping they contribute something useful, and you spend an awful amount of time doing so, because they have some weird configuration or are in way over their heads, and by the time you get them set up, the sprint is over. Now you just wasted your own time, and that of the project’s. The only person that benefitted from that situation is the sprinter, who can use the information he gathered to thrive.

The only way I have personally been able to contribute a serious amount of work is to hide in a corner and code for hours alone while everyone else is “sprinting”. Well, that sort of seems to negate the need for a sprint, since I can do that without the hands-on interaction that a sprint is supposed to encourage. So, when no one showed up at the last sprint, I had a bitter sweet taste in my mouth, but I closed about 20 todo items.

The main benefit I have seen from a sprint is that it is typically a gathering for the “elders”. The elders might talk about the future direction of a project. In the best case scenario they write some test code to see if an experimental idea will work or not. Sprinting is an incredibly valuable time for this, but often times the elders are held captive by (what we call in the climbing world) “gumbies” who capitalize on the elders’ knowledge of whatever project you are working on. Often times I will see the elders hide in another room to have a discussion, much in the same way I have done to get code done.

Crying by rabble.

One solution we have come up with to deal with this problem is to institute a “babysitter.” This person is usually also the sprint organizer, and whether they know it

or not, their drive to have a successful sprint ties them to help the gumbies succeed, even if the chance of such an occurrence may be low.

I have played this role quite a few times, and while I am happy to answer questions at a sprint, I generally point people towards the docs and have them work through our examples. If they find a problem with the tutorials, they have just found something they can sprint on and be successful!

The other major benefit we get from sprinting is that it is an awesome venue for starting new things. Since the elders get together and sometimes create some test code, you are often left with a nice base to work off of. Many times I find myself working the next 3 or 4 evenings after a sprint trying to finish things out. This is often how new features are added to a project.

So, the question you have to ask when organizing a sprint is not only how to get people to show up, but how to get the “right” people to show up. I think for this we need to change the way we think about sprinting, and come up with a new methodology for organizing sprints.

This is part two of a four part series on sprinting. In the next segment, Sprint Organization: New Rules! I talk how we could improve sprinting using our current resources.

Posted in Open Source, Python, Turbogears | 1 Comment »

A TurboGears Weekend in review

Sunday, October 11th, 2009

A few weeks ago the Front Range Pythoneers decided to organize an “Uncon” where people show up to discuss various topics on-the-spot. This is the sort of event I really enjoy participating in, so I of course agreed to attend. At the same time, I was approached by the guys from Developer Day to do a talk on TurboGears. You can imagine the conundrum I faced, but thanks to the willingness of the DD organizers to be flexible, and some creative planning, I was able to participate in both.

Speaking at Developer Day was a new experience for me because I was talking to folks that were not necessarily versed in Python,never-mind TurboGears. The conference appeared to be somewhat Rails heavy, but it was refreshing to see organizers reaching out to the greater web community to provide a well rounded conference. The nice thing about speaking to a wider audience was that I was able to expound some of the history of the Python web, as well as describe TurboGears at a high level without worrying about boring the audience. I was quite nervous speaking at first, because I have not done so in a few months, but seemed to settle into a groove by the time I showed an example of how easy it is to inject repoze.profile into a TG application and provide a cachegrind display to find any slowdowns in your app. I hope that this example was able to express how versatile WSGI is.

I stayed the morning at DevDay and I am glad that I did. Chad Fowler gave an address on what it means to remain passionate as a developer over the life of your career. I think his idea that providing structure to your life definitely allows you to achieve amazing things. His real-life examples were poignant and well received. I’ll be checking out his book soon. The other talk that I found interesting was Jeremy Hinegardner’s talk which basically discussed the numerous non-relational persistence methods available. I thought his method for showing examples of the different methods was great. For each one he had a simple succinct example that showed the benefits for the given persistence framework. He allowed the audience to choose from the frameworks he discussed in his talk. Jeremy was an engaging speaker, and I would not hesitate to sit in on one of his talks in the future.

Google in Boulder, CO by krossbow. After a bit of DD-provided BeauJos, I headed over to the UnCon. They too were having pizza provided by Google. Google Boulder was a great sized venue for the 40 people that attended. It was exciting to see so many new faces in attendance. It seemed to me that the “regulars” were doing a lot of demoing, while the new folks watched on, but there was also a lot of discussion that happened. I showed how to use repoze.profile and runsnakerun to

analyze the results. Zooko immediately installed runsnakerun and tried it on his app. It is always nice to have immediate gratification for having taught someone something, even more so when the person voluntarily tries what you think is “so cool.” I got to show off some of the work I am doing for www.getmvp.com, since much of it is prototypical of the Extension Solution that I hope to provide with a combination of Pylons and TG. Also on display was TW2. It was great to show how simply one could express all MVC elements of a widget in one complete package.

Sunday I ran the first TurboGears WorkShop. If you follow my blog, you may have read a few posts about how I think we can improve sprinting, but I’ve come to the realization that our less-than-stellar sprint performance is really due to a need for improvement in the organization at large. I have decided to add a WorkShop Series to our tireless effort for improvement of TurboGears, both from a technical aspect, and one of the community. I was up late on Wednesday creating a basic tutorial-type plan for Sunday, and I finished up with about 80 pages of documentation to provide workshop goers, basically by selecting items from the TurboGears documentation. My goal for the sprint was to provide sprinters with a working example of TG at the end of the day, with a little bit of work accomplished customizing the Admin. I asked sprinters to bring their own databases, to utilize sqlautocode as an example database for their new application, and while no one provided the class with one, we were still able to succeed with one that I provided as a backup. 5/6 people succeeded in this, and while there were some rough edges, I think I have an idea that is workable for a 3-6 hour WorkShop that will succeed with a little bit of polish.

I am still formulating the ideas for TurboGears workshops. I have started to contact folks I know throughout the country, in order to provide venues for these workshops. So far I have Boston, Dallas, San Fancisco, Atlanta, and Boulder (Denver) lined up. I think with little effort, I could also add Ann Arbor, and probably Washington D.C. The idea behind a workshop is that you arrive with a varying amount of knowledge in TG, and you leave with a greater knowledge than you arrived with. You are encouraged to bring an existing project to hack upon, or to create a new one that we can play with. I will provide a rough outline of what we might do in the tutorial, but if the group decides to go off in a different direction, that’s okay too. If you are interested in participating in one of these WorkShops as a mentor, or providing venue space, accommodations, etc. I would love to hear from you. Right now I am in the organization phase, expect a blog post announcing the official plan in the near future.

Thanks! Without the efforts of a number of individuals this weekend would have been much less successful than it actually was. I want to thank Ben Scofield for inviting me to talk at developer day, and for shuffling the schedule so I could participate in both conferences. Greg Holling put in a great effort to organize the Uncon, and Google provided an awesome venue for us to use. Three volunteers from Google Boulder provided their time, and even gave a tour of the facility to conference goers. They weren’t even Python developers… Jim Baker and Matt Boersma both showed up to provide access to Bivio so that we could have our first-ever TG WorkShop. Lastly, I’d like to thank Bruce Eckel for making the trip down from the mountains to provide his unique perspective.

Tags: Turbogears, WorkShop
Posted in Open Source, Python, Software, Turbogears, Uncategorized, pycon | 86 Comments »

TurboGears 2.1a1 released

Friday, October 2nd, 2009

Introduction

We are ready to start testing the next version of TurboGears: 2.1. This release is the first of what will be a series of alpha and beta releases before we move into production. 2.1 is not a huge departure from the 2.0 codebase, rather, it’s efforts are to clean up and speed up the existing codebase, both conceptually and technically. So, if you have some time, install it today and give it a whirl!

Major Differences (Things that affect present 2.0 users)

==================================================

Rendering

The item that will affect most 2.0 users is the renderer. Json rendering is now not a special hard-coded case, so you will need to add it to your default config. Most 2.0 apps will have to add the following line to their app_cfg.py if you are using @expose(‘json’) at all::

base_config.renderers.append(‘json’)

If you should forget to do this, you will get an error message reminding you to do so.

TurboJson

Support for TurboJson has been removed. We have not found many people using this, and in fact, if you still need it, you can still put it into your TG application. For the most part SimpleJson does a good job of rendering Json for us, and because it is a part of the python default library in 2.6, it makes sense to utilize it. This allowed us to remove about 8 package dependencies.

Minor Differences (Things that affect folks familiar with how TG already works)

==================================================================

Dispatch

The dispatch mechanism has been completely refactored. This means that pesky things like requiring *args at the end of a RestController.edit are now not required. The new dispatcher is much faster, up to 200% faster for RestController dispatching. It also has the flexibility to add a _dispatch() method to your Controller class to drive the dispatch. This is not yet documented, but it’s on the high priority list for documentation.

Setup-app

Setup-app has been broken into two pieces, schema creation and bootstrapping. This will not affect existing TurboGears apps, but if you were familiar with a certain way quickstart works, this has changed. There are now schema.py and bootstrap.py modules in the websetup directory of your quickstart. These isolate database schema creation from adding bootstrap data to the database. This helps when you are writing scripts that modify a database, but do not add the bootstrap data (useful for testing). Having them in separate modules makes the separation of concerns more apparent.

Catwalk

While the branding was nice, we decided to drop the Catwalk name in order to make it easier to follow tutorials that utilize tgext.admin. Therefore, in quickstarts, you will no longer see catwalk, just tgext.admin calls. This also allowed us to drop yet another dependency. Catwalk will still remain a viable package, but it will no longer be maintained. The good news is that nearly all the functionality for the admin resides withing tgext.admin. Catwalk was simply a configuration of the former.

New Features

===============

Documentation

A concerted effort has been made to increase the depth, breadth, and organization of the docs. Areas of focus have been: Organization of the docs themselves, Configuration, Tutorials, and Utilities. Part of the 2.1 push will be to get the docs to a level of completion never seen before, and an organized effort is taking place to ensure this aspect of the project’s success.

Mako!!

TurboGears2 has always supported Mako, but we are now providing template support in our quickstart. When you quickstart a new project, it will ask you if you want to use mako templates instead of Genshi. Mako is usually about 3x as fast as Genshi for complex rendered pages, and up to 10x as fast for simple ones. Careful attention has been given the Admin to make sure it still works with Mako (it does), so you can be certain if you make the choice to use mako, it will work out of the box.

local:

In order to handle template inheritance gracefully, an identifier for the local project has been added to the template lookup in Mako. This allows the admin to inherit your local project’s master.mak file. An inhertance clause in Mako that uses local would looks something like::

<%inherit file=”local:templates.master”/>

Genshi has support for this automatically, but it is not explicit, and we are looking at ways to support this explicitly before 2.1 goes to final.

ToscaWidgets2 Support

ToscaWidgets2 recently made a 2.0a1 release. We have included in TurboGears2.1 the ability to easily configure your application for TW2, along with other added support for this next-generation widget framework.

Thanks

=======

This release comes not without considerable effort on the part of the TurboGears team. I would like to thank Michael Pedersen for his undying effort with the docs. Michael helped to collect about 190 todo items for our docs, and squashed a countless number of them. We now have about 130 items todo on the docs, but that number is ever-decreasing with his and other’s effort. Thanks to those folks who have contributed to the DocSprint, and who still continue to contribute, including Michael Fletcher, Jorge Vargas, and Seth Davis. If you use TurboGears, and find you need to dig into the source code to figure stuff out, please help us make the docs better by contributing to a DocSprint or sending us a pull request.

Thanks to Jorge for straightening out the Json rendering issue. Also, by removing TurboJson from the stack of required packages, we have opened the door for TurboGears to run on AppEngine and Jython. This would not be possible without Jorge’s effort.

Thanks also to Mark Ramm, Christopher Ardnt, Florent Aide, Alberto Valverde, Paul Johnston, Christoph Zwerschke, and Lee McFadden for their continued support of TG.

Finally, I just wanted to send a thank you to the folks who have contributed to the TG codebase by association. Mike Bayer, Jason Kirtland, Ben Bangert, Philip Jenvey, Chris McDonough, and last but not least Ian Bicking. Thanks for all of your effort making possible this great conglomeration of parts.

Posted in Python, Turbogears | 70 Comments »

TG’s Killer Features: SQLAlchemy. Obvious, no?

Monday, September 28th, 2009

So, as I stated in a previous entry, I’ve been using SQLAlchemy for more than three years now. If you know anything about me, you’d find it amazing that I have contributed little more than a patch here or there. Why is that? I think that’s because every time I try to find something that I need SQLAlchemy to do, it already does it. I have spent so little time trying to make SQLAlchemy do what I want it to do, and so _much_ time getting work done with it.

Here is a good example. One of my clients is a

company developing sports management software by the name of MVP. Although I work on their next-gen stuff, I was called on by them to promote the students on the older system to the next grade over in the summer time. Their database system had well over 40 tables, requiring no less then 10 of them to have modification. I fired up sa, reflected the changes, wrote the changes in python, with simple loops around what records I needed to change, an it was done. In one hour’s time I had a happy client, and a school system with a functioning system. Two things about this are amazing. 1. At the time, I had practically zilch in the Postgres experience department. 2. I only needed to learn the schema of the tables I was modifying, and I could do this with introspection. The fact that I was able to do this task in less than an hour was only made possible by my knowledge of SQLAlchemy’s table-based architecture. I did not need to know the nuances of Postgres’ SQL language (I was more familiar with MySQL at the time), or a in-depth knowledge of the database schema. I was able to pick and prod until the job was done, and it was painless.

I really like the fact that SQLAlchemy’s approach to making an ORM is layered. I can drop to whatever layer I need to to meet the requirements of my project. SQL, table objects, and mapped objects all have their place in the grand scheme of things, and I have used them all to varying degrees. What SQLAlchemy achieves with this layering is the ability to adapt to existing projects, schemas, etc, and make considerably useful software, in a short amount of time. By figuring out the nuances of different SQL dialects, it frees me up to focus on the task at hand, and provide products to my clients that work in a broad range of situations.

But this post is supposed to be about TurboGears, right? For me, SQLAlchemy is more than a tool I use for TurboGears projects, or even web applications, it’s something I can use any time I have to access a relational database. It is the ultimate base for writing tools that help me get my job done. That job might be providing a system to allow schools to schedule matches against each other, or enabling scientists to access their data directly using objects at a python prompt, instead of assembling arcane sql strings to gather data. SQLAlchemy is such a good basis in fact, that it makes building tools to help do my job even easier.

I have had the opportunity to contribute to sqlalchemy-migrate. If you don’t know anything about the project, consider this: You have a production database you cannot break. You have 5 minutes downtime to modify the schema, update records, etc. You need to be able to back out the changes and bring the system back up if everything breaks. Migrate lets you do all of this, and provides a versioning system to ensure that such a process moves smoothly. I have used migrate with postgres, found some rough edges and fixed them. SA’s table mapping makes this possible, migrate just adds a layer to make versioning and some table creation processes easier. I only hope that someday some of the migrate code makes it’s way back into the SA codebase.

I do lots of testing. I made numerous schema changes today to one of my databases at www.nrel.gov in fact. I test the database schema for

matching against my definitions with some yet-to-be-released software. There are about 3000 tests. With tables numbering in the fifty-semod, I needed something to make data entry easier for my tests. I wrote bootalchemy to do this. You pass it some YAML, and the models module, and it performs all the entries for you. It does a bit of introspection to determine dates, and it has reference pointing so you can inter-connect your objects within your yaml (using & and * like my old friend C). This has vastly decreased the amount of time it takes me to create new test data for new tests. Again, this is possible only with the framework that SA provides me.

Lately I have been interested in providing a broader base of scientists at NREL access to their data using python as a medium. Scientists (especially physicists) are really good at conceptualizing data. They use crazy tools like Matlab, and R and all sorts of proprietary tools to manipulate their data. They aren’t afraid of a command prompt. My idea is to give them something like Sage, but with direct access to their data as mapped objects. I also want to be able to show up at a scientist’s desk with laptop in tow, connect to one of their existing databases, and spit out a admin-style web interface based on TG in a few minutes. These notions have driven me to contribute sqlautocode. At this point sqlautocode will spew out a page (or 7) of python code that provides you with Declarative Objects and an interactive prompt. sqlautocde works as a library, so you can use saautocode’s output to directly in memory without generating any code at all. All this is possible with SA, and I don’t know of anything else that can do all of this.

I have focused on the technical here, but beyone that is a great team of individuals like Mike Bayer and Jason Kirtland who put in long hours and answer questions promptly on the mailing list. I often wonder how these guys get any sleep. The thing that excites me most about SA is that it will soon release version 0.6, which to me means that these guys have 4 more versions of increased functionality before they consider it “done.” So, as you can see, there is more to SA than just the ORM it provides. It’s framework gives you freedom to expand your horizons and get your job done, by focusing it’s task on the challenges that relational databases all have, so you don’t have to. This is what makes SA a killer feature of TG.

The third segment in this series is: TG’s Killer Feature: The Admin (Yes, We do, and it rocks)

Posted in Python, Software, Turbogears | 47 Comments »

Best of Breed: TG is still the best choice for people who hate hitting the wall.

Monday, September 21st, 2009

We get a lot of refugees who come to TG from other frameworks where they got themselves to a place they could not get out of. I have worked with some of these frameworks in the past. Things are awesome in the beginning. You work on some less complex stuff, maybe change a template around, or the theme for a site. Then you have some technical detail you need changed and BAM! You just hit the wall. Now you are forced to dig, or suck it up and go on IRC and ask noob question. Many times I find myself getting shut down, told to RTFM, or whatever.

With TG I never really felt the wall. TW was a bit of a hump to get over, but seriously, things are so un-coupled in TG that using a technology you are already familiar with, or swapping out the standard stack is not something totally unheard of. I for one do not like working with Genshi. I’ll do it, and I support it for major OSS work I do, but really I prefer Mako. Mako is fast, works in a non-xml way (which makes it great for writing form-emails for instance). What this means is that TG has good support for Mako in the “standardized” parts like the admin.

The thing is, we really did not have to add much to TG to allow it to work with Mako. I think one of the ingenious things that TG supports is dotted template lookup. What this does is allow you to pull data from any package, because the lookup occurs using pkg_resources. Beautiful, now we have the ability to move templates into their own succinct packages. Also, we support non-dotted template lookup for template languages like Jinja, and adding support for dotted lookup wouldn’t be too hard to do in the future.

The new TGAdmin interface is another example where hitting the wall is just not something that happens. TGAdmin is built on new technologies for TG2.0. Namely RestController, Sprox, and lookup. It creates a custom controller for each of your model classes, and therefore you have a good place to start from when you get going with your application. This is great for demoing. With SQLAutocode, Sprox, and TGAdmin you can literally hook into your client’s database (MySQL, Postgres, sqlite or MSSQL) and generate web forms where you can have an infinitely scrolling tableview, edit, and create forms. From there you have the ability to customize further, by hiding table/form fields, changing the look and feel of any component in the chain. This is made possible by Sprox’s configuration interface. You can also modify the controller code for each model by adding controller methods to your default controller in the admin’s config object. All of this customization is outlined in a tutorial.

The great thing with the new TGAdmin is that since it is based on Sprox, you can re-use the knowledge of sprox externally to the admin, and also bring any existing knowledge you may have about ToscaWidgets to the table. You can even use sprox outside of TurboGears for any other python web-based applications that you have that also use SQLAlchemy (read: Pylons).

To get back to the TurboGears discussion, a lot of folks don’t want everything that TG brings to the table. Some folks need different auth/auth models, have no use for a widget library, or even a relational database connection. TG can serve you too, and it’s still fast. Recently SourceForge moved over to TG for their main website. This requires a connection to MongoDB on the back end (read: no SQLAlchemy), zero ToscaWidgets, and a different authentication method. According to Mark Ramm, they had started with a different framework which lends itself to the Jinja2 templating engine. As far as I know, Mark was able to meet all of these goals, and eliminate 9/10ths of the server load they had with their previous system, written in something non-python. This just goes to show how flexible TG was to be able to meet all of these needs, and still provide a technically sufficient solution. Maybe he will comment more about this in the future.

Now, there are those folks out there who have decided to role their own. And WSGI definitely supports, if not encourages this behavior. But I just have to ask: Who’s agenda are you really fulfilling? Even if people like your code and you have a small following, are you helping to further the benefits of your projects, or your career in general? Are you helping your customer in the long run if you leave the project, or leaving them with a dead-end piece of code? Will they be able to find someone to replace you, should you decide to leave?

TG has an active community of folks who are willing and able to help get your contributions into the main code branch. We have embraced mercurial, and use it’s abilities to offer a lessened barrier for those who want to contribute. So, please, before you go make your own framework, see if you can help us make TG better, and your reward is that you now have a community of folks that will help maintain your work, even if you decide not to.

Basically what I am saying here is that TG has an even learning curve. Sure, as you get more involved, the problems will get harder, but you wont end up having to re-write half of the framework just to get it to do what you want. And, if you do find something that needs work to meet your needs, you have options to participate in the development of TG.

In 2008 TurboGears ran a sprint series to flesh out the 2.0 release. We successfully released in Spring of 2009, thanks to the hard work and dedication of a number of folks who saw the process through. This was a great opportunity for folks who wanted to be actively involved with a web framework to jump in. It is very likely in the coming months that TG 2.1 will see the same sort of community out-reach as we prepare to move from a development cycle, to a release one. So, look forward to that, find out how you can contribute, and by all means, give us feedback as to what you really want from a Best of Breed framework.

Posted in Python, Software, Sprox, Turbogears | 6 Comments »

Best of Breed: TG’s job is hard. Here’s why.

Thursday, September 17th, 2009

In 2005 Kevin Dangoor made the decision to glue together a few existing technologies into something that would be useful for the creation of web pages. This was marketed as the “Best of Breed” selection. The challenge with Best of Breed is that the best is constantly in flux. Also, making that claim requires TG to put in a serious amount of effort in _finding_ and evaluating the best of breed technology and then integrating them with TG. This is a job I LOVE doing. Even better than that, when I find something that doesn’t cut the mustard when it comes to being the best of things, I get to write it.

For example, when SQLAlchemy blazed on the scene, and TG was one of the first frameworks to support it. My own frustrations with SQLObject lead to an early adoption of SQLAlchemy, and I never looked back. This illustrates TG’s challenge succinctly. In order to choose SA, I had to give up ModelDesigner and Catwalk. For me, I just want to write code, not use a [buggy] web page to manipulate data or move pictures around to auto-generate my code.

In 2007 at Pycon, I tried in vain to make Catwalk work with SA. It just wouldn’t happen. This was the spark that lit the fire which has become the new TGAdmin, driven by Sprox. The interesting thing here is that if the “Next ORM” is found, a reasonably small amount of code has to be written to make this happen. We already have some successes with this with Sprox.

Let’s look at the template language choices you have as a TG developer. TG started with Kid, and while this is a decent XML-oriented framework, if you used it in the early days if you are like me, you probably hit your head on the keyboard a few times for each complex page you tried to write. www.percious.com still runs kid tho, and that’s important to note. TG has not only to support new users who want to use new technologies. We have to support those sites that are still running but want to migrate over to newer technology. And I think we do a pretty good job of that, but again, this is a more difficult job than say a Dj-framework that has absolute control over the template language.

So, TG 1.1 will use SA and Genshi, the entire community decided to make the decision I made 3 years ago, that the usage of SA outweighs the benefits from Catwalk and ModelDesigner. Hooray! Again, framework decisions made by committee are challenging. This is what I LOVE about TG. I didn’t have to wait 3 years for the community to catch up, I added SA to my stack of tools, and went happily on my way.

Even the framework is not immune to TG’s “Best of Breed” mentality. In 2007, while I was futzing with Catwalk, Mark Ramm hid in a room with Ben Bangert for a few hours and created what would become TG2.0. We had been struggling with the changes that CherryPy 3.0 represented, and even though in some ways CP3 is a better back end server, we decided to use Pylons for our server level stuff because of the collaboration potential there. Pylons is indeed a great platform for server-level development.

TG’s team spends an unbelievable amount of time evaluating what _could_ be the best of breed, and some things make the cut, some don’t. Sometimes we even add stuff because it’s neat and we want to support it for that person who thinks it is the best of breed. Sometimes folks come to us with a request to support what they believe is the best of breed, and we do the best to enable them to provide their solution as yet another way of using TurboGears. ToscaWidgets is probably the best example of this. If you look at the repository, you will see over 15 JavaScript library wrappers, and quite a few other libs that make creating web content easier. Again, TG allows the developer to choose which JS library fits them the best. For 2.0 we decided to leave that choice up to the developer.

In short, TG makes the choices so that you don’t have to. We spend a lot of time examining new technologies, and exploring what _might_ work for you. All of our developers use TG in real world applications that differ from giant source code repositories to scientific database management applications. TG handles a diverse set of application in stride.

The next part in this three part series is entitled: “Best of Breed: TG is still the best choice for people who hate hitting the wall.” This will further express the flexibility of TurboGears and describe a bit about what we are doing to make it even easier to get started with TG.

Posted in Python, SQLAlchemy, Sprox, Turbogears | 38 Comments »

Best of Breed: TurboGears is alive and breathing. We are even thriving.

Tuesday, September 15th, 2009

I think a lot of people wonder what’s happened to TurboGears. Where is TG going? Where has it gone? In a recent mailing list post, we were blasted for our documentation, or lack there-of. People seem sort of frustrated that they have a great tool in TG2.0, but have to spend so much time isolating their own technical problems that they fail to see that there is considerable documentation in most areas, but that the docs have a few sore spots here and there.

2008 was all about making TG2 _work_. We’re past that now. Most things pretty much work, some things work really well. Other things need some attention. Now that we’ve got the hard part of actually designing a functioning framework, we can focus on documentation, and using that valuable framework we have written to push the envelope of what TG can do.

One of the “things” we need provide to the user community is better documentation. In the past few weeks I have seen more drive in our community to improve the docs than ever before. Michael Pedersen has taken over responsibility for our documentation. I cannot thank him enough for his work, both in reviewing, reorganizing, adding to, and fixing errors in our existing documentation. His kind of no-sacrifices attitude towards the docs means that we won’t just have “something” up there, we will have what it takes for developers to create web applications using TurboGears.

ToscaWidgets is a sore spot for a lot of folks. I feel your pain. Lot’s of folks say you don’t really need TW to do what it does because you are just creating HTML forms, what is so hard about that. Well, I’ll tell you that I could not have written Sprox without it’s flexibility. Here’s the good news: TW has been re-written from the ground up by Paul Johnston in the past few months. I’ve been helping in this process, providing the tests that will make it more stable than the previous version, and making sure the codebase is not so complex a feeble mind like my own cannot comprehend it. I spent some time benchmarking it, and making sure it’s as fast as it can be. TW2 is 2x as fast as TW. It approaches the speed of simpler frameworks that _only_ produce html (they don’t do resource injection, parameter cascading, etc.)

On other fronts, Jorge Vargas and I have been working on integrating MongoDB with Sprox. This will become the “killer app” for sprox 0.7. For me, this represents proof of concept for Sprox. We have successfully integrated the basic workings of MongoDB into Sprox, which means I generalized in the right places enough for this to work. The result is a TG Admin that will work for MongoDB or for SQLAlchemy equally.

So yeah, there’s still a lot of activity on the TG front, and if you pop into IRC you can feel free to chat up at least one of the TG dev team at almost any hour. Also, we are having a DocSprint Sept. 25-27 (with a main emphasis on Sept. 26), in Boulder, CO and worldwide remotely. We will be addressing the over 100 todo items that Michael has so graciously gathered for us.

This is the first part in a 3 part serious on TurboGears. The next part is entitled: “TG’s Job is hard. Here’s why.” which will discuss various philosophical challenges with running a project like TurboGears.

Posted in Python, Software, Sprox, Turbogears | 10 Comments »

Pycon 2009 Recap

Friday, April 3rd, 2009

It felt like this year Pycon was executed to near perfection. Many struggles I had with last years Pycon were addressed both by the organizers, and some creative thinking.
In this post I will recap everything that happened from my perspective.

WSGI House

I gathered a few close friends from the TG team and a couple of wildcards for perspective to share a house for the continuum of the conference. Having a house gave us a place to go home to at night and meet with friends, often staying up late talking about issues surrounding our favorite software. Having a focused group I feel is important because you spend less time off on wild tangents. The first (and pretty much only) rule of the house was that you pay the same amount whether you stay one night or nine. At least one of our members was encouraged by this rule to stay for the sprints which he hadn’t done before. Success!

Tutorials

For me, tutorials got off to a shaky start, but we seemed to recover nicely. TurboGears has a lot of momentum right now, and it makes it hard to come up with a succinct tutorial when there is so much functionality to cover. I think we were able to recover and that our students managed to soak in enough knowledge from our proverbial fire hose to create some useful applications. I think we have a good start on a new book.

I was extremely impressed with the quality of students who were attending my ToscaWidgets tutorial. Every single student finished every example. I chose Pylons to give the tutorial, and although it is a little harder to integrate TW in the stream than does TurboGears2, it installed quickly and flawlessly. Overall, I think the tutorial was a success.

Talks

This year I did not focus on attending the talks, but instead chose wisely based on speaker and topic and allowed my feet to do the walking if the talk became uninteresting. I definitely missed some talks, but the AV team has done an incredible job putting the talks up on blip.tv so that I can review them later.

This year I did not miss Raymond Hettinger’s talk on AI in python and was enthralled by a speaker who could successfully put a page of code on the screen and keep my interest. I showed up to support Philip Jenvey in his talk on Pylons on Jython but was impressed by his ability to provide a succinct example on where Jython really shines. I am hoping that more people take a second look at this really well done presentation.

Now, I am a SQLAlchemy supporter through and through, but find the domain of database mapping an interesting echosystem. While the ORM panel was littered by advertising chatter from one of the panelists who did not even write an ORM, an obvious dis-inclusion was Robert Brewer who wrote Dejavu, a very nice way to map persistent resources of different types for use in an “objecty” way. Bob’s talk was especially interesting and makes me wonder if SQLAlchemy could leverage some of the work with AST that Bob beautifully displayed with some of the most amazing one-handed keyboarding I have ever seen.

Open Space

Well, I said I was going to give a talk at the Open Space, and ended up not doing so. Part of the problem was the utter lack of projectors in the OS rooms, and part of it was a reluctance to break up the collaborative/discussive vibe that was going on in these sessions. WSGIers hammered out a 2.0 spec, which involved a discussion I only monitored in passing. I was disappointed by the lack of people who showed up for the GSoC BOF, but I think the economy held back a lot of students from attending Pycon. It was also nice to allow my feet to walk around and see what was up in different projects. I met one guy who took REST way to far and got to express some of my dissatisfaction with one of the available tools. On a more positive note, the TG BOF was well-attended and it was nice to see so many users wondering what was up in TG land.

Sprints

This year I refused to let the noobs get me down and actually wrote some code. I am sorry if I did not act as a good host of the TG project, but we have some important milestones coming up and I just wanted to get work done on that. Sprinting remains a cornerstone of our development process and I will see if we can’t get our monthly
sprints happening again in 2009. I was however able to completely re-engineer our dispatch system, and while it is not currently 100% complete, it should be finished in a matter of days. RestController now supports variable arguments for get_one, delete, and put, as well as supporting lookup and default. Anyone can actually now create their own dispatch mechanism, since this functionality has been generalized. Simply subclass Dispatcher, override _dispatch() and go to town. I look forward
to seeing what kind of crazy code this brings to TG land. A lot of discussion has been had on how to make “plugins” or “extensions” for TG, and you can rest assured that we will have this functionality soon.

Thanks

Thanks to all of my house mates who put up with my “mothering”. Thanks to all of you who tolerated my “um”s at my talk on Sphinx, and especially to Georg Brandl who answered some questions. Thanks to the organizers, volunteers and staff that came together to create what has been my best Pycon to date.

Tags: 2009, open space, pycon, sphinx, sprints, Turbogears
Posted in Python, Sprint, Turbogears, Uncategorized, pycon | 25 Comments »

Pycon 2009

Wednesday, February 4th, 2009

So, Pycon registration has been up for a few days, I will be speaking both on and off-podium (read: open space) and providing assistance to and presenting tutorials. Here is a run down of what I am planning in case you wanted a little bit more in-depth information.

Tutorials:

Turbogears2 Beginner and Intermediate:

I will be assisting Mark Ramm by giving individuals help installing and using the new TurboGears2 framework. Mark is an experienced tutorial presenter, an expert in the technology, and in general a fun character to spend a few hours with. When you leave his tutorials you should expect to have a working version of TG2 on your machine, along with an understanding of Model, View, and Controller paradigms. Middleware, Forms, and REST will also be covered. One note, if you are getting started with TG2, it’s best to have it installed and running if you plan to attend only the Intermediate Section. We will not be going over installation in the second-half.

Toscawidgets: Test Driven Modular Ajax:

I am presenting this tutorial which will describe how to use the valuable Toscawidgets package to create web content. If you are currently use WSGI technology, and are interested in creating reusable, modular web content, this is a perfect way to get started. I will show you how to configure TW middleware to work with pylons (which is applicable to other frameworks like repoze.bfg, paste, or even plone/Grok). I will then describe how you might use this middleware to generate web forms. The last few hours of class will be devoted to using the JavaScript utilities of TW to create an Ajaxified website, and test it using YUITest.

The Big F’ing Tutorial: Development Using the repoze.bfg Web Framework

I will assist/present with Chris McDonough about this up-and-coming framework who’s goals are to utilize bits of the zope 3 framework, wsgi, and new technologies to make a lighting-fast web server. Those of you who are familiar with Zope technologies may be interested to find how nicely some of the familiar bits of zope are integrated with wsgi with repoze.bfg.

Presentations:

Using Sphinx and Doctests to provide Robust Documentation

This is a 1/2 hour slot which describes how you can integrate tested documentation with your source code… with sanity! I go over a quick install of Sphinx, and use some screencasts to demenstrate how to add, run, and display doctests using it.

Open Space:Agile Development with SQLAlchemy and Python Testing Tools

I really enjoy giving this talk, and even though it was not accepted as a formal talk, I will find a venue by way of Open Space to express my knowledge of Testing, SA, and Nose. I have given this talk a few times now, and it’s fairly polished. My presentation, while on some dry topics, won’t put you to sleep. Carefully prepared screencasts and photograph-punctuated slides makes the 45 minutes breeze by. Questioneers/Hecklers welcome!

Sprint Topics

I want to spend some time with the Dispatch of TG2, and probably push Sprox further a bit. If you are just starting with TG, please feel free to contribute. Sprinting is a great way to learn a lot from the experts in the domain. We usually do a meet-greet-install the night before the sprints. Oh, and I’ve been known to provide refreshments to all of our sprinting hordes (read: FREE BEER).

So, I hope to see all of you there! If you see me in the hall, feel free to introduce yourself and tell me what you are using Python for!

Tags: Agile, pycon, Python, Sprint, Sprox, tg2, toscawidgets, Turbogears, Tutorial
Posted in Agile, Python, SQLAlchemy, Software, Sprint, Sprox, Testing, Turbogears, grok, nose, nosetests, pycon | 49 Comments »