Hey guys, I’ve been busy for a bit, but I made up this new resume for the TG tour, looking for feedback if you’d like to comment.
cheers.
-chris
Archive for the ‘Turbogears’ Category
New Resume
Saturday, January 2nd, 2010TurboGears Black Friday Sprint
Friday, November 13th, 2009If you are from the US, you may be familiar with the custom of racing to your local commercialization hub before the sun rises and beating your fellow man over the head to get that cute little fifi doll your daughter wants for christmas. This endeavor is often fueled by the previous day’s binge on tryptophan-laced poultry.
TurboGears wants you to know there _is_ an alternative to this ruckus behavior. While it may not offer a method of transference of the latest flu virus strain, you may come hang out with the leaders of our community on irc://#turbogears@irc.freenode.net . and ask questions and help your fellow man. In the spirit of thanksgiving, you may decide it would be good to return the 1000s of lines of code contributed by helping us document that mess*!
November 27, 2009 starting around 9am MST, ending when Chris passes out, we will convene and work on the 134 todo items left in our documentation. If some of you are near Denver/Boulder that day, I would be happy to organize an on-site meetup, please let me know. So, the day after thanksgiving, why not act gluttonous for one more day (with your writing) and lets see if we cant close another 60 todo Items as we did on the last doc push. Be you an advanced super 37173 user, or someone completely new to TG, we have got something for you to do! Lets work together to make the formal 2.1 release of TG the best yet, with the highest level of completed documentation evar!
cheers.
-chris
* the term “mess” is not used literally in this sense.
Open Source and San Francisco
Tuesday, November 3rd, 2009It’s October again, and that means Open Source folks are meeting at Google in Mountain View, CA. Last year I had such an enlightening experience that I wanted to do my best to make it even better. One thing that I
have realized is that it is hard to participate in a conference without some free time. Your brain functions better if you arrive, settle in, get your bearings, and then start the real work.
On Thursday I arrived in San Fran, and immediately sprinted to the coast. I live 1000+ miles from the ocean, and although I see myself as a mountain person, there is something compelling about swimming in the cold surf, emerging with a layer of brine. After a few hours of work in a coffee shop overlooking the sea, I headed into the interior to meet with my TurboGears pals and get the conference started.
Friday Jorge and I toured San Francisco. We decided to rent some bikes an ride across the Golden Gate Bridge. I like this kind of self-guided tour, and we definitely got lost but had a good time in the process. We arrived in Sausalito after crossing the bridge, and grabbed a bit to eat before taking the ferry home. Later on we met up with Mark and the rest of the Google Summer of Code folks for some socializing.
Friday night we finally hashed out the json/return type challenges with TG. Sticking with 2.0 implementation, you may only return a dict from your controller method if you expect the normal TG rendering to occur. However, if you return an iterable object, the Pylons machinery will take over and stream your content. This represents a problem because if you return a list and set the exposed renderer to json, it will return your dict in string form. To protect the developer from making the mistake of creating a CRSF attack, this behavior will now raise a meaningful exception. Having Mark and Jorge around to make this decision definitely sped up the implementation.
Saturday we arrived bright and early at Google and got started with the conference. The mentor summit is a self-organizing conference, which means the talks are determined on the spot rather than pre-planned. More and more I am finding myself drawn to these types of meetings. They provide a much more interactive approach to conferencing as apposed to the eyes forward norm.
The best session I attended was Recruiting and Retaining Awesome People I had proposed a sprint-related session, and decided it would be best to merge with Donnie Berkholz since his session was similar in scope. After all, sprinting is just one concept in the scope of recruitment. We had a large number of attendees, and had to re-arrange the eye’s forward setup the room had because it did not allow for the discussion we both wanted to generate.
There was excellent discussion, and I took a few new ideas away from this session. To highlight a few:
Toxic People – Placing a label on a person as toxic can be dangerous both to the people involved in the labeling, and to the community at large. If you can isolate the toxic behavior and catalyze a new response from the person, someone who is viewed as toxic may then become a valuable contributor to your project. Don’t be quick to label a person unless they simply do not respond to requests for changing of toxic behavior.
Code Review – Providing prompt and complete code review is a reward for the contributors work. I think this is especially true in OSS where we write our code in a public domain, and are therefore expecting that others will look at our work. The work in OSS is more than just the final product, it’s the code itself. By providing feedback on a patch, pull request, or documentation augmentation, you are growing a potential long term contributor, even if your review is critical. Reviewing code allows you to set expectations for your code project so you can further mold the codebase to meet your organization’s needs.
Socializing – The modus operandi for most technical IRC channels is to focus closely on the topic. This keeps the signal to noise ratio high, but makes the room feel stale. By interacting with your users in a more personable manner, it’s easier to build the interpersonal relationships that are required for contribution longevity in a project.
Thanks to Toshio for taking such detailed notes for this session.
For some reason I was having a hard time being motivated by the sessions. It seemed odd to me that there were only 1/2 of the session rooms booked at any one time on Saturday. I discussed this with Jim Baker, one of my Python colleagues in Boulder, CO. He responded by providing a “How to write a Python Interpreter” session that was remarkably well attended. The group was a mix of Python users, developers, and those interested in interpreters in general. One of the session members was a major contributor to PyPy. It seems obvious to me that python is going to move to the next level soon in terms of performance. One of the things discussed was the usage of language constructs to provide the interpreter information about a class, method, module, in order to constrain execution and therefore provide faster execution. Consider the use of __slots__ in Python classes. If you know there are only certain attributes in a class, you have additional information that can be used by the interpreter to limit frame size, etc. This is great for memory limiting techniques. Now, apply this idea to performance techniques, and you can see that there are many opportunities to give the Python interpreter hints about execution that would allow it to do less work and therefore increase speed.
When I wasn’t participating in sessions, I was doing my best to share experiences, theories, etc. in the hallway track with other attendees. I was also working on the TurboGears 2.1a2 release. It’s funny, because our release process is more manual than I like, and therefore takes a few hours to complete, but if you split it up 10 minutes here, 30 minutes there, you can get a remarkable amount done even in the midst of something else major going on. I’m not sure if this time usage technique has any merit, but I have used it at previous conferences. It feels very easy to capitalize on the energy from the conference and get some work done, I’m not exactly sure why yet, but I’ll have to think about it some more. In any event, 2.1a2 is released and ready for testing/consumption. I think we are going to have 1 more alpha before we lock down the api. Mark wants to change lookup to _lookup, so we have to add that and deprecate, deprecate, deprecate.
Sunday was more of the same in terms of sessions, but the attendees seemed to step it up and provide some interesting topics. My favorite talk of the day was one regarding turning open source into a sustainable profession. A number of topics were discussed, but this general forum lead us to group open source financial success into three categories:
Software Product Licensing – Turn your OSS project into something that is salable, either by providing a pay-for layer on top, or licensing it in such a way that corporations are responsible to pay for what they use, still allowing the OSS developers to have their way without incurring the costs.
Sell Your Organization – If your organization can offer value to an industry, it is very likely that an industry leader will want to purchase the rights to that intellectual property wholesale. MySQL is the example of this that stands out in my mind the most.
Software as a Service – Consulting is one of the main ways developers can benefit financially from their work in the open domain.
I actually would like to explore a few of these in depth, and I think the next open-space conference I attend I will try and set up a session about at least Software as a Service to explore ideas about how to give away your software and thrive simultaneously.
Google continues to provide a valuable service to the community with the Google Summer of Code program. I am convinced their motives for running such a program are noble. One of the sessions I participated in was the Google Highly Open Participation (GHOP) session, and basically it was just an informative session for me. I wanted to know more about the program, and the result is that TurboGears will likely work with the PSF in the future to participate in this program.
One of the reasons I get a lot out of conference like the Mentor Sum is that there isa large variety of technologies in attendance. This moves me out of my normally Python-centric interactions with other developers. While I think that Python is the best way to express myself as a developer, it is also interesting to see how other developers solve similar problems in their own domain.
TG’s Killer Features: The Admin (Yes, TG Does, and it rocks)
Tuesday, October 20th, 2009
TurboGears 1 had Catwalk. This provided a crud-based interface for your models. It was directly tied to SQLObject, which meant when you moved on to SQLAlchemy, you couldn’t take it with you. Since TG2 decided to only support SA (SQLALchemy) out of the box, we lost our only Admin-like competitor, and this really hurt the project. Although many people didn’t use Catwalk beyond trying it out, many hung on to the fact that they _needed_ this kind of interface when making the decision about what framework to use. Since other frameworks have this sort of interface, TG really lost out.
When I first set out to build an Admin interface for TG, I didn’t really think about what kind of impact it could have on a developer’s ability to code. I simply wanted to provide the developer with a way to get at their database tables. DBSprockets was the result. It _sort of_ worked, but the problem was that it was table-based, and therefore lacked the ability to leverage the developer provided mapping that every TG project utilizing the SA ORM (Object Relational Mapper) contains. This was mildly successful, and while my first attempt to make it easier to use outside of an admin interface (something called “primitives” ) was less than successful, it became clear to me that the auto-generated forms could be useful in their own right.
Spurred on by an unsuccessful attempt to use the peak-rules based form generator called Rum, I was determined to make it easier to auto-generate forms, and make something easy enough for the masses to grok. Sprox was born. Sprox allowed me to drop much of the cruft that DBSprockets contained, and allowed me to provide a much easier way to define what I wanted customized in a form in a declarative manner. I was inspired by SA’s declarative extension, so much of the framework is modeled after that design style.
The one design methodology I kept from DBSprockets was the database abstraction layer. I still have the idea that we should be able to support our SQLObject users, and we also need to be able to support the plethora of persistence solutions that have yet to be discovered or created. This abstraction layer probably seems unnecessary, considering that SA itself is an abstraction layer, but this layer allows you to provide access to numerous ways of interacting with a persistence mechanism. In this way, Sprox is able to talk to both MySQL, and MongoDB. Preliminary support for this has begun in the Sprox 0.7 fork, to be released in early 2010.
Sprox is really the bricks in the house that is the TurboGears Admin. After some trials with what I called Catwalk2, I decided that it would be easy enough to provide a framework for developers to use Sprox, as well as develop a configurable admin. In 2.0 the auto-configured TG Admin was called Catwalk2, in 2.1 we have dropped Catwalk altogether and just provided TGAdmin configured, which makes it easier to follow documentation about how to change the configuration.
The Turbogears Admin System is made up of 4 main components. These connect together to provide a system that you can customize at every layer
Sprox – We already have discussed this, it provides a way to generate forms from DB Schema.
CrudRestController – This provides a restful interface to each one of the models in your Application’s persistence layer. It utilizes TG’s RestController to enable RESTful urls to work with your models.
AdminConfig – Allows the developer to override settings in the admin to customize those things which may be different from what Sprox provides as a default. It also allows you to override the default behavior of CrudRestController, in the event that you need to modify the way your models interact with the database.
AdminController – Combines the above 3 things to provide a set of controllers for manipulating your models.
One of the benefits of using a configuration-based Admin is that this allows the developer to override functionality at any layer. The developer is not confined to providing an all or nothing solution. With the AdminConfig, you can limit the models, swap out forms with Sprox autogenerated ones or provide your own customized ToscaWidgets forms. The developer may choose to override one method of the crud, or provide a different set of authorization predicates. Templates are overridable, so you are not stuck with the default look-and feel. The admin supports both Mako and Genshi quickstarts, which means that your project’s master template is supported regardless of which template language you have decided to use (Mako is much faster). You are also not limited by the number of admins you choose to create within your TG application. This would allow you to create an admin-level admin, or simple crud based web application for user-level users in the same application.
The most compelling thing about the Admin for me is the fact that each layer can be used independently. Sometimes you want to provide your users with _just_ a crud interface on one model. It is simple enough to pull out the CrudRestController and use it as a stand-alone in your project. Often times I will want to provide just one form that might have many fields in it. I may just use sprox by itself for that. The admin system allows you to customize at the layer you need to modify without getting in your way.
Here is the use-case I thought about when I designed the Admin. You meet with a customer who already has a database. You connect to their database, extract the necessary schema information. You then take this schema information, and generate a TG application that allows you to see their database. The customer then provides feedback about what their interface should look like for each model while you peruse their data in the Admin. Some things you fix on the fly, and can show the customer how it looks on the spot. Other things you take notes on, and work on later. Perhaps they have a complex validation scheme, or require form that has a drastically different look than: fieldname [value].
Later you return to the customer after creating CRCs for each model requested. Something is wrong though, the customer needs the database interaction to work differently in the back end. You modify the controller methods of your CRCs to do so. The customer is pleased with the quick turn around time, you are happy because you didn’t spend your week writing the same crud interface over and over and over.
Use case or not, I have been able to successfully accomplish the goals of my clients in a shorter time frame by modifying only those things that are different from the intelligent set of defaults that Sprox and CRC provide. Having crud, and customizable forms and tables often gets me 90% of the way there, and the last 10% is where I want to work anyway. The fact that Sprox provides infinitely scrolling tables is nice too. 
Sprint Organization: But can we do more?
Tuesday, October 13th, 2009
The ultimate sprint incentive would be for companies to put up a bounty to fix bugs, or otherwise provide support for an open source project. This is the toughest thing to sell I think, but if Google can do it, why can’t other organizations? I have seen this attempted in the past with mixed results. I think it was a half baked idea, and that we can do better than that. The challenge is metrics, and making sure that everyone gets their fair share. I mean, at that point, sprinting becomes payment for work done, the sprinters are just hired guns for a weekend.
I think the challenge has been, and will always be metrics. How do you measure the work and who did it, and how much is each thing done worth? I think the sprint organizer is the greatest asset in this situation. He usually has the best idea about what needs to be done, what has been done, and ultimately who did what quantity/quality of work. So you can use him as your metric definer.
My idea is this: A sponsoring company provides the OSS organization with a certain amount of money, and what they would like to see achieved in an organized sprint. This money is given to the organization as a donation regardless of the outcome of the sprint. The donation may also be given to the organization with no stipulation of task, but the sprint organizer must choose a topic of interest in order to guide the sprint.
The sprinters all agree to a set method for dissemination of funding provided by the sponsoring organization. My suggestion would be to split all of the funding equally, but in a capitalist nation, it is hard to justify giving the same funding for a person who did a little work on a piece of documentation vs. someone who spent 40 hours putting together a full tutorial. Another method would be to allow the sprint organizer to disseminate the funding as he sees fit, but the sprinters would have to agree to this before work is started. The work would have to be completed by a certain deadline to obtain the funding, because project organizers do not have time to track down who did what 4 months after the fact, it’s just not practical. Finishing a week after a weekend sprint seems reasonable to me.
I am sure to see lots of comments about these ideas because when money gets involved, everyone gets uppity. (one of the great ideas behind OSS is that there is no money paid for the actual software)
So, there exists possibilities for setting up a sort of Round-up style sprint. The goal is to provide sprinters incentive for providing bug fixes, or even to document a part of the code that is otherwise hard to get anyone to document. I would love to open a discussion about this topic, as I see it as a completely different business model when it comes to software development.
But you have to ask yourself, “What’s in it for the company?” Well, first I think you have to ask yourself why Google has been running programs where they pay for OSS development, no questions asked, for 5 years now. But here are some insights while you ponder that. The easiest reason to understand is that the sponsoring company has some bugs in OSS software they use that they need fixed, and a team of experts can fix them in a weekend, or their staff can submit bug fixes, go through the rigamarole of OSS contribution, and things get done in months.
The next less obvious reason for a company to provide funding for a sprint is that they are using OSS software that could use a little help in the documentation department. By paying for documentation, they are getting a cut-rate deal on the experts that usually created the software providing documentation for those things that may be crucial to their business’s success. This also reduces their dependency on any one employee who may have in-depth knowledge on a particular software package. By ensuring the OSS software that the company uses is well documented, the company ensures that the intellectual know-how for that piece of software remains with the project, not with the employee that may leave at some later date.
The last simple reason I can think for a company to sponsor OSS is for recruiting reasons. Typically the people involved in the sprinting process are the ones that know the most about it. It is also be a chance to evaluate an employees enthusiasm for work in general. Those who are sprinters are more likely to work well with others and stay with projects for the long haul, in my experience.
If software development is described as herding cats, gathering and directing sprinters is like herding stray cats. I once had a stray cat visit me when I was in my potato-tuna days, and I gave it some of my tuna, because it looked in worse shape than was I. And you know what, that cat always returned for the tuna. Now, I’m not much of a cat person (current cat count: zero) but I think that if software developers are truly analogous to cats, they might show up at your door once, but it’s much easier to get them to come and visit regularly if you give them some tuna once in a while.
Sprint Organization: Is our Current Model Broken?
Tuesday, October 13th, 2009How does one entice people to thrive by sitting and writing documentation? Rather, how do you reach out to a greater audience of developers that you could enlist for the betterment of your project? The classic model has been, organize a theme, have the “elders” show up, and get some work done. This sort of works.
The problem with this model is that I find a ton of people who basically show up to the sprint for a free tutorial. You get them up and running with your source code, hoping they contribute something useful, and you spend an awful amount of time doing so, because they have some weird configuration or are in way over their heads, and by the time you get them set up, the sprint is over. Now you just wasted your own time, and that of the project’s. The only person that benefitted from that situation is the sprinter, who can use the information he gathered to thrive.
The only way I have personally been able to contribute a serious amount of work is to hide in a corner and code for hours alone while everyone else is “sprinting”. Well, that sort of seems to negate the need for a sprint, since I can do that without the hands-on interaction that a sprint is supposed to encourage. So, when no one showed up at the last sprint, I had a bitter sweet taste in my mouth, but I closed about 20 todo items.
The main benefit I have seen from a sprint is that it is typically a gathering for the “elders”. The elders might talk about the future direction of a project. In the best case scenario they write some test code to see if an experimental idea will work or not. Sprinting is an incredibly valuable time for this, but often times the elders are held captive by (what we call in the climbing world) “gumbies” who capitalize on the elders’ knowledge of whatever project you are working on. Often times I will see the elders hide in another room to have a discussion, much in the same way I have done to get code done.
One solution we have come up with to deal with this problem is to institute a “babysitter.” This person is usually also the sprint organizer, and whether they know it
or not, their drive to have a successful sprint ties them to help the gumbies succeed, even if the chance of such an occurrence may be low.
I have played this role quite a few times, and while I am happy to answer questions at a sprint, I generally point people towards the docs and have them work through our examples. If they find a problem with the tutorials, they have just found something they can sprint on and be successful!
The other major benefit we get from sprinting is that it is an awesome venue for starting new things. Since the elders get together and sometimes create some test code, you are often left with a nice base to work off of. Many times I find myself working the next 3 or 4 evenings after a sprint trying to finish things out. This is often how new features are added to a project.
So, the question you have to ask when organizing a sprint is not only how to get people to show up, but how to get the “right” people to show up. I think for this we need to change the way we think about sprinting, and come up with a new methodology for organizing sprints.
This is part two of a four part series on sprinting. In the next segment, Sprint Organization: New Rules! I talk how we could improve sprinting using our current resources.
A TurboGears Weekend in review
Sunday, October 11th, 2009A few weeks ago the Front Range Pythoneers decided to organize an “Uncon” where people show up to discuss various topics on-the-spot. This is the sort of event I really enjoy participating in, so I of course agreed to attend. At the same time, I was approached by the guys from Developer Day to do a talk on TurboGears. You can imagine the conundrum I faced, but thanks to the willingness of the DD organizers to be flexible, and some creative planning, I was able to participate in both.
Speaking at Developer Day was a new experience for me because I was talking to folks that were not necessarily versed in Python,never-mind TurboGears. The conference appeared to be somewhat Rails heavy, but it was refreshing to see organizers reaching out to the greater web community to provide a well rounded conference. The nice thing about speaking to a wider audience was that I was able to expound some of the history of the Python web, as well as describe TurboGears at a high level without worrying about boring the audience. I was quite nervous speaking at first, because I have not done so in a few months, but seemed to settle into a groove by the time I showed an example of how easy it is to inject repoze.profile into a TG application and provide a cachegrind display to find any slowdowns in your app. I hope that this example was able to express how versatile WSGI is.
I stayed the morning at DevDay and I am glad that I did. Chad Fowler gave an address on what it means to remain passionate as a developer over the life of your career. I think his idea that providing structure to your life definitely allows you to achieve amazing things. His real-life examples were poignant and well received. I’ll be checking out his book soon. The other talk that I found interesting was Jeremy Hinegardner’s talk which basically discussed the numerous non-relational persistence methods available. I thought his method for showing examples of the different methods was great. For each one he had a simple succinct example that showed the benefits for the given persistence framework. He allowed the audience to choose from the frameworks he discussed in his talk. Jeremy was an engaging speaker, and I would not hesitate to sit in on one of his talks in the future.
After a bit of DD-provided BeauJos, I headed over to the UnCon. They too were having pizza provided by Google. Google Boulder was a great sized venue for the 40 people that attended. It was exciting to see so many new faces in attendance. It seemed to me that the “regulars” were doing a lot of demoing, while the new folks watched on, but there was also a lot of discussion that happened. I showed how to use repoze.profile and runsnakerun to
analyze the results. Zooko immediately installed runsnakerun and tried it on his app. It is always nice to have immediate gratification for having taught someone something, even more so when the person voluntarily tries what you think is “so cool.” I got to show off some of the work I am doing for www.getmvp.com, since much of it is prototypical of the Extension Solution that I hope to provide with a combination of Pylons and TG. Also on display was TW2. It was great to show how simply one could express all MVC elements of a widget in one complete package.
Sunday I ran the first TurboGears WorkShop. If you follow my blog, you may have read a few posts about how I think we can improve sprinting, but I’ve come to the realization that our less-than-stellar sprint performance is really due to a need for improvement in the organization at large. I have decided to add a WorkShop Series to our tireless effort for improvement of TurboGears, both from a technical aspect, and one of the community. I was up late on Wednesday creating a basic tutorial-type plan for Sunday, and I finished up with about 80 pages of documentation to provide workshop goers, basically by selecting items from the TurboGears documentation. My goal for the sprint was to provide sprinters with a working example of TG at the end of the day, with a little bit of work accomplished customizing the Admin. I asked sprinters to bring their own databases, to utilize sqlautocode as an example database for their new application, and while no one provided the class with one, we were still able to succeed with one that I provided as a backup. 5/6 people succeeded in this, and while there were some rough edges, I think I have an idea that is workable for a 3-6 hour WorkShop that will succeed with a little bit of polish.
I am still formulating the ideas for TurboGears workshops. I have started to contact folks I know throughout the country, in order to provide venues for these workshops. So far I have Boston, Dallas, San Fancisco, Atlanta, and Boulder (Denver) lined up. I think with little effort, I could also add Ann Arbor, and probably Washington D.C. The idea behind a workshop is that you arrive with a varying amount of knowledge in TG, and you leave with a greater knowledge than you arrived with. You are encouraged to bring an existing project to hack upon, or to create a new one that we can play with. I will provide a rough outline of what we might do in the tutorial, but if the group decides to go off in a different direction, that’s okay too. If you are interested in participating in one of these WorkShops as a mentor, or providing venue space, accommodations, etc. I would love to hear from you. Right now I am in the organization phase, expect a blog post announcing the official plan in the near future.
Thanks! Without the efforts of a number of individuals this weekend would have been much less successful than it actually was. I want to thank Ben Scofield for inviting me to talk at developer day, and for shuffling the schedule so I could participate in both conferences. Greg Holling put in a great effort to organize the Uncon, and Google provided an awesome venue for us to use. Three volunteers from Google Boulder provided their time, and even gave a tour of the facility to conference goers. They weren’t even Python developers… Jim Baker and Matt Boersma both showed up to provide access to Bivio so that we could have our first-ever TG WorkShop. Lastly, I’d like to thank Bruce Eckel for making the trip down from the mountains to provide his unique perspective.
Sprint Organization: Are you Thriving or Surviving?
Monday, October 5th, 2009
I’ve been thinking about what makes it so hard to organize and hold a sprint. TurboGears’ sprints were successful in 2008, but I think they could have been more so. We have recently had a wildly successful sprint, but most of the burden for that sprint lay on one person, the sprint organizer. Michael did an awesome job, but the sprint would have been more successful had we been able to entice more sprinters, we were [unbelievably] organized to scale to about 30 sprinters, but only about 6 of us really showed up. Getting people excited enough about what you are doing is the hardest part of sprinting, and also the least interesting thing to do from a developers perspective. We just don’t like marketing all that much.
Last Saturday was a beautiful day. I sat inside a local coffee shop within eyeshot of the Flat Irons, and I did not wonder why no one had shown up to accompany me. Why would anyone want to give up a saturday to sit behind a monitor, when they could be out climbing, sailing, picnicking, hiking, etc. etc. etc. Boulder, CO is an unbelievably beautiful place, and in the fall when the days are warm and the nights are cool, it is so easy to take advantage of our surplus of good weather. And yet, I sit there in the coffee shop, writing documentation, of all things. This is certainly the drudgery of software development, and at the same time a necessary evil.
When no one showed up for an hour, and it was clear that I was destined to spend the day working alone I started to think about why, in fact, I myself was sitting in the coffee shop, working away. And I thought for a bit; it’s easy to generalize when you’ve had a few chai’s and you want to be distracted for a minute from some mundane technical writing. It occurred to me that I could simply stop work altogether, here at the sprint, and in life, and then my savings would run out, and I’d be poor again. Having to spend a summer eating potatoes for lunch, saving the tuna for dinner is not something I want to experience a second time. Compounding this desire to survive is the fact that I have three other people who require my drive to obtain subsistence. So, is sprinting about survival?
Well, here in lies the dichotomy. For the project, sprinting is required for survival. Most OSS projects I have been involved in need a shot in the arm once in a while to keep all of the “elders” motivated to work when the sprint is over. Meanwhile, the individual sprinters in no way need the sprint to survive. There’s always another project, another way to solve your technical problems so you can move on. Sprinting, for the individual is about thriving.
With the millions of OSS projects out there, there is an unbelievable number of people who find another way of solving their particular problems that they want to share with everyone. These people who share are putting their ideas, and hard work out there so that one day, they will thrive. Thriving might mean other people accepting your ideas, and utilizing your work, but it might also boil down to the fact that we need sustenance to survive, and we want to provide a better life for ourselves and our loved ones. This is an incredibly motivating factor. I have found often times in life I must suffer a bit in order to thrive. Yes, I ate potatoes and tuna for a summer. But I didn’t settle for a fast-food job to make ends meet. I continued to work to find a technical job where I could use the skills I learned my first two years of school, and pay for the last two years. And you know what? My loans stopped accruing that very summer.
This is part of a four-part series on sprinting. The next part is entitled: Sprinting: Is our current model broken?
TurboGears 2.1a1 released
Friday, October 2nd, 2009Introduction
We are ready to start testing the next version of TurboGears: 2.1. This release is the first of what will be a series of alpha and beta releases before we move into production. 2.1 is not a huge departure from the 2.0 codebase, rather, it’s efforts are to clean up and speed up the existing codebase, both conceptually and technically. So, if you have some time, install it today and give it a whirl!
Major Differences (Things that affect present 2.0 users)
==================================================
Rendering
The item that will affect most 2.0 users is the renderer. Json rendering is now not a special hard-coded case, so you will need to add it to your default config. Most 2.0 apps will have to add the following line to their app_cfg.py if you are using @expose(‘json’) at all::
base_config.renderers.append(‘json’)
If you should forget to do this, you will get an error message reminding you to do so.
TurboJson
Support for TurboJson has been removed. We have not found many people using this, and in fact, if you still need it, you can still put it into your TG application. For the most part SimpleJson does a good job of rendering Json for us, and because it is a part of the python default library in 2.6, it makes sense to utilize it. This allowed us to remove about 8 package dependencies.
Minor Differences (Things that affect folks familiar with how TG already works)
==================================================================
Dispatch
The dispatch mechanism has been completely refactored. This means that pesky things like requiring *args at the end of a RestController.edit are now not required. The new dispatcher is much faster, up to 200% faster for RestController dispatching. It also has the flexibility to add a _dispatch() method to your Controller class to drive the dispatch. This is not yet documented, but it’s on the high priority list for documentation.
Setup-app
Setup-app has been broken into two pieces, schema creation and bootstrapping. This will not affect existing TurboGears apps, but if you were familiar with a certain way quickstart works, this has changed. There are now schema.py and bootstrap.py modules in the websetup directory of your quickstart. These isolate database schema creation from adding bootstrap data to the database. This helps when you are writing scripts that modify a database, but do not add the bootstrap data (useful for testing). Having them in separate modules makes the separation of concerns more apparent.
Catwalk
While the branding was nice, we decided to drop the Catwalk name in order to make it easier to follow tutorials that utilize tgext.admin. Therefore, in quickstarts, you will no longer see catwalk, just tgext.admin calls. This also allowed us to drop yet another dependency. Catwalk will still remain a viable package, but it will no longer be maintained. The good news is that nearly all the functionality for the admin resides withing tgext.admin. Catwalk was simply a configuration of the former.
New Features
===============
Documentation
A concerted effort has been made to increase the depth, breadth, and organization of the docs. Areas of focus have been: Organization of the docs themselves, Configuration, Tutorials, and Utilities. Part of the 2.1 push will be to get the docs to a level of completion never seen before, and an organized effort is taking place to ensure this aspect of the project’s success.
Mako!!
TurboGears2 has always supported Mako, but we are now providing template support in our quickstart. When you quickstart a new project, it will ask you if you want to use mako templates instead of Genshi. Mako is usually about 3x as fast as Genshi for complex rendered pages, and up to 10x as fast for simple ones. Careful attention has been given the Admin to make sure it still works with Mako (it does), so you can be certain if you make the choice to use mako, it will work out of the box.
local:
In order to handle template inheritance gracefully, an identifier for the local project has been added to the template lookup in Mako. This allows the admin to inherit your local project’s master.mak file. An inhertance clause in Mako that uses local would looks something like::
<%inherit file=”local:templates.master”/>
Genshi has support for this automatically, but it is not explicit, and we are looking at ways to support this explicitly before 2.1 goes to final.
ToscaWidgets2 Support
ToscaWidgets2 recently made a 2.0a1 release. We have included in TurboGears2.1 the ability to easily configure your application for TW2, along with other added support for this next-generation widget framework.
Thanks
=======
This release comes not without considerable effort on the part of the TurboGears team. I would like to thank Michael Pedersen for his undying effort with the docs. Michael helped to collect about 190 todo items for our docs, and squashed a countless number of them. We now have about 130 items todo on the docs, but that number is ever-decreasing with his and other’s effort. Thanks to those folks who have contributed to the DocSprint, and who still continue to contribute, including Michael Fletcher, Jorge Vargas, and Seth Davis. If you use TurboGears, and find you need to dig into the source code to figure stuff out, please help us make the docs better by contributing to a DocSprint or sending us a pull request.
Thanks to Jorge for straightening out the Json rendering issue. Also, by removing TurboJson from the stack of required packages, we have opened the door for TurboGears to run on AppEngine and Jython. This would not be possible without Jorge’s effort.
Thanks also to Mark Ramm, Christopher Ardnt, Florent Aide, Alberto Valverde, Paul Johnston, Christoph Zwerschke, and Lee McFadden for their continued support of TG.
Finally, I just wanted to send a thank you to the folks who have contributed to the TG codebase by association. Mike Bayer, Jason Kirtland, Ben Bangert, Philip Jenvey, Chris McDonough, and last but not least Ian Bicking. Thanks for all of your effort making possible this great conglomeration of parts.
TG’s Killer Features: SQLAlchemy. Obvious, no?
Monday, September 28th, 2009So, as I stated in a previous entry, I’ve been using SQLAlchemy for more than three years now. If you know anything about me, you’d find it amazing that I have contributed little more than a patch here or there. Why is that? I think that’s because every time I try to find something that I need SQLAlchemy to do, it already does it. I have spent so little time trying to make SQLAlchemy do what I want it to do, and so _much_ time getting work done with it.
Here is a good example. One o
f my clients is a
company developing sports management software by the name of MVP. Although I work on their next-gen stuff, I was called on by them to promote the students on the older system to the next grade over in the summer time. Their database system had well over 40 tables, requiring no less then 10 of them to have modification. I fired up sa, reflected the changes, wrote the changes in python, with simple loops around what records I needed to change, an it was done. In one hour’s time I had a happy client, and a school system with a functioning system. Two things about this are amazing. 1. At the time, I had practically zilch in the Postgres experience department. 2. I only needed to learn the schema of the tables I was modifying, and I could do this with introspection. The fact that I was able to do this task in less than an hour was only made possible by my knowledge of SQLAlchemy’s table-based architecture. I did not need to know the nuances of Postgres’ SQL language (I was more familiar with MySQL at the time), or a in-depth knowledge of the database schema. I was able to pick and prod until the job was done, and it was painless.
I really like the fact that SQLAlchemy’s approach to making an ORM is layered. I can drop to whatever layer I need to to meet the requirements of my project. SQL, table objects, and mapped objects all have their place in the grand scheme of things, and I have used them all to varying degrees. What SQLAlchemy achieves with this layering is the ability to adapt to existing projects, schemas, etc, and make considerably useful software, in a short amount of time. By figuring out the nuances of different SQL dialects, it frees me up to focus on the task at hand, and provide products to my clients that work in a broad range of situations.
But this post is supposed to be about TurboGears, right? For me, SQLAlchemy is more than a tool I use for TurboGears projects, or even web applications, it’s something I can use any time I have to access a relational database. It is the ultimate base for writing tools that help me get my job done. That job might be providing a system to allow schools to schedule matches against each other, or enabling scientists to access their data directly using objects at a python prompt, instead of assembling arcane sql strings to gather data. SQLAlchemy is such a good basis in fact, that it makes building tools to help do my job even easier.
I have had the opportunity to contribute to sqlalchemy-migrate. If you don’t know anything about the project, consider this: You have a production database you cannot break. You have 5 minutes downtime to modify the schema, update records, etc. You need to be able to back out the changes and bring the system back up if everything breaks. Migrate lets you do all of this, and provides a versioning system to ensure that such a process moves smoothly. I have used migrate with postgres, found some rough edges and fixed them. SA’s table mapping makes this possible, migrate just adds a layer to make versioning and some table creation processes easier. I only hope that someday some of the migrate code makes it’s way back into the SA codebase.
I do lots of testing. I made numerous schema changes today to one of my databases at www.nrel.gov in fact. I test the database schema for
matching against my definitions with some yet-to-be-released software. There are about 3000 tests. With tables numbering in the fifty-semod, I needed something to make data entry easier for my tests. I wrote bootalchemy to do this. You pass it some YAML, and the models module, and it performs all the entries for you. It does a bit of introspection to determine dates, and it has reference pointing so you can inter-connect your objects within your yaml (using & and * like my old friend C). This has vastly decreased the amount of time it takes me to create new test data for new tests. Again, this is possible only with the framework that SA provides me.
Lately I have been interested in providing a broader base of scientists at NREL access to their data using python as a medium. Scientists (especially physicists) are really good at conceptualizing data. They use crazy tools like Matlab, and R and all sorts of proprietary tools to manipulate their data. They aren’t afraid of a command prompt. My idea is to give them something like Sage, but with direct access to their data as mapped objects. I also want to be able to show up at a scientist’s desk with laptop in tow, connect to one of their existing databases, and spit out a admin-style web interface based on TG in a few minutes. These notions have driven me to contribute sqlautocode. At this point sqlautocode will spew out a page (or 7) of python code that provides you with Declarative Objects and an interactive prompt. sqlautocde works as a library, so you can use saautocode’s output to directly in memory without generating any code at all. All this is possible with SA, and I don’t know of anything else that can do all of this.
I have focused on the technical here, but beyone that is a great team of individuals like Mike Bayer and Jason Kirtland who put in long hours and answer questions promptly on the mailing list. I often wonder how these guys get any sleep. The thing that excites me most about SA is that it will soon release version 0.6, which to me means that these guys have 4 more versions of increased functionality before they consider it “done.” So, as you can see, there is more to SA than just the ORM it provides. It’s framework gives you freedom to expand your horizons and get your job done, by focusing it’s task on the challenges that relational databases all have, so you don’t have to. This is what makes SA a killer feature of TG.
The third segment in this series is: TG’s Killer Feature: The Admin (Yes, We do, and it rocks)