Archive for the ‘Sprint’ Category

Sprint Organization: But can we do more?

Tuesday, October 13th, 2009

The ultimate sprint incentive would be for companies to put up a bounty to fix bugs, or otherwise provide support for an open source project.  This is the toughest thing to sell I think, but if Google can do it, why can’t other organizations?  I have seen this attempted in the past with mixed results.  I think it was a half baked idea, and that we can do better than that.  The challenge is metrics, and making sure that everyone gets their fair share.  I mean, at that point, sprinting becomes payment for work done, the sprinters are just hired guns for a weekend.

Autograph by kugelfish.

I think the challenge has been, and will always be metrics.  How do you measure the work and who did it, and how much is each thing done worth?  I think the sprint organizer is the greatest asset in this situation.  He usually has the best idea about what needs to be done, what has been done, and ultimately who did what quantity/quality of work.  So you can use him as your metric definer.

My idea is this:  A sponsoring company provides the OSS organization with a certain amount of money, and what they would like to see achieved in an organized sprint.  This money is given to the organization as a donation regardless of the outcome of the sprint.  The donation may also be given to the organization with no stipulation of task, but the sprint organizer must choose a topic of interest in order to guide the sprint.

The sprinters all agree to a set method for dissemination of funding provided by the sponsoring organization.  My suggestion would be to split all of the funding equally, but in a capitalist nation, it is hard to justify giving the same funding for a person who did a little work on a piece of documentation vs. someone who spent 40 hours putting together a full tutorial.  Another method would be to allow the sprint organizer to disseminate the funding as he sees fit, but the sprinters would have to agree to this before work is started.  The work would have to be completed by a certain deadline to obtain the funding, because project organizers do not have time to track down who did what 4 months after the fact, it’s just not practical.  Finishing a week after a weekend sprint seems reasonable to me.

I am sure to see lots of comments about these ideas because when money gets involved, everyone gets uppity. (one of the great ideas behind OSS is that there is no money paid for the actual software)

So, there exists possibilities for setting up a sort of Round-up style sprint.  The goal is to provide sprinters incentive for providing bug fixes, or even to document a part of the code that is otherwise hard to get anyone to document.  I would love to open a discussion about this topic, as I see it as a completely different business model when it comes to software development.

google-logo.jpg

But you have to ask yourself, “What’s in it for the company?”  Well, first I think you have to ask yourself why Google has been running programs where they pay for OSS development, no questions asked, for 5 years now.  But here are some insights while you ponder that.  The easiest reason to understand is that the sponsoring company has some bugs in OSS software they use that they need fixed, and a team of experts can fix them in a weekend, or their staff can submit bug fixes, go through the rigamarole of OSS contribution, and things get done in months.

The next less obvious reason for a company to provide funding for a sprint is that they are using OSS software that could use a little help in the documentation department.  By paying for documentation, they are getting a cut-rate deal on the experts that usually created the software providing documentation for those things that may be crucial to their business’s success.  This also reduces their dependency on any one employee who may have in-depth knowledge on a particular software package.  By ensuring the OSS software that the company uses is well documented, the company ensures that the intellectual know-how for that piece of software remains with the project, not with the employee that may leave at some later date.

The last simple reason I can think for a company to sponsor OSS is for recruiting reasons.  Typically the people involved in the sprinting process are the ones that know the most about it.  It is also be a chance to evaluate an employees enthusiasm for work in general.  Those who are sprinters are more likely to work well with others and stay with projects for the long haul, in my experience.

If software development is described as herding cats, gathering and directing sprinters is like herding stray cats.  I once had a stray cat visit me when I was in my potato-tuna days, and I gave it some of my tuna, because it looked in worse shape than was I.  And you know what, that cat always returned for the tuna.  Now, I’m not much of a cat person (current cat count: zero) but I think that if software developers are truly analogous to cats, they might show up at your door once, but it’s much easier to get them to come and visit regularly if you give them some tuna once in a while.

Pycon 2009 Recap

Friday, April 3rd, 2009

It felt like this year Pycon was executed to near perfection. Many struggles I had with last years Pycon were addressed both by the organizers, and some creative thinking.
In this post I will recap everything that happened from my perspective.

WSGI House

I gathered a few close friends from the TG team and a couple of wildcards for perspective to share a house for the continuum of the conference.  Having a house gave us a place to go home to at night and meet with friends, often staying up late talking about issues surrounding our favorite software.  Having a focused group I feel is important because you spend less time off on wild tangents.  The first (and pretty much only) rule of the house was that you pay the same amount whether you stay one night or nine.  At least one of our members was encouraged by this rule to stay for the sprints which he hadn’t done before. Success!

Tutorials

For me, tutorials got off to a shaky start, but we seemed to recover nicely.  TurboGears has a lot of momentum right now, and it makes it hard to come up with a succinct tutorial when there is so much functionality to cover.  I think we were able to recover and that our students managed to soak in enough knowledge from our proverbial fire hose to create some useful applications.  I think we have a good start on a new book.

I was extremely impressed with the quality of students who were attending my ToscaWidgets tutorial.  Every single student finished every example.  I chose Pylons to give the tutorial, and although it is a little harder to integrate TW in the stream than does TurboGears2, it installed quickly and flawlessly.  Overall, I think the tutorial was a success.

Talks

This year I did not focus on attending the talks, but instead chose wisely based on speaker and topic and allowed my feet to do the walking if the talk became uninteresting.  I definitely missed some talks, but the AV team has done an incredible job putting the talks up on blip.tv so that I can review them later.

This year I did not miss Raymond Hettinger’s talk on AI in python and was enthralled by a speaker who could successfully put a page of code on the screen and keep my interest.  I showed up to support Philip Jenvey in his talk on Pylons on Jython but was impressed by his ability to provide a succinct example on where Jython really shines.  I am hoping that more people take a second look at this really well done presentation.

Now, I am a SQLAlchemy supporter through and through, but find the domain of database mapping an interesting echosystem.  While the ORM panel was littered by advertising chatter from one of the panelists who did not even write an ORM, an obvious dis-inclusion was Robert Brewer who wrote Dejavu, a very nice way to map persistent resources of different types for use in an “objecty” way.  Bob’s talk was especially interesting and makes me wonder if SQLAlchemy could leverage some of the work with AST that Bob beautifully displayed with some of the most amazing one-handed keyboarding I have ever seen.

Open Space

Well, I said I was going to give a talk at the Open Space, and ended up not doing so.  Part of the problem was the utter lack of projectors in the OS rooms, and part of it was a reluctance to break up the collaborative/discussive vibe that was going on in these sessions.  WSGIers hammered out a 2.0 spec, which involved a discussion I only monitored in passing.  I was disappointed by the lack of people who showed up for the GSoC BOF, but I think the economy held back a lot of students from attending Pycon.  It was also nice to allow my feet to walk around and see what was up in different projects.  I met one guy who took REST way to far and got to express some of my dissatisfaction with one of the available tools.  On a more positive note, the TG BOF was well-attended  and it was nice to see so many users wondering what was up in TG land.

Sprints

This year I refused to let the noobs get me down and actually wrote some code.  I am sorry if I did not act as a good host of the TG project, but we have some important milestones coming up and I just wanted to get work done on that.  Sprinting remains a cornerstone of our development process and I will see if we can’t get our monthly
sprints happening again in 2009.  I was however able to completely re-engineer our dispatch system, and while it is not currently 100% complete, it should be finished in a matter of days.  RestController now supports variable arguments for get_one, delete, and put, as well as supporting lookup and default.  Anyone can actually now create their own dispatch mechanism, since this functionality has been generalized.  Simply subclass Dispatcher, override _dispatch() and go to town.  I look forward
to seeing what kind of crazy code this brings to TG land.  A lot of discussion has been had on how to make “plugins” or “extensions” for TG, and you can rest assured that we will have this functionality soon.

Thanks

Thanks to all of my house mates who put up with my “mothering”.  Thanks to all of you who tolerated my “um”s at my talk on Sphinx, and especially to Georg Brandl who answered some questions.  Thanks to the organizers, volunteers and staff that came together to create what has been my best Pycon to date.

Pycon 2009

Wednesday, February 4th, 2009

So, Pycon registration has been up for a few days, I will be speaking both on and off-podium (read: open space) and providing assistance to and presenting tutorials.  Here is a run down of what I am planning in case you wanted a little bit more in-depth information.

Tutorials:

Turbogears2 Beginner and Intermediate:

I will be assisting Mark Ramm by giving individuals help installing and using the new TurboGears2 framework.  Mark is an experienced tutorial presenter, an expert in the technology, and in general a fun character to spend a few hours with.  When you leave his tutorials you should expect to have a working version of TG2 on your machine, along with an understanding of Model, View, and Controller paradigms.  Middleware, Forms, and REST will also be covered.  One note, if you are getting started with TG2, it’s best to have it installed and running if you plan to attend only the Intermediate Section.  We will not be going over installation in the second-half.

 Toscawidgets: Test Driven Modular Ajax:

I am presenting this tutorial which will describe how to use the valuable Toscawidgets package to create web content.  If you are currently use WSGI technology, and are interested in creating reusable, modular web content, this is a perfect way to get started.  I will show you how to configure TW middleware to work with pylons (which is applicable to other frameworks like repoze.bfg, paste, or even plone/Grok).  I will then describe how you might use this middleware to generate web forms.  The last few hours of class will be devoted to using the JavaScript utilities of TW to create an Ajaxified website, and test it using YUITest.

The Big F’ing Tutorial: Development Using the repoze.bfg Web Framework

I will assist/present with Chris McDonough about this up-and-coming framework who’s goals are to utilize bits of the zope 3 framework, wsgi, and new technologies to make a lighting-fast web server.  Those of you who are familiar with Zope technologies may be interested to find how nicely some of the familiar bits of zope are integrated with wsgi with repoze.bfg.

 Presentations:

Using Sphinx and Doctests to provide Robust Documentation

This is a 1/2 hour slot which describes how you can integrate tested documentation with your source code… with sanity!  I go over a quick install of Sphinx, and use some screencasts to demenstrate how to add, run, and display doctests using it.

Open Space:Agile Development with SQLAlchemy and Python Testing Tools

I really enjoy giving this talk, and even though it was not accepted as a formal talk, I will find a venue by way of Open Space to express my knowledge of Testing, SA, and Nose.  I have given this talk a few times now, and it’s fairly polished.  My presentation, while on some dry topics, won’t put you to sleep.  Carefully prepared screencasts and photograph-punctuated slides makes the 45 minutes breeze by.  Questioneers/Hecklers welcome!

 Sprint Topics

I want to spend some time with the Dispatch of TG2, and probably push Sprox further a bit.  If you are just starting with TG, please feel free to contribute.  Sprinting is a great way to learn a lot from the experts in the domain.  We usually do a meet-greet-install the night before the sprints.  Oh, and I’ve been known to provide refreshments to all of our sprinting hordes (read: FREE BEER).

So, I hope to see all of you there!  If you see me in the hall, feel free to introduce yourself and tell me what you are using Python for!

Coding Binge

Tuesday, January 27th, 2009

I haven’t written to the blog in a while.  Quite frankly, I’ve been busy.  In the last 30 days, I have released 3 software new packages, updated 1, deprecated 1, participated in a sprint that lasted a virtual 2 weeks, closed countless tickets, and pushed forward TG2 functionality.

TG2b4 was released last Saturday.  This was mostly a bug-fix release, but b3 is where the new functionality really came into the scene.  TG2b3 is the first build to include Sprox, a new library for schema-generated widget generation.  Sprox is the offspring of DBSprockets.  I decided I liked the declarative part of DBSprockets so much I wanted to spin it off as it’s own entity.  Sprox looses DBSprocket’s table-based dependency, utilizing the mapping provided by SQLAlchemy.  I realized that much of DBSprocket’s code was doing precisely what SQLSoup was designed to do, and decided to focus my efforts on making and extremely configurable widget base.  The result was a considerable removal of the cruft that was associated with DBSprockets.  Sprox releases with an excellent documentation base provided by Sphinx.

There has been a bit of resistance to Sprox, people were/are confused/upset about my providing yet more options for schema based widget generation.  The fact is I have yet to find anything that performs as well as Sprox from a developer/speed standpoint, and I needed to provide our TurboGears user base with a better way to administrate their site, and also allow them to use that tool component-wise in their system.  I think this method for developing widgets is well done in other frameworks, and we need a solid answer to this problem.  Sprox is just that.

The next step was to re-work Catwalk to use Sprox.  This took a little effort, and I put in RESTful URLs while I was at it, but struggled with making the URLs work within TG2’s dispatch system.  The result was as close to REST as you can get without conforming to a set standard.

The result of hacking REST into Catwalk got me thinking, and I decided to implement content-type dispatch as well as RESTful dipatch in TG2.  I went back for another round on Catwalk, and converted it over to the standard.

I’ve also been toying around with Dojo at NREL.  I’m pretty much done with ExtJS due to licensing issues, a not-so-hot codebase, and weak support from IRC.  It’s bad when you go on to ask a question on the channel as a 6 month-user of a software project and end up spending all your time answering everyone else’s questions (as the most experienced person in the room).  Something must be said for an organization that does not push paid consulting as a primary focus on their website…  #dojo has been an exceptional resource for getting my work done.  Those guys know their software, and lend a great hand to help you with it.

Back to the topic at hand… I was able to shoe-horn Dojo into Sprox with little effort, and implemented DojoCatwalk, which worked, but was ultimately not what I wanted.  What I really wanted was configurability.  I started work on tgext.admin, which was supposed to provide enough functionality to replace tgcrud, a command to auto-create crud in your own TG application.  To support tgext.admin, I created a new package called tgext.crud, which provided a CrudRestController, which is a simple way of providing crud for any object in your model.  AdminController combines this functionality with that of Mark’s lookup code to provide a fast, configurable set of tables/forms/etc for all objects in your model.  AdminController takes a declarative AdminConfig as input which provides a consistent way to create your administrative toolset.  Did I mention it does Dojo tables with ajax loading?  Yeah.

I’m not done with this binge yet.  Catwalk is going to mutate one more time before I’m through with it.  It is going to become a default-configured AdminController specifically designed to work within the context of a quickstarted TG2 application.  I had one blocker ticket which was solved last weekend, so it’s time to get Catwalk good and finished.

Google Summer of Code Mentor Summit Recap

Wednesday, October 29th, 2008

I spent last weekend in Sunnyvale, CA at the GSoC mentor summit.  This was a great opportunity to rub elbows with a number of experts in various OSS projects.  It was great to make connections with people in and out of the Python community.  First off, I just want to thank Google, and Leslie Hawthorne for having us. Google was an excellent host, and I would jump at the chance to attend a Google-hosted event again.  Google is using GSoC to promote the open source community.  By providing funding for students to work for a summer on software instead of flipping burgers, they hope to save the world, one open source project at a time.  I have my own take on what Google’s true intentions are, but I don’t really want to discuss that in this blog entry.  That is to say, I don’t think Google’s intentions are in any way evil, but they are definitely self-serving.  In any event, it is remarkable the amount of time, energy, and money that goes into this project, which is at this time one of a kind.  Here’s hoping that other companies will consider the not-so-small buy in for sponsorship which could lead to even more OSS development.   

As for the summit itself, there were lots of talks.  Since it was run like an open-space conference, it took a little while to work up some speed the first day, but I never found myself want of something to do.  I was not surprised that the talk on monetizing your OSS project was wildly successful.  I was surprised at how easy it was for sessions to get off on tangents, but it was interesting to see how people in similar fields were solving projects.  I attended a session on Scientific Computing (since I work for NREL) and it seemed that the biggest problem the room faced was packaging/deployment of software.  I appears this problem cross-cuts platforms, language, and even hardware barriers.  It also seems that you either have a choice to make: stick with old software that is stable, or upgrade to new stuff that almost always breaks other things.  Sound familiar?  I wonder how long it will be before everyone runs isolated virtual machine images for each of their projects… I bet some already are.  

By day two I was pretty exhausted of participating in sessions, and I just wanted to hack with some like-minded folks.  It turned out that at least 8 people at the summit were hacking on making their application run on Python 2.6, so we all got together and shared the good vibe while we solved our individual problems.  The main barrier for TurboGears 2 running on Python 2.6 was ToscaWidgets and TurboJson.  First off, peak-rules needed some minor tweaking to get a few of the bugs fixed.  Thanks to Philip Eby for implementing my patch (in a slightly different way) and making it all work.  ToscaWidgets was also breaking with the SimpleJson trunk, but this turned out to be an implementation error, and I think I have it mostly sorted out.  

Yesterday evening Mark and I finished up packaging the !first beta! of Turbogears2.  I put a few custom eggs in the index from my GSoC hacking, and suddenly we have a Python 2.6 compatible release.  Also, of interesting note Gustavo Narea cranked out tgext.authorization which handles most of the security for TurboGears2.  Now I just need to get cranking and fix DBSprockets Primitives to work with Rum, and then fix silverplate to support Rum also.  (Perhaps this is my way of telling people to move in the Rum direction).  

Mark and I will be attending PyWorks in November, and doing a sprint then.  So, if you are in town, or interested in sprinting remotely, please don’t hesitate to sign up.  I will be speaking at PyWorks on using PasteScript, Virtualenv, SQLAlchemy, and Nosetests to create robust database applications. Do you think there is anything else I could cram in a 45 minute talk?  I will also be helping Mark out at his TG2 tutorial, so if you aren’t signed up for PyWorks, get crackin’ !

Plonecon Recap

Monday, October 13th, 2008

Last week I attended Plonecon in Washington D.C.  We use Plone quite a bit at NREL, and I  wanted toploneconf come up to speed on the state of the art, and see if I could find some folks to collaborate with on the Scientific Data Management front.   I was not only surprised by the number of people in attendance, but also by the progress being made in forefront of Plone development.

Of course, one of the big news items was what is known in the community as the “Plone tax”, which is a reference to the 30+ seconds it takes to start Plone.  It was announced that this has been reduced to 6 seconds, which in my mind is still too much, but I am glad they are making progress in this area.  [edit] I believe the speedups were achieved in the trunk of plone, not for 3.2.   Plone 3.2 is supposed to be all eggs, which I think is a great benefit to the community.  I have already been using Plone in this manner with Repoze, which packaged Plone as an egg a few months ago.

I was impressed by the changes to the user interface which were proposed by Alexander Limi in his talk on the future of Plone’s user experience.  It looks like portlet/viewlet terminology is being supplanted with the term “widget” which I think is more standard in the web design world. I think the new user interface is more intuitive, simplifying the page down to those components which will be most important, while still making available all of the functionality of Plone.

I was also impressed with Kapil Thangavelu’s content mirror.  While I don’t think his project which takes Plone content and injects it into an sql database will be immediately useful to me, I think it offers a nice path for someone who wants to convert their Zope/Plone site over to an relational-database based site.  It also provides a nice bridge for someone who wants to use existing relational database tools to mine data penned up within a zope database.

Coming from a TG background,  I was of course interested in Repoze.  Chris McDonough gave a compelling talk on repoze, specifically repoze.bfg and what is motivation was for creating yet another framework. We ran a BOF together, and had quite a bit of response.  I demo’d using Toscawidgets within Plone (running on repoze.plone).

Craig Swank and I also demo’d one of our repoze.plone application for a group of people interested in laboratory informatics systems.  I started up a google group for this, and I sincerely hope that our small community can find a way to collaborate efforts in order to increase our productivity in a way only OSS can.

My Blog’s Wordle

Tuesday, June 17th, 2008

wordle

http://wordle.net/

Sprints, Vacation, Refreshment

Thursday, June 5th, 2008

It’s been a while since I have blogged for a few reasons. One, I took a vacation. No work, open source or otherwise for 11 days. I barely even checked my email. One thing that is great about living in Colorado: I don’t have to go anywhere on vacation, and my family will come to me! I took my sister up the First Flatiron, my dad fishing in Cheeseman Canyon, and all of us took a nice stroll through Chautauqua Park. We ate out almost every night and enjoyed catching up since I have moved 2000 miles away.

Previous to that, I managed to win a skirmish in the framework war at work, and spent an obscene amount of time (on and off the clock) bringing my first production-level TG2 site. This was a really fun project because my coworker and I were given the reigns and let loose on implementation. In 3 hours we had a working TG2 site with data in an EXT grid, with database schema expressed as a tree. In 3 weeks we had a complete reporting system (using EXT) with object-based security. We were also able to implement a file upload manager, and a select shuttle (for assigning groups to secure objects) thanks to the help of Sanjiv Singh, my GSoC protege. Thanks to all of you who have helped myself, and TG2 get off the ground.

One of the reasons I was so absent from the OSS community was that I was trying to wrap my head around what was going on with the EXT licensing, which changed under my feet as I was developing our in house app. The move from LGPL to GPL/Commercial was a shock to the community, and in my opinion based primarily on one person’s greed. My personal choice is to finish up whatever EXT support I have to do for our in-house application, and never look back at it again. I will be moving on to Dojo, and have advised my GSoC students and other people involved with TurboGears to do the same.

SilverPlate Demo site Now that I am back from vacation/new project hell, I have been able to release a new module for TurboGears, tg.ext.silverplate, which is a plugin for TG2 providing User Management and Profile pages which are customizable. This all fits under the TGTools domain, which has become a home for tg.ext.repoze.who (Authentication for TG2) and tg.ext.geo (An upcoming library for Geographical support in TG2). If you get a chance, you might want to check out the new TGTools googlecode site.

I also did a new release of tg.ext.repoze.who, which provides authentication/authorization to TG2. The only functionality I as able to contribute was a change to the name-spacing, as well as a full test of it’s functionality. Right now I am working on an LDAP bridge which will allow your TG2 site to authenticate against LDAP, and then dump you into the TurboGears Domain for all of the authorization stuff (groups, permissions, etc.) I should have this completed in a few days. Most of the work for tg.ext.repoze.who was done at the last world-wide WSGI-Turbogears sprint. It looks like TG is going to do a release in a few days.

This weekend I am headed to ABQ for some sprinting with clearwired on their next-generation web application. I am hoping this time around TG2 is mature enough in their eyes to take advantage of everything it has to offer, since they have currently chosen to work simply with Pylons as a framework. Either way, I will be working with some of the best people in the business, and we should be able to push TG foreward one way or another.