Archive for the ‘Python’ Category

Sprints, Vacation, Refreshment

Thursday, June 5th, 2008

It’s been a while since I have blogged for a few reasons. One, I took a vacation. No work, open source or otherwise for 11 days. I barely even checked my email. One thing that is great about living in Colorado: I don’t have to go anywhere on vacation, and my family will come to me! I took my sister up the First Flatiron, my dad fishing in Cheeseman Canyon, and all of us took a nice stroll through Chautauqua Park. We ate out almost every night and enjoyed catching up since I have moved 2000 miles away.

Previous to that, I managed to win a skirmish in the framework war at work, and spent an obscene amount of time (on and off the clock) bringing my first production-level TG2 site. This was a really fun project because my coworker and I were given the reigns and let loose on implementation. In 3 hours we had a working TG2 site with data in an EXT grid, with database schema expressed as a tree. In 3 weeks we had a complete reporting system (using EXT) with object-based security. We were also able to implement a file upload manager, and a select shuttle (for assigning groups to secure objects) thanks to the help of Sanjiv Singh, my GSoC protege. Thanks to all of you who have helped myself, and TG2 get off the ground.

One of the reasons I was so absent from the OSS community was that I was trying to wrap my head around what was going on with the EXT licensing, which changed under my feet as I was developing our in house app. The move from LGPL to GPL/Commercial was a shock to the community, and in my opinion based primarily on one person’s greed. My personal choice is to finish up whatever EXT support I have to do for our in-house application, and never look back at it again. I will be moving on to Dojo, and have advised my GSoC students and other people involved with TurboGears to do the same.

SilverPlate Demo site Now that I am back from vacation/new project hell, I have been able to release a new module for TurboGears, tg.ext.silverplate, which is a plugin for TG2 providing User Management and Profile pages which are customizable. This all fits under the TGTools domain, which has become a home for tg.ext.repoze.who (Authentication for TG2) and tg.ext.geo (An upcoming library for Geographical support in TG2). If you get a chance, you might want to check out the new TGTools googlecode site.

I also did a new release of tg.ext.repoze.who, which provides authentication/authorization to TG2. The only functionality I as able to contribute was a change to the name-spacing, as well as a full test of it’s functionality. Right now I am working on an LDAP bridge which will allow your TG2 site to authenticate against LDAP, and then dump you into the TurboGears Domain for all of the authorization stuff (groups, permissions, etc.) I should have this completed in a few days. Most of the work for tg.ext.repoze.who was done at the last world-wide WSGI-Turbogears sprint. It looks like TG is going to do a release in a few days.

This weekend I am headed to ABQ for some sprinting with clearwired on their next-generation web application. I am hoping this time around TG2 is mature enough in their eyes to take advantage of everything it has to offer, since they have currently chosen to work simply with Pylons as a framework. Either way, I will be working with some of the best people in the business, and we should be able to push TG foreward one way or another.

Shell History

Thursday, April 10th, 2008

$ history|awk ‘{a[$2]++ } END{for(i in a){print a[i] ” ” i}}’ |sort -rn|head
241 nosetests
97 ls
85 cd
20 svn
15 python
7 vi
7 rm
5 easy_install
4 pylint
3 history|awk

What Open Source means in a migratory world.

Friday, April 4th, 2008

Are you a software engineer? Programmer? QA specialist? How long have you been at your current position? How long do you expect to be? I bet it’s less than 5 years.

Why would that be an important issue for open source? Most of the time you spend most of your time working OSS projects at home, right? But I bet someday you’d like to get paid for that work, unless like some OSS developers you base your happiness on the proliferation of your software.

So one day you are going to pack up and leave your company, and all of it’s proprietary software behind. And if you are developing Python, that probably means you are leaving a gaping, expensive hole for your former employer. But what if you could keep working for that employer, after you left? What if what you were doing at the new job was so transferable that you could be productive from day 1?

This is the promise that is OSS. And while this promise is probably idealist, it haImage taken bu Alan D. Wilson, and modified by Diliff s at least one tentacle sticking out from reality. The fact is, workers are migratory these days. Personally I think this is because short-sighted investors value fiscal gains over a longer payoff, but we do live in the day of day-trading. The fact is, many people don’t see programmers for what they really are, a mixture of engineer, mathematician and artist. I can always seem to find someone who will pay me more money to do the same job I already have. When something I just can’t turn down for my family’s sake comes up, I have to take it. For this reason programmers, and even the greater workforce migrates.

Employers have already realized that employees are migratory. They have all but done-away with pension plans in exchange for portability: the 401k. Who cares what kind of health insurance we offer? 80/20 is now the norm, because employers don’t expect to have to pay in the long run, and employees accept it too, because they don’t expect to be working at the company for the long haul, when health insurance becomes more of a factor.

But lets get back to the original question. What does a migratory worker mean to open source. Portability of work. I can take my Django/Turbogears/Zope knowledge to any other company using these technologies and pick up right where I left off. If I am one of the few developing software in the OSS world (as apposed to just using it) I can actually contribute more to my OSS project on my companies dime. The question is, are the companies really losing out by providing their software to the world?

Definitely not. In fact, companies stand to gain much more by choosing (and contributing to) something “free” and open. They retain some value from their employee after said person has left the company.  Bug patches, new features, etc. all pulled from the same pot they were putting their goodies in before. Since the software is open and free, more people have access to it in the age where replication of information is almost cost-less. That means they have a larger work force to choose from, and more experts and shared expertise to gain from.

So then the question becomes, “As a company, why should I contribute to open source?”  The fact is you are going to get back more than you give.  Your employees will gain greater expertise by helping to build the tools that they use.  Additionally, the company is helping to keep the project’s momentum going, which ensures the OSS project’s longevity.  It is no secret that “feature complete” projects often fall by the wayside, regardless of quality.  By playing an active role in a project’s development, you are ensuring that that project will be available, and have a good support system even if your employees have moved on.

If you are a company who is battling with the decision to contribute to, or even use open source, I implore you to look seriously at your options, your [migratory] workforce, and your ability to develop a product for the long haul. I bet you find more value in free software than the sticker price. Software, more than ever, is about the people, not the product.

TurboGears and Google Summer of Code

Wednesday, March 12th, 2008

TurboGears is undergoing a monumental effort to participate in GSoC.  Ok, maybe not monumental, but at least 5 of the developers have been hard at working putting together an application even Google wouldn’t turn down. Even if we are not accepted, we are planning on participating by way of PSF, as we have done in previous Google Highly Open Participation contests.  We have developed a number of ideas which students can choose from, or students are welcome to come up with their own TurboGears ideas, and I am sure that one of our mentors will be able to match up with you.      

My own ideas revolve around DBSprockets and TwTools, which is not surprising since I am the owner of said projects.  The largest project and the one which I have the most desire to see put into action is that of a TurboGears CMS.  There had been some work done last year by the guys at Pagoda which produced a brilliant screencast, but the project seems stalled out and it would be nice to see it revived.  Furthermore, the solution I proposed is intended to be much more modular, so you could pick apart portions of the CMS and put them in your existing applications.  I think this would be the most flexible solution, and also one which would employ much that DBSprockets, TwTools, and Toscawidgets have to offer.  If you are a student who is interested in working on this project, please don’t hesitate to drop me a line.  You can also track me down at Pycon for the remainder of the week, and at the TurboGears sprint next week.  If you are a student who is eager to get started feel free to participate in our sprint, remotely or in person.   

One of the great things I see coming from this mini-project is that we now have a very nice set of concrete ideas about how to make TurboGears better.  Whether or not students participate in the development we still get a huge benefit from the creation of ideas, and it gives the development team and possibly new developers a target to make TurboGears the best it can be.  My hope is that these ideas will not only bring a students to our project, but also bring some developers out of the woodwork who may have started similar projects and would like to contribute.  The bottom line is, if you aren’t a contributor, and want to be, here is a great place to start.

Announcing DBSprokets 0.2 Release Candidate

Sunday, February 17th, 2008

I am pleased to announce the first DBSprockets 0.2 Release Candidate. This release comes shipped as 100% code-coverage tested for all API modules (non-dbmechanic).

Many thanks to my beta-testers and developers who have submitted patches, especially Nathan McBride, Michael Brickenstein, and Jason Kirtland. Thanks to Alberto Valverde, who’s work with Toscawidgets is invaluable to this project.

The biggest part of v0.2 is probably Primitives, which give developers an easy way to generate web content from database definitions. If you have a sql database, and want fast web content, this might be the
ticket. A simple few simple calls to table reflection with SQLAlchemy, and you can use makeForm, makeTable, and getTableData to generate web content.

The second notable module in this release is the DBMechanic, which allows you to do all of your database crud with about 3 lines of code added to your Controller.

Supported on this release are TG1.0 and TG2.0-preview with both Primitives and DBMechanic. Primitives have also been verified in Grok, which took a certain amount of work with respect to wsgi and Toscawidgets. Mysql and sqlite are supported with this release Posgres has been tested and is shown to work with dbsprockets.

There are no known Issues to date.

Everything you need to know to get up and running with DBSprockets is here:
http://code.google.com/p/dbsprockets/
Thanks,
chris

Python Frameworks [in] compatibility.

Wednesday, February 13th, 2008

This is a response to Mark Ramm’s post entitled: “Site Components” in Django and TG2 .

First off I wanted to commend Mark for his insightful post. Mark certainly has considerable perspective on both frameworks, and has a great ability to divulge the best of both. He is often pointing me in directions so that I might make DBSprockets better. One day I even received a link to a Ruby on Rails application, ActiveRecord. He certainly can think outside of the box when it comes to solving the world’s Python Framework dilemmas, and is not afraid to express himself openly about his opinions.

“TG2, like Django will define a set of tools that can be used in building re-usable web site components. TG2 users should be able to powerful, reusable components, with SQLAlchemy, Genshi, ToscaWidgets, and the whole TG2 toolchain. ”

–Mark Ramm

I am so glad that Mark pushed TG2 in this direction. I am glad that he pushed DBSprockets in this direction. This week I
worked on getting DBSprockets to work within a Grok application. With not too much effort I was able to get Genshi, SQLAlchemy, and Toscawidgets working within a Grok environment. DBSprockets followed suit. I was amazed at how little work it was to get Toscawidgets working in Grok despite the complexity with which it interfaces the web framework. Granted, I did have to get Grok working through WSGI, and I have Repoze to thank for that. But in the end, all I had to do was easy_install the correct packages, and modify 4 lines in the .ini file, and poof, Toscawidgets in Zope. Who’d a thought a year ago (before the Pylons/TG “merger”) something like this was possible?

So what is my point? When I started out, I asked Mark for commit writes to TG so that I could build DBSprockets into it. The response was, “Well, no go off and do the google code thing and get back to us.” What a great move that was, because now I am able to support many more frameworks and have a much broader user base. And Django… you’re next.

Turbogears Worldwide Sprint

Monday, February 4th, 2008

The last TG2 sprint was seen as a resounding success, despite the fact that coordinating such an event is quite a challenge.

On February 23rd Mark Ramm will be visiting Colorado and plans on coordinating yet another sprint. I will probably still be working on dbsprockets which is likely to be a very important component of the new framework. If you are a python developer who is interested in Turbogears and would like to contribute, don’t hesitate to contact me.

Pycon08 Registration is open!

Monday, January 21st, 2008

I’m very excited to attend this year’s Pycon . In past years, pycon has been a place for me to learn about the multitude of open-source products that are available, and see how to use them. Turbogears, Twisted, even Iterators were topics that I tackled during previous visits to Pycon. I am certain that this year will be no different.

This year, my third time attending the conference I was unable to find a tutorial that spiked my interest, so I decided to help Mark Ramm and Ben Bangert on their tutorial on both TG2 and Pylons, which I have been participating in the development of. Mark told me that DBSprockets is going to be showcased as a competitor to Django’s newforms. I definitely don’t want to miss that. Mark and Ben are both very knowledgeable about Pylons and TG, and their tutorial should be an excellent way to learn how to use these cutting edge technologies.

I have found that one of the best ways to learn new things is to sprint with other developers. I will be sprinting on TG for this year’s Pycon, and I encourage anyone who has not sprinted before to join a sprint for the topic of their choice. You will learn more than any tutorial or talk can teach you, and best of all you get to contribute.

So, hopefully I’ll see you there. Drop me a line if you are looking for a climbing partner at the conference, I usually try to start up a birds of a feather for those who are vertically addicted.

Turbogears 2 Sprint

Monday, January 14th, 2008

I thought I would take a break from rambling about agile software development to talk a bit about the TG2 sprint which I participated in just this last weekend. Well, maybe I’ll talk a little about agile…

If you have never sprinted before, you should, at least once. What I like best about sprinting is you typically have all the experts right there at your finger tips if you get stuck. Everyone has a unified goal of developing sofware in a certain domain. This creates a great work environment, and one in which you can get a great amount done in a short period of time.

This last Turbogears sprint was interesting because it was a truely international crowd. What was particularly neat about this was that there was an extended amount of time that the sprint was happening, because people from different time zones were working at different times. This is probably more difficult for the sprint coordinator, Mark Ramm, but he pulled off coordination seamlessly. One of the greatest tools in our arsenal was IRC. Mark also had a cut-and-paste webpage, which was incredibly useful for sharing snipets.

While my project is pretty independent of TG, it was great to have discussions with other members on-line about low-level decisions regarding the architecture of TG2. Normally these conversations take a few days over message boards. So, thanks to some sprinting DBSprockets is entering it’s Beta phase, and I am hoping to get more users now that I have created a primitive way to manipulate Sprockets.

In any event, I hope that Mark will be coordinating another sprint in February, I am pretty sure he will, and if you want to come over to Boulder, CO I would be happy to make you feel welcome.

-chris

Agile Python Methods

Sunday, December 23rd, 2007

So I have been working for NREL for about 4 months now. It has been a great place to work, and I have been given quite a bit of freedom regarding my methods for programming. Previously I had been working for Pratt and Whitney in East Hartford, CT. I had an excellent mentor by the name of John Camara there. He showed me many great things, and I wish that I had taken more time to develop my agile programming skills in East Hartford, but sometimes you need to step out on your own to realize the value in such things.

Much of my time at NREL has been spent refactoring the code which was up until this point prototype software placed in a production setting. There was little to no testing of the code, and it was hard to understand, modify and most of all, trust. Through the process of refactoring, I have determined a set of best practices which I am going to share over the course of a few blog entries. This is what I call “The Way. ” Everything from design methodologies to how code is commented will be covered. Most importantly, the use of modern testing tools will be discussed, with an emphasis on Unit Testing. Let’s begin.

First off, any good engineer knows you design something first and then start to work on it later. I’m not going to go into detail on that, but it is definitely important. I usually create a rough UML diagram for what I am working on, showing the relationship between the objects I am going to be creating. I find Omnigraffle and incredibly intuitive way to create these diagrams. The result is professional enough to show to a client or use in a presentation. What I don’t do is muddle up the design with a bunch of attributes and method definitions. I find that these things change frequently in the implementation stage, and are for the most part just details.

Here is an example from my most recent open source project:

For the python programmer, setuptools and paster have changed our lives. We now have a consistent way to develop distributable software packages, create templates for application of our libraries, and even publish our products in a general repository. Installing packages and creating dependencies on those packages which we choose to utilize is now as simple as typing one command into a prompt.

While not all of my products are open source, I use paster to create a package to develop in. Once you have easy_install installed, simply easy_install paste.

Creating a new package is as easy as:

paster create MyPackage

Paste will then prompt you with a series of questions about your new package and then set up a directory complete with an egg folder and a place for you to develop (mypackage). Once this is done you want to “install” your package so it is ready for you devlop using:

python setup.py develop

What this does is give python the correct set of links to your package. This way pathing issues are solved, and any module in your package that includes items from within itself can start with “myproject.xxx” when doing imports. This is especially valuable when you put all of your tests in a separate folder because you can run all your tests from a parent directory without worrying about running certain tests from the correct directories, it all just works.

Which brings me to the final topic about setting up my project to work. Once the project is created using paste, and installed using setup.py (a derivative of setuptools) I create a “test” folder within “myproject” which contains my tests. It is at this point, after a rough design and a project setup and install that I begin my code development.