Rails Day 2006

On Saturday, I participated in Rails Day 2006. It's a 24-hour contest between 1-3 person teams to develop cool Ruby on Rails web apps. I set out to build a visual database query building system (that doesn't require knowledge of SQL) that also allows people to post and share these queries with other users of the system. This would allow people to quickly and easily build queries and share them with others.

The itch I was trying to scratch is that many people in the nationwide network of non-profits I work for use Access to create databases for the data they're working with because they can easily create queries to run on that data. I would rather they put their data in our centralized MySQL server for many (hopefully obvious) reasons.

However, asking them to learn SQL is a bit much, and connecting Access (or OpenOffice.org Base) to MySQL is a bit daunting, especially since there is no way in hades I'm going to open up the MySQL port to the Internet (so they would need to setup a VPN client, an ODBC data source, Access connections to all the tables, etc.).

If I had a web app that just gave them the visual query builder piece, it would be pretty easy. Combine that with the collaborative ability to share the queries, and you've got a big win in terms of empowering people to help themselves and help others work with the database.

Anyway, I built Emerald (as I called the tool) during Rails Day. I don't have it hosted anywhere, unfortunately, so it's not easy to try out. (You could check it out from Rails Day's svn repo and run it locally on your computer, if you really wanted to.) The app, as it stands right now, is not a solution to the problem I set out to solve. If anything, it's proof that a) ActiveRecord (Rails' ORM database access layer) is not a good tool for this sort of job, i.e. generic database access that adapts itself to your schema, not the other way around or b) I don't really know what I'm doing. :)

ActiveRecord proved very tricky to adapt to this purpose. One of the mantras of Rails is "convention over configuration" and usually it's very nice. For example, if you have a table whose name is the plural form of the class you're instantiating (let's say the table is called "people") and the primary key's name is just "id", then your model class looks like this:

class Person < ActiveRecord::Base

end

That's it. You now have full ORM access to your "people" table using instances of Person objects. It's very handy.

However, when trying to adapt it to work with existing database schema, the convention was almost never correct, and namespace conflicts abounded. These are possibly not insurmountable problems, but in a 24 hour contest, it caused me to have to spend way too much time adapting the tool to the job. This became rather frustrating by the end of the day.

I was fully aware of the fact that I was trying to use ActiveRecord in a non-standard way, so these difficulties were not entirely unexpected. I wanted to push the boundaries a little of both my knowledge of Ruby (and Rails) and what was possible to coax ActiveRecord into doing. It was a labor of love, as I am a big fan of Ruby, Rails, and ActiveRecord. Overall, it was a very fun experience and I can't wait for next year! (But I'll probably pick a project that's more suited to ActiveRecord's purpose.)

Posting this from Flock

First Post!

Slashdot just posted a link to this nifty little Firefox-based web browser. It has built-in blogging tools, del.icio.us and flickr integration, and other cool features. It's billed as the "Web 2.0 browser." Click the image below to check it out.

So far, I like it. Some are claiming it's all hype, and that you can get all these features in Firefox via extensions. While I have no doubt that that's true, I for one enjoy having it all integrated in one package, without having to hunt down a million extensions. Leave comments with your impressions of it.

Blogged with Flock

UPDATE: Flock is still in beta, and it shows. I've already run into a few rough spots with it. I'll be sticking with Firefox for now, but keeping an eye on Flock's development.

YearlyKos '06 wrap-up

I'm finally back in Denver from YearlyKos. It was great. I loved meeting and talking to all the bloggers. Everyone was very nice and approachable. If you want to build more access to the netroots, you need to go to this conference.

Some comments on a few of the speakers at YearlyKos:

• Mark Warner - +5 points after his funny and inspiring speech; Keep an eye on this guy for '08. The universal health care for everyone under 18 idea that he and other Dems are talking about now is great. Something we can actually achieve, *much* harder for the other side to speak out against than "HillaryCare" (as long as we keep getting the messaging right), and a significant step towards real universal health care (we'll get there).
• Howard Dean - between his speech at yKos and what Markos and Jerome write about him in Crashing the Gate, +12 points. We should be very, very glad he's at the top of the DNC now.
• Harry Reid - Give 'Em Hell Harry indeed! +10 points for being a model of Democratic backbone, and proving that that's not an oxymoron. Watching him struggle to contain his excitement when someone yelled, "Shut it down, Harry!" was priceless. He's introducing legislation that will require the administration to backup everything it claims about Iran w/ real intelligence.
• Barbara Boxer - Her speech was very good, but apparently she was overheard defending Lieberman to a reporter, and then wondering why so many people at YearlyKos were not very... enthusiastic about ol' Joe. Barb, come on, Jomentum is a Republicrat!

If you missed YearlyKos, go to their site (logo above is a link) and you get can video and audio of most of the conference (at least the keynotes).

Where do we go from Net Neutrality?

Still at YearlyKos. A questioner in a Net Neutrality panel brought up an excellent point. After we win on the Net Neutrality issue, we need to use that momentum to start fighting things like the DMCA and the "copyright extremism" (quoting the very smart Matt Stoller of MyDD who is on the panel) of DRM. This is a culture war; it's a free speech issue. This information belongs to all of us. Onward!

Open-source TV

I'm at the YearlyKos netroots conference in Vegas and Tiffiniy Cheng at The Participatory Culture Foundation ran a session on online video. She showed us the Democracy Internet TV Platform. It's working now, you can download the Democracy Player and see what's already online. It's basically RSS video feeds combined with an iTunes Music Store-like interface. Oh, and it's all free and open source. The software is GPL'ed and runs on Windows, Mac, and Linux. I dig it.

Linux Software Installation Made Easy

Software on Linux

Software installation should be one of those areas where Linux shines. I mean, almost everything you need is free! Freely available, free to download, free to run, free to modify, free free free.

Many distributions have gone to great lengths to make it easy to find and install new software. My personal favorite is Debian's (and thus off-shoots like Ubuntu's) APT/dpkg system. APT, along with whatever front-end you prefer using, makes it pretty trivial to find and install lots of great programs.

But, there are two problems...

1. Typically you can't get the latest version of whatever program you're installing because APT / RPM / whatever pulls from your distro's repository, tagged with its version. It doesn't matter if Firefox 1.5 is out if the latest version of your distro came with 1.0.7. Sure you can download and compile the latest source from the upstream vendor, but then your package system has no idea what you've done. Everytime you do this, you make your package system a little more useless and your /usr/local/ hierarchy becomes a den of unpatched security holes, orphaned files, and in other ways a throwback to pre-package system Linux life. Not a place I want to be.
2. Extensions, plugins, and libraries that are managed with their own package management systems (e.g. CPAN, Ruby gems, etc.) are typically only partially represented in the distro's package repositories, if at all. These systems, more than others, contain lots of little packages that need to be more up-to-date and don't really need much distro-customization. Additionally, it's sort of silly to duplicate CPAN in an APT repository. What a duplication of effort!

Gentoo to the rescue?

One distro (at least) partially solves the 2nd problem in a unique way. Gentoo comes with a script that loads Perl modules from CPAN and records them in emerge's package metadata so the package system thinks you merged an ebuild that just happened to contain the latest CPAN version of whatever Perl module you wanted. However, you have to check whether or not there is an official ebuild before using this script, and it's still a different system that's not obvious unless you read about it in a forum or something.

As far as problem #1 goes, Linux is way behind the times. Think about what you do when running Windows or Mac OS X when a new version of Firefox comes out. You go to mozilla.com and you download it and install it (well, on OS X you drag it to the hard drive, which is infinitely superior to both Linux and Windows' complex installation systems, but more on that in another post). Why can't we do that in Linux? Yeah, yeah, I know we can, but I want it to be kept track of by the package system. If it were Windows or OS X, uninstallation would be no more painful than installation. Not so on Linux.

Gentoo kinda gets around this problem by relying on the upstream source code as the "base" of its package installations (because they aren't binary packages until you compile the ebuild on your system). Ebuilds are just recipes that have compilation instructions, package metadata, and some patches for the source. This allows Gentoo to be a little more nimble in terms of keeping up to date with the latest releases. However, the more complex the software or the more major the version bump, the less true this is. Additionally, (in my experience) you lose some of the stability and system robustness of running a set of programs and libraries that are well tested together.

I used to run Gentoo for its up-to-date-ness. It was nice only having to wait a few days (usually) to be able to install the latest version of a new software package. New version, package system knows about it, we're all good.

But, the instability finally got to me. Package versions were changing all over the place, constantly. Half the time my system couldn't tell where in the world the most basic libraries could have gone (hint: they were right where they'd always been). Things that used to work would suddenly break. If I wanted this, I would run Windows, sheesh!

Turning the problem into an advantage

So, I switched to Ubuntu. I missed Debian's reliability, but not its staleness. Ubuntu seemed a happy medium, and indeed it is. In fact, it's so close to software nirvana, I think there's only one more thing it needs to fully achieve it.

Both problem #1 and problem #2 can be solved with a little standardization. No, I'm not talking about standardizing on One Package System To Rule Them All. That ain't gonna happen anytime soon. No, what we need is a little standardized metadata that gives whatever package system you're using enough info to know you installed something not from the distro and how to uninstall it later (as well as fun things like automatically uninstalling it when/if a newer version comes up for availability in the official repositories). This metadata should be defined by someone like freedesktop.org, and then dpkg, rpm, emerge, etc. should be modified to handle it.

Brave New World

Let's say it's far into the future and my vision has been fully realized. I'm running the "Blubbering Bass" version of Ubuntu and it came with Firefox 3.1. However, the super sweet Firefox 4.0 has just come out and I want it! All my friends running Mac OS 11 and Windows XP We're Sorry Vista Sucked So Bad Edition can just download it and install it. But, so can I! The file that mozilla.com is offering for Linux is just a regular .tgz source tarball. However, it has a special file inside called "install" which I can either run from a terminal or my distro can run for me when I right-click the tarball in my desktop environment and select "Install..." from the menu.

This nifty file would put some metadata about this program in a standard place that my package manager knows to check. It would say, hey, I'm installing Firefox 4.0 into this directory. My package manager would then know Firefox 4.0 existed in the system and where it lived. It would then be able to uninstall it and possibly monitor for security problems with the version I installed.

Additionally, when I ran the CPAN shell or the Ruby gem installer, these programs would also put this metadata in the right place so that my package manager knows about what crazy stuff I've installed with them.

This would make Linux the best of both worlds in the software installation department. When something new comes out, download it and install it, just like your proprietary-OS-using friends. Same goes for CPAN modules and Ruby gems. If you start having issues with some of the programs, you can run a quick little "revert to distro version" function that will get you back on track with the tried-and-true version of the program in question.

Obviously this is only the tip of the iceberg here; you could imagine extending this system to allow the installation of RPM's on Debian-based systems (although alien already does a pretty good job here), or other crazy things like that. But I'll not get into all that for now.