Unfortunately it appears to be yet another Australian site reinventing the open-data-catalogue-wheel. So far we have:

• data.gov.au: custom WordPress site
• data.brisbane.qld.gov.au: custom WordPress site
• data.nsw.gov.au: custom PHP (Drupal?) site
• data.mosman.nsw.gov.au: custom proprietary site
• data.vic.gov.au: custom proprietary site

Open Source Data Catalogues

How come none are using and contributing to an open source data catalogue like CKAN or Open Data Catalog? These projects are so much more advanced than the sites that have been deployed in Australia and it’s no wonder – CKAN’s maturity is because it has been developed for over four years.

Not only that but using an open source open data catalogue simply means it’s cheaper for Australian governments to provide these services. If data.gov.au had used and contributed to an open source data catalogue we’d probably have had the vast majority of Australian local customisations done (if any were needed) and subsequent catalogues by Victoria, NSW and now councils would border on trivial to set up and deploy.

Doing It Right

The open source catalogues I mention above also get some key things right that Australian sites deployed so far don’t – namely open and public data requests and an API (oh the irony of open data catalogues not having a good any API!).

Open and public requests are critical and would highlight the lacklustre data released to date in Australia. The Brisbane catalogue mentioned above is a prime example, there’s a clear and demonstrable need for development application data to be opened and freed but it hasn’t been chosen as one of the items to be included in the data that launched with the site. If there was a way for data developers to make public requests and not a glorified contact form we might see this data opened at some stage. As it stands, and while there’s a scraper gathering the data that simply ignores the onerous conditions, I doubt we’ll see this.

How do we fix it?

However there’s no point in moaning without suggesting a way to fix things. One way the open data community in Australia could promote the use more advanced open source data catalogues would be to simply set up our own one. A system like CKAN can slurp in data from all of the existing catalogues, thereby providing a comprehensive centralised source of all Australian open data (not just one level of government and not just community projects like OpenAustralia). This would also provide an API to that data and a central place where everyone can log requests for open data and track what data is really in demand by open data hackers in Australia.

Maybe someone should take this on as a project for #odhd? ;)

I wrote a little post about creating a PlanningAlerts scraper for the Northern Territory for my mum.

We ran an OpenAustralia hackfest last weekend and I wrote a blog post that was cross posted on the Official Google Australia Blog.

I wrote a guest post for ScraperWiki about our hackfest.

And I gave a talk at the hackfest:

and wrapped up the hackfest:

ScraperWiki also featured my ACMA scraper a while back (I must get back to that project):

We’ve written a bunch of scrapers that are being used by PlanningAlerts and I’ve also created a few for other bits of data I am interested in or that I’ve been commissioned by clients to write.

For the second time, my creations have been featured on the front page, isn’t that nice? :)

Given that I’ve not delved too deep into Drupal and certainly never developed on it, I thought I had better find out a bit about it before the conference.

I decided to write a module similar to oa4wp that displays your MP’s most recent speeches in a block. After a few hours hacking on Sunday afternoon, here it is:

OpenAustralia Drupal in a content block

I was pleased to get something usable in such a short amount of time and found the development reasonably straight-forward, despite the documentation for developers not really being there yet for version 7.

The block also works quite well as a sidebar, depending on what you want to use it for.

Install it, just like any other module, by downloading it and putting it in its own directory in the sites/all/modules folder of your Drupal site.

OpenAustralia DevLive, the ready-to-run OpenAustralia.org and PlanningAlerts development environment, is built with SUSE Studio and you can now find it published in SUSE Gallery.

This means you can download it directly from SUSE Gallery and more importantly, clone it to customise it or to improve it. I haven’t been giving it much love lately ;( so it’d be great if someone added a few new things to it. There’s a bunch of things that could be done to improve it and some bugs to fix, which should probably be added to the ticket tracker.

Unfortunately I wasn’t able to attend the hackfest as I was busy jumping out of a plane that weekend but I nonetheless managed to get a new release of DevLive ready that includes a development copy of PlanningAlerts.

I’ve added the relevant details to the DevLive page, where you can also find download links for v0.1.0 of the appliance.

If you’re planning to take a look, I think I should alert you (haw haw!) to the fact that the code is moving at a rapid pace and it might be a good idea to pull an updated copy of the code from Github to make sure you’re seeing the latest-and-greatest code.

If you have any problems with this release or any feature requests, leave them as a comment over on the DevLive page.

I’ve created another incremental build of OpenAustralia DevLive. Version 0.0.5 simply makes the web application’s directory the twfy git source tree so you can now use the source control features of git (like resetting after making changes).

You can download the 429 MB VMWare image from SUSE Studio for the next week or so (let me know if you want it after this as I may have to initiate a rebuild to make the download available again).

Git Repository

All of the files used to create OpenAustralia DevLive are now hosted in a GitHub repository so you can now build your own if you’d like.

After the appliance boots, build the search index by opening a terminal and running:

The 416 MB VMWare image can be downloaded from SUSE Studio for the next week or so. If you have any issues downloading or using the appliance, please leave a comment.

It is built using SUSE Studio and openSUSE 11.1.

Downloads

The appliance is available in two formats, as a VMWare appliance (that can be run using the free VirtualBox software or the no cost VMWare Player) or a live CD. At this stage only a full desktop version is available, although it will be trivial to build a headless server-only appliance if demand warrants it.

You can use the link at the bottom of this post to download the latest version of the VMWare appliance.

How-to

This was all made possible by SUSE Studio and the fantastic development team behind it. Here’s a high-level overview of how it was made and some hints on using SUSE Studio:

The appliance is based on the openSUSE 11.1 GNOME Desktop template, from there I added the following software:

• lamp_server pattern
• libmysqlclient-devel
• MozillaFirefox-branding-openSUSE (pulls in Firefox)
• perl-DBD-mysql
• perl-Error
• perl-HTML-Parser
• perl-XML-RSS-LibXML
• perl-XML-Twig

Most of these are dependencies listed in the OpenAustralia install instructions.

After the Software comes the Configuration tab of SUSE Studio:

SUSE Studio Configuration tab

Beyond the basics that need configuration in this screen, I exported a copy of the OpenAustralia database schema and uploaded it using the Server tab. I also found that the Apache server was not configured to start automatically so I added a line to the post-build Scripts tab:

to have the server start at runlevels 2, 3 and 5. I did this as chkconfig creates symlinks and the overlay files feature of SUSE Studio doesn’t create these (more on that now).

The Overlay Files tab is where things start to get interesting, from here you can upload files that get overlaid on the filesystem of our appliance. This is how the OpenAustralia files get created in /srv/www/openaustralia, for example. For this appliance there are a bunch of custom files:

SUSE Studio Overlay Files tab

• /etc/hosts – points dev.openaustralia.org to the local machine so you can use Firefox to look at the dev site
• /etc/php5/apache2/php.ini – set short_open_tag = On
• Add in the TWFY and libs source code into /srv/www/openaustralia
• Add a custom config/general OpenAustralia configuration file
• Add the OpenAustralia Apache configuration and enable the rewrite module

That’s it, now all you need to do is visit the Build tab to build a VMWare image, Disk/USB image, Live CD or Xen virtual machine.

Settings

Some important settings you’ll need:

• User user auto logs in to a desktop. Just fire up Firefox to see the dev site
• Unix users root and user – password oa
• MySQL – database openaustralia, user openaustralia, password openaustralia

Future

As already stated it will be trivial for us to build headless server appliances that will be a bit lighter to download. I’m also intending to create live CDs to take to the OpenAustralia Melbourne Hackfest in the next month or so.

To-do

• Add data to the DB v0.0.2
• Firefox default start page v0.0.3
• Add Open VM tools v0.0.3
• Test binary diffs to create delta images for testers
• Have the appliance download the latest code via a script that the user can run
• Better logo
• Get the parser working
• Get a dev copy of the Public Whip working

Change Log

A big thank you to Andrew from Community Builders for hosting the appliance download.

0.0.3 – Added git, added Open VM Tools (removed Perl-Error due to a conflict). Firefox bookmarks and start page changed

0.0.2 – Added data to DB

0.0.1 – First alpha release. Basic web app running

This post has been edited