Robin Smidsrød

During the summer season of 2012 (northern hemisphere) I've had the pleasure of being the coordinator of the iPXE network boot loader's participation in the Google Summer of Code program. This is a summary of what happened during the summer.

What is iPXE?

iPXE is a network boot loader. What does that actually mean? Normally when you boot your computer, control is handed over to the boot sector on a local hard drive in your computer and an operating system is loaded. But that is not the only option you have. For a lot of years a standard called PXE (developed by Intel), has been available which allows x86-based computers to boot from the network (using DHCP and TFTP). iPXE's main goal is to extend that ability to not only boot from TFTP (which is quite slow), but allow computers to boot from HTTP, FTP, iSCSI, AoE and other more esoteric sources. We also have a goal of being the main network boot loader for EFI-based systems (almost working, get in touch for details). We already support more than 60 different network cards, and cards without a native driver are also supported using a feature called chainloading.

First time participating in GSoC

This is the first year iPXE participates in the GSoC program as an organization. Most of the mentors involved have participated earlier years as part of other organizations, such as kernel.org and Etherboot. It was the first time for me, though, and it was quite fun! My job was to keep all the students and mentors on track and ensure no deadlines were forgotten and that communication between everyone flowed smoothly. In total we had three students, which I understand is quite a lot for a first year organization. Thank you to Google for believing in us.

Students and projects

Marin Hannache from EPITA University in France wrote protocol support for downloading boot files using NFS. His work is awaiting review and integration into mainline. From what I've tested it still has some issues, but it is working, and I'm most eager to get it in, as it will allow booting directly from an NFS3 server, not even needing a web server. This should be great news for people using small NAS boxes to do their booting, as they don't always have a way to serve files using HTTP.

Adrian Jamróz from Jan Kochanowski University in Poland reimplemented the drivers for the VIA Rhine and VIA Velocity family of network cards. The existing drivers were using a deprecated API and was much bigger in size than they really needed to be. Code size is always a big challenge for a project like iPXE, which commonly is burned into ROM chips that have a limit of 64KB. His code was very good and is just awaiting final review and integration. I've personally tested it on one of my Rhine2-based cards, and it seems to work reliably.

Daniel Wyatt from Lord Fairfax Community College in the United States reimplemented the driver for the Broadcom NetExtreme II family of network cards. The problem with the old bnx2 driver was the same as the VIA driver. It used a very large proprietary firmware blob to run the MIPS-based microcontrollers on the card. Most of the summer was used to understand how we could minimize this firmware without losing any functionality. In the end the student got something working, but we're still having issues with the legality of including the firmware inside the iPXE project. The integration with mainline is currently on the fence awaiting some kind of confirmation from Broadcom that we can actually distribute the firmware blob with iPXE. Luckily we have people from Broadcom participating in the community, so it shouldn't be too impossible to get this kind of approval. I guess this student got a crash course in how licensing affects what you can do within an open-source project.

Other tidbits

All of the pages linked to above were maintained by the students as they were working on their code during the summer. You can also find links to specific Git repositories that include all of their code if you're interested in testing it out before it is integrated into mainline. Most of the day-to-day collaboration was done using IRC on the #ipxe channel on irc.freenode.net. Below you can read some of the interesting things said during the summer by the students:

• 20120608-010814 <Mareo> wow, I'm in the black magic part of iPXE codebase :D
• 20120703-121824 <Mareo> this gsoc is awesome :D
• 20120712-203713 <dewyatt> the bnx2 datasheet is horrible :(
• 20120720-233044 <dewyatt> mcb30: okay...I'm eager to get started on the fun stuff
• 20120728-160956 <budrys> meteger: but I think it can't be left as is, it's a dirty hack for me
• 20120815-172646 <dewyatt> mcb30: mouse sent me some ROMs that have firmware in them. This made me take another look at the newest PXE ROM I have and I found compressed firmware. I hackishly extracted this by pausing the broadcom PXE ROM during DHCP and reading from the Debug UART. I can't do this for RV2P and I don't really know the bounds of the MIPS firmware so it's still a work in progress.
• 20120816-083313 <dewyatt> I don't know if I can legally commit this code really :/

Thanks to everyone

It is great to have taken part in this journey and seen how much the students have learned over the summer. I want to thank our three mentors, Michael Brown, Joshua Oreman and Thomas Miletich and all the other knowledgable people on the #ipxe IRC channel for the great work they've provided during the summer. Participating in GSoC has been a great experience for us all, and I hope we'll be able to participate next year as well.

If I've whet your appetite to learn more about network booting, I invite you to join us on IRC, participate in the iPXE forums or join the mailing-list.

Until next time,
Robin Smidsrød,
iPXE GSoC 2012 coordinator

I've just published a new project on GitHub, named znc-log-viewer, that might be useful for IRC junkies.

If you use the ZNC IRC bouncer to keep you logged on to IRC all the time, you might've enabled the log module to avoid missing messages when you're not connected to the bouncer.

Searching those log files have always been a hassle, because they are located deep inside the .znc directory, and there are a lot of files stored in one big directory, which makes it hard to get an overview when you want to pull out your trusty friend, grep.

If you actually bothered to use grep, you would also notice that a single log file has a lot of server messages in it, that is generally not interesting when you're trying to figure out what has happened since the last time you were online. In my case, I just want access to the conversations that have been going on.

So if this problem is something you can relate to, feel free try out my new tool. If you find any problems with it, please open up an issue on the GitHub page.

After I wrote that big 5-part article on WWW::LastFM / XML::Rabbit I noticed several things that could be improved. One of them was the fact that loading configuration information from a file in the home directory was a pretty generic thing to do. I decided to factor that code out and release it as a separate CPAN module that can be used by any Moose class.

If you look at the code used to read the configuration information in the article mentioned above you'll see that it has improved quite a bit. The code below is shorter, but does the same thing as the old code.

package WWW::LastFM;
use Moose;
use namespace::autoclean;

# Read configuration from ~/.lastfm.ini,
# available in $self->config
has 'config_filename' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);
sub _build_config_filename { '.lastfm.ini' }
with 'Config::Role';

# And here we have our api key and secret
has 'api_key' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);
sub _build_api_key {
    return (shift)->config->{'API'}->{'key'};
}

has 'api_secret' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);
sub _build_api_secret {
    return (shift)->config->{'API'}->{'secret'};
}
...

The reason this approach is quite nice, is because it allows you to load e.g. username and password from a configuration file with the ability to override it in the constructor. You can also quite easily override the location to search for the configuration file in your tests by mocking the location returned by File::HomeDir->my_data(). Alternatively, you can specify either the config_file or config_files constructor parameters to explicitly specify the path to your configuration file (or an array reference of them).

I usually start my test case with code like this to override where to look for the configuration file for the test environment. Then I just put a copy of my configuration file inside the t folder with the same file name as usual.

# Mock home/config dir location
{
    use File::HomeDir;
    no warnings 'redefine';
    sub File::HomeDir::my_data { return 't'; }
}

The configuration file is loaded via Config::Any->load_files(), so you can use any format supported by that module. Your config attribute will always be represented as a hash reference.

If you find this article or the CPAN module Config::Role useful I'd be very happy if you share it with your friends on Facebook, Google+, Twitter or other service you use. If you like what you see, follow me on any of the networks at the top of the page of my blog and subscribe to my feed to keep informed on when I post something new.

• Part 1 of 5
• Part 2 of 5
• Part 3 of 5
• Part 4 of 5
• Part 5 of 5

In the previous article we started implementing the geo.getEvents Last.FM API call. In this article we will complete the implementation of the Event class and create a small application to display events for a specific location in a compact format.

Implementing the <event/> XML chunk extractor

So let's move on to the WWW::LastFM::Response::Event class. This is one of the core classes for our particular problem domain. Everything up to here has just been stuff needed to build an extensible framework for sending our queries to the Last.FM API and extract values from the response data. Let's get on with it. Create lib/WWW/LastFM/Response/Event.pm with this content:

package WWW::LastFM::Response::Event;
use XML::Rabbit;

use HTML::FormatText;
use DateTime::Format::HTTP;

# Last.FM-specific data
has_xpath_value 'id'  => './id';
has_xpath_value 'tag' => './tag';
has_xpath_value 'url' => './url';

# The actual event-related data
has_xpath_value 'title'            => './title';
has_xpath_value 'description_html' => './description';
has_xpath_value 'website'          => './website';
has_xpath_value 'start_date'       => './startDate';
has_xpath_value 'headliner'        => './artists/headliner';
has_xpath_value_list '_artists'    => './artists/artist',
    handles => {
        'artists'      => 'elements',
    }
;
has_xpath_object 'venue'     => './venue'    => 'WWW::LastFM::Response::Venue';
has_xpath_value_list '_tags' => './tags/tag',
    handles => {
        'tags' => 'elements',
    },
;

# Some various attendance-related data
has_xpath_value 'ticket_count' => './tickets';
has_xpath_value 'attendance'   => './attendance';
has_xpath_value 'cancelled'    => './cancelled';
has_xpath_value 'review_count' => './reviews';

# Event image URLs
has_xpath_value 'image_s'  => './image[@size="small"]';
has_xpath_value 'image_m'  => './image[@size="medium"]';
has_xpath_value 'image_l'  => './image[@size="large"]';
has_xpath_value 'image_xl' => './image[@size="extralarge"]';

# The description value, converted to plain text
has 'description' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);

sub _build_description {
    my ($self) = @_;
    return HTML::FormatText->format_string(
        $self->description_html,
        leftmargin  =>  0,
        rightmargin => 79,
    );
}

# Convert the start_date to a DateTime class which is much more useful
has 'date' => (
    is         => 'ro',
    isa        => 'DateTime',
    lazy_build => 1,
);

sub _build_date {
    my ($self) = @_;
    return DateTime::Format::HTTP->parse_datetime(
        $self->start_date
    );
}

finalize_class();

There shouldn't be anything surprising in this piece of code. The only thing you haven't seen before is the use of XPath attribute values as selectors. We've already covered all the various declarations, so let's just continue with WWW::LastFM::Response::Venue, so that we can have a look at how to deal with the namespaced latitude/longtitude information. Create lib/WWW/LastFM/Response/Venue.pm with this content:

package WWW::LastFM::Response::Venue;
use XML::Rabbit;

# Last.FM-specific data
has_xpath_value 'id'  => './id';
has_xpath_value 'url' => './url';

# The actual venue-related data
has_xpath_value 'name'         => './name';
has_xpath_value 'website'      => './website';
has_xpath_value 'phone_number' => './phonenumber';

# Location-related data
has_xpath_value 'city'        => './location/city';
has_xpath_value 'country'     => './location/country';
has_xpath_value 'street'      => './location/street';
has_xpath_value 'postal_code' => './location/postalcode';
has_xpath_value 'latitude'    => './location/geo:point/geo:lat';
has_xpath_value 'longitude'   => './location/geo:point/geo:long';

# Venue image URLs
has_xpath_value 'image_s'  => './image[@size="small"]';
has_xpath_value 'image_m'  => './image[@size="medium"]';
has_xpath_value 'image_l'  => './image[@size="large"]';
has_xpath_value 'image_xl' => './image[@size="extralarge"]';

# Convenience attribute for combined address
has 'address' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);

sub _build_address {
    my ($self) = @_;
    return $self->name . ', '
         . ( $self->street ? $self->street . ', ' : "" )
         . ( $self->postal_code ? $self->postal_code . ' ' : "" )
         . $self->city . ', '
         . $self->country;
}

finalize_class();

This is just more of the same things we've been doing. Notice the simplicity in dealing with namespaces in the XPath queries for latitude and longitude? It's really that simple.

lastfm_events.pl, a simple WWW::LastFM client

Let's finish it up by making a fairly simple command line app to search for events. Create bin/lastfm_events.pl with the following content:

#!/usr/bin/env perl

use strict;
use warnings;
use rlib;
use feature qw(say);

use WWW::LastFM;
use Getopt::Long;

# PODNAME: lastfm_eventss.pl
# ABSTRACT: Search the Last.FM API for events/concerts

STDOUT->binmode(":utf8");

# Read command line flags
my $limit;
my $distance;
my $help;
GetOptions(
    "limit=i"    => \$limit,
    "distance=i" => \$distance,
    "help"       => \$help,
);

die <<"EOM" if $help;
Usage: $0 [<options>] [<location>]
Options:
    --limit <num>   ; Number of events to return
    --distance <km> ; How large area to search
EOM

my $location = shift;

# Perform query
my $event_list = WWW::LastFM->new->geo->get_events(
    ( defined $limit    ? ( limit    => $limit )    : () ),
    ( defined $distance ? ( distance => $distance ) : () ),
    ( defined $location ? ( location => $location ) : () ),
)->events;
my @events = $event_list->all;

# Format and output results
if ( @events > 0 ) {
    foreach my $event ( sort { $a->date cmp $b->date } @events ) {
        my $formatted_date = DateTime::Format::HTTP->format_datetime($event->date);
        my $headliner = $event->title ne $event->headliner ? " with " . $event->headliner : "";
        my @additional_artists = grep { ! $event->headliner } $event->artists;
        say $formatted_date . ": "
          . $event->title
          . $headliner
          . ( scalar @additional_artists ? " and " . join(", ", @additional_artists ) : "" )
          . " @ " . $event->venue->address;
    }
}
else {
    say STDERR "No events found for location '" . $event_list->location . "'";
}

1;

Let's have some fun with it!

$ bin/lastfm_events.pl --distance 50 --limit 50 Tønsberg
Fri, 04 Nov 2011 19:30:00 GMT: Melissa Horn @ Kulturhuset Bølgen, Larvik, Norway
Fri, 18 Nov 2011 19:00:00 GMT: Konsert, Bjørn Eidsvåg med band  with Bjørn Eidsvåg @ Parkteatret, Moss, Norway
Fri, 16 Dec 2011 19:00:00 GMT: Helene Bøksle på Domkirkefestivalen i Tønsberg Domkirke with Helene Bøksle @ Tønsberg Domkirke, Tønsberg, Norway
Fri, 16 Dec 2011 19:31:01 GMT: Ljungblut @ Total, Stoltenbergsgata 46, 3110 Tønsberg, Norway
Sat, 17 Dec 2011 19:34:01 GMT: Ljungblut @ Total, Stoltenbergsgata 46, 3110 Tønsberg, Norway

This shows the events happening around where I live. Not really much to be cheerful about. Hopefully you'll get some more interesting results for where you live.

Conclusion

I wrote this article mostly to showcase how easy it is extract useful information from XML documents with XML::Rabbit and augment it with additional Moose magic. I don't really go to concerts very often, but I figured it was a useful example that some of you would find interesting (instead of just using some hand-crafted example XML).

If you want to see additional API calls implemented I urge you to get in touch with me and help me to improve it. I've shown you how I've done it this far. With some help from various Perl learning resources I'm sure you'd be able to whip up a patch with some additional features (or tests). Please follow me on Twitter or GitHub or subscribe to my blog if you want to stay informed with what I'm doing.

• Part 1 of 5
• Part 2 of 5
• Part 3 of 5
• Part 4 of 5
• Part 5 of 5

In the previous article we finished implementing the geo.getMetros Last.FM API call with XML::Rabbit. That particular API call doesn't include a lot of interesting information, but it is useful to know which worldwide locations the Last.FM service knows about now that we're going to implement the geo.getEvents API call.

Implementing the geo.getEvents API call

Now that we know which locations are valid, we can actually ask for events for a specific location. Add another method towards the end of WWW::LastFM::API::Geo, like this:

# http://www.last.fm/api/show?service=270
# All strings must be binary encoded utf8 strings
#
# lat (Optional) : Specifies a latitude value to retrieve events for (service returns nearby events by default)
# location (Optional) : Specifies a location to retrieve events for (service returns nearby events by default)
# long (Optional) : Specifies a longitude value to retrieve events for (service returns nearby events by default)
# distance (Optional) : Find events within a specified radius (in kilometres)
# limit (Optional) : The number of results to fetch per page. Defaults to 10.
# page (Optional) : The page number to fetch. Defaults to first page.

sub get_events {
    my ($self, %opts) = @_;
    my $params = "";
    foreach my $key ( keys %opts ) {
        $params .= '&' . $key . '=' . uri_escape( $opts{$key} );
    }
    my $xml = $self->lastfm->get(
           $self->lastfm->api_root_url
        . '?method=geo.getEvents'
        . $params
        . '&api_key=' . $self->lastfm->api_key
    );
    return WWW::LastFM::Response->new( xml => $xml );
}

If you've paid attention, you will notice that this method is quite similar to get_metros. We will not go into refactoring it in this article, but it should be quite easy to refactor the common parts into a generic method on the (non-existing) WWW::LastFM::API class and turn that into either a role or a base class for the rest of the WWW::LastFM::API::* classes we might create in the future.

Implementing the <events/> XML chunk extractor

So let's try it out and see what we get. Again, we will use a fairly simple one-liner.

$ perl -Ilib -MWWW::LastFM -E 'STDOUT->binmode(":utf8"); say WWW::LastFM->new->geo->get_events( limit => 1 )->dump_document_xml()'
<?xml version="1.0" encoding="utf-8"?>
<lfm status="ok">
<events xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" location="Oslo, Norway"
        page="1" perPage="1" totalPages="163" total="163" festivalsonly="0">
    <event xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">
  <id>1996187</id>
  <title>The Jezabels</title>
  <artists>
    <artist>The Jezabels</artist>
    <headliner>The Jezabels</headliner>
  </artists>
    <venue>
    <id>9014441</id>
    <name>John Dee</name>
    <location>
      <city>Oslo</city>
      <country>Norway</country>
      <street>Torggata 16</street>
      <postalcode>0181</postalcode>
      <geo:point>
         <geo:lat>59.915823</geo:lat>
         <geo:long>10.751034</geo:long>
      </geo:point>
    </location>
    <url>http://www.last.fm/venue/9014441+John+Dee</url>
    <website>http://www.rockefeller.no/</website>
    <phonenumber>22 20 32 32</phonenumber>
    <image size="small"/>
        <image size="medium"/>
        <image size="large"/>
        <image size="extralarge"/>
        <image size="mega"/>
  </venue>    <startDate>Sun, 25 Sep 2011 20:00:00</startDate>
  <description><![CDATA[<div class="bbcode">The Jezabels begynte å spille sammen i 2007
  etter å ha møttes på Universitet i Sydney, Australia, og har etter det gitt ut tre
  ep-plater som har fått god mottagelse over alt i verden. Bandet har blant annet vært
  nominert til Best Breakthrough Artist og Best Single / EP på Independent Music Awards
  i 2010 og vært support til Tegan &amp; Sara. Det er litt vanskelig å beskrive popmusikken
  deres; det er som en blanding mellom indiepop og indierock, med litt disko. Bra er det
  i hvert fall! The Jezabels har ikke gitt ut noen skive i Norge ennå, men hvis du har
  sett Imsdals (forferdlige) reklamefilme, har du hørt låten ”Easy To Love”. Debutskiven
  kommer under 2011 og den 25. september kan du se bandet på John DEE!<br /><br />Billetter
  er i salg via Billettservice!</div>]]></description>
  <image size="small">http://userserve-ak.last.fm/serve/34/48219593.png</image>
  <image size="medium">http://userserve-ak.last.fm/serve/64/48219593.png</image>
  <image size="large">http://userserve-ak.last.fm/serve/126/48219593.png</image>
  <image size="extralarge">http://userserve-ak.last.fm/serve/252/48219593.png</image>
  <attendance>6</attendance>
  <reviews>0</reviews>
  <tag>lastfm:event=1996187</tag>
  <url>http://www.last.fm/event/1996187+The+Jezabels</url>
  <website>https://www.facebook.com/event.php?eid=172860196109875</website>
    <tickets>
  </tickets>
    <cancelled>0</cancelled>
    <tags>
      <tag>australian</tag>
      <tag>indie</tag>
      <tag>rock</tag>
      <tag>female vocalists</tag>
    </tags>
    </event></events></lfm>

Now we're starting to see what we're interested in!

But wait, what is that xmlns:geo attribute at the top there? Will that cause a problem? Yes it will. - That is an XML namespace, and if we want to be able to pull out information from those <geo:point/> tags we'll need to specify that we want to deal with this particular namespace. Luckily, dealing with XML namespaces is quite easy with XML::Rabbit. The first thing we need to do is to declare that we want to create queries that use this namespace. Add the following piece of code before the BUILD method in lib/WWW/LastFM/Response.pm to take care of that challenge:

add_xpath_namespace 'geo' => 'http://www.w3.org/2003/01/geo/wgs84_pos#';

Now we can add a new accessor for the events in WWW::LastFM::Response. Add the following piece of code at the bottom of lib/WWW/LastFM/Response.pm:

has_xpath_object 'events' => '/lfm/events' => 'WWW::LastFM::Response::EventList';

As this points to another class, we'll have to create that one too. The contents of lib/WWW/LastFM/Response/EventList.pm should be this:

package WWW::LastFM::Response::EventList;
use XML::Rabbit;

has_xpath_value 'page'       => './@page';
has_xpath_value 'page_limit' => './@perPage';
has_xpath_value 'page_count' => './@totalPages';

has_xpath_value 'location'       => './@location';
has_xpath_value 'festivals_only' => './@festivalsonly';

has_xpath_object_map 'map' => './event',
    './id'  => 'WWW::LastFM::Response::Event',
    handles => {
        'get' => 'get',
        'ids' => 'keys',
        'all' => 'values',
    },
;

finalize_class();

Everything here should be easy to understand, as we're only declaring simple values. The has_xpath_object_map declaration might require some clarification, though. We're declaring a Moose attribute called map which is a hash reference where the event id from the XML document is the key, and an object of class WWW::LastFM::Response::Event is the value. The native trait delegations for hashes are somewhat different than the ones for arrays. As this object is not designed for data modification, we've only added the typical read accessors. Whether you will use ids and get or all depends a bit on your application needs. We're only going to use all in our application.

In the next and final article we'll flesh out the implementation of the Event class. We will also create a small application that allows us to display events for a specific location in a compact format. See you soon!