If you use the ZNC IRC bouncer to keep you logged on to IRC all the time, you might've enabled the log module to avoid missing messages when you're not connected to the bouncer.

Searching those log files have always been a hassle, because they are located deep inside the .znc directory, and there are a lot of files stored in one big directory, which makes it hard to get an overview when you want to pull out your trusty friend, grep.

If you actually bothered to use grep, you would also notice that a single log file has a lot of server messages in it, that is generally not interesting when you're trying to figure out what has happened since the last time you were online. In my case, I just want access to the conversations that have been going on.

So if this problem is something you can relate to, feel free try out my new tool. If you find any problems with it, please open up an issue on the GitHub page.

But while I was setting it up I had some very odd problem. When I plugged everything in and wanted to boot it, it just didn't want to boot any devices (but the EFI BIOS loaded without problems), it reset instead.

At first I thought I had a power problem, so I went out and bought a bigger PSU. But it still reset instead of booting. Then I called a friend and he thought the SATA controller might be busted. So he suggested unplugging everything and trying one part at the time.

After doing a lot of cross-testing I figured out that it was the USB cable to my APC Smart-UPS 1000 that stopped the boot from processing. So obviously either my UPS or motherboard firmware is incompatible with each other. Luckily I was not dependent on having the UPS USB cable connected, so I just disconnected it to avoid the problem. With that component disconnected everything worked as expected.

Moral of the story: Test with one component at the time if you're experiencing difficulties.

During this process obviously all files had kept the old SIDs, and as they weren't matching his new SID (on the new computer) I was having problems accessing the files. Lots of stuff didn't work. I was getting "No access" messages left and right when I tried to change the permissions of those copied files. That is when it dawned on me that even though his user account was registered as an administrator, it wasn't able to change the permission bits on those files.

To avoid having to mess around forever to reset permissions to default inherited ACL on all those files (around 30000 or so of them) I dug a bit around on the Internet and found out that there is a command in Windows 7 called icacls.exe that works very much in the same way as chmod on Linux, just that it supports NTFS ACLs.

After reading around I found a forum post that mentioned it, and was redirected to Microsoft's knowledge base article on icacls.exe. After looking over that one I found this command line that solved everything.

cd C:\Users\username
icacls * /reset /T

The /T switch makes it run recursively. /reset, as the KB article explains, replaces the ACL on each file and folder with the default inherited ACL that just uses the ACL on the parent folder.

The last thing that was required was to run this command inside a cmd.exe shell with elevated privileges. You can do that by finding the Command Prompt icon in your start menu and right-clicking "Run as administrator".

Be aware that mucking around with ACLs, if you don't know what you're doing, can seriously break your system in more ways than you can think of. It can also open you up to various attacks because of more relaxed permissions.

If you like what you just read, please subscribe to the feed or follow me on Twitter or any of the other networks I'm registered in to be updated when I post something new.

If you look at the code used to read the configuration information in the article mentioned above you'll see that it has improved quite a bit. The code below is shorter, but does the same thing as the old code.

package WWW::LastFM;
use Moose;
use namespace::autoclean;

# Read configuration from ~/.lastfm.ini,
# available in $self->config
has 'config_filename' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);
sub _build_config_filename { '.lastfm.ini' }
with 'Config::Role';

# And here we have our api key and secret
has 'api_key' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);
sub _build_api_key {
    return (shift)->config->{'API'}->{'key'};
}

has 'api_secret' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);
sub _build_api_secret {
    return (shift)->config->{'API'}->{'secret'};
}
...

The reason this approach is quite nice, is because it allows you to load e.g. username and password from a configuration file with the ability to override it in the constructor. You can also quite easily override the location to search for the configuration file in your tests by mocking the location returned by File::HomeDir->my_data(). Alternatively, you can specify either the config_file or config_files constructor parameters to explicitly specify the path to your configuration file (or an array reference of them).

I usually start my test case with code like this to override where to look for the configuration file for the test environment. Then I just put a copy of my configuration file inside the t folder with the same file name as usual.

# Mock home/config dir location
{
    use File::HomeDir;
    no warnings 'redefine';
    sub File::HomeDir::my_data { return 't'; }
}

The configuration file is loaded via Config::Any->load_files(), so you can use any format supported by that module. Your config attribute will always be represented as a hash reference.

If you find this article or the CPAN module Config::Role useful I'd be very happy if you share it with your friends on Facebook, Google+, Twitter or other service you use. If you like what you see, follow me on any of the networks at the top of the page of my blog and subscribe to my feed to keep informed on when I post something new.

I recently read an article in Linux Journal #210 that talked about a new feature in the Linux kernel called ipset. It allows you to create sets that store IP addresses which can dynamically be added to and removed from. This sets can be configured as selectors in iptables rules, so that you can perform actions on any IP address in the set, like dropping the packets. You can also create iptables rules that add or remove an IP from a set.

In Debian it is straight-forward to install ipset. Type in these commands to install ipset and the required kernel module. It should ensure it is automatically recompiled whenever you do a kernel upgrade.

$ apt-get install ipset xtables-addons-source
$ module-assistant auto-install xtables-addons

I asked around on IRC if it was possible to make sshd execute a shell script once the amount of logins from an IP went above some configurable threshold, but apparently it is not possible. Someone pointed me at fail2ban which is a system that scans your syslog looking for failed logins and turn them into blocking iptables rules (and probably more, I didn't look very closely). I thought that this was too slow, as I wanted something that triggered as soon as multiple failed logins for a user reached a certain threshold, but I wanted the block to be temporary, not permanent.

That is when I remembered that syslog (rsyslog in my case) can be configured to run the log messages through a program. I put together the program below that reads incoming messages from rsyslog with auth.info facility/level (default for sshd on Debian) and does the necessary things to ensure the offending IP is added to my autoblock ipset if it triggers a certain amount of failed logins in a short period. The autoblock ipset uses the iptree storage module that has a --timeout parameter to automatically purge entries from the set after a given time. I set it to 3600 seconds (1 hour).

What is also quite cool, is that I have a rule that will immediately put the IP of anyone that tries to connect to my SMTP port (I don't run a mail server on my firewall) on the autoblock list. This particular bit happens completely inside netfilter, so its effect is immediate. Bye bye spammers! Try to connect to my (non-existent) email server and you're instantly blocked for an hour. And you didn't even know what hit you. :)

The reason I like the temporary block is that sometimes I port-scan my own server or do other strange things with it from remote computers, and the fact that I know the block will be lifted after an hour means I can continue without having to get physical access to server to remove the block. I can just smack myself in the forehead and wait an hour and continue whatever stupid thing I was doing.

PS: Remember that lots of IRC servers like to port-scan your IP when you connect to them, so you might need to put up some exceptions for those if you're an active IRC user and your particular IRC server likes to probe the SMTP port.

In the previous article we started implementing the geo.getEvents Last.FM API call. In this article we will complete the implementation of the Event class and create a small application to display events for a specific location in a compact format.

Implementing the <event/> XML chunk extractor

So let's move on to the WWW::LastFM::Response::Event class. This is one of the core classes for our particular problem domain. Everything up to here has just been stuff needed to build an extensible framework for sending our queries to the Last.FM API and extract values from the response data. Let's get on with it. Create lib/WWW/LastFM/Response/Event.pm with this content:

package WWW::LastFM::Response::Event;
use XML::Rabbit;

use HTML::FormatText;
use DateTime::Format::HTTP;

# Last.FM-specific data
has_xpath_value 'id'  => './id';
has_xpath_value 'tag' => './tag';
has_xpath_value 'url' => './url';

# The actual event-related data
has_xpath_value 'title'            => './title';
has_xpath_value 'description_html' => './description';
has_xpath_value 'website'          => './website';
has_xpath_value 'start_date'       => './startDate';
has_xpath_value 'headliner'        => './artists/headliner';
has_xpath_value_list '_artists'    => './artists/artist',
    handles => {
        'artists'      => 'elements',
    }
;
has_xpath_object 'venue'     => './venue'    => 'WWW::LastFM::Response::Venue';
has_xpath_value_list '_tags' => './tags/tag',
    handles => {
        'tags' => 'elements',
    },
;

# Some various attendance-related data
has_xpath_value 'ticket_count' => './tickets';
has_xpath_value 'attendance'   => './attendance';
has_xpath_value 'cancelled'    => './cancelled';
has_xpath_value 'review_count' => './reviews';

# Event image URLs
has_xpath_value 'image_s'  => './image[@size="small"]';
has_xpath_value 'image_m'  => './image[@size="medium"]';
has_xpath_value 'image_l'  => './image[@size="large"]';
has_xpath_value 'image_xl' => './image[@size="extralarge"]';

# The description value, converted to plain text
has 'description' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);

sub _build_description {
    my ($self) = @_;
    return HTML::FormatText->format_string(
        $self->description_html,
        leftmargin  =>  0,
        rightmargin => 79,
    );
}

# Convert the start_date to a DateTime class which is much more useful
has 'date' => (
    is         => 'ro',
    isa        => 'DateTime',
    lazy_build => 1,
);

sub _build_date {
    my ($self) = @_;
    return DateTime::Format::HTTP->parse_datetime(
        $self->start_date
    );
}

finalize_class();

There shouldn't be anything surprising in this piece of code. The only thing you haven't seen before is the use of XPath attribute values as selectors. We've already covered all the various declarations, so let's just continue with WWW::LastFM::Response::Venue, so that we can have a look at how to deal with the namespaced latitude/longtitude information. Create lib/WWW/LastFM/Response/Venue.pm with this content:

package WWW::LastFM::Response::Venue;
use XML::Rabbit;

# Last.FM-specific data
has_xpath_value 'id'  => './id';
has_xpath_value 'url' => './url';

# The actual venue-related data
has_xpath_value 'name'         => './name';
has_xpath_value 'website'      => './website';
has_xpath_value 'phone_number' => './phonenumber';

# Location-related data
has_xpath_value 'city'        => './location/city';
has_xpath_value 'country'     => './location/country';
has_xpath_value 'street'      => './location/street';
has_xpath_value 'postal_code' => './location/postalcode';
has_xpath_value 'latitude'    => './location/geo:point/geo:lat';
has_xpath_value 'longitude'   => './location/geo:point/geo:long';

# Venue image URLs
has_xpath_value 'image_s'  => './image[@size="small"]';
has_xpath_value 'image_m'  => './image[@size="medium"]';
has_xpath_value 'image_l'  => './image[@size="large"]';
has_xpath_value 'image_xl' => './image[@size="extralarge"]';

# Convenience attribute for combined address
has 'address' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);

sub _build_address {
    my ($self) = @_;
    return $self->name . ', '
         . ( $self->street ? $self->street . ', ' : "" )
         . ( $self->postal_code ? $self->postal_code . ' ' : "" )
         . $self->city . ', '
         . $self->country;
}

finalize_class();

This is just more of the same things we've been doing. Notice the simplicity in dealing with namespaces in the XPath queries for latitude and longitude? It's really that simple.

lastfm_events.pl, a simple WWW::LastFM client

Let's finish it up by making a fairly simple command line app to search for events. Create bin/lastfm_events.pl with the following content:

#!/usr/bin/env perl

use strict;
use warnings;
use rlib;
use feature qw(say);

use WWW::LastFM;
use Getopt::Long;

# PODNAME: lastfm_eventss.pl
# ABSTRACT: Search the Last.FM API for events/concerts

STDOUT->binmode(":utf8");

# Read command line flags
my $limit;
my $distance;
my $help;
GetOptions(
    "limit=i"    => \$limit,
    "distance=i" => \$distance,
    "help"       => \$help,
);

die <<"EOM" if $help;
Usage: $0 [<options>] [<location>]
Options:
    --limit <num>   ; Number of events to return
    --distance <km> ; How large area to search
EOM

my $location = shift;

# Perform query
my $event_list = WWW::LastFM->new->geo->get_events(
    ( defined $limit    ? ( limit    => $limit )    : () ),
    ( defined $distance ? ( distance => $distance ) : () ),
    ( defined $location ? ( location => $location ) : () ),
)->events;
my @events = $event_list->all;

# Format and output results
if ( @events > 0 ) {
    foreach my $event ( sort { $a->date cmp $b->date } @events ) {
        my $formatted_date = DateTime::Format::HTTP->format_datetime($event->date);
        my $headliner = $event->title ne $event->headliner ? " with " . $event->headliner : "";
        my @additional_artists = grep { ! $event->headliner } $event->artists;
        say $formatted_date . ": "
          . $event->title
          . $headliner
          . ( scalar @additional_artists ? " and " . join(", ", @additional_artists ) : "" )
          . " @ " . $event->venue->address;
    }
}
else {
    say STDERR "No events found for location '" . $event_list->location . "'";
}

1;

Let's have some fun with it!

$ bin/lastfm_events.pl --distance 50 --limit 50 Tønsberg
Fri, 04 Nov 2011 19:30:00 GMT: Melissa Horn @ Kulturhuset Bølgen, Larvik, Norway
Fri, 18 Nov 2011 19:00:00 GMT: Konsert, Bjørn Eidsvåg med band  with Bjørn Eidsvåg @ Parkteatret, Moss, Norway
Fri, 16 Dec 2011 19:00:00 GMT: Helene Bøksle på Domkirkefestivalen i Tønsberg Domkirke with Helene Bøksle @ Tønsberg Domkirke, Tønsberg, Norway
Fri, 16 Dec 2011 19:31:01 GMT: Ljungblut @ Total, Stoltenbergsgata 46, 3110 Tønsberg, Norway
Sat, 17 Dec 2011 19:34:01 GMT: Ljungblut @ Total, Stoltenbergsgata 46, 3110 Tønsberg, Norway

This shows the events happening around where I live. Not really much to be cheerful about. Hopefully you'll get some more interesting results for where you live.

Conclusion

I wrote this article mostly to showcase how easy it is extract useful information from XML documents with XML::Rabbit and augment it with additional Moose magic. I don't really go to concerts very often, but I figured it was a useful example that some of you would find interesting (instead of just using some hand-crafted example XML).

If you want to see additional API calls implemented I urge you to get in touch with me and help me to improve it. I've shown you how I've done it this far. With some help from various Perl learning resources I'm sure you'd be able to whip up a patch with some additional features (or tests). Please follow me on Twitter or GitHub or subscribe to my blog if you want to stay informed with what I'm doing.

In the previous article we finished implementing the geo.getMetros Last.FM API call with XML::Rabbit. That particular API call doesn't include a lot of interesting information, but it is useful to know which worldwide locations the Last.FM service knows about now that we're going to implement the geo.getEvents API call.

Implementing the geo.getEvents API call

Now that we know which locations are valid, we can actually ask for events for a specific location. Add another method towards the end of WWW::LastFM::API::Geo, like this:

# http://www.last.fm/api/show?service=270
# All strings must be binary encoded utf8 strings
#
# lat (Optional) : Specifies a latitude value to retrieve events for (service returns nearby events by default)
# location (Optional) : Specifies a location to retrieve events for (service returns nearby events by default)
# long (Optional) : Specifies a longitude value to retrieve events for (service returns nearby events by default)
# distance (Optional) : Find events within a specified radius (in kilometres)
# limit (Optional) : The number of results to fetch per page. Defaults to 10.
# page (Optional) : The page number to fetch. Defaults to first page.

sub get_events {
    my ($self, %opts) = @_;
    my $params = "";
    foreach my $key ( keys %opts ) {
        $params .= '&' . $key . '=' . uri_escape( $opts{$key} );
    }
    my $xml = $self->lastfm->get(
           $self->lastfm->api_root_url
        . '?method=geo.getEvents'
        . $params
        . '&api_key=' . $self->lastfm->api_key
    );
    return WWW::LastFM::Response->new( xml => $xml );
}

If you've paid attention, you will notice that this method is quite similar to get_metros. We will not go into refactoring it in this article, but it should be quite easy to refactor the common parts into a generic method on the (non-existing) WWW::LastFM::API class and turn that into either a role or a base class for the rest of the WWW::LastFM::API::* classes we might create in the future.

Implementing the <events/> XML chunk extractor

So let's try it out and see what we get. Again, we will use a fairly simple one-liner.

$ perl -Ilib -MWWW::LastFM -E 'STDOUT->binmode(":utf8"); say WWW::LastFM->new->geo->get_events( limit => 1 )->dump_document_xml()'
<?xml version="1.0" encoding="utf-8"?>
<lfm status="ok">
<events xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" location="Oslo, Norway"
        page="1" perPage="1" totalPages="163" total="163" festivalsonly="0">
    <event xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">
  <id>1996187</id>
  <title>The Jezabels</title>
  <artists>
    <artist>The Jezabels</artist>
    <headliner>The Jezabels</headliner>
  </artists>
    <venue>
    <id>9014441</id>
    <name>John Dee</name>
    <location>
      <city>Oslo</city>
      <country>Norway</country>
      <street>Torggata 16</street>
      <postalcode>0181</postalcode>
      <geo:point>
         <geo:lat>59.915823</geo:lat>
         <geo:long>10.751034</geo:long>
      </geo:point>
    </location>
    <url>http://www.last.fm/venue/9014441+John+Dee</url>
    <website>http://www.rockefeller.no/</website>
    <phonenumber>22 20 32 32</phonenumber>
    <image size="small"/>
        <image size="medium"/>
        <image size="large"/>
        <image size="extralarge"/>
        <image size="mega"/>
  </venue>    <startDate>Sun, 25 Sep 2011 20:00:00</startDate>
  <description><![CDATA[<div class="bbcode">The Jezabels begynte å spille sammen i 2007
  etter å ha møttes på Universitet i Sydney, Australia, og har etter det gitt ut tre
  ep-plater som har fått god mottagelse over alt i verden. Bandet har blant annet vært
  nominert til Best Breakthrough Artist og Best Single / EP på Independent Music Awards
  i 2010 og vært support til Tegan &amp; Sara. Det er litt vanskelig å beskrive popmusikken
  deres; det er som en blanding mellom indiepop og indierock, med litt disko. Bra er det
  i hvert fall! The Jezabels har ikke gitt ut noen skive i Norge ennå, men hvis du har
  sett Imsdals (forferdlige) reklamefilme, har du hørt låten ”Easy To Love”. Debutskiven
  kommer under 2011 og den 25. september kan du se bandet på John DEE!<br /><br />Billetter
  er i salg via Billettservice!</div>]]></description>
  <image size="small">http://userserve-ak.last.fm/serve/34/48219593.png</image>
  <image size="medium">http://userserve-ak.last.fm/serve/64/48219593.png</image>
  <image size="large">http://userserve-ak.last.fm/serve/126/48219593.png</image>
  <image size="extralarge">http://userserve-ak.last.fm/serve/252/48219593.png</image>
  <attendance>6</attendance>
  <reviews>0</reviews>
  <tag>lastfm:event=1996187</tag>
  <url>http://www.last.fm/event/1996187+The+Jezabels</url>
  <website>https://www.facebook.com/event.php?eid=172860196109875</website>
    <tickets>
  </tickets>
    <cancelled>0</cancelled>
    <tags>
      <tag>australian</tag>
      <tag>indie</tag>
      <tag>rock</tag>
      <tag>female vocalists</tag>
    </tags>
    </event></events></lfm>

Now we're starting to see what we're interested in!

But wait, what is that xmlns:geo attribute at the top there? Will that cause a problem? Yes it will. - That is an XML namespace, and if we want to be able to pull out information from those <geo:point/> tags we'll need to specify that we want to deal with this particular namespace. Luckily, dealing with XML namespaces is quite easy with XML::Rabbit. The first thing we need to do is to declare that we want to create queries that use this namespace. Add the following piece of code before the BUILD method in lib/WWW/LastFM/Response.pm to take care of that challenge:

add_xpath_namespace 'geo' => 'http://www.w3.org/2003/01/geo/wgs84_pos#';

Now we can add a new accessor for the events in WWW::LastFM::Response. Add the following piece of code at the bottom of lib/WWW/LastFM/Response.pm:

has_xpath_object 'events' => '/lfm/events' => 'WWW::LastFM::Response::EventList';

As this points to another class, we'll have to create that one too. The contents of lib/WWW/LastFM/Response/EventList.pm should be this:

package WWW::LastFM::Response::EventList;
use XML::Rabbit;

has_xpath_value 'page'       => './@page';
has_xpath_value 'page_limit' => './@perPage';
has_xpath_value 'page_count' => './@totalPages';

has_xpath_value 'location'       => './@location';
has_xpath_value 'festivals_only' => './@festivalsonly';

has_xpath_object_map 'map' => './event',
    './id'  => 'WWW::LastFM::Response::Event',
    handles => {
        'get' => 'get',
        'ids' => 'keys',
        'all' => 'values',
    },
;

finalize_class();

Everything here should be easy to understand, as we're only declaring simple values. The has_xpath_object_map declaration might require some clarification, though. We're declaring a Moose attribute called map which is a hash reference where the event id from the XML document is the key, and an object of class WWW::LastFM::Response::Event is the value. The native trait delegations for hashes are somewhat different than the ones for arrays. As this object is not designed for data modification, we've only added the typical read accessors. Whether you will use ids and get or all depends a bit on your application needs. We're only going to use all in our application.

In the next and final article we'll flesh out the implementation of the Event class. We will also create a small application that allows us to display events for a specific location in a compact format. See you soon!

In the previous article we looked at how XML::Rabbit does its magic to give us a very compact syntax for creating Moose attributes that mirror simple XML document values. In this article we'll look at some of the other sugar functions available in XML::Rabbit.

Implementing the <metros/> XML chunk extractor

So let's move on to the implementation of WWW::LastFM::Response::MetroList. Now we're starting to get to the juicy parts of the application. Use this code for lib/WWW/LastFM/Response/MetroList.pm:

package WWW::LastFM::Response::MetroList;
use XML::Rabbit;

use List::MoreUtils qw(uniq);

has_xpath_object_list '_locations' => './metro' => 'WWW::LastFM::Response::Metro',
    handles => {
        'locations'        => 'elements',
        'filter_locations' => 'grep',
    },
;

has_xpath_value_list '_countries_with_duplicates' => './metro/country';

has '_unique_countries' => (
    is         => 'ro',
    isa        => 'ArrayRef[Str]',
    traits     => ['Array'],
    handles    => {
        'countries'        => 'elements',
        'filter_countries' => 'grep',
    },
    lazy_build => 1,
);

sub _build__unique_countries {
    my ($self) = @_;
    return [
        uniq(
             @{ $self->_countries_with_duplicates }
        )
    ];
}

finalize_class();

Here you have another declaration method called has_xpath_object_list. It is similar to has_xpath_object, which we've already covered, but it returns an array of objects instead of just a single object. We also add some Moose native delegations for arrays. This makes the API more user friendly, as we don't need to dereference the array references all the time.

The next declaration is almost the same, but it works with strings instead of objects. As the XML document contains lots of duplicates, we create a separate attribute that takes care of getting rid of the duplicates and presents the API we're interested in.

Now we complete this with the implementation of WWW::LastFM::Response::Metro. Create the file lib/WWW/LastFM/Response/Metro.pm with the following content:

package WWW::LastFM::Response::Metro;
use XML::Rabbit;

has_xpath_value 'name'    => './name';
has_xpath_value 'country' => './country';

has 'country_and_name' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);

sub _build_country_and_name {
    my ($self) = @_;
    return $self->country . ": " . $self->name;
}

has 'name_and_country' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);

sub _build_name_and_country {
    my ($self) = @_;
    return $self->name . ", " . $self->country;
}

finalize_class();

This is a pretty straight-forward class. You have two string values from the <metro/> XML element extracted. Look at the use of relative XPath queries, which avoids having to construct elaborate queries based on the root of the XML document. As a convenience, I added two calculated attributes.

We can still test how it works with a one-liner, but it is starting to get a bit long.

$ perl -Ilib -MWWW::LastFM -E 'STDOUT->binmode(":utf8"); say join("\n", sort map { $_->name_and_country } WWW::LastFM->new->geo->get_metros->metros->filter_locations(sub { $_->country_and_name =~ /Norway/ }) )'
Bergen, Norway
Oslo, Norway

lastfm_locations.pl, a simple WWW::LastFM client

Let's create a script we can use to query locations available in the Last.FM service. Create bin/lastfm_locations.pl with this content:

#!/usr/bin/env perl

use strict;
use warnings;
use rlib;
use feature qw(say);

use WWW::LastFM;
use Encode qw(decode encode);

my $filter = shift;
my $filter_utf8 = $filter ? Encode::decode('UTF-8', $filter) : "";

my @locations = sort
        map { $_->name_and_country }
        WWW::LastFM->new->geo->get_metros->metros->filter_locations(
            sub { $_->country_and_name =~ /\Q$filter_utf8\E/i }
        )
;

if ( @locations > 0 ) {
    say Encode::encode('UTF-8', join("\n", @locations) );
}
else {
    say "No locations found matching '$filter'";
}

This is very similar to the one-liner, but with a bit more error checking and Unicode handling to ensure you can specify command line parameters in UTF8. I guess I've forgotten to mention this until now, but I have assumed that you are using a terminal software that uses UTF8 encoding. The only common operating system I know that doesn't do that by default nowadays is Windows. I'll leave dealing with that challenge for another article.

You should now have a complete application that allows you to ask a remote HTTP-based API for some information and display it in the way you want to. Adding more API calls is just a matter of adding additional method calls in WWW::LastFM::API::Geo and creating a new response class to handle the XML output. In the next article we will add the API call we're really interested in, geo.getEvents. Stay tuned!

In the previous article we created a simple framework for making HTTP requests to the Last.FM API. In this article we'll go into detail on how XML::Rabbit can help us to extract the information we want from the XML output we saw in the previous article. I've added that XML output here, for your convenience.

<?xml version="1.0" encoding="utf-8"?>
<lfm status="ok">
<metros>
    <metro>
        <name>Sydney</name>
                <country>Australia</country>
    </metro>
...snip...
    <metro>
        <name>Wichita</name>
                <country>United States</country>
    </metro>
</metros></lfm> 

Implementing the geo.getMetros API call

Add the following code at the top of lib/WWW/LastFM.pm to load the WWW::LastFM::API::Geo module (which we'll flesh out in a moment).

use WWW::LastFM::API::Geo;

Next we add another attribute so we can get easy access to the Geo class. Add at the bottom:

# API modules
has 'geo' => (
    is         => 'ro',
    isa        => 'WWW::LastFM::API::Geo',
    lazy_build => 1,
);

sub _build_geo {
    my ($self) = @_;
    return WWW::LastFM::API::Geo->new( lastfm => $self );
}

The next part is to create the basic skeleton that allows us to make API calls with convenience. Add the following code to lib/WWW/LastFM/API/Geo.pm:

package WWW::LastFM::API::Geo;
use Moose;
use namespace::autoclean;

use URI::Escape;

has 'lastfm' => (
    is       => 'ro',
    isa      => 'WWW::LastFM',
    required => 1,
);

# http://www.last.fm/api/show?service=435
# $country must be a binary encoded utf8 string

sub get_metros {
    my ($self, $country) = @_;
    return $self->lastfm->get(
           $self->lastfm->api_root_url
        . '?method=geo.getMetros'
        . ( $country ? '&country=' . uri_escape($country) : "" )
        . '&api_key=' . $self->lastfm->api_key
    );
}

__PACKAGE__->meta->make_immutable();

1;

We perform another one-liner to verify that we're on the right track.

$ perl -Ilib -MWWW::LastFM -E 'say WWW::LastFM->new->geo->get_metros'

What is interesting to see is that we've forwarded the instance of WWW::LastFM into the API class to avoid using global variables to hold our shared state. But let's see if we can't do something with that ugly XML.

Dealing with the root element in the response XML

Let's have a closer look at the XML returned.

<?xml version="1.0" encoding="utf-8"?>
<lfm status="ok">
<metros>
    <metro>
        <name>Sydney</name>
                <country>Australia</country>
    </metro>
... <snip> ...
    <metro>
        <name>Wichita</name>
                <country>United States</country>
    </metro>
</metros></lfm>

What we can see is that Last.FM always wraps its response in a <lfm/> root element, as described in API REST requests. It holds one attribute named status, and it's value is either ok or failed. So we're dealing with a boolean here. If the request failed it will contain an error code, like this:

<?xml version="1.0" encoding="utf-8"?>
<lfm status="failed">
    <error code="10">Invalid API Key</error>
</lfm>

So let's see if we can encapsulate this behaviour in a class with a nice API. Let's first change our get_metros method to return this response class instead of some boring XML. Edit lib/WWW/LastFM/API/Geo.pm and add this after the other imports:

use WWW::LastFM::Response;

And change the get_metros method into this:

sub get_metros {
    my ($self, $country) = @_;
    my $xml = $self->lastfm->get(
           $self->lastfm->api_root_url
        . '?method=geo.getMetros'
        . ( $country ? '&country=' . uri_escape($country) : "" )
        . '&api_key=' . $self->lastfm->api_key
    );
    return WWW::LastFM::Response->new( xml => $xml );
}

So what should this WWW::LastFM::Response class look like? What we know is that it should require some XML to work on, and if it doesn't get that it should blow up with an error message (we want it to throw an exception of some kind). We also want to know if the response failed or not, and if it failed, we should throw an exception as well. If everything is okay, we should be able to get to those metros. Create lib/WWW/LastFM/Response.pm with this content:

package WWW::LastFM::Response;
use XML::Rabbit::Root 0.1.0;

sub BUILD {
    my ($self) = @_;
    return if $self->is_success;
    confess("Last.FM response error " . $self->error_code . ": " . $self->error);
}

has 'is_success' => (
    is         => 'ro',
    isa        => 'Bool',
    lazy_build => 1,
);

sub _build_is_success {
    my ($self) = @_;
    return unless $self->status eq 'ok';
    return 1;
}

has_xpath_value 'status'     => '/lfm/@status';
has_xpath_value 'error'      => '/lfm/error';
has_xpath_value 'error_code' => '/lfm/error/@code';

has_xpath_object 'metros' => '/lfm/metros' => 'WWW::LastFM::Response::MetroList';

finalize_class();

You can test it immediately to see if it works as expected:

$ perl -Ilib -MWWW::LastFM -E 'say WWW::LastFM->new->geo->get_metros->status'

You should get a string that says ok (unless you get an error because the Last.FM API server refuses to answer). If you want to force an error condition, try it out with an invalid API key.

$ perl -Ilib -MWWW::LastFM -E 'say WWW::LastFM->new( api_key => "1234" )->geo->get_metros->status'
Last.FM response error 10: Invalid API key - You must be granted a valid key by last.fm at ...

I snipped the rest of the stack trace, as it is of no interest at this point. We know where we made a mistake.

How XML::Rabbit::Root simplifies dealing with the XML document data

Let's get into detail on how that class definition works.

Using XML::Rabbit::Root does a whole lot of things behind the scenes. It is the equivalent of doing this:

use Moose;
with "XML::Rabbit::RootNode";
use namespace::autoclean;
use XML::Rabbit::Sugar;

The finalize_class() at the bottom is the equivalent of __PACKAGE__->meta->make_immutable(); 1;, which ensures the class executes as fast as possible during runtime and that the file loaded returns a true value, as required by the perl parser. So with that out of the way, let's take a look at the meat of the class and what that imported sugar, has_xpath_value and has_xpath_object, represents.

The first thing to notice is that there is no xml attribute declared in the class. This parameter comes from XML::Rabbit::Role::Document. It is automatically available, and you must specify either file, fh, xml or dom, based on what kind of format your XML document is in. As we have it in a string, we use the xml parameter.

The BUILD method is a special method Moose will automatically call after it has constructed an instance. In our case, we use it to verify the state of our instance. If the attribute is_success is true we just return, as everything is okay. If it is not, we throw an exception with the error code and text from the XML. This is how we can easily make an unwanted value in an attribute cause the entire construction of the object to fail.

The is_success attribute is pretty straight-forward. We take a string value that represents the status and checks if it is negative (that is, NOT the value ok). If that is the case, we return false, otherwise things must be okay and we can return a true value. Notice that I use a guard clause to fail early. This is a best practice to avoid the dreaded arrow anti-pattern.

So what does the has_xpath_value 'status' => '//lfm/@status'; declaration actually mean? Let's have a look in XML::Rabbit::Sugar and see if we can't figure it out. It says:

has_xpath_value($attr_name, $xpath_query, @moose_params)

Extracts a single string according to the xpath query specified. The attribute isa parameter is automatically set to Str. The attribute native trait is automatically set to String.

Okay, so this code creates a Moose attribute with an isa parameter of Str that will actually represent the value of the status attribute on the lfm root element in the XML document. Take a look at an XPath tutorial if you're unfamiliar with the syntax. If you need some additional Moose attribute parameters, you can specify them at the end. In our case there is no need for any. The error and error_code attribute is just the same thing.

So what is this has_xpath_object thing? It represents exactly the same thing as has_xpath_value, but instead of being a string value, it is an object of the specified class. The metros attribute will represent the list of metros returned in the XML document. I was thinking about naming the attribute metro_list, to avoid nouns in plural, but decided to keep it in plural because it more closely matches the XML document layout. To avoid putting too much responsibility inside WWW::LastFM::Response, I've delegated dealing with this list of metros to another class. This follows object orientation best practices, which states that each class should have a clear and defined purpose.

In the next article we'll dive into the details on how this MetroList class is implemented.

In this series of articles I'm going to implement a client to the Last.FM web services API which allows us to find concerts and other events in your local area. We'll use a CPAN module I've created called XML::Rabbit to deal with all the mundane details of XML document handling.

Most of you probably don't like XML a lot. JSON is the new kid on the block, and dealing with all the crummy details of XML encoding, parsing and such is boring. Writing long incantations of XML::LibXML code to extract the required data is just something you would prefer not to do (unless you get paid for it, maybe not even then). I'm going to show you a way to deal with XML that requires a lot less boilerplate code than you're most likely used to. Hopefully it will make dealing with XML-based APIs a lot more fun for you. In the process of showing you how to deal with XML with ease I'll also implement a simple, but extensible, framework for communicating with the Last.FM API. You can use the same framework design to build client libraries against other HTTP-based APIs. My hope is that the code I show you will inspire you to work with me on this particular Last.FM API or create clients for other interesting APIs.

I have done my best to follow good Perl programming practices, as advocated by chromatic's Modern Perl book, Damian Conway's Perl Best Practices book, and the Moose Manual. I've also separated as much responsibility as possible into separate classes, attributes and methods. This should make it easier to create good tests that require a minimum amount of mocking to test both positive and negative failure scenarios.

Getting a Last.FM API key and secret

If you want to follow along, you'll need to get an API key from http://www.last.fm/api/account. Just fill out the required fields with something useful to your person and get your API key and secret. When you have that, stick it in a file in your home directory called .lastfm.ini.

The contents should be something like this:

[API]
key = 012345678948bb4c75ff9608aac4fe83
secret = abcdef57349f6ad7e9959a63aa472

That should ensure that the configurable information is tucked away in a personal file outside any code repository.

On Windows the correct directory will probably be C:\Users\<username>\AppData\Local. Run the following command to figure out the correct location:

$ perl -MFile::HomeDir -E "say File::HomeDir->my_data";

If you don't feel like registering, you can actually use the API key used in the examples on the Last.FM API page, currently b25b959554ed76058ac220b7b2e0a026. I'm not sure how long it will work, but as we're not going to touch any of the authenticated API calls you won't need the API secret quite yet.

Setting up your environment

If you want to follow along without typing in all the code, look at the WWW-LastFM github repository. The project uses Dist::Zilla, so installing all the dependencies should be as easy as running the following commands:

$ cpan Dist::Zilla
$ git clone git://github.com/robinsmidsrod/WWW-LastFM.git
$ cd WWW-LastFM
$ dzil authordeps | xargs cpan # or dzil authordeps | cpanm
$ dzil listdeps | xargs cpan   # or dzil listdeps | cpanm

The entry point class

Okay, now we're ready to dive in!

Let's create some basic code to read that config file and get access to our API key and secret. Create the file lib/WWW/LastFM.pm with this content:

package WWW::LastFM;
use Moose;
use namespace::autoclean;

use File::HomeDir;
use Path::Class::Dir;
use Config::Any;

# Sometimes we like some extra debugging output
has 'debug' => (
    is      => 'ro',
    isa     => 'Bool',
    default => 1,
);

# Standard stuff to read our config file
has 'config_file' => (
    is         => 'ro',
    isa        => 'Path::Class::File',
    lazy_build => 1,
);

sub _build_config_file {
    my ($self) = @_;
    my $home = File::HomeDir->my_data;
    my $conf_file = Path::Class::Dir->new($home)->file('.lastfm.ini');
    return $conf_file;
}

# This is where our config file data ends up
has 'config' => (
    is         => 'ro',
    isa        => 'HashRef',
    lazy_build => 1,
);

sub _build_config {
    my ($self) = @_;
    my $cfg = Config::Any->load_files({
        use_ext => 1,
        files   => [ $self->config_file ],
    });
    foreach my $config_entry ( @{ $cfg } ) {
        my ($filename, $config) = %{ $config_entry };
        warn("Loaded config from file: $filename\n") if $self->debug;
        return $config;
    }
    return {};
}

# And here we have our api key and secret
has 'api_key' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);

sub _build_api_key { return (shift)->config->{'API'}->{'key'}; }

has 'api_secret' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);

sub _build_api_secret { return (shift)->config->{'API'}->{'secret'}; }

__PACKAGE__->meta->make_immutable();

1;

At this point you should be able to do read your API key from your config file with this small one-liner.

$ perl -Ilib -MWWW::LastFM -E 'say WWW::LastFM->new->api_key'

This code is not really anything special. It's just a bunch of best-of-breed modules used to read an INI file in a dynamic directory based on your platform and putting its contents into a hash reference. Notice the use of lazy Moose attributes to transform your data from a filename, .lastfm.ini, to a specific value, api_key, one step at a time. Also notice that we do not need to use strict and warnings, as Moose takes care of that for us.

The next thing we need is a way to make requests, so let's use the tried and true LWP::UserAgent. Add this at the top of the file after the other imports:

use LWP::UserAgent;

our $VERSION = "0.0.1";

And then continue and add the api_root_url and ua attribute that contains our HTTP client.

# All access to the Last.FM API starts with this URL
has 'api_root_url' => (
    is      => 'ro',
    isa     => 'Str',
    default => 'http://ws.audioscrobbler.com/2.0/',
);

# And finally our HTTP client that we will use to make requests
has 'ua' => (
    is         => 'ro',
    isa        => 'LWP::UserAgent',
    lazy_build => 1,
);

sub _build_ua {
    my ($self) = @_;
    return LWP::UserAgent->new( agent => 'WWW::LastFM/' . $VERSION );
}

# A utility method for making requests, returns raw XML
# or throws exception if no content was generated
sub get {
    my ($self, $url) = @_;
    confess("No URL specified") unless $url;
    my $response = $self->ua->get($url);
    my $content = $response->content;
    confess("HTTP error: " . $response->status_line) unless defined $content;
    confess("HTTP error: " . $response->status_line) if $response->code >= 500;
    return $content;
}

We've also added a basic get method that fetches the data on the specified URL and returns the raw content (bytes). This should make it trivial to make API calls and get back XML we can work with.

The geo.getMetros API call

To test out that our HTTP client works, we can make another one-liner that fetches the locations (metros) the Last.FM service knows about.

$ perl -Ilib -MWWW::LastFM -E 'my $lfm = WWW::LastFM->new; say $lfm->get($lfm->api_root_url . "?method=geo.getMetros&api_key=" . $lfm->api_key)'

You should get some XML spewed out on your screen that looks something like this:

<?xml version="1.0" encoding="utf-8"?>
<lfm status="ok">
<metros>
    <metro>
        <name>Sydney</name>
                <country>Australia</country>
    </metro>
...snip...
    <metro>
        <name>Wichita</name>
                <country>United States</country>
    </metro>
</metros></lfm>

Let's see if we can't expose that information in a better way. If you have a closer look at Last.FM's API you'll notice that they divide their API calls into sub-sections. In the next part we'll make a separate class for the specific API calls we want to provide, in this case the geo sub-section.