Robin Smidsrød

I recently upgraded my main computer to a new shiny Intel Z68-based motherboard with a 2nd generation Core i7. A really nice piece of hardware, I'd say.

But while I was setting it up I had some very odd problem. When I plugged everything in and wanted to boot it, it just didn't want to boot any devices (but the EFI BIOS loaded without problems), it reset instead.

At first I thought I had a power problem, so I went out and bought a bigger PSU. But it still reset instead of booting. Then I called a friend and he thought the SATA controller might be busted. So he suggested unplugging everything and trying one part at the time.

After doing a lot of cross-testing I figured out that it was the USB cable to my APC Smart-UPS 1000 that stopped the boot from processing. So obviously either my UPS or motherboard firmware is incompatible with each other. Luckily I was not dependent on having the UPS USB cable connected, so I just disconnected it to avoid the problem. With that component disconnected everything worked as expected.

Moral of the story: Test with one component at the time if you're experiencing difficulties.

Yesterday I helped my father to install his new computer with Windows 7. For several reasons I shall not mention, it was in the best interest to take the hard drive of the old (broken) computer and copy all the data from it to the new. What he (and me) wanted was to keep all the settings and other stuff and just reinstall his programs. That meant copying the entire AppData folder under his user account in addition to his normal files.

During this process obviously all files had kept the old SIDs, and as they weren't matching his new SID (on the new computer) I was having problems accessing the files. Lots of stuff didn't work. I was getting "No access" messages left and right when I tried to change the permissions of those copied files. That is when it dawned on me that even though his user account was registered as an administrator, it wasn't able to change the permission bits on those files.

To avoid having to mess around forever to reset permissions to default inherited ACL on all those files (around 30000 or so of them) I dug a bit around on the Internet and found out that there is a command in Windows 7 called icacls.exe that works very much in the same way as chmod on Linux, just that it supports NTFS ACLs.

After reading around I found a forum post that mentioned it, and was redirected to Microsoft's knowledge base article on icacls.exe. After looking over that one I found this command line that solved everything.

cd C:\Users\username
icacls * /reset /T

The /T switch makes it run recursively. /reset, as the KB article explains, replaces the ACL on each file and folder with the default inherited ACL that just uses the ACL on the parent folder.

The last thing that was required was to run this command inside a cmd.exe shell with elevated privileges. You can do that by finding the Command Prompt icon in your start menu and right-clicking "Run as administrator".

Be aware that mucking around with ACLs, if you don't know what you're doing, can seriously break your system in more ways than you can think of. It can also open you up to various attacks because of more relaxed permissions.

If you like what you just read, please subscribe to the feed or follow me on Twitter or any of the other networks I'm registered in to be updated when I post something new.

After I wrote that big 5-part article on WWW::LastFM / XML::Rabbit I noticed several things that could be improved. One of them was the fact that loading configuration information from a file in the home directory was a pretty generic thing to do. I decided to factor that code out and release it as a separate CPAN module that can be used by any Moose class.

If you look at the code used to read the configuration information in the article mentioned above you'll see that it has improved quite a bit. The code below is shorter, but does the same thing as the old code.

package WWW::LastFM;
use Moose;
use namespace::autoclean;

# Read configuration from ~/.lastfm.ini,
# available in $self->config
has 'config_filename' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);
sub _build_config_filename { '.lastfm.ini' }
with 'Config::Role';

# And here we have our api key and secret
has 'api_key' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);
sub _build_api_key {
    return (shift)->config->{'API'}->{'key'};
}

has 'api_secret' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);
sub _build_api_secret {
    return (shift)->config->{'API'}->{'secret'};
}
...

The reason this approach is quite nice, is because it allows you to load e.g. username and password from a configuration file with the ability to override it in the constructor. You can also quite easily override the location to search for the configuration file in your tests by mocking the location returned by File::HomeDir->my_data(). Alternatively, you can specify either the config_file or config_files constructor parameters to explicitly specify the path to your configuration file (or an array reference of them).

I usually start my test case with code like this to override where to look for the configuration file for the test environment. Then I just put a copy of my configuration file inside the t folder with the same file name as usual.

# Mock home/config dir location
{
    use File::HomeDir;
    no warnings 'redefine';
    sub File::HomeDir::my_data { return 't'; }
}

The configuration file is loaded via Config::Any->load_files(), so you can use any format supported by that module. Your config attribute will always be represented as a hash reference.

If you find this article or the CPAN module Config::Role useful I'd be very happy if you share it with your friends on Facebook, Google+, Twitter or other service you use. If you like what you see, follow me on any of the networks at the top of the page of my blog and subscribe to my feed to keep informed on when I post something new.

Have you ever noticed that the sshd on your publicly facing machines gets bombarded with dictionary attacks several times per day? This problem is mostly an annoyance, as it fills up the logs with lots of User authentication failed, wrong password for <username> messages. There are of course several ways to work around this problem, and the most common one is to run sshd on another port than 22. I find that approach cumbersome, because it means you'll always have to configure your client software to connect to a non-standard port, and in lots of cases a firewall at your location might be blocking the traffic as well. Isn't there a way to block these bothersome users instead?

I recently read an article in Linux Journal #210 that talked about a new feature in the Linux kernel called ipset. It allows you to create sets that store IP addresses which can dynamically be added to and removed from. This sets can be configured as selectors in iptables rules, so that you can perform actions on any IP address in the set, like dropping the packets. You can also create iptables rules that add or remove an IP from a set.

In Debian it is straight-forward to install ipset. Type in these commands to install ipset and the required kernel module. It should ensure it is automatically recompiled whenever you do a kernel upgrade.

$ apt-get install ipset xtables-addons-source
$ module-assistant auto-install xtables-addons

I asked around on IRC if it was possible to make sshd execute a shell script once the amount of logins from an IP went above some configurable threshold, but apparently it is not possible. Someone pointed me at fail2ban which is a system that scans your syslog looking for failed logins and turn them into blocking iptables rules (and probably more, I didn't look very closely). I thought that this was too slow, as I wanted something that triggered as soon as multiple failed logins for a user reached a certain threshold, but I wanted the block to be temporary, not permanent.

That is when I remembered that syslog (rsyslog in my case) can be configured to run the log messages through a program. I put together the program below that reads incoming messages from rsyslog with auth.info facility/level (default for sshd on Debian) and does the necessary things to ensure the offending IP is added to my autoblock ipset if it triggers a certain amount of failed logins in a short period. The autoblock ipset uses the iptree storage module that has a --timeout parameter to automatically purge entries from the set after a given time. I set it to 3600 seconds (1 hour).

What is also quite cool, is that I have a rule that will immediately put the IP of of anyone that tries to connect to my SMTP port (I don't run a mail server on my firewall) on the autoblock list. This particular bit happens completely inside netfilter, so its effect is immediate. Bye bye spammers! Try to connect to my (non-existent) email server and you're instantly blocked for an hour. And you didn't even know what hit you. :)

The reason I like the temporary block is that sometimes I port-scan my own server or do other strange things with it from remote computers, and the fact that I know the block will be lifted after an hour means I can continue without having to get physical access to server to remove the block. I can just smack myself in the forehead and wait an hour and continue whatever stupid thing I was doing.

PS: Remember that lots of IRC servers like to port-scan your IP when you connect to them, so you might need to put up some exceptions for those if you're an active IRC user and your particular IRC server likes to probe the SMTP port.

• Part 1 of 5
• Part 2 of 5
• Part 3 of 5
• Part 4 of 5
• Part 5 of 5

In the previous article we started implementing the geo.getEvents Last.FM API call. In this article we will complete the implementation of the Event class and create a small application to display events for a specific location in a compact format.

Implementing the <event/> XML chunk extractor

So let's move on to the WWW::LastFM::Response::Event class. This is one of the core classes for our particular problem domain. Everything up to here has just been stuff needed to build an extensible framework for sending our queries to the Last.FM API and extract values from the response data. Let's get on with it. Create lib/WWW/LastFM/Response/Event.pm with this content:

package WWW::LastFM::Response::Event;
use XML::Rabbit;

use HTML::FormatText;
use DateTime::Format::HTTP;

# Last.FM-specific data
has_xpath_value 'id'  => './id';
has_xpath_value 'tag' => './tag';
has_xpath_value 'url' => './url';

# The actual event-related data
has_xpath_value 'title'            => './title';
has_xpath_value 'description_html' => './description';
has_xpath_value 'website'          => './website';
has_xpath_value 'start_date'       => './startDate';
has_xpath_value 'headliner'        => './artists/headliner';
has_xpath_value_list '_artists'    => './artists/artist',
    handles => {
        'artists'      => 'elements',
    }
;
has_xpath_object 'venue'     => './venue'    => 'WWW::LastFM::Response::Venue';
has_xpath_value_list '_tags' => './tags/tag',
    handles => {
        'tags' => 'elements',
    },
;

# Some various attendance-related data
has_xpath_value 'ticket_count' => './tickets';
has_xpath_value 'attendance'   => './attendance';
has_xpath_value 'cancelled'    => './cancelled';
has_xpath_value 'review_count' => './reviews';

# Event image URLs
has_xpath_value 'image_s'  => './image[@size="small"]';
has_xpath_value 'image_m'  => './image[@size="medium"]';
has_xpath_value 'image_l'  => './image[@size="large"]';
has_xpath_value 'image_xl' => './image[@size="extralarge"]';

# The description value, converted to plain text
has 'description' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);

sub _build_description {
    my ($self) = @_;
    return HTML::FormatText->format_string(
        $self->description_html,
        leftmargin  =>  0,
        rightmargin => 79,
    );
}

# Convert the start_date to a DateTime class which is much more useful
has 'date' => (
    is         => 'ro',
    isa        => 'DateTime',
    lazy_build => 1,
);

sub _build_date {
    my ($self) = @_;
    return DateTime::Format::HTTP->parse_datetime(
        $self->start_date
    );
}

finalize_class();

There shouldn't be anything surprising in this piece of code. The only thing you haven't seen before is the use of XPath attribute values as selectors. We've already covered all the various declarations, so let's just continue with WWW::LastFM::Response::Venue, so that we can have a look at how to deal with the namespaced latitude/longtitude information. Create lib/WWW/LastFM/Response/Venue.pm with this content:

package WWW::LastFM::Response::Venue;
use XML::Rabbit;

# Last.FM-specific data
has_xpath_value 'id'  => './id';
has_xpath_value 'url' => './url';

# The actual venue-related data
has_xpath_value 'name'         => './name';
has_xpath_value 'website'      => './website';
has_xpath_value 'phone_number' => './phonenumber';

# Location-related data
has_xpath_value 'city'        => './location/city';
has_xpath_value 'country'     => './location/country';
has_xpath_value 'street'      => './location/street';
has_xpath_value 'postal_code' => './location/postalcode';
has_xpath_value 'latitude'    => './location/geo:point/geo:lat';
has_xpath_value 'longitude'   => './location/geo:point/geo:long';

# Venue image URLs
has_xpath_value 'image_s'  => './image[@size="small"]';
has_xpath_value 'image_m'  => './image[@size="medium"]';
has_xpath_value 'image_l'  => './image[@size="large"]';
has_xpath_value 'image_xl' => './image[@size="extralarge"]';

# Convenience attribute for combined address
has 'address' => (
    is         => 'ro',
    isa        => 'Str',
    lazy_build => 1,
);

sub _build_address {
    my ($self) = @_;
    return $self->name . ', '
         . ( $self->street ? $self->street . ', ' : "" )
         . ( $self->postal_code ? $self->postal_code . ' ' : "" )
         . $self->city . ', '
         . $self->country;
}

finalize_class();

This is just more of the same things we've been doing. Notice the simplicity in dealing with namespaces in the XPath queries for latitude and longitude? It's really that simple.

lastfm_events.pl, a simple WWW::LastFM client

Let's finish it up by making a fairly simple command line app to search for events. Create bin/lastfm_events.pl with the following content:

#!/usr/bin/env perl

use strict;
use warnings;
use rlib;
use feature qw(say);

use WWW::LastFM;
use Getopt::Long;

# PODNAME: lastfm_eventss.pl
# ABSTRACT: Search the Last.FM API for events/concerts

STDOUT->binmode(":utf8");

# Read command line flags
my $limit;
my $distance;
my $help;
GetOptions(
    "limit=i"    => \$limit,
    "distance=i" => \$distance,
    "help"       => \$help,
);

die <<"EOM" if $help;
Usage: $0 [<options>] [<location>]
Options:
    --limit <num>   ; Number of events to return
    --distance <km> ; How large area to search
EOM

my $location = shift;

# Perform query
my $event_list = WWW::LastFM->new->geo->get_events(
    ( defined $limit    ? ( limit    => $limit )    : () ),
    ( defined $distance ? ( distance => $distance ) : () ),
    ( defined $location ? ( location => $location ) : () ),
)->events;
my @events = $event_list->all;

# Format and output results
if ( @events > 0 ) {
    foreach my $event ( sort { $a->date cmp $b->date } @events ) {
        my $formatted_date = DateTime::Format::HTTP->format_datetime($event->date);
        my $headliner = $event->title ne $event->headliner ? " with " . $event->headliner : "";
        my @additional_artists = grep { ! $event->headliner } $event->artists;
        say $formatted_date . ": "
          . $event->title
          . $headliner
          . ( scalar @additional_artists ? " and " . join(", ", @additional_artists ) : "" )
          . " @ " . $event->venue->address;
    }
}
else {
    say STDERR "No events found for location '" . $event_list->location . "'";
}

1;

Let's have some fun with it!

$ bin/lastfm_events.pl --distance 50 --limit 50 Tønsberg
Fri, 04 Nov 2011 19:30:00 GMT: Melissa Horn @ Kulturhuset Bølgen, Larvik, Norway
Fri, 18 Nov 2011 19:00:00 GMT: Konsert, Bjørn Eidsvåg med band  with Bjørn Eidsvåg @ Parkteatret, Moss, Norway
Fri, 16 Dec 2011 19:00:00 GMT: Helene Bøksle på Domkirkefestivalen i Tønsberg Domkirke with Helene Bøksle @ Tønsberg Domkirke, Tønsberg, Norway
Fri, 16 Dec 2011 19:31:01 GMT: Ljungblut @ Total, Stoltenbergsgata 46, 3110 Tønsberg, Norway
Sat, 17 Dec 2011 19:34:01 GMT: Ljungblut @ Total, Stoltenbergsgata 46, 3110 Tønsberg, Norway

This shows the events happening around where I live. Not really much to be cheerful about. Hopefully you'll get some more interesting results for where you live.

Conclusion

I wrote this article mostly to showcase how easy it is extract useful information from XML documents with XML::Rabbit and augment it with additional Moose magic. I don't really go to concerts very often, but I figured it was a useful example that some of you would find interesting (instead of just using some hand-crafted example XML).

If you want to see additional API calls implemented I urge you to get in touch with me and help me to improve it. I've shown you how I've done it this far. With some help from various Perl learning resources I'm sure you'd be able to whip up a patch with some additional features (or tests). Please follow me on Twitter or GitHub or subscribe to my blog if you want to stay informed with what I'm doing.