The other day I received this email from a user of one of my CPAN modules.
This is what makes participating in an Open Source community really worth it! The tool he's talking about is XML::Rabbit.
Thanks a lot, Jack!
Just a note to say thank you. I had been trying to build a system (without Moose) that could do something like this. I then decided that Moose would be better support for my classes. I ran into chromatic the other day and bought his book because it looked like it had a bit more on Moose. After playing some more with Moose, I did a search that took me to chromatic's page on Rabbit.
I cannot remember what the search was. However, I'm glad that I found it. My one-line summary for Rabbit is that it is a client side version of JQuery.
Perl keeps popping back into my life. Almost fifteen years ago, I was using Perl 4. I have not used it in over ten years. Last year, I decided that Perl would be good for translating a database, and I wanted classes. I was very happy to see Perl 5 was on target.
Now at a new company, I am faced with thousands of lines of Perl (mostly version 4) with sometimes 15K in one file!), I've been breaking the code into packages and classes. However, as I thought about it, I needed something more abstract. I need to solve some meta-programming issues to lead the company towards the future. Today, I find Rabbit -- can life get much better?
Once I've done some conversion of our XML API and can process it with Rabbit, I hope to add some ideas. If you have some thoughts on what you think would be useful to add, I would not mind thinking about them as I solve the bigger problem that I am working on.
Recently, I was participating in the Moving to Moose hackathon at Preikestolen near Stavanger, Norway together with a lot of other Perl hackers. I was one of the few people that actually drove to the venue, and it took quite some time to get there. During the trip I stopped a couple of times to enjoy the wonderful Norwegian nature.
After about 8 hours of driving I arrived at the lodge (last one to arrive). People were quite eager for me to arrive, because I was the one transporting the T-shirts for the event. Shortly after I arrived the T-shirts were taken out of my car and they were handed out to everyone. The T-shirt design was quite stylish, and the customized QR code on the back had an interesting twist. There were 16 different QR codes available, so scanning each one of them gave a funny different tidbit about the Perl / Moose community. The one I liked the the most was Perl ninjas know more than one way to do it.
Right after the T-shirts were handed out it was time for the opening dinner at Preikestolen Mountain Lodge. I was still exhausted after the driving, but I enjoyed the start of the event quite a lot. The picture below shows the location we would use to eat our meals for the next 5 days.
As this was my second "big" Perl event, I knew some of the people attending already, which was nice. But there were several people I had communicated quite a lot with over the years using IRC, and it was nice to finally be able to meet them in person.
The rest of the evening was used to get to know people, unpack stuff and trying to relax a bit before the big hike on Sunday.
The big hike to the top
Waking up on Sunday was nice, and the rain was blissfully absent, as you can see from the photo below:
After a nice breakfast we got started on the hike about 9:45. As the writer of the info column on the hackathon news site about the hike, I walked up front. I quickly understood that I was in terrible shape. At the first peak I was already exhausted, which didn't look good. But I continued and at the second stop I made a short video to document the trip.
After some more time we got to the worst part of the hike, the really steep climb that lasts for a few hundred meters. In the picture below you can actually see quite a lot of the participants of the hackathon.
From what I can tell, we can see Stevan, Greg, Patrick, Jonathan, Toby, Shawn, Karl Rune, Eystein and Ruben. So we actually have proof that we did it. Pay close attention to the text on Stevan's T-shirt: The Internet was down, so I thought I'd go outside today
Eventually we reached the top. After 3 hours me, Arne Sommer and Chris Prather were the last ones to actually get to the top, and there was quite a nice view that awaited us.
I was completely soaked, so it was really nice to have a dry T-shirt to change into once I got to the top. A little while after we arrived it was time for a photo-shoot of the entire group at the edge of the 600 meter cliff.
While I was busy relaxing and eating my lunch the rest of the group was busy paying attention to the helicopter guy filming us from different angles. By the time the rest of the group was ready to get back down I still handn't quite recovered, so me and Arne decided to stay a bit longer before we walked down again. I got a few decent photos from the plateau.
After a long and ardeous walk back down we finally reached the initial peak were you can see Preikestolhytta, the hotel, in all its glory. A truly welcome sight for sore legs. The trip back down took about 3 hours.
When we got down to the cabin we were welcomed by the rest of the people sitting around doing different things. It was quite obvious that most people were resting after a long trip. I decided to do the same thing until it was time for dinner.
The rest of the evening was spent chatting with people about the hike and other social topics. I was able to briefly get started on a skeleton for my Unicode::CLDR module together with Arne, but it was obvious that my brain wasn't functioning properly after such a long day of substantial physical activity. I decided to hit the sack early after an eventful day.
Actually getting to work
After breakfast on Monday Shawn M. Moore (sartak) decided to present his YAPC talk about Moose role patterns. It turned into more of a discussion, and most of the time between breakfast and lunch was spent on that topic. After lunch Patrick Michaud gave an adhoc talk on Perl 6 NQP, which was quite interesting for someone who has little or no experience with how the Perl 6 virtual machine works in general. It's nice to see so much work is being done on making it possible to run Perl 6 on other VMs like JVM and CLR.
While the other people were busy hacking on RDF, p5-mop and Perl 6 NQP I started working on my Unicode::CLDR module more in depth. Unicode::CLDR is a module that aims to present all the information in the very large Unicode Common Locale Data Repository in a simple and understandable Perl API. The CLDR contains information about languages, regions, scripts, even regular expressions for validating zip codes in different countries of the world. With this information at hand, a lot of modules in the Locale:: namespace on CPAN could be vastly simplified. The current method some of these modules use to gather data is to scrape Wikipedia or other sources, so a data set maintained by approximately 160 active contributors would be a much better source.
I asked around for good ideas on how the API should look, and got some input from different people. After some time it became apparent that the CLDR dataset is structured in a tricky way. The separate XML files actually inherit from each other, something that makes it impossible to load just one super-large DOM into memory and start extracting data from it. The English files are named en.xml, en_US.xml and en_US_POSIX.xml. The en_US_POSIX file is basically just a stub file that contains pointers to the other files, and most of the useful information is located in en.xml. One of the variations from base English are located in en_US.xml.
If I wanted to know the name of the language nb (Norwegian) in English I would do something like this:
my $cldr = Unicode::CLDR->new();
my $en_us = $cldr->get_locale('en_US');
my $lang_nb = $en_us->get_language('nb');
say $lang_nb->name;
The problem is that this would give a runtime error because the language nb is not defined in the file en_US.xml, it is actually defined in en.xml (because en_US doesn't have a specific variant of that language name). Because of that $lang_nb would be undefined and calling the name method would fail. Doing something like this would actually work:
my $cldr = Unicode::CLDR->new();
my $en_us = $cldr->get_locale('en');
my $lang_nb = $en_us->get_language('nb');
say $lang_nb->name;
The challenge now is to figure out how to actually make any API request, like $locale->get_language($lang_id), fall back to the parent XML file if it doesn't find a valid response in the current file. I asked several smart people (Stevan Little, Chris Prather and Shawn M. Moore) on how they would solve this challenge, and none of them had any good solutions on how this could be done without making the API very ugly or cumbersome.
If any of you who read this article have a good idea on how you could make this API work with this cumbersome data set I'd be very happy.
My first idea was to try and merge the XML files, but that causes a problem on how to deal with duplicates and overrides between the different files. Stevan suggested trying to deal with it just like OO inheritance, but that is kind of tricky given how XML::Rabbit deals with the XML DOM in itself. I suggested using a wrapper function that would automatically call the supplied code reference and fall back up the XML inheritance chain to get all the required answers. This would still require dealing with duplicates and overlapping data. It is also not a very pretty API. If I could hide this in a role of some kind I think it could be a lot better.
For the rest of the hackathon I decided to implement more of the API without worrying about this problem, and it is already able to give you information about language names, territory names (countries) and script names (writing systems). So please do take it for a spin and see how it works for you. Until I've figured out how to deal with the inheritance problem I believe it is not
ready for CPAN.
Please help me figure it out!
Socializing
The rest of Monday evening I spoke with Stevan about crazy code requirements, how it is to work full time from home and a lot of other things. I also had a nice chat with Shawn M. Moore where he explained how the Japanese input method he uses works, and it was quite intriguing. Japanese sure is an interesting language.
On Tuesday evening I sat with Ruben Verborgh and Yuval Kogman up in the common room where they were alternating playing some blues on an acuostic guitar Yuval brought with him. It was a really nice experience, and a nice break from all the Perl hacking.
On Wednesday I decided to go for a walk in the area around the cabin we stayed and take some pictures. During my walk I met Mirjam and Reini Urban, and I took a nice picture of them overlooking the cabin.
The area sure was nice. Below you can see some of the locations in the area.
When I got back to the cabin I decided to ask some of the people hanging around in the common room what they've been up to during the hackathon, and these are their responses:
As Eystein mentioned in the video, one of the really fun activities during the evenings was to play the card game FLUXX. It is a crazy card game designed by a programmer, so it must be perfect for geeks like ourselves. I played for several hours with Chris Prather, Shawn M. Moore, Salve J. Nilsen, Augustina Blair, Florian Ragwitz and Eystein Grusd. We had a blast, and the last game lasted almost 2 WHOLE HOURS! In the end we were just waiting for it to end, as we all had to get up early to get back home in the morning.
The trip back home
On Thursday I got up around 7:30, so I had time to say goodbye to the people that were leaving on the 8 bus. After a nice breakfast and some packing of stuff I was on my way back home around 9:45. Just before I was about to leave we had some furry friends visit the cabin.
The first stop on the way back was the ferry between Oanes and Lauvik, about 15 minutes from the hotel. If you look very carefully in the picture below, you can see Preikestolen, the cliff, at the edge of the mountain between the two pylons of the bridge.
The trip back was mostly uneventful, a lot of twists and turns that force you to keep your eyes on the road instead of the beautiful landscape, as pictured here.
After 8 long hours of driving I'm finally back home and I can collapse in the bed. I had a wonderful time, met several people I've wanted to meet for quite some time, and got some useful work done. All in all it was a great event.
I'm definitely looking forward to the next event. Tusen takk, alle sammen!
I've just published a new project on GitHub, named znc-log-viewer, that might be useful for IRC junkies.
If you use the ZNC IRC bouncer to keep you logged on to IRC all the time, you might've enabled the log module to avoid missing messages when you're not connected to the bouncer.
Searching those log files have always been a hassle, because they are located deep inside the .znc directory, and there are a lot of files stored in one big directory, which makes it hard to get an overview when you want to pull out your trusty friend, grep.
If you actually bothered to use grep, you would also notice that a single log file has a lot of server messages in it, that is generally not interesting when you're trying to figure out what has happened since the last time you were online. In my case, I just want access to the conversations that have been going on.
So if this problem is something you can relate to, feel free try out my new tool. If you find any problems with it, please open up an issue on the GitHub page.
After I wrote that big 5-part article on WWW::LastFM / XML::Rabbit I noticed several things that could be improved. One of them was the fact that loading configuration information from a file in the home directory was a pretty generic thing to do. I decided to factor that code out and release it as a separate CPAN module that can be used by any Moose class.
If you look at the code used to read the configuration information in the article mentioned above you'll see that it has improved quite a bit. The code below is shorter, but does the same thing as the old code.
package WWW::LastFM;
use Moose;
use namespace::autoclean;
# Read configuration from ~/.lastfm.ini,
# available in $self->config
has 'config_filename' => (
is => 'ro',
isa => 'Str',
lazy_build => 1,
);
sub _build_config_filename { '.lastfm.ini' }
with 'Config::Role';
# And here we have our api key and secret
has 'api_key' => (
is => 'ro',
isa => 'Str',
lazy_build => 1,
);
sub _build_api_key {
return (shift)->config->{'API'}->{'key'};
}
has 'api_secret' => (
is => 'ro',
isa => 'Str',
lazy_build => 1,
);
sub _build_api_secret {
return (shift)->config->{'API'}->{'secret'};
}
...
The reason this approach is quite nice, is because it allows you to load e.g. username and password from a configuration file with the ability to override it in the constructor. You can also quite easily override the location to search for the configuration file in your tests by mocking the location returned by File::HomeDir->my_data(). Alternatively, you can specify either the config_file or config_files constructor parameters to explicitly specify the path to your configuration file (or an array reference of them).
I usually start my test case with code like this to override where to look for the configuration file for the test environment. Then I just put a copy of my configuration file inside the t folder with the same file name as usual.
# Mock home/config dir location
{
use File::HomeDir;
no warnings 'redefine';
sub File::HomeDir::my_data { return 't'; }
}
The configuration file is loaded via Config::Any->load_files(), so you can use any format supported by that module. Your config attribute will always be represented as a hash reference.
If you find this article or the CPAN module Config::Role useful I'd be very happy if you share it with your friends on Facebook, Google+, Twitter or other service you use. If you like what you see, follow me on any of the networks at the top of the page of my blog and subscribe to my feed to keep informed on when I post something new.
Have you ever noticed that the sshd on your publicly facing machines gets bombarded with dictionary attacks several times per day? This problem is mostly an annoyance, as it fills up the logs with lots of User authentication failed, wrong password for <username> messages. There are of course several ways to work around this problem, and the most common one is to run sshd on another port than 22. I find that approach cumbersome, because it means you'll always have to configure your client software to connect to a non-standard port, and in lots of cases a firewall at your location might be blocking the traffic as well. Isn't there a way to block these bothersome users instead?
I recently read an article in Linux Journal #210 that talked about a new feature in the Linux kernel called ipset. It allows you to create sets that store IP addresses which can dynamically be added to and removed from. This sets can be configured as selectors in iptables rules, so that you can perform actions on any IP address in the set, like dropping the packets. You can also create iptables rules that add or remove an IP from a set.
In Debian it is straight-forward to install ipset. Type in these commands to install ipset and the required kernel module. It should ensure it is automatically recompiled whenever you do a kernel upgrade.
I asked around on IRC if it was possible to make sshd execute a shell script once the amount of logins from an IP went above some configurable threshold, but apparently it is not possible. Someone pointed me at fail2ban which is a system that scans your syslog looking for failed logins and turn them into blocking iptables rules (and probably more, I didn't look very closely). I thought that this was too slow, as I wanted something that triggered as soon as multiple failed logins for a user reached a certain threshold, but I wanted the block to be temporary, not permanent.
That is when I remembered that syslog (rsyslog in my case) can be configured to run the log messages through a program. I put together the program below that reads incoming messages from rsyslog with auth.info facility/level (default for sshd on Debian) and does the necessary things to ensure the offending IP is added to my autoblock ipset if it triggers a certain amount of failed logins in a short period. The autoblock ipset uses the iptree storage module that has a --timeout parameter to automatically purge entries from the set after a given time. I set it to 3600 seconds (1 hour).
What is also quite cool, is that I have a rule that will immediately put the IP of anyone that tries to connect to my SMTP port (I don't run a mail server on my firewall) on the autoblock list. This particular bit happens completely inside netfilter, so its effect is immediate. Bye bye spammers! Try to connect to my (non-existent) email server and you're instantly blocked for an hour. And you didn't even know what hit you. :)
The reason I like the temporary block is that sometimes I port-scan my own server or do other strange things with it from remote computers, and the fact that I know the block will be lifted after an hour means I can continue without having to get physical access to server to remove the block. I can just smack myself in the forehead and wait an hour and continue whatever stupid thing I was doing.
PS: Remember that lots of IRC servers like to port-scan your IP when you connect to them, so you might need to put up some exceptions for those if you're an active IRC user and your particular IRC server likes to probe the SMTP port.
