The other day I received this email from a user of one of my CPAN modules.
This is what makes participating in an Open Source community really worth it! The tool he's talking about is XML::Rabbit.
Thanks a lot, Jack!
Just a note to say thank you. I had been trying to build a system (without Moose) that could do something like this. I then decided that Moose would be better support for my classes. I ran into chromatic the other day and bought his book because it looked like it had a bit more on Moose. After playing some more with Moose, I did a search that took me to chromatic's page on Rabbit.
I cannot remember what the search was. However, I'm glad that I found it. My one-line summary for Rabbit is that it is a client side version of JQuery.
Perl keeps popping back into my life. Almost fifteen years ago, I was using Perl 4. I have not used it in over ten years. Last year, I decided that Perl would be good for translating a database, and I wanted classes. I was very happy to see Perl 5 was on target.
Now at a new company, I am faced with thousands of lines of Perl (mostly version 4) with sometimes 15K in one file!), I've been breaking the code into packages and classes. However, as I thought about it, I needed something more abstract. I need to solve some meta-programming issues to lead the company towards the future. Today, I find Rabbit -- can life get much better?
Once I've done some conversion of our XML API and can process it with Rabbit, I hope to add some ideas. If you have some thoughts on what you think would be useful to add, I would not mind thinking about them as I solve the bigger problem that I am working on.
During the summer season of 2012 (northern hemisphere) I've had the pleasure of being the coordinator of the iPXE network boot loader's participation in the Google Summer of Code program. This is a summary of what happened during the summer.
What is iPXE?
iPXE is a network boot loader. What does that actually mean? Normally when you boot your computer, control is handed over to the boot sector on a local hard drive in your computer and an operating system is loaded. But that is not the only option you have. For a lot of years a standard called PXE (developed by Intel), has been available which allows x86-based computers to boot from the network (using DHCP and TFTP). iPXE's main goal is to extend that ability to not only boot from TFTP (which is quite slow), but allow computers to boot from HTTP, FTP, iSCSI, AoE and other more esoteric sources. We also have a goal of being the main network boot loader for EFI-based systems (almost working, get in touch for details). We already support more than 60 different network cards, and cards without a native driver are also supported using a feature called chainloading.
First time participating in GSoC
This is the first year iPXE participates in the GSoC program as an organization. Most of the mentors involved have participated earlier years as part of other organizations, such as kernel.org and Etherboot. It was the first time for me, though, and it was quite fun! My job was to keep all the students and mentors on track and ensure no deadlines were forgotten and that communication between everyone flowed smoothly. In total we had three students, which I understand is quite a lot for a first year organization. Thank you to Google for believing in us.
Students and projects
Marin Hannache from EPITA University in France wrote protocol support for downloading boot files using NFS. His work is awaiting review and integration into mainline. From what I've tested it still has some issues, but it is working, and I'm most eager to get it in, as it will allow booting directly from an NFS3 server, not even needing a web server. This should be great news for people using small NAS boxes to do their booting, as they don't always have a way to serve files using HTTP.
Adrian Jamróz from Jan Kochanowski University in Poland reimplemented the drivers for the VIA Rhine and VIA Velocity family of network cards. The existing drivers were using a deprecated API and was much bigger in size than they really needed to be. Code size is always a big challenge for a project like iPXE, which commonly is burned into ROM chips that have a limit of 64KB. His code was very good and is just awaiting final review and integration. I've personally tested it on one of my Rhine2-based cards, and it seems to work reliably.
Daniel Wyatt from Lord Fairfax Community College in the United States
reimplemented the driver for the Broadcom NetExtreme II family of network cards. The problem with the old bnx2 driver was the same as the VIA driver. It used a very large proprietary firmware blob to run the MIPS-based microcontrollers on the card. Most of the summer was used to understand how we could minimize this firmware without losing any functionality. In the end the student got something working, but we're still having issues with the legality of including the firmware inside the iPXE project. The integration with mainline is currently on the fence awaiting some kind of confirmation from Broadcom that we can actually distribute the firmware blob with iPXE. Luckily we have people from Broadcom participating in the community, so it shouldn't be too impossible to get this kind of approval. I guess this student got a crash course in how licensing affects what you can do within an open-source project.
Other tidbits
All of the pages linked to above were maintained by the students as they were working on their code during the summer. You can also find links to specific Git repositories that include all of their code if you're interested in testing it out before it is integrated into mainline. Most of the day-to-day collaboration was done using IRC on the #ipxe channel on irc.freenode.net. Below you can read some of the interesting things said during the summer by the students:
20120608-010814 <Mareo> wow, I'm in the black magic part of iPXE codebase :D
20120703-121824 <Mareo> this gsoc is awesome :D
20120712-203713 <dewyatt> the bnx2 datasheet is horrible :(
20120720-233044 <dewyatt> mcb30: okay...I'm eager to get started on the fun stuff
20120728-160956 <budrys> meteger: but I think it can't be left as is, it's a dirty hack for me
20120815-172646 <dewyatt> mcb30: mouse sent me some ROMs that have firmware in them. This made me take another look at the newest PXE ROM I have and I found compressed firmware. I hackishly extracted this by pausing the broadcom PXE ROM during DHCP and reading from the Debug UART. I can't do this for RV2P and I don't really know the bounds of the MIPS firmware so it's still a work in progress.
20120816-083313 <dewyatt> I don't know if I can legally commit this code really :/
Thanks to everyone
It is great to have taken part in this journey and seen how much the students have learned over the summer. I want to thank our three mentors, Michael Brown, Joshua Oreman and Thomas Miletich and all the other knowledgable people on the #ipxe IRC channel for the great work they've provided during the summer. Participating in GSoC has been a great experience for us all, and I hope we'll be able to participate next year as well.
Recently, I was participating in the Moving to Moose hackathon at Preikestolen near Stavanger, Norway together with a lot of other Perl hackers. I was one of the few people that actually drove to the venue, and it took quite some time to get there. During the trip I stopped a couple of times to enjoy the wonderful Norwegian nature.
After about 8 hours of driving I arrived at the lodge (last one to arrive). People were quite eager for me to arrive, because I was the one transporting the T-shirts for the event. Shortly after I arrived the T-shirts were taken out of my car and they were handed out to everyone. The T-shirt design was quite stylish, and the customized QR code on the back had an interesting twist. There were 16 different QR codes available, so scanning each one of them gave a funny different tidbit about the Perl / Moose community. The one I liked the the most was Perl ninjas know more than one way to do it.
Right after the T-shirts were handed out it was time for the opening dinner at Preikestolen Mountain Lodge. I was still exhausted after the driving, but I enjoyed the start of the event quite a lot. The picture below shows the location we would use to eat our meals for the next 5 days.
As this was my second "big" Perl event, I knew some of the people attending already, which was nice. But there were several people I had communicated quite a lot with over the years using IRC, and it was nice to finally be able to meet them in person.
The rest of the evening was used to get to know people, unpack stuff and trying to relax a bit before the big hike on Sunday.
The big hike to the top
Waking up on Sunday was nice, and the rain was blissfully absent, as you can see from the photo below:
After a nice breakfast we got started on the hike about 9:45. As the writer of the info column on the hackathon news site about the hike, I walked up front. I quickly understood that I was in terrible shape. At the first peak I was already exhausted, which didn't look good. But I continued and at the second stop I made a short video to document the trip.
After some more time we got to the worst part of the hike, the really steep climb that lasts for a few hundred meters. In the picture below you can actually see quite a lot of the participants of the hackathon.
From what I can tell, we can see Stevan, Greg, Patrick, Jonathan, Toby, Shawn, Karl Rune, Eystein and Ruben. So we actually have proof that we did it. Pay close attention to the text on Stevan's T-shirt: The Internet was down, so I thought I'd go outside today
Eventually we reached the top. After 3 hours me, Arne Sommer and Chris Prather were the last ones to actually get to the top, and there was quite a nice view that awaited us.
I was completely soaked, so it was really nice to have a dry T-shirt to change into once I got to the top. A little while after we arrived it was time for a photo-shoot of the entire group at the edge of the 600 meter cliff.
While I was busy relaxing and eating my lunch the rest of the group was busy paying attention to the helicopter guy filming us from different angles. By the time the rest of the group was ready to get back down I still handn't quite recovered, so me and Arne decided to stay a bit longer before we walked down again. I got a few decent photos from the plateau.
After a long and ardeous walk back down we finally reached the initial peak were you can see Preikestolhytta, the hotel, in all its glory. A truly welcome sight for sore legs. The trip back down took about 3 hours.
When we got down to the cabin we were welcomed by the rest of the people sitting around doing different things. It was quite obvious that most people were resting after a long trip. I decided to do the same thing until it was time for dinner.
The rest of the evening was spent chatting with people about the hike and other social topics. I was able to briefly get started on a skeleton for my Unicode::CLDR module together with Arne, but it was obvious that my brain wasn't functioning properly after such a long day of substantial physical activity. I decided to hit the sack early after an eventful day.
Actually getting to work
After breakfast on Monday Shawn M. Moore (sartak) decided to present his YAPC talk about Moose role patterns. It turned into more of a discussion, and most of the time between breakfast and lunch was spent on that topic. After lunch Patrick Michaud gave an adhoc talk on Perl 6 NQP, which was quite interesting for someone who has little or no experience with how the Perl 6 virtual machine works in general. It's nice to see so much work is being done on making it possible to run Perl 6 on other VMs like JVM and CLR.
While the other people were busy hacking on RDF, p5-mop and Perl 6 NQP I started working on my Unicode::CLDR module more in depth. Unicode::CLDR is a module that aims to present all the information in the very large Unicode Common Locale Data Repository in a simple and understandable Perl API. The CLDR contains information about languages, regions, scripts, even regular expressions for validating zip codes in different countries of the world. With this information at hand, a lot of modules in the Locale:: namespace on CPAN could be vastly simplified. The current method some of these modules use to gather data is to scrape Wikipedia or other sources, so a data set maintained by approximately 160 active contributors would be a much better source.
I asked around for good ideas on how the API should look, and got some input from different people. After some time it became apparent that the CLDR dataset is structured in a tricky way. The separate XML files actually inherit from each other, something that makes it impossible to load just one super-large DOM into memory and start extracting data from it. The English files are named en.xml, en_US.xml and en_US_POSIX.xml. The en_US_POSIX file is basically just a stub file that contains pointers to the other files, and most of the useful information is located in en.xml. One of the variations from base English are located in en_US.xml.
If I wanted to know the name of the language nb (Norwegian) in English I would do something like this:
my $cldr = Unicode::CLDR->new();
my $en_us = $cldr->get_locale('en_US');
my $lang_nb = $en_us->get_language('nb');
say $lang_nb->name;
The problem is that this would give a runtime error because the language nb is not defined in the file en_US.xml, it is actually defined in en.xml (because en_US doesn't have a specific variant of that language name). Because of that $lang_nb would be undefined and calling the name method would fail. Doing something like this would actually work:
my $cldr = Unicode::CLDR->new();
my $en_us = $cldr->get_locale('en');
my $lang_nb = $en_us->get_language('nb');
say $lang_nb->name;
The challenge now is to figure out how to actually make any API request, like $locale->get_language($lang_id), fall back to the parent XML file if it doesn't find a valid response in the current file. I asked several smart people (Stevan Little, Chris Prather and Shawn M. Moore) on how they would solve this challenge, and none of them had any good solutions on how this could be done without making the API very ugly or cumbersome.
If any of you who read this article have a good idea on how you could make this API work with this cumbersome data set I'd be very happy.
My first idea was to try and merge the XML files, but that causes a problem on how to deal with duplicates and overrides between the different files. Stevan suggested trying to deal with it just like OO inheritance, but that is kind of tricky given how XML::Rabbit deals with the XML DOM in itself. I suggested using a wrapper function that would automatically call the supplied code reference and fall back up the XML inheritance chain to get all the required answers. This would still require dealing with duplicates and overlapping data. It is also not a very pretty API. If I could hide this in a role of some kind I think it could be a lot better.
For the rest of the hackathon I decided to implement more of the API without worrying about this problem, and it is already able to give you information about language names, territory names (countries) and script names (writing systems). So please do take it for a spin and see how it works for you. Until I've figured out how to deal with the inheritance problem I believe it is not
ready for CPAN.
Please help me figure it out!
Socializing
The rest of Monday evening I spoke with Stevan about crazy code requirements, how it is to work full time from home and a lot of other things. I also had a nice chat with Shawn M. Moore where he explained how the Japanese input method he uses works, and it was quite intriguing. Japanese sure is an interesting language.
On Tuesday evening I sat with Ruben Verborgh and Yuval Kogman up in the common room where they were alternating playing some blues on an acuostic guitar Yuval brought with him. It was a really nice experience, and a nice break from all the Perl hacking.
On Wednesday I decided to go for a walk in the area around the cabin we stayed and take some pictures. During my walk I met Mirjam and Reini Urban, and I took a nice picture of them overlooking the cabin.
The area sure was nice. Below you can see some of the locations in the area.
When I got back to the cabin I decided to ask some of the people hanging around in the common room what they've been up to during the hackathon, and these are their responses:
As Eystein mentioned in the video, one of the really fun activities during the evenings was to play the card game FLUXX. It is a crazy card game designed by a programmer, so it must be perfect for geeks like ourselves. I played for several hours with Chris Prather, Shawn M. Moore, Salve J. Nilsen, Augustina Blair, Florian Ragwitz and Eystein Grusd. We had a blast, and the last game lasted almost 2 WHOLE HOURS! In the end we were just waiting for it to end, as we all had to get up early to get back home in the morning.
The trip back home
On Thursday I got up around 7:30, so I had time to say goodbye to the people that were leaving on the 8 bus. After a nice breakfast and some packing of stuff I was on my way back home around 9:45. Just before I was about to leave we had some furry friends visit the cabin.
The first stop on the way back was the ferry between Oanes and Lauvik, about 15 minutes from the hotel. If you look very carefully in the picture below, you can see Preikestolen, the cliff, at the edge of the mountain between the two pylons of the bridge.
The trip back was mostly uneventful, a lot of twists and turns that force you to keep your eyes on the road instead of the beautiful landscape, as pictured here.
After 8 long hours of driving I'm finally back home and I can collapse in the bed. I had a wonderful time, met several people I've wanted to meet for quite some time, and got some useful work done. All in all it was a great event.
I'm definitely looking forward to the next event. Tusen takk, alle sammen!
I've just published a new project on GitHub, named znc-log-viewer, that might be useful for IRC junkies.
If you use the ZNC IRC bouncer to keep you logged on to IRC all the time, you might've enabled the log module to avoid missing messages when you're not connected to the bouncer.
Searching those log files have always been a hassle, because they are located deep inside the .znc directory, and there are a lot of files stored in one big directory, which makes it hard to get an overview when you want to pull out your trusty friend, grep.
If you actually bothered to use grep, you would also notice that a single log file has a lot of server messages in it, that is generally not interesting when you're trying to figure out what has happened since the last time you were online. In my case, I just want access to the conversations that have been going on.
So if this problem is something you can relate to, feel free try out my new tool. If you find any problems with it, please open up an issue on the GitHub page.
I recently upgraded my main computer to a new shiny Intel Z68-based motherboard with a 2nd generation Core i7. A really nice piece of hardware, I'd say.
But while I was setting it up I had some very odd problem. When I plugged everything in and wanted to boot it, it just didn't want to boot any devices (but the EFI BIOS loaded without problems), it reset instead.
At first I thought I had a power problem, so I went out and bought a bigger PSU. But it still reset instead of booting. Then I called a friend and he thought the SATA controller might be busted. So he suggested unplugging everything and trying one part at the time.
After doing a lot of cross-testing I figured out that it was the USB cable to my APC Smart-UPS 1000 that stopped the boot from processing. So obviously either my UPS or motherboard firmware is incompatible with each other. Luckily I was not dependent on having the UPS USB cable connected, so I just disconnected it to avoid the problem. With that component disconnected everything worked as expected.
Moral of the story: Test with one component at the time if you're experiencing difficulties.
