Robin Smidsrød

A lot of people still use DSL to connect to the Internet, and most of these connections have a lower upstream bandwidth than their downstream. This can often cause latency for interactive low-bandwidth applications, like SSH and VoIP (SIP, Skype and Lync), when uploading large amounts of data because of a phenomenon called "bufferbloat". To combat this issue there are many solutions. Below I'll tell you how I've worked around this issue.

History

A long time ago, in Internet time, a script for Linux machines was developed that was called the LARTC Wondershaper. This served many as a good solution for years, but as Internet services blossomed and user behavior changed, so did our needs for controlling packet latency change. I wrote the first version of SuperShaper-SOHO, and improved Wondershaper, back in September 2004. It used the improved HTB scheduler and life was good for many years.

Enter HFSC

But then, a few days ago, I found, by accident, a GitHub gist about HFSC, Linux traffic shaping's best kept secret, which turned out to be a variant of Daniel S. Sterling's hfsc-shape script. I was intrigued, because I had been feeling lately that something didn't quite work right with the older version of my setup which I've used for years. Latency wasn't controlled as well as it was before. I started reading up on HFSC (hierarchical fair-service curve). I found the man pages tc-hfsc(8) and tc-hfsc(7), but those just confused me even more. Reading up on this just got me more and more confused, especially when they talk about the real-time ("rt") service curves. HFSC is really hard to understand! Eventually I found the kernel documentation called sch_hfsc.txt which was the best explanation I've found so far. It showed some fairly simple math on how to calculate the values I needed.

Refactoring the existing script

I started looking at my old version of the script and saw that this was no easy feat. I started looking into my existing script and figured out that I'd gotten something mixed up back in the day when I wrote the initial version. When you define filters for a classful scheduler you need to make sure that your filters are defined in a way that narrower terms are matched before the broad ones. The pre-1.6 versions of my script have this woefully wrong, which is why they don't work properly. Once I had all of this fixed (and my flow and filter definitions refactored into proper Bash functions) I was good to go.

Going from HTB to HFSC

Once I had refactored everything into nice Bash functions it wasn't actually so hard to change from HTB to HFSC. All my packet filters could stay the same, because they just ensured traffic would be put in the proper flow. I had to change my old HTB priority-based define_flow function into a HFSC-based define_ls_flow function. As you can imagine by the name, I've completely ignored the real-time ("rt") parts of the HFSC scheduler and just implemented simple link-sharing ("ls").

The core difference between HTB and HFSC, when used only with link-sharing service curves, is that the algorithm doesn't dequeue packets in terms of flow priority, but instead specify minimum bandwidth a specific flow should get if the link is fully saturated.

So with that understanding in place I defined my flows in terms of how much bandwidth the different types of traffic should get if my link was fully saturated. With this setup any flow can borrow packets from another flow as long as the link isn't saturated. But the moment it is, even tiny flows (like TCP ACK packets, ICMP/DNS traffic and VoIP) get priority over bandwidth-heavy uploads.

The end result is that as long as nothing else is happening, my cloud backup can utilize my full upstream bandwidth, but the moment I start watching YouTube/Netflix or take a VoIP call, that traffic will get priority and the cloud backup traffic will yield down to what I've set as a minimum bandwidth allowance for that flow. My SSH console sessions now work as if there were no traffic on the link even though I'm fully utilizing my upstream.

MISSION COMPLETE!

Last thoughts

I would really like to figure out how to easily define a real-time service curve, but the complexity of it baffles me. If you have some insight into this topic a pull request is most welcome, as I'd like for the script to have this feature.

As the sch_hfsc kernel documentation mentioned, and the hfsc-shape scripts I've linked to, they implement link-sharing using the upperlimit ("ul") service curve. I've avoided this, so that any flow can saturate the uplink when no other traffic is present. That's a win-win in my book.

I've also created a Telegraf script that monitors the bytes sent and the period value for each flow, which gives a good understanding of how this setup behaves. This script allowed me to quickly understand why the older version of this setup was broken. I might blog about this monitoring solution in a later article.

Have you ever experienced that the system service Media climbing to the top of your application list on the Battery tab on your Android device? I had this particular problem, and it took me a very long time to actually figure out what caused the problem. I had read on forums that it might be because of corrupted files, but the only solution presented was to go through every single file on your memory card and verify that it was indeed working. That sounded like a very time-consuming way to solve the issue, and I imagined there had to be a better way. And there was.

I unmounted the memory card from the phone and inserted it into my computer. Then I copied all of the files I could read on the memory card (there was apparently some file system corruption, so some files were unreadable) onto my computer hard drive. I then ejected the card from the computer, reinserted it on the phone and formatted it completely. Once that was done I ejected it from the phone again, put it back into the computer and copied all of the files back onto the memory card. Finally I put it back into the phone again.

As far as I can tell everything is working as it should now and the MediaScannerService is no longer draining excessive battery power. Win!

Another alternative instead of copying all the files and formatting the memory card is to run the traditional Windows chkdsk program on the entire memory card, which should also be able to fix the corrupted filesystem. Regardless of which method you decided to use, you should have something that works again. It does indeed seem like file corruption wasn't the problem (as mentioned on the forum), but FAT filesystem corruption issues.

Before I actually tried this approach, I actually ran the memory card through a level 5 (recover good sectors) recovery on SpinRite, and it didn't make any difference to the problem. I'm glad the solution I found worked for me.

So, a while ago I decided that I wanted to invite the core developers of the iPXE project to my home for an extended weekend of hacking on the project. The agenda was to try and merge as many outstanding patches and pull requests that we knew about. The work the various students did as part of GSoC 2012 was on the priority list. The three people that were able to come were Michael Brown, Thomas Miletich and Marin Hannache.

TL;DR: Skip down to the summary at the bottom.

Thursday 11th of July

After a busy day cleaning the house and making things ready for the arrival of the guests I headed down to Sandefjord airport (TRF) to pick up Michael. His plane was scheduled to arrive around midnight. He had brought a test machine padded in a large suitcase which packed the car quite full. Luckily Marin and Thomas missed the planned train from Oslo airport, so they arrived an hour later, which was just enough time to get Michael settled in before I had to pick them up. After everyone had arrived we ensured the sleeping arrangements were appropriate and test equipment was unpacked and set up. We finished up the evening around 3AM with strawberries with cream and sugar accompanied by white wine (brought by Marin). Finally people crashed and burned after a long day.

Friday 12th of July

Since the day before was quite late, people slept in a bit and woke up a little bit before noon. I had solicited food preferences in advance, which was received favorably when we had breakfast out on the patio in the sun/shade.

After the breakfast Thomas started working on the code review of the VIA Velocity Gigabit driver written from scratch by Adrian Jamróz as part of GSoC 2012. I guess we should've merged this code a long time ago, but better late than never.

Michael continued working on his code to allow iPXE scripts to pull out information from PCI configuration space. This is useful for e.g. including the PCI vendor/device IDs in embedded menus or sending it off to a server-side script somewhere, either for inventory purposes or dynamic driver loading. The code was merged and pushed out later the same day. Documentation on how to use it is expected to show up at http://ipxe.org/settings/pci eventually.

Marin worked hard on cleaning up his NFS protocol support code. There were some bugs around that caused the downloaded image name to be NULL, and the URL format was a bit odd. After some discussion we decided on a new URL format and a way to derive the required mount point. Marin worked actively on that piece for the rest of the day.

I was performing initial testing on the new Broadcom NetXtreme II driver written by Daniel Wyatt as part of GSoC 2012. Preliminary testing indicated there was something strange with the loopback test between this card and the Intel 82541pi used as a reference. Except for performance, most of the driver seemed to be working more or less as it should, but the loopback failure required more hardcore debugging. I decided to let it rest until Michael could help me out.

Some time during the day we all went to the supermarket and bought various food we were planning on eating during the weekend. That avoided the trouble of trying to translate all of the names of things from Norwegian to English.

Later in the evening I organized a traditional Norwegian barbecue with potato salad, fresh watermelon summer salad and burgers, hot dogs and some tender beef. The guys seemed to like it. After the dinner we ate marzipan cake, which is something Michael decided to buy at the supermarket. Michael also made a traditional English drink called Pimm's with lemonade to top it off. It tasted quite good, all of it, in my opinion.

After a break which included a few episodes of Futurama some more hacking got done, until we eventually succumbed to sleep deprivation.

Saturday 13th of July

Even before breakfast was served, Marin was hard at work with the changes on his NFS project. I helped him test it out and eventually he was able to finish it up and hand it over to Michael for final code review and merging.

Also before breakfast Michael rewrote the support for netX by implementing symlink support between settings blocks. The netX settings block now shows up first in the list of network adapters, and it directly points to the last opened network device. This now enables a clean way of abstracting out the netX feature to be used in e.g. autoboot and other places you can specify a network device. It also fixes a bug with ${mac}, ${chip} and ${busid} not representing the currently opened device if you have more than one network device, but always the first network device.

Thomas was still working hard on the VIA Velocity driver. It seemed like never-ending work.

Breakfast we enjoyed outside this day as well. I had bought cold burgers, heated in the microwave, and put them on the table together with the other normal things. Since the first meal of the day was quite late (2PM), everyone ate like the hungry wolves they were. :)

After looking through some iPXE configuration menu I suggested to Marin that he should add support for Home/End/PgUp/PgDown to the config TUI. It didn't take him much time to shake that out of his right arm. It's currently waiting for merging. After he did that he started looking into a shutdown command someone had mentioned they wanted. He spent some time trying to figure out how APM works, and sometime in the middle of the day Shao Miller showed up on IRC and helped him out, and he was able to complete a patch for that feature. It is currently lingering in the merge queue.

Michael started reviewing the VIA Rhine driver, first cleaning it up and then hitting several obstacles with the loopback test not giving him what he expected. He continued figuratively banging his head against the wall for the rest of the day.

Sometimes it is good to take a break when you just can't find any good answers. He started making a typical Asian dish called watercress & chicken stir-fry with rice for dinner (we couldn't find watercress in the store, but he substituted with some other appropriate vegetables). Once he was done with that we sat down and enjoyed a good meal at a late hour (almost 10PM).

I had mentioned earlier in the day that Matthew Helton wanted a way to make the iPXE version string available to scripts. Michael encouraged me to try to write the support myself, considering it was just a few lines of code in core/settings.c. While Michael was busy making dinner I had a look at it, and with some insight from Marin and Thomas I was able to push out my second commit in the C parts of the iPXE codebase. I felt so proud of myself, even though it was mostly copy and paste and two lines of changed code.

After a long day of extensive hacking and a good meal, we sat down with some ice cream for dessert. While we were eating we fired up the old MAME emulator and had some fun playing games like Street Fighter II, Mortal Kombat and Bubble Bobble with an authentic arcade joystick from X-Arcade. Michael even tried it!

Although most of us was ready for bed Michael and Thomas ended up back at the computers hacking on the troublesome VIA drivers while Marin and myself continued beating each other to a pulp in Mortal Kombat. Michael found a page with the special moves earlier which made it so much more funny.

At 3AM the last of us figured it was getting way too late and decided to just stop what we were doing and try again the next day with fresh eyes.

Sunday 14th of July

I had mentioned to the guys earlier that I was planning on making eggs and bacon for breakfast one morning, and this was indeed the day for that. Everything went well and my favorite brunch was served the way it was supposed to. We ate it indoors because the wind was a bit chilly. It seemed to go down very well after a long evening hacking the night before.

After the breakfast we went on a tour around the center of Tønsberg. We started at Slottsfjell museum where we saw the viking ship and the large whale skeletons. Both things impressed the guys. After that we walked up to Tallak and had a look around the old buildings there. The place was very busy, because a lot of construction workers were preparing for the Slottsfjell festival that is happening the next weekend. After that we walked up to the tower and had a look out over the city. For the first time in years I could see the changes in the city from an aerial view. The next stop on our route was the pier, which was reasonably active with people. We stopped at Parad-is for some long-awaited ice-cream and a walking break. Finally we walked back up to where the car was parked by the museum. All along the route me and Marin were busy hacking Ingress portals.

Once we came back Michael continued working on his VIA Rhine reviewing, in which he had an epiphany and found out there was an interrupt bug in the Intel driver which caused packets to be lost for some reason. Once that was taken care of he was able to finally push the new rhine driver.

Thomas continued working on the VIA Velocity driver, getting more and more annoyed at the poor quality of the data sheet and the odd behavior of the test hardware.

Marin said he wanted to make potatoes baked in cream for dinner one of the days, so he started preparing that after we got back, and after a good amount of time in the stove the dinner was ready. Together with some left-over watermelon salad, some sausages, burgers, grilled chicken and other goodies it was time for a feast. And boy was it true!

After the dinner everyone was feeling a bit sleepy after a long day with more than just hacking, so we sat back and watched an episode of Futurama and MacGyver, during which some of us fell asleep for shorter bits of time. It's really nice to see that people enjoy themselves and feel like they're at home. While we were cleaning the plates off the table Marin was eating out of the almost empty pot of left-over potatoes. I guess he's just as much a sucker for food as I am.

The guys went back to hacking the code, while I decided to go to bed a bit earlier than the other days.

Monday 15th of July

When I woke up I got some good news from Thomas. It seems the machine he had been using for testing the VIA Velocity driver had some hardware issues. Once he moved to another machine, all the tests passed and he was very happy with the outcome. The driver could finally be merged.

Once everyone was up and around we had breakfast out in the sun again. The weather had improved a bit from the day before.

After the breakfast Michael was busy merging all of the other outstanding patches mentioned earlier before he started in the NFS feature from Marin. After several back-and-forth bug fixes by Marin the patch was finally in a mergeable state and we could all rejoice over a job well done.

While the others were busy doing code reviewing, I tried to figure out why my dual-NIC test machine didn't want to exit properly back to my embedded menu from my main menu script. After a lot of testing we found out that autoboot had an issue with its getopt implementation which caused it to go in a loop if you specified a network adapter, but worked fine if you didn't specify an adapter list.

We had already decided on going out to eat dinner one of the days, so this was definitely the day to do that. We went to the restaurant in the city which is supposed to have the best kitchen, La Barca. Most of us ordered reindeer, and the dish was outstanding, as usual.

After we got back from the city Marin and Michael worked more on getting the file license Makefile target to work properly. They actually got it working, but it also revealed a lot of manual work that needs to be done (basically going through each source file, figuring out which license it is under and then tagging it with a single line of code - if you want to help out with this, please get it touch).

Thomas was at this point getting tired of the review work on the bnx2 driver, so he decided to start looking at a driver for the IC+ 1000 Gigabit Ethernet adapter he's had around for a while.

At 1AM we started getting hungry, so we pulled out the rest of the strawberries and enjoyed (most likely) the last desert of the hackathon.

While eating people got grandiose ideas about implementing Pong using iPXE scripting. It was pretty obvious the sleep deprivation was starting to get to us.

Tuesday 16th of July

The day had finally arrived when everyone had to pack up and get back home. People woke up quite late, which meant we had to eat breakfast in a hurry to avoid Thomas and Marin missing their train to the airport.

While I drove Thomas and Marin to the train, Michael was still hacking around on a new increment command for the iPXE scripting language. It is going to be a neat feature that should enable us to easily loop through e.g. network adapters in embedded menus and probe hardware or other information using the pci or smbios settings blocks.

When I got back we sat down and chatted a bit, watched a couple of episodes (introduced Michael to The Big Bang Theory) and I suggested to him to get in touch with Randal Schwartz to appear on FLOSS Weekly. This could give us a decent PR boost and get the word out a bit more, and hopefully grow our community.

Later I dropped Michael off at the airport and hacked some more Ingress portals on my way home.

Summary

Overall these new features and drivers where published to mainline during the hackathon:

• New hexraw variable format. Useful for using MAC addresses without separators.
• A network interface's PCI location can be used from scripts.
• PCI vendor/device IDs (and more) can be used from scripts.
• netX is now a symlink settings block to most recently opened interface.
• Version string available to scripts.
• New VIA Rhine 100Mbit driver.
• Backslash (like bash) continuation to break long script lines.
• New VIA Velocity Gigabit driver.
• Home/End/PgUp/PgDown keys now work in config.
• APM computer shutdown command, poweroff.
• Downloading of files using NFS protocol.
• Building the license targets now work (although we miss license info for lots of files).
• Several minor bug fixes that have been reported and some we discovered during the weekend.

These are the items on the agenda we weren't able to complete because of lack of time:

• Merge the bnx2 driver.
• Make ipxe.kpxe work (aka UNDI fallback).
• Make iPXE as an EFI application work.
• New Intel wireless driver.

All in all it was a great event, and I'm happy that everyone enjoyed it as much as I did hosting it. I'm quite up for doing it again some time in the future, it was a blast! You can find some photos from the event on my Flickr set for the hackathon.

If you want to participate in the iPXE community, join us on IRC, the mailing-list or the forum. If you're a beginner to network booting, you might also find the website http://networkboot.org/ useful.

If you want to get in touch with me there are links at the top of my blog with links to Twitter, Facebook and Google+. Talk to you later!

The other day I received this email from a user of one of my CPAN modules.

This is what makes participating in an Open Source community really worth it! The tool he's talking about is XML::Rabbit.

Thanks a lot, Jack!

Just a note to say thank you.  I had been trying to build a system (without Moose) that could do something like this. I then decided that Moose would be better support for my classes.   I ran into chromatic the other day and bought his book because it looked like it had a bit more on Moose.  After playing some more with Moose, I did a search that took me to chromatic's page on Rabbit.

I cannot remember what the search was.  However, I'm glad that I found it.  My one-line summary for Rabbit is that it is a client side version of JQuery.

Perl keeps popping back into my life.  Almost fifteen years ago, I was using Perl 4.  I have not used it in over ten years.  Last year, I decided that Perl would be good for translating a database, and I wanted classes. I was very happy to see Perl 5 was on target.

Now at a new company, I am faced with thousands of lines of Perl (mostly version 4) with sometimes 15K in one file!),  I've been breaking the code into packages and classes.  However, as I thought about it, I needed something more abstract. I need to solve some meta-programming issues to lead the company towards the future.  Today, I find Rabbit -- can life get much better?

Once I've done some conversion of our XML API and can process it with Rabbit, I hope to add some ideas.  If you have some thoughts on what you think would be useful to add, I would not mind thinking about them as I solve the bigger problem that I am working on.

Cheers,
Jack

During the summer season of 2012 (northern hemisphere) I've had the pleasure of being the coordinator of the iPXE network boot loader's participation in the Google Summer of Code program. This is a summary of what happened during the summer.

What is iPXE?

iPXE is a network boot loader. What does that actually mean? Normally when you boot your computer, control is handed over to the boot sector on a local hard drive in your computer and an operating system is loaded. But that is not the only option you have. For a lot of years a standard called PXE (developed by Intel), has been available which allows x86-based computers to boot from the network (using DHCP and TFTP). iPXE's main goal is to extend that ability to not only boot from TFTP (which is quite slow), but allow computers to boot from HTTP, FTP, iSCSI, AoE and other more esoteric sources. We also have a goal of being the main network boot loader for EFI-based systems (almost working, get in touch for details). We already support more than 60 different network cards, and cards without a native driver are also supported using a feature called chainloading.

First time participating in GSoC

This is the first year iPXE participates in the GSoC program as an organization. Most of the mentors involved have participated earlier years as part of other organizations, such as kernel.org and Etherboot. It was the first time for me, though, and it was quite fun! My job was to keep all the students and mentors on track and ensure no deadlines were forgotten and that communication between everyone flowed smoothly. In total we had three students, which I understand is quite a lot for a first year organization. Thank you to Google for believing in us.

Students and projects

Marin Hannache from EPITA University in France wrote protocol support for downloading boot files using NFS. His work is awaiting review and integration into mainline. From what I've tested it still has some issues, but it is working, and I'm most eager to get it in, as it will allow booting directly from an NFS3 server, not even needing a web server. This should be great news for people using small NAS boxes to do their booting, as they don't always have a way to serve files using HTTP.

Adrian Jamróz from Jan Kochanowski University in Poland reimplemented the drivers for the VIA Rhine and VIA Velocity family of network cards. The existing drivers were using a deprecated API and was much bigger in size than they really needed to be. Code size is always a big challenge for a project like iPXE, which commonly is burned into ROM chips that have a limit of 64KB. His code was very good and is just awaiting final review and integration. I've personally tested it on one of my Rhine2-based cards, and it seems to work reliably.

Daniel Wyatt from Lord Fairfax Community College in the United States reimplemented the driver for the Broadcom NetExtreme II family of network cards. The problem with the old bnx2 driver was the same as the VIA driver. It used a very large proprietary firmware blob to run the MIPS-based microcontrollers on the card. Most of the summer was used to understand how we could minimize this firmware without losing any functionality. In the end the student got something working, but we're still having issues with the legality of including the firmware inside the iPXE project. The integration with mainline is currently on the fence awaiting some kind of confirmation from Broadcom that we can actually distribute the firmware blob with iPXE. Luckily we have people from Broadcom participating in the community, so it shouldn't be too impossible to get this kind of approval. I guess this student got a crash course in how licensing affects what you can do within an open-source project.

Other tidbits

All of the pages linked to above were maintained by the students as they were working on their code during the summer. You can also find links to specific Git repositories that include all of their code if you're interested in testing it out before it is integrated into mainline. Most of the day-to-day collaboration was done using IRC on the #ipxe channel on irc.freenode.net. Below you can read some of the interesting things said during the summer by the students:

• 20120608-010814 <Mareo> wow, I'm in the black magic part of iPXE codebase :D
• 20120703-121824 <Mareo> this gsoc is awesome :D
• 20120712-203713 <dewyatt> the bnx2 datasheet is horrible :(
• 20120720-233044 <dewyatt> mcb30: okay...I'm eager to get started on the fun stuff
• 20120728-160956 <budrys> meteger: but I think it can't be left as is, it's a dirty hack for me
• 20120815-172646 <dewyatt> mcb30: mouse sent me some ROMs that have firmware in them. This made me take another look at the newest PXE ROM I have and I found compressed firmware. I hackishly extracted this by pausing the broadcom PXE ROM during DHCP and reading from the Debug UART. I can't do this for RV2P and I don't really know the bounds of the MIPS firmware so it's still a work in progress.
• 20120816-083313 <dewyatt> I don't know if I can legally commit this code really :/

Thanks to everyone

It is great to have taken part in this journey and seen how much the students have learned over the summer. I want to thank our three mentors, Michael Brown, Joshua Oreman and Thomas Miletich and all the other knowledgable people on the #ipxe IRC channel for the great work they've provided during the summer. Participating in GSoC has been a great experience for us all, and I hope we'll be able to participate next year as well.

If I've whet your appetite to learn more about network booting, I invite you to join us on IRC, participate in the iPXE forums or join the mailing-list.

Until next time,
Robin Smidsrød,
iPXE GSoC 2012 coordinator