Keeping up with updates on all the devices in my home lab usually isn’t a big deal. Windows, macOS, Android, and iOS mostly take care of themselves. Ubuntu needs the occasional touch of ‘apt upgrade’ but is otherwise painless. My printer hasn’t had new firmware in over a year (there’s definitely a vuln or two that won’t ever be patched). Given the typical monotony, I was caught off guard when the firmware update on my router failed.

A/B firmware updates

Instead of a typical WiFi router, I use a Ubiquity EdgeRouter Lite to get some fancy features like BGP routing. Being geared toward WISPs and other industry applications, firmware upgrades are made safer by maintaining two OS images. When a new update is loaded onto the device, it overwrites the inactive OS image (the one not currently running). Once the update is written, checksummed, and ready to go, the device is rebooted and the bootloader loads the new OS image. If the new OS image fails for some reason, the bootloader will fallback to the old OS image that was running previously.

When failures repeat themselves

I SSH’d to the router and ran the update command:

$ add system image https://dl.ubnt.com/firmwares/edgemax/v1.9.7/ER-e100.v1.9.7+hotfix.4.5024004.tar
Trying to get upgrade file from https://dl.ubnt.com/firmwares/edgemax/v1.9.7/ER-e100.v1.9.7+hotfix.4.5024004.tar
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 80.8M 100 80.8M 0 0 4740k 0 0:00:17 0:00:17 --:--:-- 5082k
Download suceeded
Checking upgrade image...Done
Preparing to upgrade...Failed to mount partition

Uh, oh. This is about the time that I remember that this particular router failed to boot after a power outage a few months ago. The internal storage had been corrupted. I ran a recovery procedure, got it back online, and promptly forgot about it. Seems like the internal storage is failing. What can I do about it?

Surprisingly serviceable

From the Ubiquity Community forums, I learned that the internal storage is actually a USB thumb drive and can be replaced. Perfect! I have a pile of extra USB drives that I keep around for OS installers. I grabbed an 8GB drive and promptly realized I had no idea how to load the OS onto the new drive. Sure, I could pull the old drive and copy it but that would copy any corruption already present. A little more hunting in the forums turned up mkeosimg, a script that prepares a USB drive from an EdgeRouter firmware update.

The README is pretty clear that I need a Linux machine to use it. A quick look at the source confirms that it relies on a handful of tools such as parted and mkfs.ext3. The only Linux machines I have at home at rack-mount servers running containers. Getting access to a USB port on those is a bit of a chore so I took the “easier” approach of firing up an Ubuntu VM on Virtualbox.

Why so slow?!?

Almost immediately, mkeosimg tells me it is “Creating ER-e100.v1.9.7+hotfix.4.5024004-configured.img” and then sits there for a few minutes. What the heck is it doing? Creating a zero-filled file of course! While waiting for dd to copy 8GB worth of zeros to a file inside a VM is an exciting way to spend time, I like to live a fast and dangerous life. I replaced dd with a call to fallocate. fallocate simply asks the filesystem to create a file entry but don’t actually allocate sectors for data. That is, it creates an 8GB file almost instantly because it only updates the filesystem metadata. The data sectors will be allocated as data is written to the file. Now the script runs in seconds. I have an image! I write it to a USB drive. I’m nearing the end!

Opening the case

Yes, there is a USB stick inside this router. I’m not sure I’d call it a “standard” USB stick though.

By removing all the normal casing, the router enclosure can be slightly smaller. The problem: my new USB drive has all the regular casing. A trip to the bench grinder was in order.

Now it just barely fits in the enclosure.

Well, that was fun.