So I decided to jot down some notes for myself after being given a small project to setup some devices for a school in PNG.

We had a large amount of old devices that we were scrapping- Acer B117 and B116 devices, most of which had previously been issued to students. I weeded out 80 of the least worst of them, and had a think about what we could do with them. I demoed a few Linux builds-

The folks from Aiyuran seemed most impressed with Ubuntu 18.04, and to be fair it’s a great stepping stone for a school on a budget. Long term, they were looking at using Gsuite and AWS for most of the grunt work application wise, so the OS really didn’t matter so much so long as it could run a web browser.

So we have 80 machines and a USB stick install of Ubuntu. How to make this work?

Obvious choice is to make the office juniour run from machine to machine with a handful of USB live install sticks, but I’m not that cruel. Instead, I thought back to the time I had to image some laptops at a workplace and made use of Microsoft Deployment Toolbench (MDT). For those of you not in the loop, MDT lets you build deployments of Windows, using Windows Deployment Server (WDS) to PXE boot a trimmed back Windows that can then be used to network install full blown Windows. I was able to chain boot the Windows PXE loader with pxelinux, and give my coworkers a choice of network booting a Ubuntu image or a Windows image. I figured I could do the same here, but without the WDS or MDT components.

I ransacked our store room desktop collection, and pulled out an old HP Z230. It was actually pretty grunty, with 16GB RAM and a Xeon processor. It had a RAID card that Ubuntu just did not like at all, so I ended up scrapping the RAID and installing in IDE mode. I now had a barebones, Ubuntu 18.04 LTS server.

I did some googling around for some details, to see what other people had been doing for imaging. There’s a lot of details out there for Canonical’s Metal as a Service (MaaS), and various ways to deploy earlier versions of Ubuntu, but for some reason very little around more modern distros. It was time for a little trial and error.

I briefly considered using things like FAI and Cobbler after reading this blog post but decided since I’d had such great success with it previously I would stick to Kickstart. I found a great guide on setting up a PXE boot server at Linux Hint and followed that basically word for word. It’s a fantastic guide, and it got me up and running in about 20 minutes. My setup looked a little like this-

With little to no work on my part, I was able to get this basic setup working and serving a Ubuntu Live image over NFS. I generated a kickstart config and a very basic preseed file, expecting most of the install to be handled by kickstart. Boy was I wrong.

My first problem was pretty immediate- I could dhcp boot the laptop from a USB ethernet adapter, but as soon as it got to the stage where it had to pull over files from my nfs root it failed to find the server. After some head scratching and googling, I figured out this was because the live image did not have the USB network adapter drivers for whatever reason. So I switched to a laptop with a built in NIC to troubleshoot.

Second problem was, no matter what options I set in my pxelinux.cfg/default file, it would fail to parse the kickstart config. Each and every time I would end up in a live environment. I tried sharing the kickstart file over http, I shared it by NFS- I could see in syslog the DHCP options for it were there and present, and I could even wget it from the live environment. It just wasn’t picking it up at all. I had a look at the process for adding a driver to an initrd file, and very quickly lost heart- the process seems very fiddly and annoying to test,

I took a break at this point. I was pretty bummed out that I seemed to have been stymied so early in the project. I had time off booked anyway, which was equal parts welcome and frustrating, but it let me take a step back from the problem and reassess.

When I returned to work, I took a different approach. Instead of installing from a Ubuntu live system, I found the network boot files. I replaced my vmlinuz and initrd with the linux and initrd.gz from there, and was able to quickly boot in and get to an installer. Progress!

The only issue was, this was a network install. So it required network access. In the diagram above, the 10.200 range is actually our iPad network, which meant I didn’t want to interfere too much with it since my little Z230 was running DHCP. I went down the rabbit hole on figuring out what to do with my network.

First problem was I had a default gateway set on my 10.0 adapter. Any time I wanted to connect to the internet, I had been disconnecting the Z230 from the switch, reconnecting it to test the DHCP. By removing the gateway settings on the 10.0 NIC and adding a gateway to the 10.200 NIC, I could now have both connected at the same time. Next was getting that connection shared. In Ubuntu, there is a file /etc/sysctl.conf, I simply uncommented the line net.ipv4.ip_forward=1 to allow packet forwarding.

At this stage I started messing around with ufw. I whitelisted all the ports I needed, and read a guide on getting ufw to handle NAT. But no matter what I did, it straight up would not work. DNS was resolving hostnames on my test imaging laptops, I could get and IP for outside addresses…but no traffic. Finally, my boss asked why I was swearing so much, and steered me in the right direction. Instead of ufw, he suggested I use plain iptables.

iptables -A FORWARD -i eno1 -o enx000ec6f5d408 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i enx000ec6f5d408 -o eno1 -j ACCEPT

Magic!

However, the install process is still entirely manual.

So I went back to google and did some more research. I read somewhere (lost the link sorry) another admin had issues getting kickstart to work with Ubuntu 18.04 in particular, and had instead moved on to using preseed files that they would bundle in to their install iso. So I cut out the middle man, and created a more definitive preseed file to cover what I needed. After a little tweaking, the final version of that preseed is available on my github page.

I also found that Ubuntu would prompt for a hostname prior to asking for an IP address in the installer, and if I wanted it to smoothly and automatically install I needed to add priority=critical netfcg/get_hostname=ubuntu1804 to the pxlinux default file. It was at this point I started getting dpkg errors.

Everything would run smoothly up to the point where it was installing the base system. I would get

dpkg: error processing package ubuntu-minimal (--configure)
dpkg: error processing package console-setup (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of kbd:

Unfortunately, this is apparently a pretty common and hard to diagnose issue. Most threads suggested an apt-get update && apt-get upgrade would fix this, which is not really going to work when you’re installing Ubuntu from a busybox stripped down system. I took a look at my preseed file and realised I hadn’t specified a keyboard layout, it was at this point I added the following lines.

d-i keyboard-configuration/xkb-keymap select us
d-i keyboard-configuration/layoutcode string us

A test run after this worked smoothly, although there are still two steps in the partitioning that are manual and I’ll need to iron out. However aside from this, the installer now performs all steps automatically based off the information in the preseed, and pulls packages for the system via the internet.

I found the documentation to be pretty helpful as well.

It took me a lot longer than I’d like to admit to get this whole thing running. There are a couple of things I’d like to do to make this nicer-

  1. Install apt-cacher and have the installer pull up to date packages from there
  2. Automate the install and setup of the PXE server so I can rebuild it from a script
  3. Distribute an ssh key and install sshd for remote management
  4. Setup a management server to ship with the laptops, to handle DHCP, updates, and user management

Unfortunately I have very limited time left in my current role, moving on to another employer in a completely different state and city! So my list will likely never be completed…