Rambling about stuff probably nobody else cares about

Ripped everything out of one of these cases to make a router

I’ve noticed this blog is really less about the info I’m trying to share with people, and more of a collection of me rambling on about the stuff I’ve fixed or put together.

I’m not sure what’s wrong with me. I just spent an hour going off on another blog about this case I owned I made a firewall out of. Since I put so much time and energy into it, I thought I’d bring it here and add it to the collection: http://disq.us/p/2p7i6zw

Orignal on site Baptiste Wicht Norco RPC 230 case review: https://baptiste-wicht.com/posts/2013/08/norco-rpc-230-case-review.html

I have one of those Norco RPC-230 cases, I’ve had it since 2011. It’s crossed 2 states during 4 moves, and at least 6 different configurations, so it’s seen a thing or two.

I suffered through using it longest for a TV recorder PC case, running HDHomeRun for at least a year or two, with 4x HGST 7K2000s in it, which are power-hungry, fast-spinning 7200RPM, hot, old drives. The thermal design of this case is garbage, the fans only hit 2 out of 4 of the drives, so the two coolest HDDs ran 50+ deg C 24/7 that entire time, the other two 55-60+ deg. It was a major testament to HGST’s enterprise line, as they were refurbished, so undoubtedly they were tormented mercilessly before I even got them. The case I got new, but was chipped and scraped after the first of my builds it endured. The front USB connector broke in the middle within a year after normal use. It still works, but is difficult to plug into. As far as quality, amazing drives, crap case, both still work, only one deserves praise.

I also used this case for a desktop PC for a while, with a i5-3570K 77w TDP processor. With stock CPU cooler it would idle at 65 deg C within 5 minutes, which was unacceptable. A Thermaltake Gravity cooler got it under 50 deg C, but the clearance was only a couple mm so it probably could have been more efficient had there been more room, or perhaps a shorter cooler. The Gravity is 3-pin non-PWM, though, which in this case is probably better since the case was never not too hot under any circumstances, so there’s no reason to give a PWM fan any time to decide to rev up.

I thought about drilling some holes in the side near where the CPU would usually sit, since it doesn’t have any airflow from either the side or the back to speak of, but have you ever tried to drill through sheet metal? It’s not as easy as it sounds, even with a specialized drill bit. Even the aluminum foil-like character of the RPC-230’s build quality would present a painstaking chore. I’m glad I decided not to, because I ended up putting a board in it where the CPU isn’t even in the back, it’s in the front, so it wouldn’t have made any difference for my current config, other than to look hideously abused.

Anything over 40 deg C consistently makes me nervous personally, so I was never at ease running any equipment in the Norco RPC-230 case that wasn’t meant to be low-powered, cool, designed for minimal energy consumption, etc. So I finally took a server board X9SPU-F meant for proprietary cases that are near impossible to find, and put it in the RPC with a very low-TDP CPU. I chose the RPC-230 mainly because the motherboard wouldn’t fit in anything else I owned at the time, but also because I finally had a very low-TDP processor I intended to use with it. I’d been watching prices of the E3-1220Lv2 for a couple years and they plummeted from being around $150 to around $35 at the time (and now, virtually worthless). Finally, I had the type of components that wouldn’t make you think you could fry an egg on the top of this thing.

To fit the X9SPU-F in the RPC-230, I had to rip the drive tray out of the front. There’s no use putting drives in the RPC-230 anyway, unless you want to make fried hard drive souffle. Then, I equipped this odd-shaped server board with an E3-1220Lv2 which is 17w TDP, butchered a heatsink out of a CPU cooler from a Dell SFF Optiplex 7020 and tightly zip-tied a Noctua NF-A8 to it (works great). Then, I velcroed a couple mSATA SSDs to the bottom of the case in a single 2-port mSATA to SATA adapter board for a RAID1-ish config designed to keep the system from being taken down by an IO shit-nado of network log files.

Now it’s an edge router + firewall! Has dual Intel NICs and IPMI, too, so it’s fancy. Pulls less than 40w at the wall, does up to 2.4Gbps as tested with an 82599 (possibly faster if not using FreeBSD/pfSense, which has terrible Intel NIC driver and kernel regressions, and less-than-ideal for 10GbE `sysctl` defaults).

For a network device SSD application, I’ve tended to think size and speed are secondary to being able to take a very certain and constant beating. I’ve been using these 24GB Intel 313 SLC mSATA SSDs on dual-SATA daughterboards and putting them in some kind of SDS version of RAID1 (either with mdadm or ZFS). The key is they’re SLC, which is

 hard to find and weren’t made very long for cost reasons. However, you’d be hard-pressed to find another SSD that’ll take a beating like these. They were designed to be cache devices for an old Windows 7 + RST config that never really caught on in I want to say like 2014. The plus side, is you can find a bunch of them around for cheap, and they’re pretty unlikely to die. They only get about 120MB/s read, 60MB/s write, but it doesn’t matter, and if you use a mSATA to SATA converter board like mine that has one port per drive, you can reap some benefit in throughput from running your SDS RAID1-ish mirror-type config (in this device’s case, it’s running ZFS).

And such is the tale of the long and winding life of my poor, beat-up, ill-equipped for most tasks I envisioned, but still utilized, and now humbly revered for being able to fit my bizarrely-shaped server board, Norco RPC-230.

— Good lord, I am such a nerd 🤓

Booting Fedora 36 on default btrfs partition structure from the grub prompt

One of the many logos from the many articles discussing Fedora 33’s stunning move to finally ditch XFS, after their unfortunate habit of talking much trash about btrfs for a number of years


I ran into an issue where I had an unpopulated grub menu on a Fedora 36 Workstation installation. It ended up only booting to the empty menu, but at least I could drop to a prompt to try and figure out how to get through it.

The usual booting with a live USB, mounting the affected drive to /mnt and the other partitions and bind mounts respectively, and rebuilding grub.cfg from chroot didn’t appear to be functional due to an error in the chroot environment that makes it unable to see /dev and use any of the usual tools to build the file.


All you really need to read in this article to boot from grub prompt is in the 2nd codeblock below (some people just here for the lols)

So booting from the grub prompt was necessary in order to be in an environment where /dev would be recognized and grub.cfg could be rebuilt – since grub.cfg isn’t anything that can be edited manually, unfortunately (it looks like it could be the way it’s named, but if you know anything about how grub works, you know it’s definitely not. Not even a little bit…)

I had written a while back about how to use the grub prompt in general here: The GRUB prompt: Demystified

The example I gave in the last article was done on Ubuntu. It looks like pretty much any other grub rescue operation using EXT4, XFS, etc. but it’s actually a homebrew ZFS initrd- I used to make my own ZFS built-in kernels regularly, and threw together a script for automating the process if you’re interested: https://github.com/averyfreeman/zfs-kernel-builder

But lately I’ve been dabbling with Fedora so this article’s related to Fedora. It’s upstream for some great software, what can I say. I had an NVMe from a Thinkpad I’ve had for a couple years I tried to swap into a newer Thinkpad I was upgrading to, because Thinkpads. For some reason grub.cfg got hosed in the process and I couldn’t boot from it.

Not wanting to take too much time troubleshooting, I fresh-installed a new copy of Fedora 36 to another NVMe in the new laptop and copied as much as I could from /home and a package list from repoquery -a --installed run using chroot from the old NVMe in an external enclosure (gee willikers, I sure love my external NVMe with Thunderbolt 3, by golly).

Since I still had the old installation all set up, I thought I’d try and rescue it eventually, so when I got a moment I booted it from the external NVMe enclosure on another laptop with Thunderbolt using VMware Workstation and the enclosure as a physical disk .vmdk (hypervisors are awesome).

After failing with the tried-and-true live USB chroot rescue method, at least I could get into the grub menu without issue. That poor, empty menu, so lonely feeling with its 0 boot loader entries. I knew the OS was still there, but how to get into it without the entries? What’s a nerd to do? Start messing around with the prompt to explore how to crack open the damn thing!

Since this was btrfs, of course, it was a little different, and since each distro uses a different subvolume layout, none of the info about any of them translate to the others very well. At the time of writing, I didn’t see any definitive info on how to boot from grub prompt on stock btrfs Fedora Workstation, which is kind of surprising since it’s been btrfs default for 3-4 years now, since version 33.

There’s a great guide here about Ubutntu using btrfs on “nixventure”, which I admit I’d never heard of before, but appears very thorough, and I saw one from Debian (I believe) that I’m not going to reference because it was less remarkable, but Fedora there were just a bunch of forums with people flailing about trying to figure out the same thing I was, and ending up being unsuccessful and presumably giving up from the look of their abruptly truncated threads. So that was concerning, to say the last.

I even tried “rescatux” automated rescue tool, if you can believe that, because I was being lazy and kind of running out of ideas. I can’t say I give it a glowing review, but it tries. That statement probably indicates how well it worked for my needs. I had to suck it up and adapt what I knew about using the grub prompt to the new partition layout and filesystem. Thankfully, it all turned out well in the end.

Here’s some notes I took while I was working through the process:

# Fedora's variant of grub has some helpful btrfs-specific commands:

grub> help

. . . commands . . . 
btrfs-get-default-subvol (hd0,gpt3)
btrfs-list-subvols (hd0,gpt3)
. . . more commands . . .  

# so you get information like this:

grub> btrfs-list-subvols (hd0,gpt3)

ID 256 path home
ID 257 path root
ID 258 path var

# and this:

grub> btrfs-info (hd0,gpt3)

Label: 'fedora_treygouty' uuid: 7caff388-2bb3-434a-a927-096dac2dc892
        Total devices 1 FS bytes used 298520481792

# Intuitively, I thought this would work, but I was mistaken:

grub> linux /vmlinuz-5.17.9-300 <tab-tab works> root=UUID=7caff388-2bb3-434a-a927-096dac2dc892 ro rootflags=subvol=root

# You'd probably have to write all that out by hand, so be thankful I'm telling you ahead of time it didn't work for me

TL;DR #2 – you’re getting closer …

Anyway, I’ll cut to the chase and show how I did it. Note, again, for the record, this is the bare-minimum default partition structure on an Anaconda installed Fedora 36 Workstation setup. No LUKS or other encryption, no LVM.

Note: For the record, this is the first Anaconda (RedHat’s) installer where LVM hasn’t been enabled for the default partition configuration. LVM is marginally useful for btrfs, I’ve tried it before on OpenSUSE, but comparitively it’s much less of a crutch than it is for EXT4 or XFS filesystems. So if RH’s installer says don’t bother with LVM, don’t bother… It’s RH, you know how they LOVE their LVM, if they’re not recommending it it really must not be necessary. /rant

# Quick grub prompt recap --
# First, list your storage devices:

grub> ls

(proc) (hd0) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1) (cd0) (cd0,msdos2) 

# If you're lucky like me and you only have one drive connected, /boot will be easy to find. /boot is almost always gpt2, and there's only one hard drive, so...:

grub> set root=(hd0,gpt2)

grub> ls /


# Set your vmlinuz and root device + partition. Here (hd0,gpt3) corresponds with /dev/sda3:

grub> linux /vmlinuz-5.17.9-300.fc36.x86_64 root=/dev/sda3 ro rootflags=subvol=root

# Then boot 'er up:

grub> boot

That’s really all there is to it. It’s not too bad once you know what you’re doing. Since the process is so short, you might want to poke around some more to try and get more info, or to try and give your life meaning. There’s the (hd0,gpt2)/grub/grub.cfg file, or the /loader/entries folder – you can cat anything in either of those. Type help or something. Go wild.

Coincidentally, all the loaders were in the loader directory the entire time, but none of them would load all of the things. Such a bummer when one cannot load all of the things. So, frustrating, but glad it all worked out in the end. Hope this helps someone else, too.

VMware Upgrade renders Workstation 16.2.1 useless, thanks to Houdini-like disappearance of DLLs

Librarian replies to man inquiring about biography of famous escape artist, "I'm sorry sir, all of our books on Houdini have disappeared"
Picture depicts Houdinis as books disappearing from a library, while our problem depicts missing libraries themselves. Kinda makes you think, doesn’t it?


I just came across an issue many people have been experiencing after upgrading VMware Workstation to 16.2.1 in Windows. It looks like the installer has a bug that deletes two necessary files by accident, and their absence prevents the programs from running, beginning with vmware-tray.exe, which is the precursor to all things Workstation.

The lawnmowered files are two dynamic link libraries (“DLLs”) responsible for encryption. Their names are libssl-1_1.dll and libcrypto-1_1.dll. Thankfully, I’ve managed to gather them from a reliable source and have re-packaged them for your convenience. Below is an archive containing both missing files for you to download, in hopes that you may use them to remedy this rank insipidity:

The organization responsible for storing and distributing DLL files such as these is named dll-files.com, located at https://www.dll-files.com in lovely Tilf AB, Sweden. Their website offers a surprisingly painless, hassle-free download (full disclosure: no affiliation).

I actually visited a few other sites looking for these DLL files before I found dll-files.com, but all the other sites “provided” were dubious “utilities” containing code that made my antivirus software blush a deep shade of “seriously?”.

Speaking of malware, everything I am providing today has been scanned with Malwarebytes, the best antivirus software in the biz (full disclosure: no affiliation), which reports these DLLs, and obligatory license + attribution document, are as squeaky-clean as a newborn antelope (devouring placenta does have advantages). I installed them on my laptop, the one I’m using now, and it’s been smoother sailing than vacationing on Velveeta.

Installing random files with these kinds of file names actually can be a little scary, considering all the ransomware thieves who were wreaking havoc across the US a couple years ago, but legitimate programs use cryptographic libraries just often as cyber-criminals, perhaps even more. So never you be a’ feared of them scary ol’ names, now, their bark a’ far worse than their soo-eey.

Come to think of it, in contrast to the wrath of cryptographic criminals a couple years ago, it seems that recently the media is more likely to associate “crypto” with blockchain currency. At least wealth is a more pleasant association than robbery… <dentist office music>

To install the files above after you download them, unzip the .7z file using 7zip archive software, and move the two .dll files contained therein to the folder located at C:\Program Files (x86)\VMware\VMware Workstation\<files go here>

I used an admin command prompt to move mine, but you could just as easily open your file explorer to drag-and-drop them, I’m sure. Once Meta gets off their behinds and releases the first telekinesis controller, we’ll all be able to will our files across our hard drives with sheer focus and determination, but until then it’s a flail across our mice and keyboards. Sorry to remind you we’re still savages with disappearing DLLs.

Mount Ubuntu 22.04 ZFS partitions using live ISO for disaster recovery

ZFS Send and Receive ·

My system runs ZFS and lately has been dropping to the initramfs / busybox prompt on boot. I had a hard time finding a fleshed-out guide on how to mount ZFS in a live environment for performing disaster recovery tasks like chroot and grub repair, so I thought I’d write something up.

My system was dropping to the busybox prompt after GRUB menu. I started experiencing the issue after a routine apt upgrade, I rebooted and wasn’t able to get any of my initramfs to boot. It seems a little strange, because usually the inability to boot will be limited to a new initramfs – e.g. an older version of the kernel will still have the ZFS drivers, or other necessary components to boot, while the newer versions (the ones just installed) will be lacking these necessary components for whatever reason.

First of all, burn yourself a copy of a live USB, and boot into it. Luckily, the newest version of Ubuntu (22.04 – Jammy Jellyfish) has the ZFS drivers and executables installed by default, unlike prior versions where you had to add the multiverse repo manually, download the packages, and enable the ZFS drivers using modprobe.

A peek at lsmod shows the ZFS drivers are indeed loaded, and lo-and-behold, there’s the zpool and zfs executables:

ubuntu@ubuntu:~$ lsmod | grep zfs
zfs                  3751936  29
zunicode              348160  1 zfs
zzstd                 487424  1 zfs
zlua                  155648  1 zfs
zavl                   20480  1 zfs
icp                   319488  1 zfs
zcommon               102400  2 zfs,icp
znvpair                94208  2 zfs,zcommon
spl                   122880  6 zfs,icp,zzstd,znvpair,zcommon,zavl

ubuntu@ubuntu:~$ which {zpool,zfs}

The drive I am diagnosing is the internal NVMe, so there’s no need to attach it. One question I had was how to mount the two pools, and in what order. By default, Ubuntu creates an rpool for the root partition, and a bpool for the boot partition.

Generally, on an EFI system, one would mount the root partition in a clean directory like /mnt first, and subsequently mount boot at /mnt/boot once it is provided by the previously mounted root partition, and then mount efi at /mnt/boot/efi once that’s provided by the boot partition. As you can see, the order of mounting these partitions is therefore of paramount importance, but as there are only 3 options, it’s not too complicated.

You’ll need to be root for basically all these commands. Using sudo su without a password will typically get you to a root prompt (#) in a live environment.

TL;DR – probably way more than you ever wanted to know about an lsblk device list:

First, we should identify the storage devices using lsblk -f (the -f flag includes the filesystem information, which is important for our purposes):

# lsblk -f
     squash 4.0                                                    0   100% /rofs
     squash 4.0                                                    0   100% /snap/bare/5
     squash 4.0                                                    0   100% /snap/core20/1405
     squash 4.0                                                    0   100% /snap/snapd/15177
     squash 4.0                                                    0   100% /snap/snap-store/575
     squash 4.0                                                    0   100% /snap/gtk-common-themes/1534
     squash 4.0                                                    0   100% /snap/firefox/1232
     squash 4.0                                                    0   100% /snap/snapd-desktop-integration/10
     squash 4.0                                                    0   100% /snap/gnome-3-38-2004/99
sda  iso966 Jolie Ubuntu 22.04 LTS amd64
│                       2022-04-19-10-23-19-00                              
│    iso966 Jolie Ubuntu 22.04 LTS amd64
│                       2022-04-19-10-23-19-00                     0   100% /cdrom
│    vfat   FAT12 ESP   8D6C-A9F8                                           
     ext4   1.0   writable
                        bb277d84-75cc-473b-b327-fd885d85889a   24.5G     0% /var/crash
zd0  btrfs              b6239f8a-058b-4a6c-8258-b9a7b50f6c23                
     btrfs              d6074499-b9aa-47e0-a08a-58e27c73e771                
zd32 btrfs              c68aa9ca-933a-48cb-9adb-22fd6a8ca8c8                
     btrfs              f52702bd-c805-4edc-87d1-6fb877ee6738                
│    vfat   FAT32       B045-5C3B                                           
│    swap   1           584b9b78-7d8d-4a5a-9263-d6f6a48adc6b                
│    zfs_me 5000  bpool 11241115695889536197                                
     zfs_me 5000  rpool 16130566787573079380                                
│    vfat   FAT32       EC9D-0344                                           
│    ntfs               A4EEBDB4EEBD7F5C                                    
     ntfs               989EE7E99EE7BDBE

OK, there’s a lot there, so what are we looking at? Well, the first 9 devices that say loop are snaps, since we’re on Ubuntu. Those are responsible for storing some of the programs being run by the OS. Each one gets their own virtual storage device, sometimes referred to as an “overlay”. They create a fair amount of clutter in our device list, but that’s about all. You can ignore them.

Then, /dev/sda is our copy of Ubuntu ISO we booted from – you can see how it says cdrom there, and iso9660 (the cdrom spec). It’s read-only, so we couldn’t do anything with it if we wanted to, and we don’t, so let’s move on…

There’s a device for log and crash log, so that’s kind of interesting. I imagine the live ISO makes those since you can’t write to the USB drive, seeing as the ISO is a virtual CD-ROM, and CD-ROMs are read-only. Then there’s a bunch of what are called “zvols” (the zd0, zd16, etc. devices – see those?). Those are devices created with ZFS that are isolated from the rest of the filesystem. zvols are virtual block devices you can use just like any other block device, but in this context they’re typically either formatted with a different filesystem, or mounted via iSCSI for block-level filesharing (filesystem-sharing?). You can see these ones say btrfs, they were actually created for use with container runtimes, namely podman and systemd-container, both of which support btrfs very well and ZFS either nominally or not at all.

Now we get to nvme1n1 – this is the first NVMe drive listed. Generally 0 would be listed first, but for some reason it’s listed second. n1 is the number of the drive (the second NVMe drive in the laptop), after that the partitions are listed as p1, p2, p3, and so on. Here’s the drive in isolation:

│    vfat   FAT32       B045-5C3B                                           
│    swap   1           584b9b78-7d8d-4a5a-9263-d6f6a48adc6b                
│    zfs_me 5000  bpool 11241115695889536197                                
     zfs_me 5000  rpool 16130566787573079380  

The canonical address for this drive is: /dev/nvme1n1p{1,2,3,4} . The /dev (device) folder, while not listed in this output, is important to reference, as the full path is required for mounting a partition. Typically one would only mount a single partition at a time, but you could conceivably chain them in a single command by using curly braces, as shown. This is not common, as you will probably need to mount different partitions in different locations (e.g. /mnt, /mnt/boot), and usually either in descending order, or with no pattern at all.

If you remember back at the start, I mentioned the rpool and bpool. These are seen on /dev/nvme1n1p4 and /dev/nvme1n1p3 respectively. If the disk were formatted in a block filesystem such as EXT4 (Ubuntu’s default filesystem), the root partition could be mounted by attaching /dev/nvme0n1p4 to an empty folder. The command would therefore be:

# mount /dev/nvme1n1p4 /mnt

And then you’d be able to ls /mnt and see the files contained on your newly mounted root partition. E.g.:

# ls /mnt
Qogir  boot   dev  home  lib32  libx32  mnt  proc  run   snap  sys  usr
bin    cdrom  etc  lib   lib64  media   opt  root  sbin  srv   tmp  var

But this NVMe is formatted using ZFS. So what to do? That’s the process I was having difficulty finding that inspired this blog post.

End TL;DR – here’s the ZFS-specific stuff again:

First, after you confirm that you have your ZFS modules loaded by referencing your list of loaded kernel modules, and confirming that your ZFS executables are available in PATH (here’s the syntax again so you don’t have to scroll back):

# lsmod | grep zfs 
zfs                  3751936  29
zunicode              348160  1 zfs
zzstd                 487424  1 zfs
zlua                  155648  1 zfs
zavl                   20480  1 zfs
icp                   319488  1 zfs
zcommon               102400  2 zfs,icp
znvpair                94208  2 zfs,zcommon
spl                   122880  6 zfs,icp,zzstd,znvpair,zcommon,zavl

# which {zpool,zfs}

Here’s where it’s different than your typical mount. You use zpool to import rpool, but you need to mount it using an alternate root (at /mnt) – otherwise it’ll try to mount itself over your live environment! Then confirm that the import worked.

# zpool import -f rpool -R /mnt

# ls /mnt
Qogir  boot   dev  home  lib32  libx32  mnt  proc  run   snap  sys  usr
bin    cdrom  etc  lib   lib64  media   opt  root  sbin  srv   tmp  var

OK, that went well. You can see that now we have a /mnt/boot folder, which is boot inside rpool – that’s where initramfs lives, but they’re stored in the bpool. We needed that folder to be available to mount our bpool into. So, let’s import bpool into /mnt/boot as an alternate root (if we didn’t, it’d try and overwrite our currently mounted /boot partition:

# zpool import -f bpool -R /mnt/boot

# ls /mnt/boot
config-5.15.32-xanmod1       memtest86+_multiboot.bin
config-5.15.34-xanmod1       System.map-5.15.32-xanmod1
config-5.15.36-xanmod1       System.map-5.15.34-xanmod1
config-5.17.0-xanmod1        System.map-5.15.36-xanmod1
config-5.17.1-xanmod1        System.map-5.17.0-xanmod1
config-5.17.3-xanmod1        System.map-5.17.1-xanmod1
config-5.17.5-xanmod1        System.map-5.17.3-xanmod1
config-5.17.9-xanmod1        System.map-5.17.5-xanmod1
config-5.17.9-xanmod1-x64v2  System.map-5.17.9-xanmod1
efi                          System.map-5.17.9-xanmod1-x64v2
grub                         vmlinuz
initrd.img                   vmlinuz-5.15.32-xanmod1
initrd.img-5.15.32-xanmod1   vmlinuz-5.15.34-xanmod1
initrd.img-5.15.34-xanmod1   vmlinuz-5.15.36-xanmod1
initrd.img-5.17.0-xanmod1    vmlinuz-5.17.0-xanmod1
initrd.img-5.17.1-xanmod1    vmlinuz-5.17.1-xanmod1
initrd.img-5.17.3-xanmod1    vmlinuz-5.17.3-xanmod1
initrd.img-5.17.5-xanmod1    vmlinuz-5.17.5-xanmod1
initrd.img.old               vmlinuz-5.17.9-xanmod1
memtest86+.bin               vmlinuz-5.17.9-xanmod1-x64v2
memtest86+.elf               vmlinuz.old

That looks like a bunch of initramfs files to me! Good, so that means those kickstarter runtimes that load from grub are available.

If you look in that list, you’ll also see both efi and grub folders. Both of those are empty and waiting for storage to be attached. The efi partition lives in the first partition of the same NVMe drive, and is formatted with FAT, and grub is a bind-mount (you can see it in /etc/fstab):

# mount -t msdos /dev/nvme1n1p1 /mnt/boot/efi

Can also use UUID from lsblk if prefer (just use one or other, not both): 
# mount -t msdos UUID=B045-5C3B /mnt/boot/efi

# ls /mnt/boot/efi
efi  grub  system~1  (confirm it's mounted)

# grep grub /mnt/etc/fstab
/boot/efi/grub	/boot/grub	none	defaults,bind	0	0
(we'll bind-mount this in next step)

Then you’ll want to mount a few system folders inside your drive’s filesystem so you can access them inside the chroot (required for things to work OK):

# for i in proc dev sys dev/pts; do mount -v --bind /$i /mnt/$i; done

mount: /proc bound on /mnt/proc.
mount: /dev bound on /mnt/dev.
mount: /sys bound on /mnt/sys.
mount: /dev/pts bound on /mnt/dev/pts.

# mount -v --bind /mnt/boot/efi/grub /mnt/boot/grub
mount: /mnt/boot/efi/grub bound on /mnt/boot/grub.

chrooting”: Now that all 3 partitions are mounted together in a cohesive filesystem tree, and you’ve got all your necessary bind mounts, one of the most effective ways to diagnose issues as if you’re running the affected disk, is to chroot into the filesystem. Run # chroot /mnt and now you’ll see /mnt as / (root), and you can run your programs as if you booted the computer using that drive (from the terminal, anyway):

# chroot /mnt

# apt update (failed)

# cd /etc
# ls -la resolv.conf
lrwxrwxrwx 1 root root 39 Feb 17 12:09 resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

If your network connection fails inside the chroot like mine did, go to /etc and delete resolv.conf if it’s a symlink to systemd-resolved (as shown above). Then point /etc/resolv.conf to a known good dns forwarder (e.g.,, etc.)

# echo 'nameserver' > resolv.conf

# apt update (works)

# apt list --installed | grep dkms

dkms/jammy,now 2.8.7-2ubuntu2 all [installed,automatic]
zfs-dkms/jammy-proposed,now 2.1.4-0ubuntu0.1 all [installed]

I was really hoping zfs-dkms got uninstalled somehow, because I thought that might have been why my initramfs files didn’t have zfs modules. So unfortunately I still have to keep looking to figure out what’s wrong…

Note, you’ll probably see this error a lot, but it’s safe to ignore:

ERROR couldn't connect to zsys daemon: connection error: desc = "transport: Error while dialing dial unix /run/zsysd.sock: connect: connection refused" 

Let’s try upgrading the packages and see what shakes out:

# apt upgrade 

The following packages were automatically installed and are no longer required:
  linux-headers-5.15.32-xanmod1 linux-headers-5.15.34-xanmod1
  linux-headers-5.15.36-xanmod1 linux-headers-5.17.0-xanmod1
  linux-headers-5.17.1-xanmod1 linux-headers-5.17.3-xanmod1
  linux-headers-5.17.5-xanmod1 linux-image-5.15.32-xanmod1
  linux-image-5.15.34-xanmod1 linux-image-5.15.36-xanmod1
  linux-image-5.17.0-xanmod1 linux-image-5.17.1-xanmod1
  linux-image-5.17.3-xanmod1 linux-image-5.17.5-xanmod1
Use 'sudo apt autoremove' to remove them.

That was … interesting … and then the issue presented itself next while I ran apt autoremove:

Setting up linux-image-5.17.9-xanmod1 (5.17.9-xanmod1-0~git20220518.d88d798) ...
 * dkms: running auto installation service for kernel 5.17.9-xanmod1     [ OK ] 
update-initramfs: Generating /boot/initrd.img-5.17.9-xanmod1
zstd: error 25 : Write error : No space left on device (cannot write compressed 

(emphasis added)

bpool has no space left. That’s almost certainly the problem. I’m going to remove a couple kernels and rebuild all my initramfs, that ought to do it. I’m also noticing my bpool is full of snapshots. List current snapshots with this first command, and then destroy them with the second one:

// This lists the snapshots:
# zfs list -H -o name -t snapshot | grep bpool

...auto-snapshots look like pool/BOOT/ubuntu_pd3ehl@autozsys_xxxx, 
snapshots have @ symbol - no @ symbol, not a snapshot, don't delete it!

// This destroys the snapshots:
# zfs list -H -o name -t snapshot | grep bpool | xargs -n1 zfs destroy -r 
What this does:
(list only snapshots by full name) | (list only bpool) | (delete by ea line)
It's the same as what's above, but with the delete command, destroy. 

Make sure you understand what's going on with this command, as you can delete stuff you don't want to really easily. Please be careful.  

… looks pretty good to me – much more tidy:

# ls /boot
config-5.15.0-33-generic       memtest86+.elf
config-5.15.40-xanmod1-tt      memtest86+_multiboot.bin
efi                            System.map-5.15.0-33-generic
grub                           System.map-5.15.40-xanmod1-tt
initrd.img                     vmlinuz
initrd.img-5.15.0-33-generic   vmlinuz-5.15.0-33-generic
initrd.img-5.15.40-xanmod1-tt  vmlinuz-5.15.40-xanmod1-tt
initrd.img.old                 vmlinuz.old

Install some generic kernel to make sure you have one available, check that zfs-initramfs is installed if all you’re going to use is generic kernel (or zfs-dkms if using xanmod, other 3rd-party kernel). E.g. I got rid of my xanmod kernels just so I wouldn’t have to deal with building custom dkms modules:

# apt list --installed | grep xanmod

linux-headers-5.15.40-xanmod1-tt/unknown,now 5.15.40-xanmod1-tt-0~git20220515.867e3cb amd64 [installed,automatic]
linux-image-5.15.40-xanmod1-tt/unknown,now 5.15.40-xanmod1-tt-0~git20220515.867e3cb amd64 [installed,automatic]
linux-xanmod-tt/unknown,now 5.15.40-xanmod1-tt-0 amd64 [installed]
xanmod-repository/unknown,now 1.0.5 all [installed]

# apt remove linux-headers-5.15.40-xanmod1-tt linux-image-5.15.40-xanmod1-tt xanmod-repository linux-xanmod-tt zfs-dkms
 . . . 
The following packages will be REMOVED:
  linux-headers-5.15.40-xanmod1-tt linux-image-5.15.40-xanmod1-tt
  linux-xanmod-tt xanmod-repository zfs-dkms
Do you want to continue? [Y/n] 
 . . .
# apt autoremove -y

... install a couple kernels...

# apt install -y linux-{image,headers}-5.15.0-28-generic linux-{image,headers}-5.15.0-33-generic

 . . . using versions that are most current & 2nd most current now . . . 

Then update all the initramfs one last time, just in case. I’ll probably re-install grub, too, just bc, but one thing at a time…

# update-initramfs -uvk all  

. . . lots of output . . . that's how you know it's working . . . 

Let’s re-install grub and run update-grub

# grub-install --bootloader-id=ubuntu --recheck --target=x86_64-efi --efi-directory=/boot/efi --no-floppy

Installing for x86_64-efi platform.
grub-install: warning: EFI variables cannot be set on this system.
grub-install: warning: You will have to complete the GRUB setup manually.
Installation finished. No error reported.

When you get this error, it just means you can’t set the UEFI boot order while you’re in a chroot. I also like to run update-grub for good measure (this is grub2-mkconfig -o /boot/grub/grub.cfg on most other systems if that’s more familiar sounding to you). Update-grub rebuilds the entries in your grub menu, along with their parameters detailed in /etc/default/grub.

Speaking of which, you can always take a peek at /etc/default/grub before you run this command – just in case.

# which update-grub

# cat /usr/sbin/update-grub

// update-grub:
set -e
exec grub-mkconfig -o /boot/grub/grub.cfg "$@"

# update-grub
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: vmlinuz-5.15.0-33-generic in rpool/ROOT/ubuntu_pd3ehl
Found initrd image: initrd.img-5.15.0-33-generic in rpool/ROOT/ubuntu_pd3ehl
Found linux image: vmlinuz-5.15.0-28-generic in rpool/ROOT/ubuntu_pd3ehl
Found initrd image: initrd.img-5.15.0-28-generic in rpool/ROOT/ubuntu_pd3ehl
Found linux image: vmlinuz-5.15.0-33-generic in rpool/ROOT/ubuntu_pd3ehl@autozsys_yg50xc
 . . . snapshot menu entries . . . 

Now leave the chroot now, remove the system folder redirects and bind mounts, and reboot, like so:

# exit

# for i in proc dev/pts dev sys boot/grub; do umount -v /mnt/$i; done
umount: /mnt/proc unmounted
umount: /mnt/dev/pts unmounted
umount: /mnt/dev unmounted
umount: /mnt/sys unmounted
umount: /mnt/boot/grub unmounted

# umount -v /dev/nvme1n1p1
umount: /mnt/boot/efi (/dev/nvme1n1p1) unmounted

# zpool export bpool

# zpool export rpool

One last quick thing you can do before rebooting is check out efibootmgr and see which order your system will start up in. This is a little easier and more predictable, as you can make sure you boot from the right efi file, rather than mashing the startup menu button to make sure it loads the correct disk / efi.

Some stuff I was messing with trying cover all the bases. efibootmgr reference: https://wiki.archlinux.org/title/GRUB/EFI_examples#Asus

# efibootmgr -v
Boot0000* ubuntu	HD(1,GPT,544a9120-eef7-4aae-8311-cd6ca6929213,0x800,0x100000)/File(\EFI\ubuntu\shimx64.efi)
 . . . 
# efibootmgr -B Boot0000 -b 0

# efibootmgr --create /dev/nvme1n1 --part 1 --write-signature --loader /EFI/GRUB/grubx64.efi --label "GRUB" --verbose
BootCurrent: 0002
Timeout: 0 seconds
BootOrder: 0000,0001,0002
Boot0001* UEFI: Samsung SSD 980 1TB, Partition 1	HD(1,GPT,6afa5e93-54a5-4628-978f-313a0dcfe27b,0x800,0xfa000)/File(\EFI\Microsoft\Boot\bootmgfw.efi)..BO
Boot0002* UEFI: Samsung Flash Drive DUO 1100, Partition 2	PciRoot(0x0)/Pci(0x14,0x0)/USB(16,0)/HD(2,GPT,a09db2b8-b5f6-43ae-afb1-91e0a90189a1,0x6cc954,0x2130)..BO
Boot0003  Windows Boot Manager	HD(1,GPT,6afa5e93-54a5-4628-978f-313a0dcfe27b,0x800,0xfa000)/File(\EFI\Microsoft\Boot\bootmgfw.efi)WINDOWS.........x...B.C.D.O.B.J.E.C.T.=.{.9.d.e.a.8.6.2.c.-.5.c.d.d.-.4.e.7.0.-.a.c.c.1.-.f.3.2.b.3.4.4.d.}....................
Boot0000* GRUB	HD(1,GPT,a09db2b8-b5f6-43ae-afb2-91e0a90189a1,0x40,0x6cc914)/File(\EFI\GRUB\grubx64.efi)/dev/nvme1n1

A troubleshooting tip: If you have issues using the pool names with zpool for some reason, the UUIDs are listed in lsblk. While technically interchangeable, the UUID can coax some commands into to working correctly when the name can’t.

If it doesn’t boot from the ZFS drive again, boot it into the live ISO and go through everything all over … 😉 Good luck!!

dpdk 22.03 rpm packages for Fedora 36: a dependency of openvswitch

open vswitch with DPDK: architecture and performance - ppt download

I meant to post this over a month ago, but got sidetracked, so I’m coming in a little late. Unfortunately, it looks like even though Fedora 36 has officially been released, the dpdk 22.03 rpms still aren’t available.

Back when I compiled these, I realized there were no official dpdk 22.03 rpms available in the yum repos, despite being required for openvswitch. So I compiled them so I could install openvswitch for use with virt-manager.

Dpdk on Fedora has an official maintainer, they probably just got sidetracked themselves – I can relate. But I had these packages already and wanted to help other people install openvswitch, so I posted the resulting packages in a github repo case other people want to download it.

So if you (like me) want to run openvswitch on your fancy new officially-released non-beta Fedora 36 workstation (or server, or silverblue, kinoite or iot using rpm-ostree), and you don’t want to wait or downgrade, you’d have to either have to compile dpdk 22.03 for yourself, or now you can download them from my repo.

I’ve got all the instructions to go through the build process if you’d like to get your hands dirty with compilation (it’s pretty straightforward): https://github.com/averyfreeman/dpdk-2203-for-fedora36

The .rpm files are also there if you don’t care for all the fuss of compilation. You can use them as openvswitch dependencies, just install them before you try and install openvswitch using dnf.

I should have made this post sooner so people knew the rpms were available, but at least I put a note or two in on reddit in a couple key places (now that I think about it, probably just /r/fedora). Nothing beats the officialism of posting a fancy notice on your wordpress blog, though, amirite?

Speaking of which, I think I will be looking at a way of integrating this blog with my repo – I’m hoping I can figure out a way to produce and update posts on wordpress automatically by creating a gist or a new repo. Then the two would be more tightly integrated, and could avoid this whole getting sidetracked issue…

The less people get sidetracked, the sooner they use software…

Run Windows Store (UWP) App on Startup – the EASY way

TL;DR: drag the icon from the shell:AppsFolder over to the shell:startup folder!

I am building a dedicated TV viewing VM for HDHomeRun View using Windows IoT 2021 LTSC, so I can watch TV while using a computer running Linux with the smallest possible KVM-QEMU VM I can possibly put together. That’s because, unfortunately, the app I have to use is only available in Windows, and only through the Windows Store on top of that, which in the past has created an additional layer of potential configuration difficulties due to app sandboxing, obscured location paths, etc.

Well, thankfully starting UWP apps (also called “Windows Store” apps, but herein will be referred to as UWP apps) have gotten a lot easier in newer versions of Windows. I’m not actually sure of this because I haven’t looked into any changelogs detailing new Windows shell or explorer features, but I’m assuming because I’ve gone through all sorts of trouble to get them to start up automatically in the past (this HDHomeRun one in particular), but now it seems ridiculously easy given what I had previously gone through.

I was reading through this article on how to start a UWP app on startup from HowToGeek.com. It’s a good article with lots of interesting points, but it seemed unnecessarily difficult to me. Between it, and this question answered on superuser.com, I came up with the easiest possible way to start a UWP app on startup.

  • In Windows File Explorer (the manilla folder icon), type shell:AppsFolder in the location bar and hit enter. This will bring up a list of icons, “regular” Windows applications in addition to UWP (store) apps.
  • Navigate through the icons until you find the UWP app you’re looking for.
  • In a separate Explorer window, type shell:startup and hit enter.
  • Drag the UWP app you want to start during startup into the shell:startup folder. It might ask you if you want to create a shortcut in that folder (hit yes).
  • Log out and back in to make sure it worked.

This worked for me on the first try. The hardest thing I did was remove the - shortcut string that was auto-appended to the icon. It really doesn’t get much easier than that.

In the past, I’d gone through all sorts of trouble figuring out which folder holds the app – they’re all in a hidden folder C:\Program Files\WindowsApps with folders named things like EF712BA7.HDHomeRunDVR_1.1.345.0_x64__23nna27hyxhag only accessible from administrative shell, isn’t that fun? – getting the application name and path from the AppManifest.xml file, and then creating a batch file for the startup script for the UWP app to be started with C:\Windows\System32\cmd.exe /C:, etc.

As I said, I’m not sure if using UWP apps is just getting easier on newer versions of Windows, but this is pretty darn convenient. If I was just doing something unnecessary in the past, it sure was a lot of trouble.

If you’re interested in running Windows Store for the apps you can’t find anywhere else on LTSC, check out this github repo – says last updated 3 years ago, but the script has worked for me on newer versions of LTSC for me just fine: https://github.com/kkkgo/LTSC-Add-MicrosoftStore

Recent posts to message boards

I haven’t had time to write lately, but I have made a few notable posts on message boards I thought I’d include here for reference:



Do you use both Visual Studio Code and Sublime Text 3?

Stripe Elements with Laravel (and Vue) - YouTube

If so, wouldn’t you like both their keyboard shortcuts to be consistent?

I personally am using both Visual Studio Code and Sublime Text 3 on my main dev laptop. Once I got used to all the nice features in a pseudo-IDE text editor like VS Code, it became hard to go back to editing things like .conf files in editors like vim and notepad. Wrapping text with parenthesis, quotes or brackets, simply by highlighting the word and hitting the opening character is probably my favorite feature, but things like duplicating lines, moving lines up and down, etc. are all very nice ways to save some time.

However, once my VS Code setup had enough formatting extensions like Prettier, or other handlers for package managers like npm, version control with git, color manager, gistpad, python, markdown parser, :emoji sense:, etc. etc. load times have gotten longer, and the editor feels less up to the task of opening a little .conf or .sh file just to do a little revision.

That’s where Sublime Text comes in. For one thing, it’s smaller. The fact that it doesn’t take up as much room on my screen makes it much more convenient for basic text-editing tasks. It also loads faster without so many plugins (I try to keep it down to only one or two). But it still has essentially all of the really awesome interactive qualities of VS code.

The only thing that ever really irks me is when I try to do something the way I’m used to in vscode with the keyboard and it does something else, or doesn’t work at all.

Luckily, when the keyboard shortcuts are inconsistent, ST3 is really easy to extend. I’ve forked a GitHub user’s keyboard shortcuts who started mapping ST3’s keyboard mapping and added a few of my own – for example, alt+z for enabling word wrap didn’t even have a keyboard shortcut in ST3, but now it can (!).

Whenever I run into another inconsistency, I’ll add it and push it to GH. If you’d like to see (or use) it, it’s located here: https://github.com/averyfreeman/vscode-keybindings-for-sublime

Creating a local HTTPS server for Create-React-App using nginx and certbot in WSL2: Works for BOTH npm start AND npm build

How Certbot and Letsencrypt Work (DNS and SNI-TLS automation) - Magic of  Security

Note: This article has some interesting WSL-centric material, like how to:

  • Set up bridge networking without Hyper-V
  • Find the UWP Ubuntu executable to run Linux commands outside of a WSL terminal
  • Create automatic startup tasks without systemd init system
  • Expose your (domain-enabled) dev environment to certbot/letsencrypt for automatic issuance of SSL certificates
  • Use nginx proxy for react

So even if you’re not doing the exact thing I set out to do (create an HTTPS dev & build preview server), you might still find these notes useful.

Update 6/11/2021: While the notes in this article basically work, WSL is pretty quirky for doing backend development. There are many strange variables that have no parallel in a stock Ubuntu installation.

Also, bridge.sys kept KP-ing my laptop, but I did manage to stop BSODs by replacing the Dell-released driver with the newest stock Intel driver. But I have to say, there’s nothing more frustrating/demoralizing than having the system crash while trying to write code. Hopefully you won’t have the same issue, I will leave this up in case it helps anyone with any of these steps, but I cannot in good conscience recommend it as a development platform. It’s OK, just know the risks!

If you are looking for a platform to develop software you plan to deploy to an Ubuntu VPS, I recommend using a virtual machine. I have taken to doing front-end dev in WSL and copying anything that needs additional tooling/config to a “real” Ubuntu VM *first*, getting it ready on that, and *then* pushing it to VPS.

Interestingly, if you run Alpine on your web server, WSL might actually be more suitable for backend tooling configuration, as Alpine uses OpenRC rather than systemd as its init system – it’s probably more likely that you could use it in the WSL environment similarly to how you’d run it bare metal. I have not verified this first-hand, this guess is purely informed speculation from using Alpine extensively for various servers and containers – but never for WSL. Try it, find out, and message me!

Examples of major weird / unparalleled issues I ran into re: using WSL for backend tooling are (from orig post):

  • Windows controls all the networking. That makes setting stuff up that requires networking really strange. In addition, the bridge setup I enabled has needed to be re-adopted manually after every Windows reboot, the WSL vNIC won’t stay configured to stay in the bridge persistently.
  • There’s no actual init system. That means no systemd for enabling / starting / stopping software. /etc/init.d/$INITSCRIPT is present, but has no upstart control, so =>
  • All “init” jobs are handled by Windows Task scheduler – either that, or you just don’t start scripts automatically (unless there’s another method I’m not aware of)
  • MS’s software L2 bridge (shown as Network Bridge adapter in Network Connection) which uses bridge.sys has a high propensity to cause kernel panics (and thus, BSODs) on my system. I’m using a Dell Precision 7730 w/ Intel i219-LM network adapter w/ driver version on 19043.1110. If you’ve got something different, good luck to you. If your setup is quite similar, brace yourself for BSODs.

    After updating my I219-LM ethernet adapter driver to the one available on the Intel website (instead of the one released by Dell), I have managed to run this configuration for the last day and a half and haven’t had any KPs from bridge.sys. That’s a new record!

    The new driver is version, listed as v26.3 on Intel’s website. Note: The ProSet (software) and driver are in separate packages, ProSet archive does not install driver. Hoping it holds, but feeling pretty confident about it working so far. Would be nice if these computer vendors would support their hardware longer than a couple years.

I still like WSL2 for the additional flexibility it gives me in Windows, but it’s been “liberating” to recognize its deficiencies and put aside trying to get it to do everything I want – it’s just not there yet. The sooner I realized that, the sooner I stopped trying to wrap its alien configuration methods to purposes for which they are ill suited, and got back to coding.

However, if you want to try to get it to do stuff it’s not “supposed” to do, I left these notes up in the hope that someone else would come along and build on what I learned after several days of frustrating experiences.

Orig post:

This is a heavy post. This is the kind of thing I do when I’m working through a tough problem – well organized notes are a must. Obviously, parts of this will make more sense to me than you, but I have the added benefit (shortcoming?) of having trouble making sense of my notes when I come back to them, so I try to make them as complete as possible the first time so hopefully if it feels like I’m “reading them again for the first time” they still make sense.

This was an environment I set up over a night setting up certbot with nginx for testing a create-react-app project, which sounds trivial, but there are some definite caveats and pitfalls:

  1. I’m using it for a dev environment, which is the opposite of what certbot is intended for (but it does make some things easier, if you can get it to work in the first place…). You have to set it up with a sort of “this could be for production” mindset, as it requires a lot more bootstrapping than your average dev env.
  2. I’m using WSL2, which is just weird to configure coming from a bare-metal distro, and not well documented for what I’m attempting (I don’t think there are any other attempts documented on the net thus far – including disparate forum threads, github issues, reddit, etc.). I know Microsoft really tries, and there’s nothing else quite like WSL2, so they deserve some kudos for even creating this Linux compatibility layer, but… well… if you try it, you’ll see what I mean…
  3. The networking doesn’t ever really work right without manual intervention after each boot. It’s not that big of a deal, but it further belies how WSL is really lacking in proper integration with Windows. It works, but it’s hacky – it’s not ready for prime-time.

Certbot, nginx and CRA on Ubuntu 20.04 LTS WSL2 dev environment

There’s a really comprehensive guide about deploying CRA with nginx and certbot for HTTPS over here: https://coderrocketfuel.com/article/deploy-a-create-react-app-website-to-digitalocean

Can check against this article, but it does not cover WSL – so if you’re working w/ WSL it’s only good for some things.

some baseline info — env:

ISP: comcast residential cable (dynamic IP with namecheap dynamic DNS)
Home domain name: home.com
Current home gateway IP: (fake)
LAN subnet: (real)
WSL target IP: (why not)
Windows: 21h1 (bloody)

Also have a 256GB NVMe with Ubuntu 21.04 on it I can run through VMWare Workstation, but surprisingly WSL2 is more performant (although the VMware VM has a GUI, so apples and oranges…). I like the WSL integration, but this hairy config stuff is a little tiresome. Oh well, I’m in this far now…


└─ ▶ cat /etc/os-release && uname -a
VERSION="20.04.2 LTS (Focal Fossa)"
PRETTY_NAME="Ubuntu 20.04.2 LTS"
Linux wharfrat 5.4.72-microsoft-standard-WSL2 #1 SMP Wed Oct 28 23:40:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

make sure focal-updates/universe repo enabled in /etc/apt/sources.list:

└─ ▶ grep -v "#" /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu/ focal main restricted

deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted

deb http://archive.ubuntu.com/ubuntu/ focal universe 
deb http://archive.ubuntu.com/ubuntu/ focal-updates universe # <-- this is our repo here

deb http://archive.ubuntu.com/ubuntu/ focal multiverse
deb http://archive.ubuntu.com/ubuntu/ focal-updates multiverse

deb http://archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu/ focal-security main restricted
deb http://security.ubuntu.com/ubuntu/ focal-security universe
deb http://security.ubuntu.com/ubuntu/ focal-security multiverse

Universe repo certbot package maintainer’s info:

└─ ▶ apt show certbot
Package: certbot
Version: 0.40.0-1ubuntu0.1
Priority: extra
Section: universe/web
Source: python-certbot
Origin: Ubuntu
Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com
Original-Maintainer: Debian Let's Encrypt
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 51.2 kB
Provides: letsencrypt
Depends: python3-certbot (= 0.40.0-1ubuntu0.1), python3:any
Suggests: python3-certbot-apache, python3-certbot-nginx, python-certbot-doc
Breaks: letsencrypt (<= 0.6.0)
Replaces: letsencrypt
Homepage: https://certbot.eff.org/
Download-Size: 17.9 kB
APT-Sources: http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages

Description: automatically configure HTTPS using Let's Encrypt
The objective of Certbot, Let's Encrypt, and the ACME (Automated
Certificate Management Environment) protocol is to make it possible
to set up an HTTPS server and have it automatically obtain a
browser-trusted certificate, without any human intervention. This is
accomplished by running a certificate management agent on the web

This agent is used to:
Automatically prove to the Let's Encrypt CA that you control the website 

Obtain a browser-trusted certificate and set it up on your web server
Keep track of when your certificate is going to expire, and renew it

Help you revoke the certificate if that ever becomes necessary.

This package contains the main application, including the standalone
and the manual authenticators.

first, you’ll want to set up WSL2 so it can work in bridge mode. Create a bridge in Control Panel\Network and Internet\Network Connections by ctrl-clicking vEthernet (WSL) and Ethernet (or WIFI), right-clicking and selecting ‘create bridge

I took screenshots! Look at me go…:

Control Panel\Network and Internet\Network Connections
demonstration of making a network bridge in Windows
demonstration of making a network bridge in Windows

AFAIK you have to re-add WSL NIC in control panel every time you reboot (through the right-click -> properties menu) – trying to work around this, but haven’t found a solution yet as of 8am 6-5-2021

Then go to services.msc and disable Internet Connection Sharing (ics) – find it, right click, go to properties and select ‘disabled‘.

AFAIK this admin template is for domain networks only, so if in a domain env, try doing this:

Go to group policy editor and search for ‘ics‘ keyword, there will be only one administrative template. Enable it – it’ll disable internet connection sharing on your domain

The “enable disable of ics” administrative template, likely making my hs English teacher roll in her grave.
If Microsoft would have decided that all administrative templates enable a feature, they wouldn’t have to spend 10 paragraphs explaining whether the template turns something on or off…

TBH neither one of these methods really stopped ICS from being started with Ubuntu, but with the help of the ip commands (make a script!) it’s relatively painless to get back to a workable bridge network connection after reboot – just re-add WSL NIC in control panel to the network bridge, and re-run these ip commands (coming up…). I had varying success getting the ICS service to stop in services.msc – it doesn’t really seem to matter.

Next, open Ubuntu and run these commands – choose whatever IP and gateway you’d like/need:

sudo ip addr flush dev eth0
sudo ip addr add dev eth0
ip addr show dev eth0
sudo ip link set eth0 up
sudo ip route add default via dev eth0
ip route show
ping google.com

install ifupdown – and if you want, optional ifupdown-extra scripts:

sudo apt update && sudo apt install -y ifupdown [ifupdown-extra]

and add something like this to /etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static

Add these two lines using visudo so you can start nginx without a password:

# Allow members of group sudo to execute any command

%sudo   ALL=(ALL:ALL) ALL
%sudo ALL=NOPASSWD: /etc/init.d/nginx start
%sudo ALL=NOPASSWD: /etc/init.d/networking start

Get the path of Ubuntu.exe so you can create a task in task scheduler using Ubuntu UWP install – note, you can find the exact path in Task Manager while Ubuntu window is open by finding wsl.exe or Ubuntu.exe, right-clicking and going to details, the right-clicking on Ubuntu.exe and showing file location. It’ll be something like this:

"C:\Program Files\WindowsApps\CanonicalGroupLimited.UbuntuonWindows_2004.2021.222.0_x64__79rhkp1fndgsc\ubuntu.exe"

Then you can use it to run the service from Windows and set it up in Task Scheduler to start on login, boot, etc.:

"C:\Program Files\WindowsApps\CanonicalGroupLimited.UbuntuonWindows_2004.2021.222.0_x64__79rhkp1fndgsc\ubuntu.exe" run sudo /etc/init.d/nginx start

try restarting to see if it works … (open an ubuntu window and see if you have networking – your /etc/network/interfaces IP, etc. – save those ip commands in a script, just in case!…)

install nginx and certbot w/ certbot nginx plugin and docs:

sudo apt update && sudo apt install -y nginx certbot python3-certbot-nginx python-certbot-nginx-doc

Get loopback and ethernet card info – omit loopback (no need to do for loop) if only assessing outward-facing network, or plan to use certbot instead of openssl self-signed cert:

└─ ▶ for i in eth0 lo; do ip addr show dev $i; done

# output:

4: eth0: mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:ab:6b:b6 brd ff:ff:ff:ff:ff:ff
    inet brd scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:feab:6bb6/64 scope link
       valid_lft forever preferred_lft forever
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

WSL: this address is only available on your local machine (because it’s a NAT address). You want an address on your LAN.

Start ngnix – native ubuntu:

└─ ▶ sudo systemctl enable --now ngnix

Start ngnix – WSL – the old upstart way, but without upstart (see Windows Task Manager workaround – yes, you read that right):

└─ ▶ sudo /etc/init.d/nginx start

# output:
Starting nginx nginx [ OK ]

check ip address (either lo or eth0):

Browser: http://localhost


Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
. . .

set /etc/ngnix/sites-available/default settings:

sudo vim /etc/nginx/sites-available/default

set firewall for 80 and 443

( beyond scope – see firewall-cmd, ufw, iptables, ntables, bpf, etc. documentation )

copy virtual site portion of nginx default config

cd /etc/nginx/sites-available && cat default

# (selected and copied virtual section to file called subdomain.domain.name)

vim lindabnotary.home.com # name of dev site - I pasted default virt config here

sed -i 's/#//g' lindabnotary.home.com # get rid of all the comment leaders

vim lindabnotary.com # added a couple hashes for top comment, edited site name, added https port - example:

# File: /etc/nginx/sites-available/lindbnotary.home.com

cat /etc/nginx/sites-available/lindabnotary.home.com
# You can move that to a different file under sites-available/ and symlink that to sites-enabled/ to enable it.

server {
listen 80;
listen [::]:80 ipv6only=yes;

server_name lindabnotary.home.com;

   root /var/www/lindabnotary; ### end new bits
   index index.html;

   location / {
           try_files $uri $uri/ =404;

The nice thing about certbot, is you can create a super easy cursory config file like this and when you invoke the script it’ll take care of the rest of the config for you in an optimized fashion. All it has to be able to do is connect with your endpoint via http – with the correct domain name in nginx. Which brings me to the next subject…

I have Windows AD DNS, so I created a DNS entry for my local dev server:

DNS server RSAT –> create A + PTR records: lindabnotary.home.com @,

If you don’t have a DNS server, set up dnsmasq or similar (out of scope) – one easy way off the top of my head is to create a record in pfsense/opnsense to the eth0 ip addr

The pfsense alias thing will work for local dev, but you won’t be able to create a certbot instance through its automated cert creator unless you have an externally-accessible domain name (ala one purchased through a registrar, like namecheap, godaddy, etc.). So if this is your case, either quit while you’re ahead and experiment on your VPS, get a DN through namecheap and link it up to your home’s external IP real quick, or read up on how to generate a cert using certbot manually (or using openssl to create a self-signed cert).

certbot manual instructions (note: I have no idea if this actually works the way I’m hoping it would): https://certbot.eff.org/docs/using.html#manual

I’m noticing there’s also dns-standalone plugin which is presumably for bind, etc. but probs still needs an external DN.

create symlink to dev build:

└─ ▶ cd /var/www && ln -s /home/avery/node/lindabnotary/build ./lindabnotary

└─ ▶ ls -la
total 12
drwxr-xr-x 3 root root 4096 Jun 4 22:16 .
drwxr-xr-x 14 root root 4096 Jun 4 20:26 ..
drwxr-xr-x 2 root root 4096 Jun 4 20:26 html
lrwxrwxrwx 1 root root 35 Jun 4 22:16 lindabnotary -> /home/avery/node/lindabnotary/build

create symlink between sites-available and sites-enabled:

└─ ▶ sudo ln -s /etc/nginx/sites-available/lindabnotary.home.com/etc/nginx/sites-enabled/lindabnotary.home.com

(re)start nginx:

└─ ▶ sudo /etc/init.d/nginx start

# output: 
Starting nginx nginx

navigated to dev site lindabnotary.home.com in browser

Returned: “It works!” 😂😂😂

Then once the nginx server is serving the content locally, it’s time to set it up for access on the internet.

Necessity for web access – certbot requires internet access to your site in order to work. You must be able to get the dev server to respond to an http request on the internet. Therefore, you must open a firewall port to your dev server (at least temporarily) through external firewall/gateway, etc.

I did this in OPNsense. Here’s a thread/guide if you have questions: https://forum.opnsense.org/index.php?topic=6155.0

(Through web GUI):
Location: Firewall –> NAT –> Port Forward

  • Firewall: NAT: Port Forward
  • Interface: WAN
  • TcpVersion: IPV4
  • Protocol: TCP
  • Destination: WAN address
  • Destination port range: HTTP, HTTPS
  • Redirect Target IP
  • Pool options: default
  • Description: WSL on Dell Laptop WharfRat (for dev purposes)
  • Set local tag: (optional)
  • Match local tag: (if you have one)
  • Nat reflection: Enable
  • Filter rule exception: (Will autocreate when saved)

Then, now that the external firewall rule is enabled to allow access from the internet, get your gateway’s current external ip address:

└─ ▶ curl -s https://dnsleaktest.com | grep Hello | cut -d '>' -f2 | cut -d '<' -f1


And use the IP address to see if you can access your site from outside – There’s lots of ways you could do this, but I used my cell phone’s browser after turning off WIFI.

Real quick: If you mess your domain’s config up, you can do this to start over:

└─ ▶ sudo certbot delete --cert-name lindabnotary.home.com

OK, now for the moment of truth – run this command for certbot to automatically create a certificate and configuration:

└─ ▶ sudo certbot run --nginx -d lindabnotary.home.com

Another quick note: My namecheap dynamic DNS only allows one subdomain at a time, so I did not configure the www.lindabnotary.home.com DN. The subdomain ‘lindabnotary’ is already a wildcard – it’ll link to whatever server is presented through the firewall, not a specific server identified by the subdomain, like a “real” static DN.

But if you wanted to (or it would matter in your case, unlike mine) you could just string another ‘-d www.example.com‘ on the end (presumably, you can do as many -d‘s as you want).

Here’s the output – I selected 2 for redirecting all traffic to HTTPS:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Obtaining a new certificate
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/lindabnotary.home.com

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.

1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.

Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/lindabnotary.home.com

Congratulations! You have successfully enabled

You should test your configuration at:


Congratulations! Your certificate and chain have been saved at:
Your key file has been saved at:
Your cert will expire on 2021-09-03. To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the "certonly" option. To non-interactively renew all of
your certificates, run "certbot renew"
If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le

Note: Now nginx config has been altered by certbot – here’s what it looks like after running the installation command:

You can move that to a different file under sites-available/ and symlink that to sites-enabled/ to enable it. Don’t forget to restart nginx.

   server {
       server_name lindabnotary.home.com;

       root /var/www/lindabnotary;
       index index.html;

       location / {
           try_files $uri $uri/ =404;

   listen [::]:443 ssl ipv6only=on; # managed by Certbot
   listen 443 ssl; # managed by Certbot
   ssl_certificate /etc/letsencrypt/live/lindabnotary.home.com/fullchain.pem; # managed by Certbot
   ssl_certificate_key /etc/letsencrypt/live/lindabnotary.home.com/privkey.pem; # managed by Certbot
   include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
   ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

server {
    if ($host = lindabnotary.webtool.space) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
   listen 80;
   listen [::]:80 ipv6only=on;

   server_name lindabnotary.home.com;
return 404; # managed by Certbot

Troubleshooting: I noticed after I got the lock icon locally, I still couldn’t access my site remotely. For dev purposes, this might not be a big deal – I really just need to do development work locally, I’ll worry about external HTTPS access once I get it on the VPS.

Not having port 80 access and not being able to access 443 from the internet WILL prevent certbot from updating the certificate automatically, but I could just open up port 80 access again (instead of auto-redirect) in the /etc/nginx/sites-available/lindabnotary.home.com file for automatic updates to work.

Hopefully this is just a small issue like a firewall port not being open, or something – will need to go re-check my OPNsense config…

Further dev stipulations: Now that I have my .pem file from certbot, I can link to it while running the development server, by editing the project folder’s package.json start script to utilize the ssl cert+key.

Make a dir to store the .pem files in locally, and copy them over because we need to modify the permissions

mkdir ssl
sudo cp /etc/letsencrypt/live/lindabnotary.home.com/fullchain.pem ./ssl/fullchain.pem
sudo cp /etc/letsencrypt/live/lindabnotary.home.com/privkey.pem ./ssl/privkey.pem

These are wrong permissions for production, but must be able to execute as user other than root (prod perms: 0700 dir, 0600 each file, owned by root)

sudo chown -R root:root ./ssl
sudo chmod -R 755 ssl
sudo chmod 644 ssl/*.pem

# it’s possible you could change the owner and decrease the accessibility of the .pem files, have not tried. Play around with it and find out (chown -R user:user ./ssl, chmod -R 0700, chmod 0600 ./ssl/*.pem)

Ignore that, I tried this:

avery @ wharfrat ~/node/lindabnotary
└─ ▶ sudo chmod 0653 ./ssl/.pem avery @ wharfrat ~/node/lindabnotary └─ ▶ sudo chmod -R 0754 ssl avery @ wharfrat ~/node/lindabnotary └─ ▶ sudo chmod 0653 ./ssl/.pem
avery @ wharfrat ~/node/lindabnotary
└─ ▶ npm start

> Lindabnotary.com@0.1.0 start
> bash https-dev-script.sh

EACCES: permission denied, open '/home/avery/node/lindabnotary/ssl/fullchain.pem'

Just make the ssl dir owned root, perms readable all users – it works, this is not production.

You can embed all these variables into package.json, but I opted to make a separate script

filename: ~/projectfolder/https-dev-script.sh

#!/usr/bin/env bash

export HTTPS=true
export SSL_CRT_FILE=./ssl/fullchain.pem
export SSL_KEY_FILE=./ssl/privkey.pem 

react-scripts start

Make the script runnable:

chmod +x ./https-dev-script.sh

and then modified package.json to run the script:

  "scripts": {
    "start": "bash https-dev-script.sh", # this is our mod
    "build": "HTTPS=true react-scripts build",  # not sure if HTTPS=true does anything during build - look into
    "test": "react-scripts test", # FYI you cannot put comments in json files
    "eject": "react-scripts eject"

An aside: This bash script is not runnable outside of npm start (cannot just run ./https-dev-script.sh – needs to be invoked by npm to run react-scripts).

Easiest WSL2 Bridge Network (without Hyper-V Virtual Network Manager)

Burning a Wireless Router - YouTube
Grilling a Linksys. Delicious…

I am testing certbot locally with Create-React-App, but I’m using WSL2. Little did I know, the networking for Ubuntu in WSL2 is a little strange when coming from your typical systemd-network, NetworkManager, ifupdown, <insert your favorite networking API>

Apparently, the networking is taken care of in the lxssManager service (you can reveal it / restart it, etc. in task manager – ctrl-shift-esc -> services). This is because WSL doesn’t have an init system, unless you get some kind of hack to give you a systemd-ish responsive system – I’ve heard of something called ‘genie’, but I haven’t wanted to try it, since I’m trying to do WSL the “right” way 😂

Anywho, if you want an IP from your WSL installation that’s in the same subnet as your LAN, you’re out of luck, since the IP address supplied to your WSL instance is provided through NAT. Some threads I was scanning while looking for a solution talked about installing Hyper-V Virtual Network Manager, but wanting to keep things as light as possible, I have no interest in installing Hyper-V or any of its tools just to get a bridge network setup for WSL.

So I poked around the Network Connections in the Control Panel – this can be revealed by searching for Control in the start menu (it’s been getting progressively more hidden every release since the introduction of the UWP “Settings” app). Once you’re in there, go to Control Panel\Network and Internet\Network Connections (you can paste that line straight in the address bar) and look at what you’ve got. You should see something like this:

Control Panel Network Connections – quaint, isn’t it?

Although it’s likely you won’t have a “Network Bridge”… yet.

Here’s what to do:

Highlight your Ethernet (or WIFI) connection (or both) and the “vEthernet (WSL)” adapter (can select multiple by holding down CTRL while clicking on them), then right-click and select “Create Bridge” from the menu.

That’ll take a minute, but should leave you with this thing:

network bridge software adapter (Windows 10 control panel)

One of the threads I read said you have to restart after creating this thing, but that was not my experience. If you’re having weird networking issues you can’t figure out, you might want to give that a shot.

Then once you have it bridging your WSL adapter with your choice of wired-NIC or WIFI, head over to your bash prompt and set the adapter – note, my LAN’s subnet is, you’ll want to adjust that for whatever yours is:

 sudo ip addr flush dev eth0
 sudo ip addr add dev eth0
 ip addr show dev eth0
 sudo ip link set eth0 up
 sudo ip route add default via dev eth0
 ip route show
 ping google.com

So I’ll run through these real quick:

First you flush your old NAT ip settings, then you assign an address to your WSL adapter. I just picked one at random, but I’d definitely recommend checking to make sure you don’t have the IP address assigned first (try an IP scanner, like arp-scan, or something in Windows – note: I always get horribly paltry results with arp -a for some reason).

Give your WSL eth0 adapter the IP in CIDR notation, then set it to up so you can create a route for it. Create the route to your local gateway (e.g. or whatever yours is…)

Show that your address has been created, the default route has been created, then I like to ping a local machine like my gateway, then something that requires name resolution (here I used google).

Funny thing is, that actually worked before setting up the DNS resolver file, but you have to do that next. I don’t think it was pinging using ipv6, so that doesn’t explain why it worked – probably some weird Windows networking dichotomy, but sure enough, nothing else will really work in name resolution land beyond that, so it’s time to do the next step…

You’ll want to move the /etc/resolv.conf file to /etc/wsl.conf (unless you already have a wsl.conf file, of course – but it’s not created by default). I just invoked # mv /etc/resolv.conf /etc/wsl.conf since it already had the lines I had to add to wsl.conf in it, and I was about to replace it anyway…

Make your /etc/wsl.conf file look like this:

└─ ▶ cat /etc/wsl.conf
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
generateResolvConf = false

Then create an /etc/resolv.conf file using your favorite DNS servers (,,, etc.) I have local DNS, so here’s mine:

# echo 'nameserver
> nameserver
> search webtool.space' > /etc/resolv.conf

Run something like # apt update to make sure stuff’s resolving now. In my case it was all good.

Note: These ip commands are ephemeral, so you’ll probably want to do some research into how to making them more permenant. I literally put that first block of ip commands in a script to run for when I come back after a restart, because I’m almost 100% certain all this setup will be lost.

One really interesting caveat is that I noticed the Network Bridge in the Control Panel didn’t include the “vEthernet WSL” adapter when I restarted. The bridge was still there, but only had my Ethernet adapter in it, so I had to manually add the WSL adapter again manually in properties.

There’s apparently two config files – one’s the /etc/wsl.conf file we just created, the other is C:\Users\YourUserName\.wslconfig

Unfortunately, the /etc/wsl.conf file is pretty limited as to network config – I did add the generateHosts = false line in mine, because my /etc/hosts had a bunch of weird Windows-only subdomains created by programs that are totally unnecessary to have in Linux, but there’s a lot of other neat stuff in there related to selective case sensitivity, mounting different filesystems, etc. you might want to check out.

So I’m still searching for ways to make this permanent, but once I figure it out, I’ll come back and finish this post. Might have to send an issue to the devs over on GitHub, but I’ll get this working eventually. For now, adding WSL back in the bridge if I restart and running my script isn’t that big of a deal.

Update: This post is a small part of a comprehensive project I was working on. So far, I have posted my raw notes for others to scour if they want to try and re-create the environment: https://develmonk.com/2021/06/05/raw-notes-from-creating-a-local-dev-and-build-https-server-for-create-react-app-using-nginx-and-certbot/