VMware Upgrade renders Workstation 16.2.1 useless, thanks to Houdini-like disappearance of DLLs

Librarian replies to man inquiring about biography of famous escape artist, "I'm sorry sir, all of our books on Houdini have disappeared"
Picture depicts Houdinis as books disappearing from a library, while our problem depicts missing libraries themselves. Kinda makes you think, doesn’t it?

Hey,

I just came across an issue many people have been experiencing after upgrading VMware Workstation to 16.2.1 in Windows. It looks like the installer has a bug that deletes two necessary files by accident, and their absence prevents the programs from running, beginning with vmware-tray.exe, which is the precursor to all things Workstation.

The lawnmowered files are two dynamic link libraries (“DLLs”) responsible for encryption. Their names are libssl-1_1.dll and libcrypto-1_1.dll. Thankfully, I’ve managed to gather them from a reliable source and have re-packaged them for your convenience. Below is an archive containing both missing files for you to download, in hopes that you may use them to remedy this rank insipidity:

The organization responsible for storing and distributing DLL files such as these is named dll-files.com, located at https://www.dll-files.com in lovely Tilf AB, Sweden. Their website offers a surprisingly painless, hassle-free download (full disclosure: no affiliation).

I actually visited a few other sites looking for these DLL files before I found dll-files.com, but all the other sites “provided” were dubious “utilities” containing code that made my antivirus software blush a deep shade of “seriously?”.

Speaking of malware, everything I am providing today has been scanned with Malwarebytes, the best antivirus software in the biz (full disclosure: no affiliation), which reports these DLLs, and obligatory license + attribution document, are as squeaky-clean as a newborn antelope (devouring placenta does have advantages). I installed them on my laptop, the one I’m using now, and it’s been smoother sailing than vacationing on Velveeta.

Installing random files with these kinds of file names actually can be a little scary, considering all the ransomware thieves who were wreaking havoc across the US a couple years ago, but legitimate programs use cryptographic libraries just often as cyber-criminals, perhaps even more. So never you be a’ feared of them scary ol’ names, now, their bark a’ far worse than their soo-eey.

Come to think of it, in contrast to the wrath of cryptographic criminals a couple years ago, it seems that recently the media is more likely to associate “crypto” with blockchain currency. At least wealth is a more pleasant association than robbery… <dentist office music>

To install the files above after you download them, unzip the .7z file using 7zip archive software, and move the two .dll files contained therein to the folder located at C:\Program Files (x86)\VMware\VMware Workstation\<files go here>

I used an admin command prompt to move mine, but you could just as easily open your file explorer to drag-and-drop them, I’m sure. Once Meta gets off their behinds and releases the first telekinesis controller, we’ll all be able to will our files across our hard drives with sheer focus and determination, but until then it’s a flail across our mice and keyboards. Sorry to remind you we’re still savages with disappearing DLLs.

Mount Ubuntu 22.04 ZFS partitions using live ISO for disaster recovery

ZFS Send and Receive ·

My system runs ZFS and lately has been dropping to the initramfs / busybox prompt on boot. I had a hard time finding a fleshed-out guide on how to mount ZFS in a live environment for performing disaster recovery tasks like chroot and grub repair, so I thought I’d write something up.

My system was dropping to the busybox prompt after GRUB menu. I started experiencing the issue after a routine apt upgrade, I rebooted and wasn’t able to get any of my initramfs to boot. It seems a little strange, because usually the inability to boot will be limited to a new initramfs – e.g. an older version of the kernel will still have the ZFS drivers, or other necessary components to boot, while the newer versions (the ones just installed) will be lacking these necessary components for whatever reason.

First of all, burn yourself a copy of a live USB, and boot into it. Luckily, the newest version of Ubuntu (22.04 – Jammy Jellyfish) has the ZFS drivers and executables installed by default, unlike prior versions where you had to add the multiverse repo manually, download the packages, and enable the ZFS drivers using modprobe.

A peek at lsmod shows the ZFS drivers are indeed loaded, and lo-and-behold, there’s the zpool and zfs executables:

ubuntu@ubuntu:~$ lsmod | grep zfs
zfs                  3751936  29
zunicode              348160  1 zfs
zzstd                 487424  1 zfs
zlua                  155648  1 zfs
zavl                   20480  1 zfs
icp                   319488  1 zfs
zcommon               102400  2 zfs,icp
znvpair                94208  2 zfs,zcommon
spl                   122880  6 zfs,icp,zzstd,znvpair,zcommon,zavl

ubuntu@ubuntu:~$ which {zpool,zfs}
/usr/sbin/zpool
/usr/sbin/zfs

The drive I am diagnosing is the internal NVMe, so there’s no need to attach it. One question I had was how to mount the two pools, and in what order. By default, Ubuntu creates an rpool for the root partition, and a bpool for the boot partition.

Generally, on an EFI system, one would mount the root partition in a clean directory like /mnt first, and subsequently mount boot at /mnt/boot once it is provided by the previously mounted root partition, and then mount efi at /mnt/boot/efi once that’s provided by the boot partition. As you can see, the order of mounting these partitions is therefore of paramount importance, but as there are only 3 options, it’s not too complicated.

You’ll need to be root for basically all these commands. Using sudo su without a password will typically get you to a root prompt (#) in a live environment.

TL;DR – probably way more than you ever wanted to know about an lsblk device list:

First, we should identify the storage devices using lsblk -f (the -f flag includes the filesystem information, which is important for our purposes):

# lsblk -f
NAME FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0
     squash 4.0                                                    0   100% /rofs
loop1
     squash 4.0                                                    0   100% /snap/bare/5
loop2
     squash 4.0                                                    0   100% /snap/core20/1405
loop3
     squash 4.0                                                    0   100% /snap/snapd/15177
loop4
     squash 4.0                                                    0   100% /snap/snap-store/575
loop5
     squash 4.0                                                    0   100% /snap/gtk-common-themes/1534
loop6
     squash 4.0                                                    0   100% /snap/firefox/1232
loop7
     squash 4.0                                                    0   100% /snap/snapd-desktop-integration/10
loop8
     squash 4.0                                                    0   100% /snap/gnome-3-38-2004/99
sda  iso966 Jolie Ubuntu 22.04 LTS amd64
│                       2022-04-19-10-23-19-00                              
├─sda1
│    iso966 Jolie Ubuntu 22.04 LTS amd64
│                       2022-04-19-10-23-19-00                     0   100% /cdrom
├─sda2
│    vfat   FAT12 ESP   8D6C-A9F8                                           
├─sda3
│                                                                           
└─sda4
     ext4   1.0   writable
                        bb277d84-75cc-473b-b327-fd885d85889a   24.5G     0% /var/crash
                                                                            /var/log
zd0  btrfs              b6239f8a-058b-4a6c-8258-b9a7b50f6c23                
zd16                                                                        
└─zd16p1
     btrfs              d6074499-b9aa-47e0-a08a-58e27c73e771                
zd32 btrfs              c68aa9ca-933a-48cb-9adb-22fd6a8ca8c8                
zd48                                                                        
└─zd48p1
     btrfs              f52702bd-c805-4edc-87d1-6fb877ee6738                
nvme1n1
│                                                                           
├─nvme1n1p1
│    vfat   FAT32       B045-5C3B                                           
├─nvme1n1p2
│    swap   1           584b9b78-7d8d-4a5a-9263-d6f6a48adc6b                
├─nvme1n1p3
│    zfs_me 5000  bpool 11241115695889536197                                
└─nvme1n1p4
     zfs_me 5000  rpool 16130566787573079380                                
nvme0n1
│                                                                           
├─nvme0n1p1
│    vfat   FAT32       EC9D-0344                                           
├─nvme0n1p2
│                                                                           
├─nvme0n1p3
│    ntfs               A4EEBDB4EEBD7F5C                                    
└─nvme0n1p4
     ntfs               989EE7E99EE7BDBECode language: PHP (php)

OK, there’s a lot there, so what are we looking at? Well, the first 9 devices that say loop are snaps, since we’re on Ubuntu. Those are responsible for storing some of the programs being run by the OS. Each one gets their own virtual storage device, sometimes referred to as an “overlay”. They create a fair amount of clutter in our device list, but that’s about all. You can ignore them.

Then, /dev/sda is our copy of Ubuntu ISO we booted from – you can see how it says cdrom there, and iso9660 (the cdrom spec). It’s read-only, so we couldn’t do anything with it if we wanted to, and we don’t, so let’s move on…

There’s a device for log and crash log, so that’s kind of interesting. I imagine the live ISO makes those since you can’t write to the USB drive, seeing as the ISO is a virtual CD-ROM, and CD-ROMs are read-only. Then there’s a bunch of what are called “zvols” (the zd0, zd16, etc. devices – see those?). Those are devices created with ZFS that are isolated from the rest of the filesystem. zvols are virtual block devices you can use just like any other block device, but in this context they’re typically either formatted with a different filesystem, or mounted via iSCSI for block-level filesharing (filesystem-sharing?). You can see these ones say btrfs, they were actually created for use with container runtimes, namely podman and systemd-container, both of which support btrfs very well and ZFS either nominally or not at all.

Now we get to nvme1n1 – this is the first NVMe drive listed. Generally 0 would be listed first, but for some reason it’s listed second. n1 is the number of the drive (the second NVMe drive in the laptop), after that the partitions are listed as p1, p2, p3, and so on. Here’s the drive in isolation:

nvme1n1
│                                                                           
├─nvme1n1p1
│    vfat   FAT32       B045-5C3B                                           
├─nvme1n1p2
│    swap   1           584b9b78-7d8d-4a5a-9263-d6f6a48adc6b                
├─nvme1n1p3
│    zfs_me 5000  bpool 11241115695889536197                                
└─nvme1n1p4
     zfs_me 5000  rpool 16130566787573079380  

The canonical address for this drive is: /dev/nvme1n1p{1,2,3,4} . The /dev (device) folder, while not listed in this output, is important to reference, as the full path is required for mounting a partition. Typically one would only mount a single partition at a time, but you could conceivably chain them in a single command by using curly braces, as shown. This is not common, as you will probably need to mount different partitions in different locations (e.g. /mnt, /mnt/boot), and usually either in descending order, or with no pattern at all.

If you remember back at the start, I mentioned the rpool and bpool. These are seen on /dev/nvme1n1p4 and /dev/nvme1n1p3 respectively. If the disk were formatted in a block filesystem such as EXT4 (Ubuntu’s default filesystem), the root partition could be mounted by attaching /dev/nvme0n1p4 to an empty folder. The command would therefore be:

# mount /dev/nvme1n1p4 /mntCode language: PHP (php)

And then you’d be able to ls /mnt and see the files contained on your newly mounted root partition. E.g.:

# ls /mnt
Qogir  boot   dev  home  lib32  libx32  mnt  proc  run   snap  sys  usr
bin    cdrom  etc  lib   lib64  media   opt  root  sbin  srv   tmp  varCode language: PHP (php)

But this NVMe is formatted using ZFS. So what to do? That’s the process I was having difficulty finding that inspired this blog post.

End TL;DR – here’s the ZFS-specific stuff again:

First, after you confirm that you have your ZFS modules loaded by referencing your list of loaded kernel modules, and confirming that your ZFS executables are available in PATH (here’s the syntax again so you don’t have to scroll back):

# lsmod | grep zfs 
zfs                  3751936  29
zunicode              348160  1 zfs
zzstd                 487424  1 zfs
zlua                  155648  1 zfs
zavl                   20480  1 zfs
icp                   319488  1 zfs
zcommon               102400  2 zfs,icp
znvpair                94208  2 zfs,zcommon
spl                   122880  6 zfs,icp,zzstd,znvpair,zcommon,zavl

# which {zpool,zfs}
/usr/sbin/zpool
/usr/sbin/zfsCode language: PHP (php)

Here’s where it’s different than your typical mount. You use zpool to import rpool, but you need to mount it using an alternate root (at /mnt) – otherwise it’ll try to mount itself over your live environment! Then confirm that the import worked.

# zpool import -f rpool -R /mnt

# ls /mnt
Qogir  boot   dev  home  lib32  libx32  mnt  proc  run   snap  sys  usr
bin    cdrom  etc  lib   lib64  media   opt  root  sbin  srv   tmp  varCode language: PHP (php)

OK, that went well. You can see that now we have a /mnt/boot folder, which is boot inside rpool – that’s where initramfs lives, but they’re stored in the bpool. We needed that folder to be available to mount our bpool into. So, let’s import bpool into /mnt/boot as an alternate root (if we didn’t, it’d try and overwrite our currently mounted /boot partition:

# zpool import -f bpool -R /mnt/boot

# ls /mnt/boot
config-5.15.32-xanmod1       memtest86+_multiboot.bin
config-5.15.34-xanmod1       System.map-5.15.32-xanmod1
config-5.15.36-xanmod1       System.map-5.15.34-xanmod1
config-5.17.0-xanmod1        System.map-5.15.36-xanmod1
config-5.17.1-xanmod1        System.map-5.17.0-xanmod1
config-5.17.3-xanmod1        System.map-5.17.1-xanmod1
config-5.17.5-xanmod1        System.map-5.17.3-xanmod1
config-5.17.9-xanmod1        System.map-5.17.5-xanmod1
config-5.17.9-xanmod1-x64v2  System.map-5.17.9-xanmod1
efi                          System.map-5.17.9-xanmod1-x64v2
grub                         vmlinuz
initrd.img                   vmlinuz-5.15.32-xanmod1
initrd.img-5.15.32-xanmod1   vmlinuz-5.15.34-xanmod1
initrd.img-5.15.34-xanmod1   vmlinuz-5.15.36-xanmod1
initrd.img-5.17.0-xanmod1    vmlinuz-5.17.0-xanmod1
initrd.img-5.17.1-xanmod1    vmlinuz-5.17.1-xanmod1
initrd.img-5.17.3-xanmod1    vmlinuz-5.17.3-xanmod1
initrd.img-5.17.5-xanmod1    vmlinuz-5.17.5-xanmod1
initrd.img.old               vmlinuz-5.17.9-xanmod1
memtest86+.bin               vmlinuz-5.17.9-xanmod1-x64v2
memtest86+.elf               vmlinuz.oldCode language: PHP (php)

That looks like a bunch of initramfs files to me! Good, so that means those kickstarter runtimes that load from grub are available.

If you look in that list, you’ll also see both efi and grub folders. Both of those are empty and waiting for storage to be attached. The efi partition lives in the first partition of the same NVMe drive, and is formatted with FAT, and grub is a bind-mount (you can see it in /etc/fstab):

# mount -t msdos /dev/nvme1n1p1 /mnt/boot/efi

Can also use UUID from lsblk if prefer (just use one or other, not both): 
# mount -t msdos UUID=B045-5C3B /mnt/boot/efi

# ls /mnt/boot/efi
efi  grub  system~1  (confirm it's mounted)

# grep grub /mnt/etc/fstab
/boot/efi/grub	/boot/grub	none	defaults,bind	0	0
(we'll bind-mount this in next step)Code language: PHP (php)

Then you’ll want to mount a few system folders inside your drive’s filesystem so you can access them inside the chroot (required for things to work OK):

# for i in proc dev sys dev/pts; do mount -v --bind /$i /mnt/$i; done

mount: /proc bound on /mnt/proc.
mount: /dev bound on /mnt/dev.
mount: /sys bound on /mnt/sys.
mount: /dev/pts bound on /mnt/dev/pts.

# mount -v --bind /mnt/boot/efi/grub /mnt/boot/grub
mount: /mnt/boot/efi/grub bound on /mnt/boot/grub.Code language: PHP (php)

chrooting”: Now that all 3 partitions are mounted together in a cohesive filesystem tree, and you’ve got all your necessary bind mounts, one of the most effective ways to diagnose issues as if you’re running the affected disk, is to chroot into the filesystem. Run # chroot /mnt and now you’ll see /mnt as / (root), and you can run your programs as if you booted the computer using that drive (from the terminal, anyway):

# chroot /mnt

# apt update (failed)

# cd /etc
# ls -la resolv.conf
lrwxrwxrwx 1 root root 39 Feb 17 12:09 resolv.conf -> ../run/systemd/resolve/stub-resolv.confCode language: PHP (php)

If your network connection fails inside the chroot like mine did, go to /etc and delete resolv.conf if it’s a symlink to systemd-resolved (as shown above). Then point /etc/resolv.conf to a known good dns forwarder (e.g. 1.1.1.1, 8.8.8.8, etc.)

# echo 'nameserver 8.8.8.8' > resolv.conf

# apt update (works)

# apt list --installed | grep dkms

dkms/jammy,now 2.8.7-2ubuntu2 all [installed,automatic]
zfs-dkms/jammy-proposed,now 2.1.4-0ubuntu0.1 all [installed]Code language: PHP (php)

I was really hoping zfs-dkms got uninstalled somehow, because I thought that might have been why my initramfs files didn’t have zfs modules. So unfortunately I still have to keep looking to figure out what’s wrong…

Note, you’ll probably see this error a lot, but it’s safe to ignore:

ERROR couldn't connect to zsys daemon: connection error: desc = "transport: Error while dialing dial unix /run/zsysd.sock: connect: connection refused" 

Let’s try upgrading the packages and see what shakes out:

# apt upgrade 

The following packages were automatically installed and are no longer required:
  linux-headers-5.15.32-xanmod1 linux-headers-5.15.34-xanmod1
  linux-headers-5.15.36-xanmod1 linux-headers-5.17.0-xanmod1
  linux-headers-5.17.1-xanmod1 linux-headers-5.17.3-xanmod1
  linux-headers-5.17.5-xanmod1 linux-image-5.15.32-xanmod1
  linux-image-5.15.34-xanmod1 linux-image-5.15.36-xanmod1
  linux-image-5.17.0-xanmod1 linux-image-5.17.1-xanmod1
  linux-image-5.17.3-xanmod1 linux-image-5.17.5-xanmod1
Use 'sudo apt autoremove' to remove them.Code language: PHP (php)

That was … interesting … and then the issue presented itself next while I ran apt autoremove:

Setting up linux-image-5.17.9-xanmod1 (5.17.9-xanmod1-0~git20220518.d88d798) ...
 * dkms: running auto installation service for kernel 5.17.9-xanmod1     [ OK ] 
update-initramfs: Generating /boot/initrd.img-5.17.9-xanmod1
<span style="text-decoration: underline;"><strong>zstd: error 25 : Write error : No space left on device (cannot write compressed</strong> </span>
block) 

(emphasis added)Code language: HTML, XML (xml)

bpool has no space left. That’s almost certainly the problem. I’m going to remove a couple kernels and rebuild all my initramfs, that ought to do it. I’m also noticing my bpool is full of snapshots. List current snapshots with this first command, and then destroy them with the second one:

// This lists the snapshots:
# zfs list -H -o name -t snapshot | grep bpool

...auto-snapshots look like pool/BOOT/ubuntu_pd3ehl<strong>@autozsys_xxxx</strong>, 
<strong>snapshots have @ symbol - no @ symbol, not a snapshot, don't delete it!</strong>

// This destroys the snapshots:
# zfs list -H -o name -t snapshot | grep bpool | xargs -n1 zfs destroy -r 
What this does:
(list only snapshots by full name) | (list only bpool) | (delete by ea line)
It's the same as what's above, but with the delete command, destroy. 

Make sure you understand what's going on with this command, as you can delete stuff you don't want to <em>really</em> easily. Please be careful.  Code language: PHP (php)

… looks pretty good to me – much more tidy:

# ls /boot
config-5.15.0-33-generic       memtest86+.elf
config-5.15.40-xanmod1-tt      memtest86+_multiboot.bin
efi                            System.map-5.15.0-33-generic
grub                           System.map-5.15.40-xanmod1-tt
initrd.img                     vmlinuz
initrd.img-5.15.0-33-generic   vmlinuz-5.15.0-33-generic
initrd.img-5.15.40-xanmod1-tt  vmlinuz-5.15.40-xanmod1-tt
initrd.img.old                 vmlinuz.old
memtest86+.binCode language: PHP (php)

Install some generic kernel to make sure you have one available, check that zfs-initramfs is installed if all you’re going to use is generic kernel (or zfs-dkms if using xanmod, other 3rd-party kernel). E.g. I got rid of my xanmod kernels just so I wouldn’t have to deal with building custom dkms modules:

# apt list --installed | grep xanmod

linux-headers-5.15.40-xanmod1-tt/unknown,now 5.15.40-xanmod1-tt-0~git20220515.867e3cb amd64 [installed,automatic]
linux-image-5.15.40-xanmod1-tt/unknown,now 5.15.40-xanmod1-tt-0~git20220515.867e3cb amd64 [installed,automatic]
linux-xanmod-tt/unknown,now 5.15.40-xanmod1-tt-0 amd64 [installed]
xanmod-repository/unknown,now 1.0.5 all [installed]

# apt remove linux-headers-5.15.40-xanmod1-tt linux-image-5.15.40-xanmod1-tt xanmod-repository linux-xanmod-tt zfs-dkms
 . . . 
The following packages will be REMOVED:
  linux-headers-5.15.40-xanmod1-tt linux-image-5.15.40-xanmod1-tt
  linux-xanmod-tt xanmod-repository zfs-dkms
Do you want to continue? [Y/n] 
 . . .
# apt autoremove -y

... install a couple kernels...

# apt install -y linux-{image,headers}-5.15.0-28-generic linux-{image,headers}-5.15.0-33-generic

 . . . using versions that are most current & 2nd most current now . . . 
 Code language: PHP (php)

Then update all the initramfs one last time, just in case. I’ll probably re-install grub, too, just bc, but one thing at a time…

# update-initramfs -uvk all  

. . . lots of output . . . that's how you know it's working . . . Code language: PHP (php)

Let’s re-install grub and run update-grub

# grub-install --bootloader-id=ubuntu --recheck --target=x86_64-efi --efi-directory=/boot/efi --no-floppy

Installing for x86_64-efi platform.
grub-install: warning: EFI variables cannot be set on this system.
grub-install: warning: You will have to complete the GRUB setup manually.
Installation finished. No error reported.Code language: PHP (php)

When you get this error, it just means you can’t set the UEFI boot order while you’re in a chroot. I also like to run update-grub for good measure (this is grub2-mkconfig -o /boot/grub/grub.cfg on most other systems if that’s more familiar sounding to you). Update-grub rebuilds the entries in your grub menu, along with their parameters detailed in /etc/default/grub.

Speaking of which, you can always take a peek at /etc/default/grub before you run this command – just in case.

# which update-grub
/usr/sbin/update-grub

# cat /usr/sbin/update-grub

// update-grub:
#!/bin/sh
set -e
exec grub-mkconfig -o /boot/grub/grub.cfg "$@"

# update-grub
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: vmlinuz-5.15.0-33-generic in rpool/ROOT/ubuntu_pd3ehl
Found initrd image: initrd.img-5.15.0-33-generic in rpool/ROOT/ubuntu_pd3ehl
Found linux image: vmlinuz-5.15.0-28-generic in rpool/ROOT/ubuntu_pd3ehl
Found initrd image: initrd.img-5.15.0-28-generic in rpool/ROOT/ubuntu_pd3ehl
Found linux image: vmlinuz-5.15.0-33-generic in rpool/ROOT/ubuntu_pd3ehl@autozsys_yg50xc
 . . . snapshot menu entries . . . Code language: PHP (php)

Now leave the chroot now, remove the system folder redirects and bind mounts, and reboot, like so:

# exit

# for i in proc dev/pts dev sys boot/grub; do umount -v /mnt/$i; done
umount: /mnt/proc unmounted
umount: /mnt/dev/pts unmounted
umount: /mnt/dev unmounted
umount: /mnt/sys unmounted
umount: /mnt/boot/grub unmounted

# umount -v /dev/nvme1n1p1
umount: /mnt/boot/efi (/dev/nvme1n1p1) unmounted

# zpool export bpool

# zpool export rpoolCode language: PHP (php)

One last quick thing you can do before rebooting is check out efibootmgr and see which order your system will start up in. This is a little easier and more predictable, as you can make sure you boot from the right efi file, rather than mashing the startup menu button to make sure it loads the correct disk / efi.

Some stuff I was messing with trying cover all the bases. efibootmgr reference: https://wiki.archlinux.org/title/GRUB/EFI_examples#Asus

# efibootmgr -v
Boot0000* ubuntu	HD(1,GPT,544a9120-eef7-4aae-8311-cd6ca6929213,0x800,0x100000)/File(\EFI\ubuntu\shimx64.efi)
 . . . 
# efibootmgr -B Boot0000 -b 0

# efibootmgr --create /dev/nvme1n1 --part 1 --write-signature --loader /EFI/GRUB/grubx64.efi --label "GRUB" --verbose
BootCurrent: 0002
Timeout: 0 seconds
BootOrder: 0000,0001,0002
Boot0001* UEFI: Samsung SSD 980 1TB, Partition 1	HD(1,GPT,6afa5e93-54a5-4628-978f-313a0dcfe27b,0x800,0xfa000)/File(\EFI\Microsoft\Boot\bootmgfw.efi)..BO
Boot0002* UEFI: Samsung Flash Drive DUO 1100, Partition 2	PciRoot(0x0)/Pci(0x14,0x0)/USB(16,0)/HD(2,GPT,a09db2b8-b5f6-43ae-afb1-91e0a90189a1,0x6cc954,0x2130)..BO
Boot0003  Windows Boot Manager	HD(1,GPT,6afa5e93-54a5-4628-978f-313a0dcfe27b,0x800,0xfa000)/File(\EFI\Microsoft\Boot\bootmgfw.efi)WINDOWS.........x...B.C.D.O.B.J.E.C.T.=.{.9.d.e.a.8.6.2.c.-.5.c.d.d.-.4.e.7.0.-.a.c.c.1.-.f.3.2.b.3.4.4.d.4.7.9.5.}....................
Boot0000* GRUB	HD(1,GPT,a09db2b8-b5f6-43ae-afb2-91e0a90189a1,0x40,0x6cc914)/File(\EFI\GRUB\grubx64.efi)/dev/nvme1n1Code language: PHP (php)

A troubleshooting tip: If you have issues using the pool names with zpool for some reason, the UUIDs are listed in lsblk. While technically interchangeable, the UUID can coax some commands into to working correctly when the name can’t.

If it doesn’t boot from the ZFS drive again, boot it into the live ISO and go through everything all over … 😉 Good luck!!

dpdk 22.03 rpm packages for Fedora 36: a dependency of openvswitch

open vswitch with DPDK: architecture and performance - ppt download

I meant to post this over a month ago, but got sidetracked, so I’m coming in a little late. Unfortunately, it looks like even though Fedora 36 has officially been released, the dpdk 22.03 rpms still aren’t available.

Back when I compiled these, I realized there were no official dpdk 22.03 rpms available in the yum repos, despite being required for openvswitch. So I compiled them so I could install openvswitch for use with virt-manager.

Dpdk on Fedora has an official maintainer, they probably just got sidetracked themselves – I can relate. But I had these packages already and wanted to help other people install openvswitch, so I posted the resulting packages in a github repo case other people want to download it.

So if you (like me) want to run openvswitch on your fancy new officially-released non-beta Fedora 36 workstation (or server, or silverblue, kinoite or iot using rpm-ostree), and you don’t want to wait or downgrade, you’d have to either have to compile dpdk 22.03 for yourself, or now you can download them from my repo.

I’ve got all the instructions to go through the build process if you’d like to get your hands dirty with compilation (it’s pretty straightforward): https://github.com/averyfreeman/dpdk-2203-for-fedora36

The .rpm files are also there if you don’t care for all the fuss of compilation. You can use them as openvswitch dependencies, just install them before you try and install openvswitch using dnf.

I should have made this post sooner so people knew the rpms were available, but at least I put a note or two in on reddit in a couple key places (now that I think about it, probably just /r/fedora). Nothing beats the officialism of posting a fancy notice on your wordpress blog, though, amirite?

Speaking of which, I think I will be looking at a way of integrating this blog with my repo – I’m hoping I can figure out a way to produce and update posts on wordpress automatically by creating a gist or a new repo. Then the two would be more tightly integrated, and could avoid this whole getting sidetracked issue…

The less people get sidetracked, the sooner they use software…

Run Windows Store (UWP) App on Startup – the EASY way

TL;DR: drag the icon from the shell:AppsFolder over to the shell:startup folder!

I am building a dedicated TV viewing VM for HDHomeRun View using Windows IoT 2021 LTSC, so I can watch TV while using a computer running Linux with the smallest possible KVM-QEMU VM I can possibly put together. That’s because, unfortunately, the app I have to use is only available in Windows, and only through the Windows Store on top of that, which in the past has created an additional layer of potential configuration difficulties due to app sandboxing, obscured location paths, etc.

Well, thankfully starting UWP apps (also called “Windows Store” apps, but herein will be referred to as UWP apps) have gotten a lot easier in newer versions of Windows. I’m not actually sure of this because I haven’t looked into any changelogs detailing new Windows shell or explorer features, but I’m assuming because I’ve gone through all sorts of trouble to get them to start up automatically in the past (this HDHomeRun one in particular), but now it seems ridiculously easy given what I had previously gone through.

I was reading through this article on how to start a UWP app on startup from HowToGeek.com. It’s a good article with lots of interesting points, but it seemed unnecessarily difficult to me. Between it, and this question answered on superuser.com, I came up with the easiest possible way to start a UWP app on startup.

  • In Windows File Explorer (the manilla folder icon), type shell:AppsFolder in the location bar and hit enter. This will bring up a list of icons, “regular” Windows applications in addition to UWP (store) apps.
  • Navigate through the icons until you find the UWP app you’re looking for.
  • In a separate Explorer window, type shell:startup and hit enter.
  • Drag the UWP app you want to start during startup into the shell:startup folder. It might ask you if you want to create a shortcut in that folder (hit yes).
  • Log out and back in to make sure it worked.

This worked for me on the first try. The hardest thing I did was remove the - shortcut string that was auto-appended to the icon. It really doesn’t get much easier than that.

In the past, I’d gone through all sorts of trouble figuring out which folder holds the app – they’re all in a hidden folder C:\Program Files\WindowsApps with folders named things like EF712BA7.HDHomeRunDVR_1.1.345.0_x64__23nna27hyxhag only accessible from administrative shell, isn’t that fun? – getting the application name and path from the AppManifest.xml file, and then creating a batch file for the startup script for the UWP app to be started with C:\Windows\System32\cmd.exe /C:, etc.

As I said, I’m not sure if using UWP apps is just getting easier on newer versions of Windows, but this is pretty darn convenient. If I was just doing something unnecessary in the past, it sure was a lot of trouble.

If you’re interested in running Windows Store for the apps you can’t find anywhere else on LTSC, check out this github repo – says last updated 3 years ago, but the script has worked for me on newer versions of LTSC for me just fine: https://github.com/kkkgo/LTSC-Add-MicrosoftStore

Recent posts to message boards

I haven’t had time to write lately, but I have made a few notable posts on message boards I thought I’d include here for reference:

https://www.truenas.com/community/threads/adding-slog-freenas-will-not-boot-with-m-2-optane-800p-passthrough-in-esxi-6-7-vm.79206/#post-674140

https://communities.vmware.com/t5/VMware-Workstation-Pro/Convert-hard-drive-into-VMWare-virtual-machine/m-p/2885130#M173194

Do you use both Visual Studio Code and Sublime Text 3?

Stripe Elements with Laravel (and Vue) - YouTube

If so, wouldn’t you like both their keyboard shortcuts to be consistent?

I personally am using both Visual Studio Code and Sublime Text 3 on my main dev laptop. Once I got used to all the nice features in a pseudo-IDE text editor like VS Code, it became hard to go back to editing things like .conf files in editors like vim and notepad. Wrapping text with parenthesis, quotes or brackets, simply by highlighting the word and hitting the opening character is probably my favorite feature, but things like duplicating lines, moving lines up and down, etc. are all very nice ways to save some time.

However, once my VS Code setup had enough formatting extensions like Prettier, or other handlers for package managers like npm, version control with git, color manager, gistpad, python, markdown parser, :emoji sense:, etc. etc. load times have gotten longer, and the editor feels less up to the task of opening a little .conf or .sh file just to do a little revision.

That’s where Sublime Text comes in. For one thing, it’s smaller. The fact that it doesn’t take up as much room on my screen makes it much more convenient for basic text-editing tasks. It also loads faster without so many plugins (I try to keep it down to only one or two). But it still has essentially all of the really awesome interactive qualities of VS code.

The only thing that ever really irks me is when I try to do something the way I’m used to in vscode with the keyboard and it does something else, or doesn’t work at all.

Luckily, when the keyboard shortcuts are inconsistent, ST3 is really easy to extend. I’ve forked a GitHub user’s keyboard shortcuts who started mapping ST3’s keyboard mapping and added a few of my own – for example, alt+z for enabling word wrap didn’t even have a keyboard shortcut in ST3, but now it can (!).

Whenever I run into another inconsistency, I’ll add it and push it to GH. If you’d like to see (or use) it, it’s located here: https://github.com/averyfreeman/vscode-keybindings-for-sublime

Creating a local HTTPS server for Create-React-App using nginx and certbot in WSL2: Works for BOTH npm start AND npm build

How Certbot and Letsencrypt Work (DNS and SNI-TLS automation) - Magic of  Security

Note: This article has some interesting WSL-centric material, like how to:

  • Set up bridge networking without Hyper-V
  • Find the UWP Ubuntu executable to run Linux commands outside of a WSL terminal
  • Create automatic startup tasks without systemd init system
  • Expose your (domain-enabled) dev environment to certbot/letsencrypt for automatic issuance of SSL certificates
  • Use nginx proxy for react

So even if you’re not doing the exact thing I set out to do (create an HTTPS dev & build preview server), you might still find these notes useful.

Update 6/11/2021: While the notes in this article basically work, WSL is pretty quirky for doing backend development. There are many strange variables that have no parallel in a stock Ubuntu installation.

Also, bridge.sys kept KP-ing my laptop, but I did manage to stop BSODs by replacing the Dell-released driver with the newest stock Intel driver. But I have to say, there’s nothing more frustrating/demoralizing than having the system crash while trying to write code. Hopefully you won’t have the same issue, I will leave this up in case it helps anyone with any of these steps, but I cannot in good conscience recommend it as a development platform. It’s OK, just know the risks!

If you are looking for a platform to develop software you plan to deploy to an Ubuntu VPS, I recommend using a virtual machine. I have taken to doing front-end dev in WSL and copying anything that needs additional tooling/config to a “real” Ubuntu VM *first*, getting it ready on that, and *then* pushing it to VPS.

Interestingly, if you run Alpine on your web server, WSL might actually be more suitable for backend tooling configuration, as Alpine uses OpenRC rather than systemd as its init system – it’s probably more likely that you could use it in the WSL environment similarly to how you’d run it bare metal. I have not verified this first-hand, this guess is purely informed speculation from using Alpine extensively for various servers and containers – but never for WSL. Try it, find out, and message me!

Examples of major weird / unparalleled issues I ran into re: using WSL for backend tooling are (from orig post):

  • Windows controls all the networking. That makes setting stuff up that requires networking really strange. In addition, the bridge setup I enabled has needed to be re-adopted manually after every Windows reboot, the WSL vNIC won’t stay configured to stay in the bridge persistently.
  • There’s no actual init system. That means no systemd for enabling / starting / stopping software. /etc/init.d/$INITSCRIPT is present, but has no upstart control, so =>
  • All “init” jobs are handled by Windows Task scheduler – either that, or you just don’t start scripts automatically (unless there’s another method I’m not aware of)
  • MS’s software L2 bridge (shown as Network Bridge adapter in Network Connection) which uses bridge.sys has a high propensity to cause kernel panics (and thus, BSODs) on my system. I’m using a Dell Precision 7730 w/ Intel i219-LM network adapter w/ driver version 12.18.9.10 on 19043.1110. If you’ve got something different, good luck to you. If your setup is quite similar, brace yourself for BSODs.

    After updating my I219-LM ethernet adapter driver to the one available on the Intel website (instead of the one released by Dell), I have managed to run this configuration for the last day and a half and haven’t had any KPs from bridge.sys. That’s a new record!

    The new driver is version 12.19.1.37, listed as v26.3 on Intel’s website. Note: The ProSet (software) and driver are in separate packages, ProSet archive does not install driver. Hoping it holds, but feeling pretty confident about it working so far. Would be nice if these computer vendors would support their hardware longer than a couple years.

I still like WSL2 for the additional flexibility it gives me in Windows, but it’s been “liberating” to recognize its deficiencies and put aside trying to get it to do everything I want – it’s just not there yet. The sooner I realized that, the sooner I stopped trying to wrap its alien configuration methods to purposes for which they are ill suited, and got back to coding.

However, if you want to try to get it to do stuff it’s not “supposed” to do, I left these notes up in the hope that someone else would come along and build on what I learned after several days of frustrating experiences.

Orig post:

This is a heavy post. This is the kind of thing I do when I’m working through a tough problem – well organized notes are a must. Obviously, parts of this will make more sense to me than you, but I have the added benefit (shortcoming?) of having trouble making sense of my notes when I come back to them, so I try to make them as complete as possible the first time so hopefully if it feels like I’m “reading them again for the first time” they still make sense.

This was an environment I set up over a night setting up certbot with nginx for testing a create-react-app project, which sounds trivial, but there are some definite caveats and pitfalls:

  1. I’m using it for a dev environment, which is the opposite of what certbot is intended for (but it does make some things easier, if you can get it to work in the first place…). You have to set it up with a sort of “this could be for production” mindset, as it requires a lot more bootstrapping than your average dev env.
  2. I’m using WSL2, which is just weird to configure coming from a bare-metal distro, and not well documented for what I’m attempting (I don’t think there are any other attempts documented on the net thus far – including disparate forum threads, github issues, reddit, etc.). I know Microsoft really tries, and there’s nothing else quite like WSL2, so they deserve some kudos for even creating this Linux compatibility layer, but… well… if you try it, you’ll see what I mean…
  3. The networking doesn’t ever really work right without manual intervention after each boot. It’s not that big of a deal, but it further belies how WSL is really lacking in proper integration with Windows. It works, but it’s hacky – it’s not ready for prime-time.

Certbot, nginx and CRA on Ubuntu 20.04 LTS WSL2 dev environment

There’s a really comprehensive guide about deploying CRA with nginx and certbot for HTTPS over here: https://coderrocketfuel.com/article/deploy-a-create-react-app-website-to-digitalocean

Can check against this article, but it does not cover WSL – so if you’re working w/ WSL it’s only good for some things.

some baseline info — env:

ISP: comcast residential cable (dynamic IP with namecheap dynamic DNS)
Home domain name: home.com
Current home gateway IP: 67.100.93.10 (fake)
LAN subnet: 192.168.1.0/24 (real)
WSL target IP: 192.168.1.244 (why not)
Windows: 21h1 (bloody)

Also have a 256GB NVMe with Ubuntu 21.04 on it I can run through VMWare Workstation, but surprisingly WSL2 is more performant (although the VMware VM has a GUI, so apples and oranges…). I like the WSL integration, but this hairy config stuff is a little tiresome. Oh well, I’m in this far now…

WuSLbuntu:

└─ ▶ cat /etc/os-release && uname -a
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Linux wharfrat 5.4.72-microsoft-standard-WSL2 #1 SMP Wed Oct 28 23:40:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

make sure focal-updates/universe repo enabled in /etc/apt/sources.list:

└─ ▶ grep -v "#" /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu/ focal main restricted

deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted

deb http://archive.ubuntu.com/ubuntu/ focal universe 
deb http://archive.ubuntu.com/ubuntu/ focal-updates universe # <-- this is our repo here

deb http://archive.ubuntu.com/ubuntu/ focal multiverse
deb http://archive.ubuntu.com/ubuntu/ focal-updates multiverse

deb http://archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu/ focal-security main restricted
deb http://security.ubuntu.com/ubuntu/ focal-security universe
deb http://security.ubuntu.com/ubuntu/ focal-security multiverse

Universe repo certbot package maintainer’s info:

└─ ▶ apt show certbot
Package: certbot
Version: 0.40.0-1ubuntu0.1
Priority: extra
Section: universe/web
Source: python-certbot
Origin: Ubuntu
Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com
Original-Maintainer: Debian Let's Encrypt
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 51.2 kB
Provides: letsencrypt
Depends: python3-certbot (= 0.40.0-1ubuntu0.1), python3:any
Suggests: python3-certbot-apache, python3-certbot-nginx, python-certbot-doc
Breaks: letsencrypt (<= 0.6.0)
Replaces: letsencrypt
Homepage: https://certbot.eff.org/
Download-Size: 17.9 kB
APT-Sources: http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages

Description: automatically configure HTTPS using Let's Encrypt
The objective of Certbot, Let's Encrypt, and the ACME (Automated
Certificate Management Environment) protocol is to make it possible
to set up an HTTPS server and have it automatically obtain a
browser-trusted certificate, without any human intervention. This is
accomplished by running a certificate management agent on the web
server.

This agent is used to:
Automatically prove to the Let's Encrypt CA that you control the website 

Obtain a browser-trusted certificate and set it up on your web server
Keep track of when your certificate is going to expire, and renew it

Help you revoke the certificate if that ever becomes necessary.

This package contains the main application, including the standalone
and the manual authenticators.

first, you’ll want to set up WSL2 so it can work in bridge mode. Create a bridge in Control Panel\Network and Internet\Network Connections by ctrl-clicking vEthernet (WSL) and Ethernet (or WIFI), right-clicking and selecting ‘create bridge

I took screenshots! Look at me go…:

Control Panel\Network and Internet\Network Connections
demonstration of making a network bridge in Windows
demonstration of making a network bridge in Windows

AFAIK you have to re-add WSL NIC in control panel every time you reboot (through the right-click -> properties menu) – trying to work around this, but haven’t found a solution yet as of 8am 6-5-2021

Then go to services.msc and disable Internet Connection Sharing (ics) – find it, right click, go to properties and select ‘disabled‘.

AFAIK this admin template is for domain networks only, so if in a domain env, try doing this:

Go to group policy editor and search for ‘ics‘ keyword, there will be only one administrative template. Enable it – it’ll disable internet connection sharing on your domain

The “enable disable of ics” administrative template, likely making my hs English teacher roll in her grave.
If Microsoft would have decided that all administrative templates enable a feature, they wouldn’t have to spend 10 paragraphs explaining whether the template turns something on or off…

TBH neither one of these methods really stopped ICS from being started with Ubuntu, but with the help of the ip commands (make a script!) it’s relatively painless to get back to a workable bridge network connection after reboot – just re-add WSL NIC in control panel to the network bridge, and re-run these ip commands (coming up…). I had varying success getting the ICS service to stop in services.msc – it doesn’t really seem to matter.

Next, open Ubuntu and run these commands – choose whatever IP and gateway you’d like/need:

sudo ip addr flush dev eth0
sudo ip addr add 192.168.1.244/24 dev eth0
ip addr show dev eth0
sudo ip link set eth0 up
sudo ip route add default via 192.168.1.1 dev eth0
ip route show
ping 192.168.1.1
ping google.com

install ifupdown – and if you want, optional ifupdown-extra scripts:

sudo apt update && sudo apt install -y ifupdown [ifupdown-extra]

and add something like this to /etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 192.168.1.244/24
gateway 192.168.1.1

Add these two lines using visudo so you can start nginx without a password:

# Allow members of group sudo to execute any command

%sudo   ALL=(ALL:ALL) ALL
%sudo ALL=NOPASSWD: /etc/init.d/nginx start
%sudo ALL=NOPASSWD: /etc/init.d/networking start

Get the path of Ubuntu.exe so you can create a task in task scheduler using Ubuntu UWP install – note, you can find the exact path in Task Manager while Ubuntu window is open by finding wsl.exe or Ubuntu.exe, right-clicking and going to details, the right-clicking on Ubuntu.exe and showing file location. It’ll be something like this:

"C:\Program Files\WindowsApps\CanonicalGroupLimited.UbuntuonWindows_2004.2021.222.0_x64__79rhkp1fndgsc\ubuntu.exe"

Then you can use it to run the service from Windows and set it up in Task Scheduler to start on login, boot, etc.:

"C:\Program Files\WindowsApps\CanonicalGroupLimited.UbuntuonWindows_2004.2021.222.0_x64__79rhkp1fndgsc\ubuntu.exe" run sudo /etc/init.d/nginx start

try restarting to see if it works … (open an ubuntu window and see if you have networking – your /etc/network/interfaces IP, etc. – save those ip commands in a script, just in case!…)

install nginx and certbot w/ certbot nginx plugin and docs:

sudo apt update && sudo apt install -y nginx certbot python3-certbot-nginx python-certbot-nginx-doc

Get loopback and ethernet card info – omit loopback (no need to do for loop) if only assessing outward-facing network, or plan to use certbot instead of openssl self-signed cert:

└─ ▶ for i in eth0 lo; do ip addr show dev $i; done

# output:

4: eth0: mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:ab:6b:b6 brd ff:ff:ff:ff:ff:ff
    inet 172.24.219.149/20 brd 172.24.223.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:feab:6bb6/64 scope link
       valid_lft forever preferred_lft forever
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

WSL: this address is only available on your local machine (because it’s a NAT address). You want an address on your LAN.

Start ngnix – native ubuntu:

└─ ▶ sudo systemctl enable --now ngnix

Start ngnix – WSL – the old upstart way, but without upstart (see Windows Task Manager workaround – yes, you read that right):

└─ ▶ sudo /etc/init.d/nginx start

# output:
Starting nginx nginx [ OK ]

check ip address (either lo or eth0):

Browser: http://localhost

Received:

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
. . .

set /etc/ngnix/sites-available/default settings:

sudo vim /etc/nginx/sites-available/default

set firewall for 80 and 443

( beyond scope – see firewall-cmd, ufw, iptables, ntables, bpf, etc. documentation )

copy virtual site portion of nginx default config

cd /etc/nginx/sites-available && cat default

# (selected and copied virtual section to file called subdomain.domain.name)

vim lindabnotary.home.com # name of dev site - I pasted default virt config here

sed -i 's/#//g' lindabnotary.home.com # get rid of all the comment leaders

vim lindabnotary.com # added a couple hashes for top comment, edited site name, added https port - example:


# File: /etc/nginx/sites-available/lindbnotary.home.com

cat /etc/nginx/sites-available/lindabnotary.home.com
 
# You can move that to a different file under sites-available/ and symlink that to sites-enabled/ to enable it.

server {
listen 80;
listen [::]:80 ipv6only=yes;

server_name lindabnotary.home.com;

   root /var/www/lindabnotary; ### end new bits
   index index.html;

   location / {
           try_files $uri $uri/ =404;
   }
}

The nice thing about certbot, is you can create a super easy cursory config file like this and when you invoke the script it’ll take care of the rest of the config for you in an optimized fashion. All it has to be able to do is connect with your endpoint via http – with the correct domain name in nginx. Which brings me to the next subject…

I have Windows AD DNS, so I created a DNS entry for my local dev server:

DNS server RSAT –> create A + PTR records: lindabnotary.home.com @ 192.168.1.244, 244.1.168.192.in-addr.arpa

If you don’t have a DNS server, set up dnsmasq or similar (out of scope) – one easy way off the top of my head is to create a record in pfsense/opnsense to the eth0 ip addr

The pfsense alias thing will work for local dev, but you won’t be able to create a certbot instance through its automated cert creator unless you have an externally-accessible domain name (ala one purchased through a registrar, like namecheap, godaddy, etc.). So if this is your case, either quit while you’re ahead and experiment on your VPS, get a DN through namecheap and link it up to your home’s external IP real quick, or read up on how to generate a cert using certbot manually (or using openssl to create a self-signed cert).

certbot manual instructions (note: I have no idea if this actually works the way I’m hoping it would): https://certbot.eff.org/docs/using.html#manual

I’m noticing there’s also dns-standalone plugin which is presumably for bind, etc. but probs still needs an external DN.

create symlink to dev build:

└─ ▶ cd /var/www && ln -s /home/avery/node/lindabnotary/build ./lindabnotary

└─ ▶ ls -la
total 12
drwxr-xr-x 3 root root 4096 Jun 4 22:16 .
drwxr-xr-x 14 root root 4096 Jun 4 20:26 ..
drwxr-xr-x 2 root root 4096 Jun 4 20:26 html
lrwxrwxrwx 1 root root 35 Jun 4 22:16 lindabnotary -> /home/avery/node/lindabnotary/build

create symlink between sites-available and sites-enabled:

└─ ▶ sudo ln -s /etc/nginx/sites-available/lindabnotary.home.com/etc/nginx/sites-enabled/lindabnotary.home.com

(re)start nginx:

└─ ▶ sudo /etc/init.d/nginx start

# output: 
Starting nginx nginx

navigated to dev site lindabnotary.home.com in browser

Returned: “It works!” 😂😂😂

Then once the nginx server is serving the content locally, it’s time to set it up for access on the internet.

Necessity for web access – certbot requires internet access to your site in order to work. You must be able to get the dev server to respond to an http request on the internet. Therefore, you must open a firewall port to your dev server (at least temporarily) through external firewall/gateway, etc.

I did this in OPNsense. Here’s a thread/guide if you have questions: https://forum.opnsense.org/index.php?topic=6155.0

(Through web GUI):
Location: Firewall –> NAT –> Port Forward

  • Firewall: NAT: Port Forward
  • Interface: WAN
  • TcpVersion: IPV4
  • Protocol: TCP
  • Destination: WAN address
  • Destination port range: HTTP, HTTPS
  • Redirect Target IP 192.168.1.244
  • Pool options: default
  • Description: WSL on Dell Laptop WharfRat (for dev purposes)
  • Set local tag: (optional)
  • Match local tag: (if you have one)
  • Nat reflection: Enable
  • Filter rule exception: (Will autocreate when saved)

Then, now that the external firewall rule is enabled to allow access from the internet, get your gateway’s current external ip address:

└─ ▶ curl -s https://dnsleaktest.com | grep Hello | cut -d '>' -f2 | cut -d '<' -f1

Returned:
Hello 67.100.93.10

And use the IP address to see if you can access your site from outside – There’s lots of ways you could do this, but I used my cell phone’s browser after turning off WIFI.

Real quick: If you mess your domain’s config up, you can do this to start over:

└─ ▶ sudo certbot delete --cert-name lindabnotary.home.com

OK, now for the moment of truth – run this command for certbot to automatically create a certificate and configuration:

└─ ▶ sudo certbot run --nginx -d lindabnotary.home.com

Another quick note: My namecheap dynamic DNS only allows one subdomain at a time, so I did not configure the www.lindabnotary.home.com DN. The subdomain ‘lindabnotary’ is already a wildcard – it’ll link to whatever server is presented through the firewall, not a specific server identified by the subdomain, like a “real” static DN.

But if you wanted to (or it would matter in your case, unlike mine) you could just string another ‘-d www.example.com‘ on the end (presumably, you can do as many -d‘s as you want).

Here’s the output – I selected 2 for redirecting all traffic to HTTPS:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Obtaining a new certificate
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/lindabnotary.home.com

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.

1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.

Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/lindabnotary.home.com

Congratulations! You have successfully enabled
https://lindabnotary.home.com

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=lindabnotary.home.com

IMPORTANT NOTES:

Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/lindabnotary.home.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/lindabnotary.home.com/privkey.pem
Your cert will expire on 2021-09-03. To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the "certonly" option. To non-interactively renew all of
your certificates, run "certbot renew"
If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le

Note: Now nginx config has been altered by certbot – here’s what it looks like after running the installation command:

You can move that to a different file under sites-available/ and symlink that to sites-enabled/ to enable it. Don’t forget to restart nginx.

   server {
       server_name lindabnotary.home.com;

       root /var/www/lindabnotary;
       index index.html;

       location / {
           try_files $uri $uri/ =404;
   }

   listen [::]:443 ssl ipv6only=on; # managed by Certbot
   listen 443 ssl; # managed by Certbot
   ssl_certificate /etc/letsencrypt/live/lindabnotary.home.com/fullchain.pem; # managed by Certbot
   ssl_certificate_key /etc/letsencrypt/live/lindabnotary.home.com/privkey.pem; # managed by Certbot
   include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
   ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
   }


server {
    if ($host = lindabnotary.webtool.space) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
    
   listen 80;
   listen [::]:80 ipv6only=on;

   server_name lindabnotary.home.com;
return 404; # managed by Certbot

Troubleshooting: I noticed after I got the lock icon locally, I still couldn’t access my site remotely. For dev purposes, this might not be a big deal – I really just need to do development work locally, I’ll worry about external HTTPS access once I get it on the VPS.

Not having port 80 access and not being able to access 443 from the internet WILL prevent certbot from updating the certificate automatically, but I could just open up port 80 access again (instead of auto-redirect) in the /etc/nginx/sites-available/lindabnotary.home.com file for automatic updates to work.

Hopefully this is just a small issue like a firewall port not being open, or something – will need to go re-check my OPNsense config…

Further dev stipulations: Now that I have my .pem file from certbot, I can link to it while running the development server, by editing the project folder’s package.json start script to utilize the ssl cert+key.

Make a dir to store the .pem files in locally, and copy them over because we need to modify the permissions

mkdir ssl
sudo cp /etc/letsencrypt/live/lindabnotary.home.com/fullchain.pem ./ssl/fullchain.pem
sudo cp /etc/letsencrypt/live/lindabnotary.home.com/privkey.pem ./ssl/privkey.pem

These are wrong permissions for production, but must be able to execute as user other than root (prod perms: 0700 dir, 0600 each file, owned by root)

sudo chown -R root:root ./ssl
sudo chmod -R 755 ssl
sudo chmod 644 ssl/*.pem

# it’s possible you could change the owner and decrease the accessibility of the .pem files, have not tried. Play around with it and find out (chown -R user:user ./ssl, chmod -R 0700, chmod 0600 ./ssl/*.pem)

Ignore that, I tried this:

avery @ wharfrat ~/node/lindabnotary
└─ ▶ sudo chmod 0653 ./ssl/.pem avery @ wharfrat ~/node/lindabnotary └─ ▶ sudo chmod -R 0754 ssl avery @ wharfrat ~/node/lindabnotary └─ ▶ sudo chmod 0653 ./ssl/.pem
avery @ wharfrat ~/node/lindabnotary
└─ ▶ npm start

> Lindabnotary.com@0.1.0 start
> bash https-dev-script.sh

EACCES: permission denied, open '/home/avery/node/lindabnotary/ssl/fullchain.pem'

Just make the ssl dir owned root, perms readable all users – it works, this is not production.

You can embed all these variables into package.json, but I opted to make a separate script

filename: ~/projectfolder/https-dev-script.sh

#!/usr/bin/env bash

export HTTPS=true
export SSL_CRT_FILE=./ssl/fullchain.pem
export SSL_KEY_FILE=./ssl/privkey.pem 

react-scripts start

Make the script runnable:

chmod +x ./https-dev-script.sh

and then modified package.json to run the script:

  "scripts": {
    "start": "bash https-dev-script.sh", # this is our mod
    "build": "HTTPS=true react-scripts build",  # not sure if HTTPS=true does anything during build - look into
    "test": "react-scripts test", # FYI you cannot put comments in json files
    "eject": "react-scripts eject"
  },

An aside: This bash script is not runnable outside of npm start (cannot just run ./https-dev-script.sh – needs to be invoked by npm to run react-scripts).

Easiest WSL2 Bridge Network (without Hyper-V Virtual Network Manager)

Burning a Wireless Router - YouTube
Grilling a Linksys. Delicious…

I am testing certbot locally with Create-React-App, but I’m using WSL2. Little did I know, the networking for Ubuntu in WSL2 is a little strange when coming from your typical systemd-network, NetworkManager, ifupdown, <insert your favorite networking API>

Apparently, the networking is taken care of in the lxssManager service (you can reveal it / restart it, etc. in task manager – ctrl-shift-esc -> services). This is because WSL doesn’t have an init system, unless you get some kind of hack to give you a systemd-ish responsive system – I’ve heard of something called ‘genie’, but I haven’t wanted to try it, since I’m trying to do WSL the “right” way 😂

Anywho, if you want an IP from your WSL installation that’s in the same subnet as your LAN, you’re out of luck, since the IP address supplied to your WSL instance is provided through NAT. Some threads I was scanning while looking for a solution talked about installing Hyper-V Virtual Network Manager, but wanting to keep things as light as possible, I have no interest in installing Hyper-V or any of its tools just to get a bridge network setup for WSL.

So I poked around the Network Connections in the Control Panel – this can be revealed by searching for Control in the start menu (it’s been getting progressively more hidden every release since the introduction of the UWP “Settings” app). Once you’re in there, go to Control Panel\Network and Internet\Network Connections (you can paste that line straight in the address bar) and look at what you’ve got. You should see something like this:

Control Panel Network Connections – quaint, isn’t it?

Although it’s likely you won’t have a “Network Bridge”… yet.


Here’s what to do:

Highlight your Ethernet (or WIFI) connection (or both) and the “vEthernet (WSL)” adapter (can select multiple by holding down CTRL while clicking on them), then right-click and select “Create Bridge” from the menu.

That’ll take a minute, but should leave you with this thing:

network bridge software adapter (Windows 10 control panel)

One of the threads I read said you have to restart after creating this thing, but that was not my experience. If you’re having weird networking issues you can’t figure out, you might want to give that a shot.

Then once you have it bridging your WSL adapter with your choice of wired-NIC or WIFI, head over to your bash prompt and set the adapter – note, my LAN’s subnet is 192.168.1.0/24, you’ll want to adjust that for whatever yours is:

 sudo ip addr flush dev eth0
 sudo ip addr add 192.168.1.244/24 dev eth0
 ip addr show dev eth0
 sudo ip link set eth0 up
 sudo ip route add default via 192.168.1.1 dev eth0
 ip route show
 ping 192.168.1.1
 ping google.com

So I’ll run through these real quick:

First you flush your old NAT ip settings, then you assign an address to your WSL adapter. I just picked one at random, but I’d definitely recommend checking to make sure you don’t have the IP address assigned first (try an IP scanner, like arp-scan, or something in Windows – note: I always get horribly paltry results with arp -a for some reason).

Give your WSL eth0 adapter the IP in CIDR notation, then set it to up so you can create a route for it. Create the route to your local gateway (e.g. 192.168.1.1 or whatever yours is…)

Show that your address has been created, the default route has been created, then I like to ping a local machine like my gateway, then something that requires name resolution (here I used google).

Funny thing is, that actually worked before setting up the DNS resolver file, but you have to do that next. I don’t think it was pinging using ipv6, so that doesn’t explain why it worked – probably some weird Windows networking dichotomy, but sure enough, nothing else will really work in name resolution land beyond that, so it’s time to do the next step…

You’ll want to move the /etc/resolv.conf file to /etc/wsl.conf (unless you already have a wsl.conf file, of course – but it’s not created by default). I just invoked # mv /etc/resolv.conf /etc/wsl.conf since it already had the lines I had to add to wsl.conf in it, and I was about to replace it anyway…

Make your /etc/wsl.conf file look like this:

└─ ▶ cat /etc/wsl.conf
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
[network]
generateResolvConf = false

Then create an /etc/resolv.conf file using your favorite DNS servers (1.1.1.1, 1.0.0.1, 8.8.8.8, etc.) I have local DNS, so here’s mine:

# echo 'nameserver 192.168.1.2
> nameserver 192.168.1.3
> search webtool.space' > /etc/resolv.conf

Run something like # apt update to make sure stuff’s resolving now. In my case it was all good.

Note: These ip commands are ephemeral, so you’ll probably want to do some research into how to making them more permenant. I literally put that first block of ip commands in a script to run for when I come back after a restart, because I’m almost 100% certain all this setup will be lost.

One really interesting caveat is that I noticed the Network Bridge in the Control Panel didn’t include the “vEthernet WSL” adapter when I restarted. The bridge was still there, but only had my Ethernet adapter in it, so I had to manually add the WSL adapter again manually in properties.

There’s apparently two config files – one’s the /etc/wsl.conf file we just created, the other is C:\Users\YourUserName\.wslconfig

Unfortunately, the /etc/wsl.conf file is pretty limited as to network config – I did add the generateHosts = false line in mine, because my /etc/hosts had a bunch of weird Windows-only subdomains created by programs that are totally unnecessary to have in Linux, but there’s a lot of other neat stuff in there related to selective case sensitivity, mounting different filesystems, etc. you might want to check out.

So I’m still searching for ways to make this permanent, but once I figure it out, I’ll come back and finish this post. Might have to send an issue to the devs over on GitHub, but I’ll get this working eventually. For now, adding WSL back in the bridge if I restart and running my script isn’t that big of a deal.

Update: This post is a small part of a comprehensive project I was working on. So far, I have posted my raw notes for others to scour if they want to try and re-create the environment: https://develmonk.com/2021/06/05/raw-notes-from-creating-a-local-dev-and-build-https-server-for-create-react-app-using-nginx-and-certbot/

Fetch a file from a Clonezilla image without restoring to disk

Dinosaurs are cute

I thought this was neat, but I got the info from a site that presented it in a way that was a bit difficult to decipher, so I wanted to pass it on in a presentation that is easier to understand.

If you’re not using CloneZilla, you’re definitely missing out. It’s a super easy way to clone hard drives, USB thumb drives, etc. into a format that’s easy to restore. It does disk-to-image, disk-to-disk, disk-to-network supporting ssh, nfs and cifs, and even has a p2p cloning framework for two networked computers using CloneZilla live USBs (!)

Needless to say, that’s definitely cool. But what if you want to grab a file out of an archive you’ve already created?

I had made a backup of a client’s computer I wanted to have in case something went south, but I needed to make sure I could get to the files in case I was asked to deliver files while the computer was in a state of (dis)repair.

Well, thankfully that’s pretty easy, since CloneZilla’s default image format is just some gzipped archives. You basically just unzip them into another file, which appears as the raw image file (e.g. .img), which at that point you can mount and traverse like any other mounted drive.

The basic structure of an image is contained inside a directory created by CloneZilla. It looks like this:

$ ls -la
total 55474810
drwxr-xr-x 2 avery avery                  43 May  5 18:57 .
drwxrwxrwx 9 avery administrators         16 May  5 21:36 ..
-rw-r--r-- 1 avery avery                 874 May  5 17:28 blkdev.list
-rw-r--r-- 1 avery avery                 706 May  5 17:28 blkid.list
-rw-r--r-- 1 avery avery               10068 May  5 18:57 clonezilla-img
-rw-r--r-- 1 avery avery                 235 May  5 18:56 dev-fs.list
-rw-r--r-- 1 avery avery                   4 May  5 18:57 disk
-rw-r--r-- 1 avery avery                 482 May  5 18:57 efi-nvram.dat
-rw-r--r-- 1 avery avery                7354 May  5 18:57 Info-dmi.txt
-rw-r--r-- 1 avery avery                 236 May  5 18:57 Info-img-id.txt
-rw-r--r-- 1 avery avery               19252 May  5 18:57 Info-lshw.txt
-rw-r--r-- 1 avery avery                2401 May  5 18:57 Info-lspci.txt
-rw-r--r-- 1 avery avery                 313 May  5 18:57 Info-OS-prober.txt
-rw-r--r-- 1 avery avery                 211 May  5 18:57 Info-packages.txt
-rw-r--r-- 1 avery avery                 114 May  5 18:57 Info-saved-by-cmd.txt
-rw-r--r-- 1 avery avery               10192 May  5 18:57 Info-smart.txt
-rw-r--r-- 1 avery avery                  20 May  5 18:57 parts
-rw------- 1 avery avery            13784650 May  5 17:28 sda1.vfat-ptcl-img.gz.aa
-rw------- 1 avery avery             7477849 May  5 17:28 sda2.dd-ptcl-img.gz.aa
-rw------- 1 avery avery          4096000000 May  5 17:36 sda3.ntfs-ptcl-img.gz.aa
-rw------- 1 avery avery          4096000000 May  5 17:42 sda3.ntfs-ptcl-img.gz.ab
-rw------- 1 avery avery          4096000000 May  5 17:48 sda3.ntfs-ptcl-img.gz.ac
-rw------- 1 avery avery          4096000000 May  5 17:54 sda3.ntfs-ptcl-img.gz.ad
-rw------- 1 avery avery          4096000000 May  5 18:00 sda3.ntfs-ptcl-img.gz.ae
-rw------- 1 avery avery          4096000000 May  5 18:07 sda3.ntfs-ptcl-img.gz.af
-rw------- 1 avery avery          4096000000 May  5 18:13 sda3.ntfs-ptcl-img.gz.ag
-rw------- 1 avery avery          4096000000 May  5 18:19 sda3.ntfs-ptcl-img.gz.ah
-rw------- 1 avery avery          4096000000 May  5 18:25 sda3.ntfs-ptcl-img.gz.ai
-rw------- 1 avery avery          4096000000 May  5 18:32 sda3.ntfs-ptcl-img.gz.aj
-rw-r--r-- 1 avery avery               16384 May  5 17:28 sda-gpt-2nd
-rw-r--r-- 1 avery avery               17920 May  5 17:28 sda-gpt.gdisk
-rw-r--r-- 1 avery avery                 790 May  5 17:28 sda-gpt.sgdisk
-rw-r--r-- 1 avery avery                 512 May  5 17:28 sda-mbr
-rw-r--r-- 1 avery avery                 621 May  5 17:28 sda-pt.parted
-rw-r--r-- 1 avery avery                 551 May  5 17:28 sda-pt.parted.compact
-rw-r--r-- 1 avery avery                 849 May  5 17:28 sda-pt.sf

They’re logically named with tags for partitions, like sda1, sda2, etc. You can see the super huge files are .gz.* files – gzip archives (or zstd, depending on what option you chose) that have to be re-combined to decompress properly, but that’s no big deal.

If you want to save a partition to an image file, make sure you have a copy of partclone installed on your system (e.g. # apt install partclone -y if on a debian-based distro). That’ll automate the restructuring of the partition from the gzip archives.

Make sure you have enough room on whatever drive you’re doing it on for the entire partition (and then some) – might want to du and df first just to be safe.

Note: if you used zstd instead of gzip for the archives, you’ll have to look up the zstd command switches and adapt the one-liner to accommodate the differences in their APIs. I haven’t tried it, but I assume the result should be the same.

Here’s a quick run-down of how the process goes:

# These instructions assume you're in a dir containing the backup dir
$ ls -la
drwxrwxrwx  9 avery                       administrators           16 May  5 21:36  .
drwxr-xr-x  6 root                        root                      7 Mar 23 00:21  ..
-rw-------  1 root                        root               14196736 
drwxr-xr-x  2 avery                       avery                    43 May  5 18:57  laptop-drive-backup 

# create the file that will be your raw image - any name will do
$ touch laptopPartition.img

# pipe the archives to gzip and use partclone to construct the .img 
$ cat ./laptop-drive-backup/sda3.ntfs-ptcl-img.gz.* | gzip -d -c | partclone.restore -C -s - -O ./laptopPartition.img

# then you can mount your .img file as a loop
$ sudo mount -o loop -t ntfs ./laptopPartition.img /mnt/someEmptyDir

# and browse it like anywhere else
$ ls -la /mnt/someEmptyDir # which is now loop-mounted .img file

Another thing to note is that loop mounts are read-only, kind of like mounting an .iso file, or more recently, a snap container. So you can copy stuff off, but you can’t put stuff on. But this is just a copy of your original .gz.* backup, so there’s really no chance in harming the original’s integrity anyway.

It’s always important to conduct yourself with the utmost (file) integrity!

Compiling and installing GitHub’s Hub Utility in msys2

Hub project source code root directory

So I’ve been liking msys2 so far. I’ve replaced the default git install with msys2’s utilities in C:\tools\msys64\usr\bin and they seem to work from Windows command prompt just fine, except for bash, which complains of cygwin1.dll mismatch, but that’s OK, I’ll just run the msys2_shell.cmd which is how it’s intended to be used anyway (inside Mintty – or, alternately ConEmu or defterm). So it’s slightly less flexible (git-bash’s bash.exe could be run from inside a standard cmd.exe terminal), but I’m OK with that since I now have man files in Windows (!) and I can update the utilities using pacman, instead of waiting for updates to come through git-bash updates (I have my doubts that it ever happens).

Why do all of this hacky 3rd-party linux sublayer stuff instead of just enabling Windows Subsystem for Linux (WSL)? Simple: I am fairly certain you cannot run WSL binaries from Windows (please correct me if I’m wrong!).

Continue reading “Compiling and installing GitHub’s Hub Utility in msys2”