   #[1]alternate [2]alternate

   [3]Xe

   [4]Blog

   [5]Contact

   [6]Resume

   [7]Talks

   [8]VODs

   [9]Signalboost

gokrazy is really cool

   Published on 09/20/2023
   An image of undefined I work for Tailscale at the time of writing this
   article. I wrote this on my own time out of my own volition.

   When you deal with Linux, you end up hearing about "distributions" as
   different "flavors" of Linux combined with a bunch of other tools. This
   is mostly true, but it's slightly missing the forest for the trees.

   Consider this famous and often misunderstood quote by Richard Stallman:

     I'd just like to interject for a moment. What you're referring to as
     Linux is in fact, GNU/Linux, or as I've recently taken to calling
     it, GNU plus Linux.Linux is not an operating system unto itself, but
     rather another free component of a fully functioning GNU system made
     useful by the GNU corelibs, shell utilities and vital system
     components comprising a full OS as defined by POSIX.

   Many pages of ink have been spilled over analyzing this quote, and a
   lot of them fall short of really getting at the heart of the matter.
   What this actually means is something like this:

   By itself, Linux is useless. It does boot the system, it does interface
   with hardware, but without a bunch of other tools, it's not very
   useful. It's like a car without a steering wheel, or a boat without a
   rudder. It does something, but it's not very useful. The real value of
   things like the GNU project, systemd, openrc and other tools in that
   vein is that they make Linux useful. They make it into a complete
   system that you can use to do things. They are the proverbial steering
   wheel and rudder in the metaphor.
   Mara is hacker
   <[10]Mara> Fun fact, if you try to boot a Linux kernel without an init
   process, it'll just panic and crash!

   Most Linux systems on the face of the planet are built with GNU tools
   and utilities. In order to compile the Linux kernel, you need to use
   [11]GCC. In order to run ls to list files in the current directory, you
   need to use [12]GNU coreutils. Every dynamically linked program uses
   [13]glibc for performing basic system interactions like writing to
   files or opening network sockets. Everything is built on top of the GNU
   toolset. This is why Stallman is so adamant about calling it GNU/Linux.
   It's not that he's trying to take credit for Linux, it's that he's
   trying to give credit to the GNU project for making Linux useful.

   However, there's a lot of room for nuance here. For example, [14]Alpine
   Linux is a Linux distribution that uses [15]musl libc instead of
   [16]glibc and [17]busybox instead of GNU coreutils. It's still a Linux
   distribution, but it doesn't use the GNU toolset. It's still a Linux
   distribution, but it's not GNU/Linux.
   Mara is hacker
   <[18]Mara> Also, for the record you can build the Linux kernel with
   clang, but that's a whole other can of worms. For one, GCC supports
   many more targets than clang likely ever will, but in general there are
   some compromises you need to make until clang implements some
   GCC-specific compiler extensions a bit better. Google, Facebook, and a
   few other companies do run LLVM compiled kernels in production though,
   so it's probably closer to viable than you think. Especially if you use
   ChromeOS or Android.

   So, what is a Linux distribution? It's a collection of tools that make
   Linux useful. It's a collection of tools that make Linux into a
   complete system. It's not a "flavor" of Linux (though this conceptually
   can exist with alternative kernels like the Zen kernel patchset), it's
   a system that just so happens to make Linux useful.

   As a counter-argument, consider the reason why Linux runs on more
   devices worldwide than there are people: [19]Android. Android does use
   the Linux kernel, but it doesn't use any GNU tools in the stack at all.
   You can't take programs that are compiled against other Linux
   distributions and run them on Android. You can't take programs that are
   compiled against Android and run them on other Linux distributions.
   Aoi is wut
   <[20]Aoi> Wait, so does this mean Android's not a Linux distribution?
   What is it then?

   I'm going to argue that Android is not a Linux distribution unto
   itself. Android is a Linux implementation. It uses the Linux kernel,
   but that's where the similarities with the rest of the ecosystem end.
   Android is its own little world where there's just enough system tools
   to get the system running, but once you get into the UI, it's a
   completely different world. It's a completely different ecosystem. It's
   a completely different operating system.
   Aoi is wut
   <[21]Aoi> So what's the difference between a Linux distribution and a
   Linux implementation?
   Cadey is enby
   <[22]Cadey> It's a bit of a fuzzy line, but I'd say that a Linux
   distribution is a collection of discrete tools that make Linux useful,
   and a Linux implementation is a cohesive collection of bespoke tools
   that make Linux into a complete system. Really, you could argue that if
   it has /bin/sh, it's a Linux distribution.

gokrazy

   [23]gokrazy is a Linux implementation that I've used off and on for a
   few years now. It's a very interesting project because everything on
   the system is written in Go save the kernel. The init process is in Go
   (and even listens over HTTP to handle updates!), every userland process
   is written in Go, and even the core system services are written in Go.

   Out of the box a gokrazy install comes with these basic tools:
     * The init process that is mandated to be the parent of all userland
       processes by the Linux kernel.
     * A [24]DHCP client that automatically configures the network
       interface.
     * A [25]NTP client that automatically sets the system clock.
     * A little tool to save randomness from the kernel to a file so that
       it can be used to seed the random number generator on boot (because
       the Raspberry Pi doesn't have a robust hardware random number
       generator)

   That's it. Everything else from the web UI to A/B update logic is
   written in Go. It boots in literal seconds, uses an insanely small
   amount of RAM out of the box, and runs with nearly zero overhead. When
   you configure your gokrazy install to run additional software, you do
   so by adding the Go command path to a configuration file and then
   updating to trigger a reboot into the new version.

   Here's an example of what my gokrazy virtual machine's file tree looks
   like:
/ # tree etc gokrazy user
etc
âââ breakglass.authorized_keys
âââ gokr-pw.txt
âââ gokrazy
â   âââ sbom.json
âââ hostname
âââ hosts
âââ http-port.txt
âââ https-port.txt
âââ localtime
âââ machine-id
âââ resolv.conf -> /tmp/resolv.conf
âââ ssl
    âââ ca-bundle.pem

gokrazy
âââ dhcp
âââ heartbeat
âââ init
âââ ntp
âââ randomd

user
âââ breakglass
âââ fbstatus
âââ qemu-guest-kragent
âââ serial-busybox
âââ tailscale
âââ tailscaled
âââ waifud-gok-agent

   That is the entire system. It's all stripped down to these few
   programs, configuration files, and one symlink for DNS resolution. This
   is a very minimal system, and it's all you need to run statically
   linked Go programs. It's very easy to deploy your own services to it
   too. It's probably the easiest platform I know of that lets you just
   deploy a Go binary and have it run as a service, automatically
   restarting when it crashes.

The tooling

   When I used gokrazy back in the day, you had to use a command line
   called gokr-packer that you passed a bunch of command line flags to
   with information about all the Go programs you wanted to run on the
   machine, configuration for those programs, and any other
   meta-information like where the update tool should push the image to.
   It was a bit of a pain to use, but it worked. Recently the [26]gok tool
   was added to the project, and this has been revolutionary when it comes
   to using and administrating gokrazy installs.

   Essentially, gok is a wrapper around the existing gokr-packer logic
   with a JSON file to store your configuration details. It's a lot easier
   to use, understand, and automate. You don't have to remember command
   line flags or maintain unwieldy scripts. You just edit a JSON file and
   push updates with gok update. It's amazingly simple.

Setting up a gokrazy machine

   As an example, I'm going to show you how to install a bunch of tailnet
   addons to a gokrazy machine. I'm also going to assume that you don't
   have a gokrazy install set up yet, so we'll need to install it. To do
   this, we'll need to do a few simple things:
     * Install the gok tool.
     * Create your gok configuration.
     * Install Tailscale on the machine.
     * Create your "seed" image with gok overwrite.
     * Boot it on your Raspberry Pi or VM.
     * Push any updates to the image to the machine with gok update.

   First, let's install the gok tool. In order to do this, you need to
   have the [27]Go toolchain installed. Once you have that, you can run go
   install to install the gok tool:
go install github.com/gokrazy/tools/cmd/gok@main

   Mara is hacker
   <[28]Mara> You may want to ensure that ~/go/bin is in your $PATH
   variable so that you can run it by the name gok instead of
   ~/go/bin/gok.

   Next, create a new gokrazy configuration with gok new:
gok new -i casa

   This will create a configuration named casa (cf: Spanish for "house")
   in ~/gokrazy/casa. This is where all of your configuration files will
   live. You can edit the configuration file with gok edit:
gok edit -i casa

   If you are making a virtual machine

   If you are making a virtual machine, you will need to override the
   kernel and firmware packages. You can do this by adding the following
   to your configuration file:
{
  // ...
  "KernelPackage": "github.com/rtr7/kernel",
  "FirmwarePackage": "github.com/rtr7/kernel",
  // ...
}

   You will need to prefix the gok overwrite and gok update commands with
   GOARCH=amd64 to ensure that Go builds x86_64 binaries instead of ARM
   binaries:
GOARCH=amd64 gok update -i casa

   If you don't do this, you will get arm64 binaries being built. This may
   require manual recovery of your virtual machine.

   Let's make our lives easier by installing [29]Tailscale on the machine.
   By default, gokrazy will announce its hostname over DHCP, which usually
   makes most consumer routers pick it up and then lets you ping it by
   name. When you have [30]MagicDNS enabled, Tailscale can take over this
   logic and prevent you from accessing the machine by name.

   However, Tailscale is written in Go and doesn't require any of the
   services that most Linux distributions provide in order to function.
   It's a perfect fit for gokrazy. You can install it with gok add:
gok add tailscale.com/cmd/tailscaled
gok add tailscale.com/cmd/tailscale

   And be sure to add the mkfs service to create a persistent partition on
   /perm:
gok add github.com/gokrazy/mkfs

   Next, fetch an [31]auth key from [32]the admin console and make sure
   you check that it's reusable. Then, add the following to your
   configuration file under the PackageConfig block:
{
    // ...
    "PackageConfig": {
        // ...
        "tailscale.com/cmd/tailscale": {
            "CommandLineFlags": [
                "up",
                 // paste your key here!
                "--authkey=tskey-auth-hunter2-hunter2hunter2hunter2"
            ]
        },
        // ...
    }
    // ...
}

   Mara is hacker
   <[33]Mara> You can pass any other [34]tailscale up flags you want here,
   such as --advertise-exit-node if you want to use your gokrazy machine
   as an [35]exit node.

   This will make your machine automatically connect to Tailscale on boot.

   Next, we need to create our "seed" image with gok overwrite. First,
   figure out what the device node for your SD card is. On Linux, you can
   do this with lsblk:
lsblk

   And then look for the one that has the same size as your SD card. In my
   case, it's /dev/sdd. Once you have that, you can run gok overwrite:
gok overwrite --full /dev/sdd

   However if you want to write the image to a file (such as if you are
   doing mass distribution or making a VM image), you need to use gok
   overwrite with a file instead of a device node. This will create a 16
   GB image:
gok overwrite -i casa --full gokrazy.img --target_storage_bytes 17179869184

   Once you have your image, you can write it to your SD card with dd (or
   [36]balenaEtcher) or import it into your virtual machine hypervisor of
   choice.

   Once you have your image written to your SD card, you can boot it on
   your Raspberry Pi or VM.
   Aoi is wut
   <[37]Aoi> Wait, so how do I log in with a shell?
   Cadey is enby
   <[38]Cadey> You don't. gokrazy doesn't have a login prompt. It's a
   single-user system. There is [39]breakglass as a tool of last resort to
   modify things, but you only have a very minimal subset of busybox to
   work with, so it should be avoided if at all possible.

   Once you have your machine booted and it responds to pings over
   Tailscale, you can open its HTTP interface in your browser. If you
   called your machine casa, you can open it at [40]http://casa. It will
   prompt you for a username and password. Your username is gokrazy, and
   the password is near the top of your config.json file. When you log in,
   you'll see a screen like this:
   [41][gokrazy-ui.jpg]

   This is the gokrazy web UI. It lets you see the status of your machine
   and any logs that are being generated by your applications. You can
   also start, stop, and restart any of your applications from here. It's
   a very simple UI, but it's fantastic for debugging and monitoring.

Tailnet addons

   Now that we have a Gokrazy system up and running, let's add some
   programs to it! I'm going to list a couple tailnet addons that give
   your tailnet superpowers. These are all written in Go, so they're a
   perfect fit for gokrazy.

   Today I'm going to show you how to install these tools into your
   tailnet:
     * [42]golink - a URL shortener at http://go
     * [43]tmemes - an internal meme generator you can host at
       http://memegen
     * [44]tclip - a pastebin you can host at http://paste

   These tools help you augment your tailnet by giving you tools that will
   make you and your team's life a lot easier. A URL shortener helps you
   link to complicated Google Docs URLs. A meme generator gives you a new
   innovative way to let off steam. A pastebin lets you share text with
   your team without having to worry about the service you're using going
   offline due to no fault of your own.

golink

   To install golink, we need to add the golink binary to the
   configuration. You can do this with gok add:
gok add github.com/tailscale/golink/cmd/golink

   Then configure it with gok edit:
{
    // ...
    "PackageConfig": {
        // ...
        "github.com/tailscale/golink/cmd/golink": {
            "CommandLineFlags": [
                "--sqlitedb=/perm/home/golink/data.db"
            ],
            "Environment": [
                // the same one from before
                "TS_AUTHKEY=tskey-auth-hunter2-hunter2hunter2hunter2"
            ],
            // don't start the service until NTP catches up
            "WaitForClock": true
        },
        // ...
    }
    // ...
}

   And finally push it with gok update:
gok update -i casa

   It'll build the image, push it out over Tailscale, trigger a reboot,
   and be back up in the span of a minute. Once it's back up, you can open
   the web UI again and see the status of your golink instance at
   [45]http://casa/status?path=%2fuser%2fgolink:
   [46][golink.jpg]

   And then you can start using short URLs at [47]http://go:
   [48][golink-ui.jpg]

   And that's it! You now have a super minimal VM running small programs
   that let you do useful things to you. You can add more programs to your
   configuration file and push them with gok update to add more
   functionality to your machine. You can even add your own programs to
   the configuration file and push them to your machine. It's a very
   simple system, but it's very powerful.

tmemes

   Google is infamous for having an internal service named [49]memegen.
   This allows Googlers to make internal-facing memes about the slings and
   arrows that impact them as highly paid programmers. This is an internal
   service inside Google that has a lot of serious investment of time and
   energy to make it the best possible experience it can be. It's to the
   point that reportedly people can keep up with how an all-hands meeting
   is going by the tone of the sarcastic memes that are being posted to
   memegen.

   The main reason this is run inside Google is to avoid information
   leaking via memes. Yes, this is an actual threat model.

   Thanks to the magic of Tailscale, you can make your own private memegen
   using [50]tmemes. tmemes is a tailnet addon that lets you post image
   macro templates and layer wisdom over it in the form of text.

   Here's an example meme:
   [51][society-if-gokrazy.jpg]

   To add tmemes to your gokrazy machine, you can use gok add:
gok add github.com/tailscale/tmemes/tmemes

   Then open your config with gok edit and add the following to your
   PackageConfig block:
{
    // ...
    "PackageConfig": {
        // ...
        "github.com/tailscale/tmemes/tmemes": {
            "Environment": [
                "TS_AUTHKEY=tskey-auth-hunter2-hunter2hunter2hunter2"
            ],
            "CommandLineFlags": [
                // change this to your desired hostname
                "--hostname=memegen",
                // change this to your username on Tailscale
                "--admin=Xe@github",
                "--store=/perm/home/tmemes"
            ],
            "WaitForClock": true
        },
        // ...
    },
    // ...
}

   And then push it with gok update:
gok update -i casa

   Then you can head to [52]http://memegen and upload a template to make
   your own dank memes.

   If you want to integrate your own tools with tmemes, you can check out
   the [53]API documentation. This should help you do whatever it is you
   want with a meme generator as a service.

tclip

   Sometimes you just need a place to paste text and get a URL pointing to
   it. [54]tclip is a tool that you can add to your tailnet and get
   exactly that. It's a very simple tool, but it's very useful. It's also
   written in Go, so it's a perfect fit for gokrazy. [55]Their recent
   update to remove Cgo dependencies makes it possible to run your tclip
   node on a gokrazy machine.

   To add tclip to your gokrazy machine, you can use gok add:
gok add github.com/tailscale-dev/tclip/cmd/tclipd

   Then open your config with gok edit and add the following to your
   PackageConfig block:
{
    // ...
    "PackageConfig": {
        // ...
        "github.com/tailscale-dev/tclip/cmd/tclipd": {
            "CommandLineFlags": [
                "--data-location=/perm/home/tclip/"
            ],
            "WaitForClock": true,
            "Environment": [
                "TS_AUTHKEY=tskey-auth-hunter2-hunter2hunter2hunter2",
                "USE_FUNNEL=true" // Remove this if you don't want to use Funnel
            ]
        },
        // ...
    }
}

   And then push it with gok update:
gok update -i casa

   And then you can start using it by heading to [56]http://paste. Install
   the command-line tool on your development workstation with go install:
go install github.com/tailscale-dev/tclip/cmd/tclip@latest

   Here's an example tclip link if you want to see what it looks like in
   practice: [57]interjection.c. It's a very simple tool, but it's very
   useful.

Conclusion

   gokrazy is insanely cool. It's the easiest way to deploy Go services to
   your homelab. It integrates seamlessly with Tailscale, and is something
   that I'm very excited to see grow and mature. I'm very excited to see
   what the future holds for gokrazy, and I'm very excited to see what
   people do with it.

   I've seen signs that they're going to be adding an automatic update
   process, and that has me very excited. I'm also excited to see what
   other services people add to the gokrazy ecosystem. I'm hoping to add a
   few of my own in the future, and I'm hoping to see what other people do
   with it.
   Mara is hacker
   <[58]Mara> Spoiler alert: [59]waifud support is coming soon to a
   homelab near you.
   (BUTTON) Share

   Facts and circumstances may have changed since publication. Please
   contact me before jumping to conclusions if something seems wrong or
   unclear.

   Tags: go, gokrazy, linux

   Copyright 2012-2023 Xe Iaso (Christine Dodrill). Any and all opinions
   listed here are my own and not representative of any of my employers,
   past, future, and/or present.

   Like what you see? Donate on [60]Patreon like [61]these awesome people!

   Served by xesite v4
   (/nix/store/5k4azk5h5ymf5r6siw9jv7d3k58qrwx7-xesite_v4-20231009/bin/xes
   ite), source code available [62]here.

References

   1. https://xeiaso.net/blog.rss
   2. https://xeiaso.net/blog.json
   3. file:///
   4. file:///blog
   5. file:///contact
   6. file:///resume
   7. file:///talks
   8. file:///vods
   9. file:///signalboost
  10. file:///characters#mara
  11. https://gcc.gnu.org/
  12. https://www.gnu.org/software/coreutils/coreutils.html
  13. https://www.gnu.org/software/libc/
  14. https://alpinelinux.org/
  15. https://musl.libc.org/
  16. https://www.gnu.org/software/libc/
  17. https://en.wikipedia.org/wiki/BusyBox
  18. file:///characters#mara
  19. https://en.wikipedia.org/wiki/Android_(operating_system)
  20. file:///characters#aoi
  21. file:///characters#aoi
  22. file:///characters#cadey
  23. https://gokrazy.org/
  24. https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol
  25. https://en.wikipedia.org/wiki/Network_Time_Protocol
  26. https://gokrazy.org/quickstart/
  27. https://golang.org/doc/install
  28. file:///characters#mara
  29. https://tailscale.com/
  30. https://tailscale.com/kb/1081/magicdns/
  31. https://tailscale.com/kb/1085/auth-keys/
  32. https://login.tailscale.com/admin/settings/keys
  33. file:///characters#mara
  34. https://tailscale.com/kb/1080/cli/#up
  35. https://tailscale.com/kb/1103/exit-nodes/?q=exit node
  36. https://etcher.balena.io/
  37. file:///characters#aoi
  38. file:///characters#cadey
  39. https://github.com/gokrazy/breakglass
  40. http://casa/
  41. https://cdn.xeiaso.net/file/christine-static/blog/2023/gokrazy/gokrazy-ui.jpg
  42. https://github.com/tailscale/golink
  43. https://github.com/tailscale/tmemes
  44. https://github.com/tailscale-dev/tclip
  45. http://casa/status?path=/user/golink
  46. https://cdn.xeiaso.net/file/christine-static/blog/2023/gokrazy/golink.jpg
  47. http://go/
  48. https://cdn.xeiaso.net/file/christine-static/blog/2023/gokrazy/golink-ui.jpg
  49. https://www.buzzfeednews.com/article/reyhan/inside-googles-internal-meme-generator
  50. https://github.com/tailscale/tmemes
  51. https://cdn.xeiaso.net/file/christine-static/blog/2023/gokrazy/society-if-gokrazy.jpg
  52. http://memegen/
  53. https://github.com/tailscale/tmemes/blob/main/docs/api.md
  54. https://tailscale.dev/blog/tclip
  55. https://tailscale.dev/blog/tclip-updates-092023
  56. http://paste/
  57. https://paste.shark-harmonic.ts.net/paste/696b9b02-90ac-4adc-a33d-d749bb6f460f
  58. file:///characters#mara
  59. https://github.com/Xe/waifud-gok-agent
  60. https://patreon.com/cadey
  61. file:///patrons
  62. https://github.com/Xe/site
