checkin

build.sh

@@ -1,2 +1,2 @@
-docker build . -t git.howdoicomputer.lol/howdoicomputer/blog:5
-docker push git.howdoicomputer.lol/howdoicomputer/blog:5
+docker build . -t git.howdoicomputer.lol/howdoicomputer/blog:7
+docker push git.howdoicomputer.lol/howdoicomputer/blog:7

ci/main.go

@@ -11,23 +11,20 @@ import (
 func main() {
 	ctx := context.Background()
 
-	// initialize Dagger client
-	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
+	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout), dagger.WithWorkdir(".."))
 	if err != nil {
 		panic(err)
 	}
 	defer client.Close()
 
-	// use a golang:1.19 container
-	// get version
-	// execute
-	golang := client.Container().From("golang:1.19").WithExec([]string{"go", "version"})
+	buildDir := client.Container().
+		From("klakegg/hugo:0.111.3-ext-onbuild").
+		Directory("/target")
 
-	version, err := golang.Stdout(ctx)
-	if err != nil {
-		panic(err)
-	}
+	ref, err := client.Container().
+		From("nginx").
+		WithDirectory("/usr/share/nginx/html", buildDir).
+		Build()
 
-	// print output
-	fmt.Println("Hello from Dagger and " + version)
+	fmt.Println(ref)
 }

content/posts/dagger-pe.md

@@ -0,0 +1,51 @@
+---
+title: "Dagger: Platform Engineer's Dream"
+date: 2023-11-05T19:53:24-08:00
+draft: true
+---
+
+# but what is dagger
+
+Dagger is a multi-language toolkit that allows you to create CI/CD pipelines that are backed by Docker. Imagine if `make` was an API where you could write targets in Go, TypeScript, Python, etc and then each target was ran inside a Docker container and the results of each container was cached.
+
+For example, let's say we have a web API written in Rust. For this hypothetical project, we want to test, build, upload, and deploy. Test would be running `cargo test`, build would be either creating a container image or an executable, upload would be uploading a container image to a registry, and deploy would be running `helm upgrade` to upgrade an existing Kubernetes `Deployment`.
+
+With Dagger, each of those steps could be written using Rust, with each step executing inside a container image built for that step, all running on a developers laptop. When that pipeline needs to be slotted into Jenkins or any other CI, all the CI system needs to be able to do is run Dagger.
+
+But we aren't just limited to a make target analogue. Dagger pipelines are written using a REAL programming language - not a DSL! That means anything is possible! Do you need to fetch a secret from a secret store? Do you need to support a custom versioning process using git tags? Do you need to communicate with internal services when deployments occur? You can do all of these things using a language that you're comfortable with. Gone are the days of hunting down long abandoned Jenkins plugins written in Groovy and riddled with security vulnerabilities just to add simple functionality to your pipelines. Write it in a language you love instead. <3
+
+# internal dsl library
+
+At a certain scale, it's not uncommon for a company to have a blessed path for developers to onboard their applications and processes into a CI runner like Jenkins, TeamCity, or GitHub Actions. This typically coalesces as a singular library that a platform engineering or release engineering team builds and then provides to the greater engineering org as a sort of an SDK for CI/CD processes. If your company is running Jenkins, for example, you might have a `jenkins-utilty-lib` that gets imported into DSL files that are stored in application repositories and that utility lib would have functions for enforcing company standards or abstracting complex release processes.
+
+Dagger can replace those DSL libraries with code that the platform engineering team favors rather than forcing them to write Groovy. But... what if they choose isn't favored by everyone else?
+
+# any language, comrade
+
+Okay, so internal DSL libraries are common and Dagger allows you to write them in any language. But what if your platform engineering team really wants to code in Go but your front-end engineers really want to code in TypeScript? What if the PE team writes some Go functions in Dagger that uploads static assets to a CDN? Does the FE team need to write Go to use those functions?
+
+Well, no.
+
+Dagger has the ability to expose pipeline steps **across languages**. As long as the output of a function is serializable, it can be consumed by TypeScript from Go (or any other pairing). In the above example, the platform engineering team *can* write build pipelines that are compatible with front-end's pipelines even if they aren't in the same language.
+
+# but what about CLI tools
+
+So, for example, let's say we have a CLI tool written in Ruby and it requires a specific set of libraries installed and a sane Ruby environment. This Ruby CLI could be some internal tool used by your company to automate some process. However, it's kind of unwieldy to have every engineer install Ruby on their machine alongside this fantasy tool's dependencies.
+
+With Dagger modules, you could easily provide a way for end users to consume your API by not having them install dependencies.
+
+Using a [Python dagger module](https://daggerverse.dev/mod/github.com/helderco/daggerverse/python@ba7984f47857794ff5aace82947aaf6b433595be) as an example:
+
+```
+dagger mod install github.com/helderco/daggerverse/python@ba7984f47857794ff5aace82947aaf6b433595be
+export DAGGER_MODULE=github.com/helderco/daggerverse/python
+dagger up http-server -n --src .
+```
+
+The above commands will install the Python Dagger module and then use it to spin up a Python container and run `http-server` to serve files.
+
+# caching, k8s, and other good stuff
+
+Because Dagger uses Docker BuildKit under the hood it also has support for caching operations and artifacts from pipeline steps. Additionally, you can run the Dagger Engine on Kubernetes and have it run your CI/CD so that the build caches are shared across multiple CI runs. This means that there is a TON of opportunity for time savings across multiple runs.
+
+Seriously, check it out. It's neat.

content/posts/homelab-1.md

@@ -14,8 +14,6 @@ This blog. The one that you are currently reading right now. Is running on a met
 
 ### yeah, but why
 
-I like solving problems and these were mine:
-
 * I have a plethora of software projects that I've written that need a home. Discord bots, daemons, websites, etc.
 * I needed a place to host a blog. Hi!
 * I want to replace the myriad online software services that I'm subscribed to. Spotify, Netflix, etc.
@@ -54,7 +52,7 @@ Here are some features of my homelab:
 
 A secret that I've withheld from you so far is that Megamind isn't my first homelab. A previous homelab that I built was a k3s cluster running on top of Raspbery Pis. It was quaint but got decommissioned when I moved apartments. I never set it up again and it currently lies in pieces in a drawer somewhere.
 
-So why Nomad? Mostly because, at the time, I was leading an infrastructure initiative at my day job to use Nomad to orchestrate applications that are deployed to heterogenous IoT devices spread across distinct geographic regions. I wanted to dog food my own proposal. Also, Nomad is *a lot* simpler to use than Kubernetes. Nomad starts as *just* an orchestrator but gives you the option to introduce complexity later on through Vault and Consul. Every concept is almost immediately grokable and it has just been a pleasure to work with.
+So why Nomad? Mostly because, at the time, I was leading an infrastructure initiative at my day job to use Nomad to orchestrate applications that are deployed to heterogenous IoT devices spread across distinct geographic regions. Nomad was a better fit than k8s for that specific use case because it was lighter-weight, easier to use, had less overhead (which was important because embedded devices don't have a lot of resources), and could run non-containerized applications that needed deep access to hardware. And so I wanted to dog food my own proposal. Also, Nomad is *a lot* simpler to use than Kubernetes. Nomad starts as *just* an orchestrator but gives you the option to introduce complexity later on through Vault and Consul. Every concept is almost immediately grokable and it has just been a pleasure to work with.
 
 ### what's runnin
 
@@ -162,6 +160,6 @@ The future is bright! It has been incredibly fluid to roll out new services to t
 
 For CI/CD, I'll most likely go with [Drone](https://www.drone.io/). Really, I just want a UI for Dagger pipelines. For an auth model, eh, I am not sure. LDAP? There is also the possiblity of using Oauth2 and whitelisting accounts.
 
-I also want to do a thorough lockdown of Gitea so that I can expose my code to the world.
+For now though, my homelab is now an integral part of my life. For consumption, I use it to store a digital library of books, music, and movies. For productivity, the homelab is a cornerstone of developing applications that I want to expose to the public internet. It's... neat.
 
-Till next time.
+[Part 2]({{< ref "/posts/homelab-2" >}} "Part 2")

content/posts/homelab-2.md

@@ -1,7 +1,7 @@
 ---
-title: "Homelab 2: Monitoring"
+title: "Homelab 2: Monitoring Boogaloo"
 date: 2023-09-01T21:15:40-07:00
-draft: true
+draft: false
 summary: How metrics and logs work for my homelab
 tags:
   - Prometheus
@@ -13,9 +13,12 @@ tags:
 
 Every good production environment needs robust monitoring in-place before it could be considered production and I feel like my homelab shouldn't be an exception.
 
-With that in mind, I created a monitoring stack that I've used professionally at companies that I've worked at and I'm pretty happy with it.
+With that in mind, I created a monitoring stack that I've used professionally and I'm pretty happy with it.
 
-## stackem
+![dash](/node_exp_dashboard.png)
+Grafana dashboard for host resource metrics
+
+## stackem'
 
 The different components of the system are thus:
 
@@ -27,17 +30,72 @@ The different components of the system are thus:
 Here is what I want to collect metrics for, from most critical to least:
 
 1. The base host resources. This includes available memory, CPU, disk space, network traffic, etc. Also includes the ZFS pool.
-2. Nomad.
-4. The services that run on the host via Nomad.
+2. Nomad itself.
+4. The services that are orchestrated by Nomad.
 
-In essence, monitoring starts for the platform used to deliver my applications and then moves upward to the end services I expose to myself and my friends.
+## tldr prometheus
+
+I won't go too deeply into how Prometheus works; I wouldn't be able to do better than the [official documentation](https://prometheus.io/docs/introduction/overview/). That being said, it's worthwhile to do a quick summary:
+
+* Applications are responsible for presenting their own metrics over HTTP via a dedicated endpoint
+* Prometheus operates on a pull based model. As in, Prometheus will reach out and HTTP GET that endpoint for each application in order to "scrape" those metrics and store them
+* Prometheus has a massive amount of support for different types of service discovery for automatically registering scrape targets
+* Prometheus is a time series database (TSDB) - it is built to store linear, time oriented data that has dimensionality expressed through key-value pairs
 
 ## base system monitoring
 
-Starting from the base, the Prometheus project supports the collection of metrics for Linux based hosts through their [node_exporter](https://github.com/prometheus/node_exporter) project. It's a Golang binary that exposes an absolute treasure trove of data including ZFS stats! It covers my first two monitoring needs.
+For monitoring the base system, the Prometheus project supports the collection of metrics for Linux based hosts through their [node_exporter](https://github.com/prometheus/node_exporter) project. It's a Golang binary that exposes an absolute treasure trove of data... including ZFS stats!
 
-While I could run the `node_exporter` via systemd, I instead opted to use the `exec` driver for Nomad. There is a Nix package for installing the exporter so the job definition just relies on executing the binary itself. Doing it this way means I get visibility into the exporter process and logs via the Nomad dashboard.
+The exporter project recommends running it outside of a container as it needs a deep access to the host system and the isolation of containers are counterintiutive to that level of access.
 
-## everything else
+While I could run the `node_exporter` via systemd, I instead opted to use the `raw_exec` driver for Nomad. There is a Nix package for installing the exporter so the job definition just relies on executing the binary itself. Doing it this way means I get visibility into the exporter process and logs via the Nomad dashboard.
 
-Nomad and Consul both expose their own metrics and so it's easy enough to add a Prometheus stat
+```
+task "node-exporter" {
+  driver = "raw_exec"
+
+  config {
+    command = "/run/current-system/sw/bin/node_exporter"
+  }
+}
+```
+
+There is some security implications for enabling the `raw_exec` driver for Nomad as it will run every process under the same user that the Nomad client is running as - which is usually root. However, it was the path of least resistance and it's a TODO to later use the `exec` driver - which uses cgroups and chroot to isolate the spawned process. This, again, is counterintuitive to the node exporter being able to collect data and would require me to make sure that the task allocation has the correct access to the right resources in order to facilitate complete data collection.
+
+## service discovery brings all the metrics to the yard
+
+Prometheus has support for a plethora of service discovery methods that can be used to help it locate endpoints for scraping. The integrations range from using simple files to integrating with service meshes. For my homelab, I opted to use the [consul](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config) service discovery integration because, well, that's what I'm running.
+
+Whenever a service is deployed to Nomad, it will register an IP address and port to Consul. Prometheus will then talk to Consul to get that address to construct a metrics endpoint and then start scraping metrics an a specified interval.
+
+The configuration for Prometheus looks like this:
+
+```
+scrape_configs:
+  - job_name: 'service_metrics'
+    consul_sd_configs:
+      - server: '{{ env "NOMAD_IP_prometheus" }}:8500'
+    relabel_configs:
+      - source_labels: [__meta_consul_service]
+        target_label: service
+```
+
+Since everything is on the same box, there is no authentication method for Consul and so the only necessary config is the server endpoint. The `$NOMAD_IP_prometheus` is the IP address for the Prometheus task - which is also the IP address for the homelab server itself because I make liberal use of the `bridge` networking type. As a quick note, bridge here means that all tasks share a networking namespace but ingress happens over a port bound to localhost for the Nomad client.
+
+The relabel config exists to take the Consul service source label - in this case, the Consul service label is the name of the service that is running on Nomad - and create a target label of `service` so that every metric is properly tagged. This means that if I deploy a `gitea` service to Nomad then it gets a `service: gitea` label in Prometheus.
+
+Additionally, I want to monitor Nomad itself and, well, it so happens that Nomad registers itself with Consul and so it gets collected as well. The only difference is that the `metrics_path` has to manually changed to `/v1/metrics`.
+
+## logs are, uh, not done
+
+Logging is incredibly import and it's on my TODO list to create a better log collection setup but for now I just use Nomad's log UI.
+
+![gitea_logs](/nomad_gitea_logs.png)
+
+## future todo
+
+Something that I'm excited to play with is using eBPF to construct fine-grained networking monitoring for the box to better monitor traffic flowing between tasks. There is a lot of [documentation](https://www.redhat.com/en/blog/monitoring-ebpf-based-metrics) for this - I just need to clear my calendar for it.
+
+Stay tuned for part 3
+
+---

deploy/job.nomad

@@ -4,8 +4,6 @@ job "blog" {
 
   group "blog" {
     network {
-      mode = "bridge"
-
       port "http" {
         to = 80
       }
@@ -33,7 +31,7 @@ job "blog" {
       driver = "docker"
 
       config {
-        image = "git.howdoicomputer.lol/howdoicomputer/blog:5"
+        image = "git.howdoicomputer.lol/howdoicomputer/blog:7"
         ports = ["http"]
       }
 

public/about/index.html

@@ -58,7 +58,7 @@
             <section class="page__aside">
                 <div class="aside__about">
 <div class="aside__about">
-    
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
 <h1 class="about__title">howdoicomputer&#39;s blog</h1>
 <p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
 </div>

public/categories/index.html

@@ -52,7 +52,7 @@
             <section class="page__aside">
                 <div class="aside__about">
 <div class="aside__about">
-    
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
 <h1 class="about__title">howdoicomputer&#39;s blog</h1>
 <p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
 </div>

public/index.html

@@ -2,7 +2,7 @@
 <html lang="en">
 
     <head>
-	<meta name="generator" content="Hugo 0.117.0"><title>howdoicomputer &ndash; howdoicomputer</title>
+	<meta name="generator" content="Hugo 0.119.0"><title>howdoicomputer &ndash; howdoicomputer</title>
 <meta name="description" content="A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making">
 
 <meta name="viewport" content="width=device-width, initial-scale=1">
@@ -41,10 +41,15 @@
             <section class="page__body">
   <ul class="contents">
     
-    
-    <li><a href="https://howdoicomputer.lol/posts/">Posts</a>
+    <li><a href="https://howdoicomputer.lol/posts/homelab-2/">Homelab 2: Monitoring Boogaloo</a>
         <div>
+          How metrics and logs work for my homelab
+        </div>
+    </li>
     
+    <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab 1: Genesis</a>
+        <div>
+          First part of a multi-part series about my current homelab
         </div>
     </li>
     
@@ -55,7 +60,7 @@
             <section class="page__aside">
                 <div class="aside__about">
 <div class="aside__about">
-    
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
 <h1 class="about__title">howdoicomputer&#39;s blog</h1>
 <p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
 </div>

public/index.xml

@@ -6,9 +6,18 @@
     <description>Recent content on howdoicomputer</description>
     <generator>Hugo -- gohugo.io</generator>
     <language>en-us</language>
-    <lastBuildDate>Sun, 27 Aug 2023 18:36:47 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/index.xml" rel="self" type="application/rss+xml" />
+    <lastBuildDate>Fri, 01 Sep 2023 21:15:40 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/index.xml" rel="self" type="application/rss+xml" />
     <item>
-      <title>Homelab: Part 1</title>
+      <title>Homelab 2: Monitoring Boogaloo</title>
+      <link>https://howdoicomputer.lol/posts/homelab-2/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/posts/homelab-2/</guid>
+      <description>How metrics and logs work for my homelab</description>
+    </item>
+    
+    <item>
+      <title>Homelab 1: Genesis</title>
       <link>https://howdoicomputer.lol/posts/homelab-1/</link>
       <pubDate>Sun, 27 Aug 2023 18:36:47 -0700</pubDate>
       

public/posts/homelab-1/index.html

@@ -1,7 +1,7 @@
 <!DOCTYPE html>
 <html lang="en">
 
-    <head><title>Homelab: Part 1 &ndash; howdoicomputer</title>
+    <head><title>Homelab 1: Genesis &ndash; howdoicomputer</title>
 <meta name="description" content="A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making">
 
 <meta name="viewport" content="width=device-width, initial-scale=1">
@@ -39,13 +39,12 @@
 
             <section class="page__body">
     <header class="content__header">
-        <h1>Homelab: Part 1</h1>
+        <h1>Homelab 1: Genesis</h1>
     </header>
     
     <div class="content__body">
-        <p>This blog. The one that you are currently reading right now. Is running on a metal box that is just barely large enough to fit the average house cat. It cost ~$800 USD to build and has been christened Megamind. The HTTP request that you initiated when you visited <a href="https://blog.howdoicomputer.lol">https://blog.howdoicomputer.lol</a> has returned digital bits from hardware that is more than likely a couple feet from where I&rsquo;m sitting. Welcome!</p>
+        <p>This blog. The one that you are currently reading right now. Is running on a metal box that is just barely large enough to fit the average house cat. It cost ~$800 USD to build and has been christened Megamind. The HTTP request that you initiated when you visited <a href="https://howdoicomputer.lol">https://howdoicomputer.lol</a> has returned digital bits from hardware that is more than likely a couple feet from where I&rsquo;m sitting. Welcome!</p>
 <h3 id="yeah-but-why">yeah, but why</h3>
-<p>I like solving problems and these were mine:</p>
 <ul>
 <li>I have a plethora of software projects that I&rsquo;ve written that need a home. Discord bots, daemons, websites, etc.</li>
 <li>I needed a place to host a blog. Hi!</li>
@@ -77,7 +76,7 @@
 <li>The operating system is NixOS and I use my laptop to do flake based deployments to the underlying OS</li>
 </ul>
 <p>A secret that I&rsquo;ve withheld from you so far is that Megamind isn&rsquo;t my first homelab. A previous homelab that I built was a k3s cluster running on top of Raspbery Pis. It was quaint but got decommissioned when I moved apartments. I never set it up again and it currently lies in pieces in a drawer somewhere.</p>
-<p>So why Nomad? Mostly because, at the time, I was leading an infrastructure initiative at my day job to use Nomad to orchestrate applications that are deployed to heterogenous IoT devices spread across distinct geographic regions. I wanted to dog food my own proposal. Also, Nomad is <em>a lot</em> simpler to use than Kubernetes. Nomad starts as <em>just</em> an orchestrator but gives you the option to introduce complexity later on through Vault and Consul. Every concept is almost immediately grokable and it has just been a pleasure to work with.</p>
+<p>So why Nomad? Mostly because, at the time, I was leading an infrastructure initiative at my day job to use Nomad to orchestrate applications that are deployed to heterogenous IoT devices spread across distinct geographic regions. Nomad was a better fit than k8s for that specific use case because it was lighter-weight, easier to use, had less overhead (which was important because embedded devices don&rsquo;t have a lot of resources), and could run non-containerized applications that needed deep access to hardware. And so I wanted to dog food my own proposal. Also, Nomad is <em>a lot</em> simpler to use than Kubernetes. Nomad starts as <em>just</em> an orchestrator but gives you the option to introduce complexity later on through Vault and Consul. Every concept is almost immediately grokable and it has just been a pleasure to work with.</p>
 <h3 id="whats-runnin">what&rsquo;s runnin</h3>
 <p>Here is what is running so far:</p>
 <ul>
@@ -128,12 +127,12 @@
 </span></span><span style="display:flex;"><span>      tags <span style="color:#f92672">=</span> [
 </span></span><span style="display:flex;"><span>        &#34;traefik.enable<span style="color:#f92672">=</span><span style="color:#66d9ef">true</span><span style="color:#960050;background-color:#1e0010">&#34;</span>,
 </span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-https.tls<span style="color:#f92672">=</span><span style="color:#66d9ef">true</span><span style="color:#960050;background-color:#1e0010">&#34;</span>,
-</span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-https.rule<span style="color:#f92672">=</span><span style="color:#66d9ef">Host</span>(<span style="color:#960050;background-color:#1e0010">`</span><span style="color:#66d9ef">blog</span>.<span style="color:#66d9ef">howdoicomputer</span>.<span style="color:#66d9ef">lol</span><span style="color:#960050;background-color:#1e0010">`</span>)<span style="color:#960050;background-color:#1e0010">&#34;</span>,
+</span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-https.rule<span style="color:#f92672">=</span><span style="color:#66d9ef">Host</span>(<span style="color:#960050;background-color:#1e0010">`</span><span style="color:#66d9ef">howdoicomputer</span>.<span style="color:#66d9ef">lol</span><span style="color:#960050;background-color:#1e0010">`</span>)<span style="color:#960050;background-color:#1e0010">&#34;</span>,
 </span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-https.tls.certresolver<span style="color:#f92672">=</span><span style="color:#66d9ef">resolver</span><span style="color:#960050;background-color:#1e0010">&#34;</span>,
-</span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-https.tls.domains[0].main<span style="color:#f92672">=</span><span style="color:#66d9ef">blog</span>.<span style="color:#66d9ef">howdoicomputer</span>.<span style="color:#66d9ef">lol</span><span style="color:#960050;background-color:#1e0010">&#34;</span>,
+</span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-https.tls.domains[0].main<span style="color:#f92672">=</span><span style="color:#66d9ef">howdoicomputer</span>.<span style="color:#66d9ef">lol</span><span style="color:#960050;background-color:#1e0010">&#34;</span>,
 </span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-https.entrypoints<span style="color:#f92672">=</span><span style="color:#66d9ef">websecure</span><span style="color:#960050;background-color:#1e0010">&#34;</span>,
 </span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-http.entrypoints<span style="color:#f92672">=</span><span style="color:#66d9ef">web</span><span style="color:#960050;background-color:#1e0010">&#34;</span>,
-</span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-http.rule<span style="color:#f92672">=</span><span style="color:#66d9ef">Host</span>(<span style="color:#960050;background-color:#1e0010">`</span><span style="color:#66d9ef">blog</span>.<span style="color:#66d9ef">howdoicomputer</span>.<span style="color:#66d9ef">lol</span><span style="color:#960050;background-color:#1e0010">`</span>)<span style="color:#960050;background-color:#1e0010">&#34;</span>,
+</span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-http.rule<span style="color:#f92672">=</span><span style="color:#66d9ef">Host</span>(<span style="color:#960050;background-color:#1e0010">`</span><span style="color:#66d9ef">howdoicomputer</span>.<span style="color:#66d9ef">lol</span><span style="color:#960050;background-color:#1e0010">`</span>)<span style="color:#960050;background-color:#1e0010">&#34;</span>,
 </span></span><span style="display:flex;"><span>        &#34;traefik.http.routers.blog-http.middlewares<span style="color:#f92672">=</span><span style="color:#66d9ef">redirecthttps</span><span style="color:#960050;background-color:#1e0010">&#34;</span>,
 </span></span><span style="display:flex;"><span>        &#34;traefik.http.middlewares.redirecthttps.redirectscheme.scheme<span style="color:#f92672">=</span><span style="color:#66d9ef">https</span><span style="color:#960050;background-color:#1e0010">&#34;</span>
 </span></span><span style="display:flex;"><span>      ]
@@ -169,8 +168,8 @@
 <h3 id="the-future">the future</h3>
 <p>The future is bright! It has been incredibly fluid to roll out new services to the server. There are two additional features that I want: a CI/CD solution and an auth model for friends who want to use services.</p>
 <p>For CI/CD, I&rsquo;ll most likely go with <a href="https://www.drone.io/">Drone</a>. Really, I just want a UI for Dagger pipelines. For an auth model, eh, I am not sure. LDAP? There is also the possiblity of using Oauth2 and whitelisting accounts.</p>
-<p>I also want to do a thorough lockdown of Gitea so that I can expose my code to the world.</p>
-<p>Till next time.</p>
+<p>For now though, my homelab is now an integral part of my life. For consumption, I use it to store a digital library of books, music, and movies. For productivity, the homelab is a cornerstone of developing applications that I want to expose to the public internet. It&rsquo;s&hellip; neat.</p>
+<p><a href="https://howdoicomputer.lol/posts/homelab-2/" title="Part 2">Part 2</a></p>
 
     </div>
     <footer class="content__footer"></footer>
@@ -180,7 +179,7 @@
             <section class="page__aside">
                 <div class="aside__about">
 <div class="aside__about">
-    
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
 <h1 class="about__title">howdoicomputer&#39;s blog</h1>
 <p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
 </div>

public/posts/homelab-2/index.html

@@ -0,0 +1,161 @@
+<!DOCTYPE html>
+<html lang="en">
+
+    <head><title>Homelab 2: Monitoring Boogaloo &ndash; howdoicomputer</title>
+<meta name="description" content="A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making">
+
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta charset="UTF-8"/>
+
+
+
+<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.1.2/css/all.min.css" integrity="sha512-1sCRPdkRXhBV2PBLUdRb4tMg1w2YPf37qatUFeS7zlBy7jJI8Lf4VHwWfZZfpXtYSLy85pkm9GaYVYMfw5BC1A==" crossorigin="anonymous" />
+
+
+<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/academicons/1.9.1/css/academicons.min.css" integrity="sha512-b1ASx0WHgVFL5ZQhTgiPWX+68KjS38Jk87jg7pe+qC7q9YkEtFq0z7xCglv7qGIs/68d3mAp+StfC8WKC5SSAg==" crossorigin="anonymous" />
+
+
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/palettes/tokyo-night-dark.css">
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/risotto.css">
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/custom.css">
+</head>
+
+    <body>
+        <div class="page">
+
+            <header class="page__header"><nav class="page__nav main-nav">
+    <ul>
+    <h1 class="page__logo"><a href="https://howdoicomputer.lol/" class="page__logo-inner">howdoicomputer</a></h1>
+    
+    
+    <li class="main-nav__item"><a class="nav-main-item" href="https://howdoicomputer.lol/about" title="">About</a></li>
+    
+    <li class="main-nav__item"><a class="nav-main-item active" href="https://howdoicomputer.lol/posts/" title="Posts">Posts</a></li>
+    
+    </ul>
+</nav>
+
+</header>
+
+            <section class="page__body">
+    <header class="content__header">
+        <h1>Homelab 2: Monitoring Boogaloo</h1>
+    </header>
+    
+    <div class="content__body">
+        <p>Every good production environment needs robust monitoring in-place before it could be considered production and I feel like my homelab shouldn&rsquo;t be an exception.</p>
+<p>With that in mind, I created a monitoring stack that I&rsquo;ve used professionally and I&rsquo;m pretty happy with it.</p>
+<p><img src="/node_exp_dashboard.png" alt="dash">
+Grafana dashboard for host resource metrics</p>
+<h2 id="stackem">stackem'</h2>
+<p>The different components of the system are thus:</p>
+<ul>
+<li>Prometheus</li>
+<li>Grafana</li>
+<li>Prometheus&rsquo;s node_exporter</li>
+<li>Consul</li>
+</ul>
+<p>Here is what I want to collect metrics for, from most critical to least:</p>
+<ol>
+<li>The base host resources. This includes available memory, CPU, disk space, network traffic, etc. Also includes the ZFS pool.</li>
+<li>Nomad itself.</li>
+<li>The services that are orchestrated by Nomad.</li>
+</ol>
+<h2 id="tldr-prometheus">tldr prometheus</h2>
+<p>I won&rsquo;t go too deeply into how Prometheus works; I wouldn&rsquo;t be able to do better than the <a href="https://prometheus.io/docs/introduction/overview/">official documentation</a>. That being said, it&rsquo;s worthwhile to do a quick summary:</p>
+<ul>
+<li>Applications are responsible for presenting their own metrics over HTTP via a dedicated endpoint</li>
+<li>Prometheus operates on a pull based model. As in, Prometheus will reach out and HTTP GET that endpoint for each application in order to &ldquo;scrape&rdquo; those metrics and store them</li>
+<li>Prometheus has a massive amount of support for different types of service discovery for automatically registering scrape targets</li>
+<li>Prometheus is a time series database (TSDB) - it is built to store linear, time oriented data that has dimensionality expressed through key-value pairs</li>
+</ul>
+<h2 id="base-system-monitoring">base system monitoring</h2>
+<p>For monitoring the base system, the Prometheus project supports the collection of metrics for Linux based hosts through their <a href="https://github.com/prometheus/node_exporter">node_exporter</a> project. It&rsquo;s a Golang binary that exposes an absolute treasure trove of data&hellip; including ZFS stats!</p>
+<p>The exporter project recommends running it outside of a container as it needs a deep access to the host system and the isolation of containers are counterintiutive to that level of access.</p>
+<p>While I could run the <code>node_exporter</code> via systemd, I instead opted to use the <code>raw_exec</code> driver for Nomad. There is a Nix package for installing the exporter so the job definition just relies on executing the binary itself. Doing it this way means I get visibility into the exporter process and logs via the Nomad dashboard.</p>
+<pre tabindex="0"><code>task &#34;node-exporter&#34; {
+  driver = &#34;raw_exec&#34;
+
+  config {
+    command = &#34;/run/current-system/sw/bin/node_exporter&#34;
+  }
+}
+</code></pre><p>There is some security implications for enabling the <code>raw_exec</code> driver for Nomad as it will run every process under the same user that the Nomad client is running as - which is usually root. However, it was the path of least resistance and it&rsquo;s a TODO to later use the <code>exec</code> driver - which uses cgroups and chroot to isolate the spawned process. This, again, is counterintuitive to the node exporter being able to collect data and would require me to make sure that the task allocation has the correct access to the right resources in order to facilitate complete data collection.</p>
+<h2 id="service-discovery-brings-all-the-metrics-to-the-yard">service discovery brings all the metrics to the yard</h2>
+<p>Prometheus has support for a plethora of service discovery methods that can be used to help it locate endpoints for scraping. The integrations range from using simple files to integrating with service meshes. For my homelab, I opted to use the <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config">consul</a> service discovery integration because, well, that&rsquo;s what I&rsquo;m running.</p>
+<p>Whenever a service is deployed to Nomad, it will register an IP address and port to Consul. Prometheus will then talk to Consul to get that address to construct a metrics endpoint and then start scraping metrics an a specified interval.</p>
+<p>The configuration for Prometheus looks like this:</p>
+<pre tabindex="0"><code>scrape_configs:
+  - job_name: &#39;service_metrics&#39;
+    consul_sd_configs:
+      - server: &#39;{{ env &#34;NOMAD_IP_prometheus&#34; }}:8500&#39;
+    relabel_configs:
+      - source_labels: [__meta_consul_service]
+        target_label: service
+</code></pre><p>Since everything is on the same box, there is no authentication method for Consul and so the only necessary config is the server endpoint. The <code>$NOMAD_IP_prometheus</code> is the IP address for the Prometheus task - which is also the IP address for the homelab server itself because I make liberal use of the <code>bridge</code> networking type. As a quick note, bridge here means that all tasks share a networking namespace but ingress happens over a port bound to localhost for the Nomad client.</p>
+<p>The relabel config exists to take the Consul service source label - in this case, the Consul service label is the name of the service that is running on Nomad - and create a target label of <code>service</code> so that every metric is properly tagged. This means that if I deploy a <code>gitea</code> service to Nomad then it gets a <code>service: gitea</code> label in Prometheus.</p>
+<p>Additionally, I want to monitor Nomad itself and, well, it so happens that Nomad registers itself with Consul and so it gets collected as well. The only difference is that the <code>metrics_path</code> has to manually changed to <code>/v1/metrics</code>.</p>
+<h2 id="logs-are-uh-not-done">logs are, uh, not done</h2>
+<p>Logging is incredibly import and it&rsquo;s on my TODO list to create a better log collection setup but for now I just use Nomad&rsquo;s log UI.</p>
+<p><img src="/nomad_gitea_logs.png" alt="gitea_logs"></p>
+<h2 id="future-todo">future todo</h2>
+<p>Something that I&rsquo;m excited to play with is using eBPF to construct fine-grained networking monitoring for the box to better monitor traffic flowing between tasks. There is a lot of <a href="https://www.redhat.com/en/blog/monitoring-ebpf-based-metrics">documentation</a> for this - I just need to clear my calendar for it.</p>
+<p>Stay tuned for part 3</p>
+<hr>
+
+    </div>
+    <footer class="content__footer"></footer>
+
+            </section>
+
+            <section class="page__aside">
+                <div class="aside__about">
+<div class="aside__about">
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
+<h1 class="about__title">howdoicomputer&#39;s blog</h1>
+<p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
+</div>
+
+
+<ul class="aside__social-links">
+    
+</ul>
+</div>
+                <hr>
+                <div class="aside__content">
+    
+    
+        <p>
+            
+            2023-09-01
+        </p>
+    
+
+    
+
+                </div>
+            </section>
+
+            <footer class="page__footer"><p>
+    
+    
+    
+    
+    
+    
+      
+    
+      
+    
+    
+    
+</p>
+<br /><br />
+<p class="copyright"></p>
+<p class="advertisement">Powered by <a href="https://gohugo.io/">hugo</a> and <a href="https://github.com/joeroe/risotto">risotto</a>.</p>
+</footer>
+
+        </div>
+    </body>
+
+</html>

public/posts/index.html

@@ -44,7 +44,10 @@
 
     <ul>
     
-        <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab: Part 1</a></li>
+        <li><a href="https://howdoicomputer.lol/posts/homelab-2/">Homelab 2: Monitoring Boogaloo</a></li>
+
+    
+        <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab 1: Genesis</a></li>
 
     
     </ul>
@@ -55,7 +58,7 @@
             <section class="page__aside">
                 <div class="aside__about">
 <div class="aside__about">
-    
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
 <h1 class="about__title">howdoicomputer&#39;s blog</h1>
 <p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
 </div>

public/posts/index.xml

@@ -6,9 +6,18 @@
     <description>Recent content in Posts on howdoicomputer</description>
     <generator>Hugo -- gohugo.io</generator>
     <language>en-us</language>
-    <lastBuildDate>Sun, 27 Aug 2023 18:36:47 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/posts/index.xml" rel="self" type="application/rss+xml" />
+    <lastBuildDate>Fri, 01 Sep 2023 21:15:40 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/posts/index.xml" rel="self" type="application/rss+xml" />
     <item>
-      <title>Homelab: Part 1</title>
+      <title>Homelab 2: Monitoring Boogaloo</title>
+      <link>https://howdoicomputer.lol/posts/homelab-2/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/posts/homelab-2/</guid>
+      <description>How metrics and logs work for my homelab</description>
+    </item>
+    
+    <item>
+      <title>Homelab 1: Genesis</title>
       <link>https://howdoicomputer.lol/posts/homelab-1/</link>
       <pubDate>Sun, 27 Aug 2023 18:36:47 -0700</pubDate>
       

public/sitemap.xml

@@ -3,28 +3,40 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://howdoicomputer.lol/tags/consul/</loc>
-    <lastmod>2023-08-27T18:36:47-07:00</lastmod>
+    <lastmod>2023-09-01T21:15:40-07:00</lastmod>
+  </url><url>
+    <loc>https://howdoicomputer.lol/tags/grafana/</loc>
+    <lastmod>2023-09-01T21:15:40-07:00</lastmod>
+  </url><url>
+    <loc>https://howdoicomputer.lol/posts/homelab-2/</loc>
+    <lastmod>2023-09-01T21:15:40-07:00</lastmod>
+  </url><url>
+    <loc>https://howdoicomputer.lol/</loc>
+    <lastmod>2023-09-01T21:15:40-07:00</lastmod>
+  </url><url>
+    <loc>https://howdoicomputer.lol/tags/monitoring/</loc>
+    <lastmod>2023-09-01T21:15:40-07:00</lastmod>
+  </url><url>
+    <loc>https://howdoicomputer.lol/tags/nomad/</loc>
+    <lastmod>2023-09-01T21:15:40-07:00</lastmod>
+  </url><url>
+    <loc>https://howdoicomputer.lol/posts/</loc>
+    <lastmod>2023-09-01T21:15:40-07:00</lastmod>
+  </url><url>
+    <loc>https://howdoicomputer.lol/tags/prometheus/</loc>
+    <lastmod>2023-09-01T21:15:40-07:00</lastmod>
+  </url><url>
+    <loc>https://howdoicomputer.lol/tags/</loc>
+    <lastmod>2023-09-01T21:15:40-07:00</lastmod>
   </url><url>
     <loc>https://howdoicomputer.lol/tags/homelab/</loc>
     <lastmod>2023-08-27T18:36:47-07:00</lastmod>
   </url><url>
     <loc>https://howdoicomputer.lol/posts/homelab-1/</loc>
     <lastmod>2023-08-27T18:36:47-07:00</lastmod>
-  </url><url>
-    <loc>https://howdoicomputer.lol/</loc>
-    <lastmod>2023-08-27T18:36:47-07:00</lastmod>
   </url><url>
     <loc>https://howdoicomputer.lol/tags/nix/</loc>
     <lastmod>2023-08-27T18:36:47-07:00</lastmod>
-  </url><url>
-    <loc>https://howdoicomputer.lol/tags/nomad/</loc>
-    <lastmod>2023-08-27T18:36:47-07:00</lastmod>
-  </url><url>
-    <loc>https://howdoicomputer.lol/posts/</loc>
-    <lastmod>2023-08-27T18:36:47-07:00</lastmod>
-  </url><url>
-    <loc>https://howdoicomputer.lol/tags/</loc>
-    <lastmod>2023-08-27T18:36:47-07:00</lastmod>
   </url><url>
     <loc>https://howdoicomputer.lol/about/</loc>
   </url><url>

public/tags/consul/index.html

@@ -44,7 +44,10 @@
 
     <ul>
     
-        <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab: Part 1</a></li>
+        <li><a href="https://howdoicomputer.lol/posts/homelab-2/">Homelab 2: Monitoring Boogaloo</a></li>
+
+    
+        <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab 1: Genesis</a></li>
 
     
     </ul>
@@ -55,7 +58,7 @@
             <section class="page__aside">
                 <div class="aside__about">
 <div class="aside__about">
-    
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
 <h1 class="about__title">howdoicomputer&#39;s blog</h1>
 <p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
 </div>

public/tags/consul/index.xml

@@ -6,9 +6,18 @@
     <description>Recent content in Consul on howdoicomputer</description>
     <generator>Hugo -- gohugo.io</generator>
     <language>en-us</language>
-    <lastBuildDate>Sun, 27 Aug 2023 18:36:47 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/consul/index.xml" rel="self" type="application/rss+xml" />
+    <lastBuildDate>Fri, 01 Sep 2023 21:15:40 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/consul/index.xml" rel="self" type="application/rss+xml" />
     <item>
-      <title>Homelab: Part 1</title>
+      <title>Homelab 2: Monitoring Boogaloo</title>
+      <link>https://howdoicomputer.lol/posts/homelab-2/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/posts/homelab-2/</guid>
+      <description>How metrics and logs work for my homelab</description>
+    </item>
+    
+    <item>
+      <title>Homelab 1: Genesis</title>
       <link>https://howdoicomputer.lol/posts/homelab-1/</link>
       <pubDate>Sun, 27 Aug 2023 18:36:47 -0700</pubDate>
       

public/tags/grafana/index.html

@@ -0,0 +1,95 @@
+<!DOCTYPE html>
+<html lang="en">
+
+    <head><title>Grafana &ndash; howdoicomputer</title>
+<meta name="description" content="A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making">
+
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta charset="UTF-8"/>
+
+
+
+<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.1.2/css/all.min.css" integrity="sha512-1sCRPdkRXhBV2PBLUdRb4tMg1w2YPf37qatUFeS7zlBy7jJI8Lf4VHwWfZZfpXtYSLy85pkm9GaYVYMfw5BC1A==" crossorigin="anonymous" />
+
+
+<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/academicons/1.9.1/css/academicons.min.css" integrity="sha512-b1ASx0WHgVFL5ZQhTgiPWX+68KjS38Jk87jg7pe+qC7q9YkEtFq0z7xCglv7qGIs/68d3mAp+StfC8WKC5SSAg==" crossorigin="anonymous" />
+
+
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/palettes/tokyo-night-dark.css">
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/risotto.css">
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/custom.css">
+</head>
+
+    <body>
+        <div class="page">
+
+            <header class="page__header"><nav class="page__nav main-nav">
+    <ul>
+    <h1 class="page__logo"><a href="https://howdoicomputer.lol/" class="page__logo-inner">howdoicomputer</a></h1>
+    
+    
+    <li class="main-nav__item"><a class="nav-main-item" href="https://howdoicomputer.lol/about" title="">About</a></li>
+    
+    <li class="main-nav__item"><a class="nav-main-item" href="https://howdoicomputer.lol/posts/" title="Posts">Posts</a></li>
+    
+    </ul>
+</nav>
+
+</header>
+
+            <section class="page__body">
+    <h1 id="grafana">Grafana</h1>
+
+    
+
+    <ul>
+    
+        <li><a href="https://howdoicomputer.lol/posts/homelab-2/">Homelab 2: Monitoring Boogaloo</a></li>
+
+    
+    </ul>
+
+
+            </section>
+
+            <section class="page__aside">
+                <div class="aside__about">
+<div class="aside__about">
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
+<h1 class="about__title">howdoicomputer&#39;s blog</h1>
+<p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
+</div>
+
+
+<ul class="aside__social-links">
+    
+</ul>
+</div>
+                <hr>
+                <div class="aside__content">
+                </div>
+            </section>
+
+            <footer class="page__footer"><p>
+    
+    
+    
+    
+    
+    
+      
+    
+      
+    
+    
+    
+</p>
+<br /><br />
+<p class="copyright"></p>
+<p class="advertisement">Powered by <a href="https://gohugo.io/">hugo</a> and <a href="https://github.com/joeroe/risotto">risotto</a>.</p>
+</footer>
+
+        </div>
+    </body>
+
+</html>

public/tags/grafana/index.xml

@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
+  <channel>
+    <title>Grafana on howdoicomputer</title>
+    <link>https://howdoicomputer.lol/tags/grafana/</link>
+    <description>Recent content in Grafana on howdoicomputer</description>
+    <generator>Hugo -- gohugo.io</generator>
+    <language>en-us</language>
+    <lastBuildDate>Fri, 01 Sep 2023 21:15:40 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/grafana/index.xml" rel="self" type="application/rss+xml" />
+    <item>
+      <title>Homelab 2: Monitoring Boogaloo</title>
+      <link>https://howdoicomputer.lol/posts/homelab-2/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/posts/homelab-2/</guid>
+      <description>How metrics and logs work for my homelab</description>
+    </item>
+    
+  </channel>
+</rss>

public/tags/homelab/index.html

@@ -44,7 +44,7 @@
 
     <ul>
     
-        <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab: Part 1</a></li>
+        <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab 1: Genesis</a></li>
 
     
     </ul>
@@ -55,7 +55,7 @@
             <section class="page__aside">
                 <div class="aside__about">
 <div class="aside__about">
-    
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
 <h1 class="about__title">howdoicomputer&#39;s blog</h1>
 <p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
 </div>

public/tags/homelab/index.xml

@@ -8,7 +8,7 @@
     <language>en-us</language>
     <lastBuildDate>Sun, 27 Aug 2023 18:36:47 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/homelab/index.xml" rel="self" type="application/rss+xml" />
     <item>
-      <title>Homelab: Part 1</title>
+      <title>Homelab 1: Genesis</title>
       <link>https://howdoicomputer.lol/posts/homelab-1/</link>
       <pubDate>Sun, 27 Aug 2023 18:36:47 -0700</pubDate>
       

public/tags/index.html

@@ -47,15 +47,24 @@
         <li><a href="https://howdoicomputer.lol/tags/consul/">Consul</a></li>
 
     
+        <li><a href="https://howdoicomputer.lol/tags/grafana/">Grafana</a></li>
+
+    
+        <li><a href="https://howdoicomputer.lol/tags/monitoring/">Monitoring</a></li>
+
+    
+        <li><a href="https://howdoicomputer.lol/tags/nomad/">Nomad</a></li>
+
+    
+        <li><a href="https://howdoicomputer.lol/tags/prometheus/">Prometheus</a></li>
+
+    
         <li><a href="https://howdoicomputer.lol/tags/homelab/">Homelab</a></li>
 
     
         <li><a href="https://howdoicomputer.lol/tags/nix/">Nix</a></li>
 
     
-        <li><a href="https://howdoicomputer.lol/tags/nomad/">Nomad</a></li>
-
-    
     </ul>
 
 
@@ -64,7 +73,7 @@
             <section class="page__aside">
                 <div class="aside__about">
 <div class="aside__about">
-    
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
 <h1 class="about__title">howdoicomputer&#39;s blog</h1>
 <p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
 </div>

public/tags/index.xml

@@ -6,16 +6,52 @@
     <description>Recent content in Tags on howdoicomputer</description>
     <generator>Hugo -- gohugo.io</generator>
     <language>en-us</language>
-    <lastBuildDate>Sun, 27 Aug 2023 18:36:47 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/index.xml" rel="self" type="application/rss+xml" />
+    <lastBuildDate>Fri, 01 Sep 2023 21:15:40 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/index.xml" rel="self" type="application/rss+xml" />
     <item>
       <title>Consul</title>
       <link>https://howdoicomputer.lol/tags/consul/</link>
-      <pubDate>Sun, 27 Aug 2023 18:36:47 -0700</pubDate>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
       
       <guid>https://howdoicomputer.lol/tags/consul/</guid>
       <description></description>
     </item>
     
+    <item>
+      <title>Grafana</title>
+      <link>https://howdoicomputer.lol/tags/grafana/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/tags/grafana/</guid>
+      <description></description>
+    </item>
+    
+    <item>
+      <title>Monitoring</title>
+      <link>https://howdoicomputer.lol/tags/monitoring/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/tags/monitoring/</guid>
+      <description></description>
+    </item>
+    
+    <item>
+      <title>Nomad</title>
+      <link>https://howdoicomputer.lol/tags/nomad/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/tags/nomad/</guid>
+      <description></description>
+    </item>
+    
+    <item>
+      <title>Prometheus</title>
+      <link>https://howdoicomputer.lol/tags/prometheus/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/tags/prometheus/</guid>
+      <description></description>
+    </item>
+    
     <item>
       <title>Homelab</title>
       <link>https://howdoicomputer.lol/tags/homelab/</link>
@@ -34,14 +70,5 @@
       <description></description>
     </item>
     
-    <item>
-      <title>Nomad</title>
-      <link>https://howdoicomputer.lol/tags/nomad/</link>
-      <pubDate>Sun, 27 Aug 2023 18:36:47 -0700</pubDate>
-      
-      <guid>https://howdoicomputer.lol/tags/nomad/</guid>
-      <description></description>
-    </item>
-    
   </channel>
 </rss>

public/tags/monitoring/index.html

@@ -0,0 +1,95 @@
+<!DOCTYPE html>
+<html lang="en">
+
+    <head><title>Monitoring &ndash; howdoicomputer</title>
+<meta name="description" content="A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making">
+
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta charset="UTF-8"/>
+
+
+
+<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.1.2/css/all.min.css" integrity="sha512-1sCRPdkRXhBV2PBLUdRb4tMg1w2YPf37qatUFeS7zlBy7jJI8Lf4VHwWfZZfpXtYSLy85pkm9GaYVYMfw5BC1A==" crossorigin="anonymous" />
+
+
+<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/academicons/1.9.1/css/academicons.min.css" integrity="sha512-b1ASx0WHgVFL5ZQhTgiPWX+68KjS38Jk87jg7pe+qC7q9YkEtFq0z7xCglv7qGIs/68d3mAp+StfC8WKC5SSAg==" crossorigin="anonymous" />
+
+
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/palettes/tokyo-night-dark.css">
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/risotto.css">
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/custom.css">
+</head>
+
+    <body>
+        <div class="page">
+
+            <header class="page__header"><nav class="page__nav main-nav">
+    <ul>
+    <h1 class="page__logo"><a href="https://howdoicomputer.lol/" class="page__logo-inner">howdoicomputer</a></h1>
+    
+    
+    <li class="main-nav__item"><a class="nav-main-item" href="https://howdoicomputer.lol/about" title="">About</a></li>
+    
+    <li class="main-nav__item"><a class="nav-main-item" href="https://howdoicomputer.lol/posts/" title="Posts">Posts</a></li>
+    
+    </ul>
+</nav>
+
+</header>
+
+            <section class="page__body">
+    <h1 id="monitoring">Monitoring</h1>
+
+    
+
+    <ul>
+    
+        <li><a href="https://howdoicomputer.lol/posts/homelab-2/">Homelab 2: Monitoring Boogaloo</a></li>
+
+    
+    </ul>
+
+
+            </section>
+
+            <section class="page__aside">
+                <div class="aside__about">
+<div class="aside__about">
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
+<h1 class="about__title">howdoicomputer&#39;s blog</h1>
+<p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
+</div>
+
+
+<ul class="aside__social-links">
+    
+</ul>
+</div>
+                <hr>
+                <div class="aside__content">
+                </div>
+            </section>
+
+            <footer class="page__footer"><p>
+    
+    
+    
+    
+    
+    
+      
+    
+      
+    
+    
+    
+</p>
+<br /><br />
+<p class="copyright"></p>
+<p class="advertisement">Powered by <a href="https://gohugo.io/">hugo</a> and <a href="https://github.com/joeroe/risotto">risotto</a>.</p>
+</footer>
+
+        </div>
+    </body>
+
+</html>

public/tags/monitoring/index.xml

@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
+  <channel>
+    <title>Monitoring on howdoicomputer</title>
+    <link>https://howdoicomputer.lol/tags/monitoring/</link>
+    <description>Recent content in Monitoring on howdoicomputer</description>
+    <generator>Hugo -- gohugo.io</generator>
+    <language>en-us</language>
+    <lastBuildDate>Fri, 01 Sep 2023 21:15:40 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/monitoring/index.xml" rel="self" type="application/rss+xml" />
+    <item>
+      <title>Homelab 2: Monitoring Boogaloo</title>
+      <link>https://howdoicomputer.lol/posts/homelab-2/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/posts/homelab-2/</guid>
+      <description>How metrics and logs work for my homelab</description>
+    </item>
+    
+  </channel>
+</rss>

public/tags/nix/index.html

@@ -44,7 +44,7 @@
 
     <ul>
     
-        <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab: Part 1</a></li>
+        <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab 1: Genesis</a></li>
 
     
     </ul>
@@ -55,7 +55,7 @@
             <section class="page__aside">
                 <div class="aside__about">
 <div class="aside__about">
-    
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
 <h1 class="about__title">howdoicomputer&#39;s blog</h1>
 <p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
 </div>

public/tags/nix/index.xml

@@ -8,7 +8,7 @@
     <language>en-us</language>
     <lastBuildDate>Sun, 27 Aug 2023 18:36:47 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/nix/index.xml" rel="self" type="application/rss+xml" />
     <item>
-      <title>Homelab: Part 1</title>
+      <title>Homelab 1: Genesis</title>
       <link>https://howdoicomputer.lol/posts/homelab-1/</link>
       <pubDate>Sun, 27 Aug 2023 18:36:47 -0700</pubDate>
       

public/tags/nomad/index.html

@@ -44,7 +44,10 @@
 
     <ul>
     
-        <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab: Part 1</a></li>
+        <li><a href="https://howdoicomputer.lol/posts/homelab-2/">Homelab 2: Monitoring Boogaloo</a></li>
+
+    
+        <li><a href="https://howdoicomputer.lol/posts/homelab-1/">Homelab 1: Genesis</a></li>
 
     
     </ul>
@@ -55,7 +58,7 @@
             <section class="page__aside">
                 <div class="aside__about">
 <div class="aside__about">
-    
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
 <h1 class="about__title">howdoicomputer&#39;s blog</h1>
 <p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
 </div>

public/tags/nomad/index.xml

@@ -6,9 +6,18 @@
     <description>Recent content in Nomad on howdoicomputer</description>
     <generator>Hugo -- gohugo.io</generator>
     <language>en-us</language>
-    <lastBuildDate>Sun, 27 Aug 2023 18:36:47 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/nomad/index.xml" rel="self" type="application/rss+xml" />
+    <lastBuildDate>Fri, 01 Sep 2023 21:15:40 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/nomad/index.xml" rel="self" type="application/rss+xml" />
     <item>
-      <title>Homelab: Part 1</title>
+      <title>Homelab 2: Monitoring Boogaloo</title>
+      <link>https://howdoicomputer.lol/posts/homelab-2/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/posts/homelab-2/</guid>
+      <description>How metrics and logs work for my homelab</description>
+    </item>
+    
+    <item>
+      <title>Homelab 1: Genesis</title>
       <link>https://howdoicomputer.lol/posts/homelab-1/</link>
       <pubDate>Sun, 27 Aug 2023 18:36:47 -0700</pubDate>
       

public/tags/prometheus/index.html

@@ -0,0 +1,95 @@
+<!DOCTYPE html>
+<html lang="en">
+
+    <head><title>Prometheus &ndash; howdoicomputer</title>
+<meta name="description" content="A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making">
+
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta charset="UTF-8"/>
+
+
+
+<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.1.2/css/all.min.css" integrity="sha512-1sCRPdkRXhBV2PBLUdRb4tMg1w2YPf37qatUFeS7zlBy7jJI8Lf4VHwWfZZfpXtYSLy85pkm9GaYVYMfw5BC1A==" crossorigin="anonymous" />
+
+
+<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/academicons/1.9.1/css/academicons.min.css" integrity="sha512-b1ASx0WHgVFL5ZQhTgiPWX+68KjS38Jk87jg7pe+qC7q9YkEtFq0z7xCglv7qGIs/68d3mAp+StfC8WKC5SSAg==" crossorigin="anonymous" />
+
+
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/palettes/tokyo-night-dark.css">
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/risotto.css">
+<link rel="stylesheet" href="https://howdoicomputer.lol/css/custom.css">
+</head>
+
+    <body>
+        <div class="page">
+
+            <header class="page__header"><nav class="page__nav main-nav">
+    <ul>
+    <h1 class="page__logo"><a href="https://howdoicomputer.lol/" class="page__logo-inner">howdoicomputer</a></h1>
+    
+    
+    <li class="main-nav__item"><a class="nav-main-item" href="https://howdoicomputer.lol/about" title="">About</a></li>
+    
+    <li class="main-nav__item"><a class="nav-main-item" href="https://howdoicomputer.lol/posts/" title="Posts">Posts</a></li>
+    
+    </ul>
+</nav>
+
+</header>
+
+            <section class="page__body">
+    <h1 id="prometheus">Prometheus</h1>
+
+    
+
+    <ul>
+    
+        <li><a href="https://howdoicomputer.lol/posts/homelab-2/">Homelab 2: Monitoring Boogaloo</a></li>
+
+    
+    </ul>
+
+
+            </section>
+
+            <section class="page__aside">
+                <div class="aside__about">
+<div class="aside__about">
+    <img class="about__logo" src="https://howdoicomputer.lol/favicon.ico" alt="Logo">
+<h1 class="about__title">howdoicomputer&#39;s blog</h1>
+<p class="about__description">A dumping ground for ideas related to making, tomfoolery, and tomfoolery related to making</p>
+</div>
+
+
+<ul class="aside__social-links">
+    
+</ul>
+</div>
+                <hr>
+                <div class="aside__content">
+                </div>
+            </section>
+
+            <footer class="page__footer"><p>
+    
+    
+    
+    
+    
+    
+      
+    
+      
+    
+    
+    
+</p>
+<br /><br />
+<p class="copyright"></p>
+<p class="advertisement">Powered by <a href="https://gohugo.io/">hugo</a> and <a href="https://github.com/joeroe/risotto">risotto</a>.</p>
+</footer>
+
+        </div>
+    </body>
+
+</html>

public/tags/prometheus/index.xml

@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
+  <channel>
+    <title>Prometheus on howdoicomputer</title>
+    <link>https://howdoicomputer.lol/tags/prometheus/</link>
+    <description>Recent content in Prometheus on howdoicomputer</description>
+    <generator>Hugo -- gohugo.io</generator>
+    <language>en-us</language>
+    <lastBuildDate>Fri, 01 Sep 2023 21:15:40 -0700</lastBuildDate><atom:link href="https://howdoicomputer.lol/tags/prometheus/index.xml" rel="self" type="application/rss+xml" />
+    <item>
+      <title>Homelab 2: Monitoring Boogaloo</title>
+      <link>https://howdoicomputer.lol/posts/homelab-2/</link>
+      <pubDate>Fri, 01 Sep 2023 21:15:40 -0700</pubDate>
+      
+      <guid>https://howdoicomputer.lol/posts/homelab-2/</guid>
+      <description>How metrics and logs work for my homelab</description>
+    </item>
+    
+  </channel>
+</rss>