4.8 KiB
| title | date | draft |
|---|---|---|
| Dagger: Platform Engineer's Dream | 2023-11-05T19:53:24-08:00 | true |
but what is dagger
Dagger is a multi-language toolkit that allows you to create CI/CD pipelines that are backed by Docker. Imagine if make was an API where you could write targets in Go, TypeScript, Python, etc and then each target was ran inside a Docker container and the results of each container was cached.
For example, let's say we have a web API written in Rust. For this hypothetical project, we want to test, build, upload, and deploy. Test would be running cargo test, build would be either creating a container image or an executable, upload would be uploading a container image to a registry, and deploy would be running helm upgrade to upgrade an existing Kubernetes Deployment.
With Dagger, each of those steps could be written using Rust, with each step executing inside a container image built for that step, all running on a developers laptop. When that pipeline needs to be slotted into Jenkins or any other CI, all the CI system needs to be able to do is run Dagger.
But we aren't just limited to a make target analogue. Dagger pipelines are written using a REAL programming language - not a DSL! That means anything is possible! Do you need to fetch a secret from a secret store? Do you need to support a custom versioning process using git tags? Do you need to communicate with internal services when deployments occur? You can do all of these things using a language that you're comfortable with. Gone are the days of hunting down long abandoned Jenkins plugins written in Groovy and riddled with security vulnerabilities just to add simple functionality to your pipelines. Write it in a language you love instead. <3
internal dsl library
At a certain scale, it's not uncommon for a company to have a blessed path for developers to onboard their applications and processes into a CI runner like Jenkins, TeamCity, or GitHub Actions. This typically coalesces as a singular library that a platform engineering or release engineering team builds and then provides to the greater engineering org as a sort of an SDK for CI/CD processes. If your company is running Jenkins, for example, you might have a jenkins-utilty-lib that gets imported into DSL files that are stored in application repositories and that utility lib would have functions for enforcing company standards or abstracting complex release processes.
Dagger can replace those DSL libraries with code that the platform engineering team favors rather than forcing them to write Groovy. But... what if they choose isn't favored by everyone else?
any language, comrade
Okay, so internal DSL libraries are common and Dagger allows you to write them in any language. But what if your platform engineering team really wants to code in Go but your front-end engineers really want to code in TypeScript? What if the PE team writes some Go functions in Dagger that uploads static assets to a CDN? Does the FE team need to write Go to use those functions?
Well, no.
Dagger has the ability to expose pipeline steps across languages. As long as the output of a function is serializable, it can be consumed by TypeScript from Go (or any other pairing). In the above example, the platform engineering team can write build pipelines that are compatible with front-end's pipelines even if they aren't in the same language.
but what about CLI tools
So, for example, let's say we have a CLI tool written in Ruby and it requires a specific set of libraries installed and a sane Ruby environment. This Ruby CLI could be some internal tool used by your company to automate some process. However, it's kind of unwieldy to have every engineer install Ruby on their machine alongside this fantasy tool's dependencies.
With Dagger modules, you could easily provide a way for end users to consume your API by not having them install dependencies.
Using a Python dagger module as an example:
dagger mod install github.com/helderco/daggerverse/python@ba7984f47857794ff5aace82947aaf6b433595be
export DAGGER_MODULE=github.com/helderco/daggerverse/python
dagger up http-server -n --src .
The above commands will install the Python Dagger module and then use it to spin up a Python container and run http-server to serve files.
caching, k8s, and other good stuff
Because Dagger uses Docker BuildKit under the hood it also has support for caching operations and artifacts from pipeline steps. Additionally, you can run the Dagger Engine on Kubernetes and have it run your CI/CD so that the build caches are shared across multiple CI runs. This means that there is a TON of opportunity for time savings across multiple runs.
Seriously, check it out. It's neat.