This is our opinionated chezmoi- and asdf-based "engineering sandbox home" setup for polyglot software development or any other "creator tasks" that are performed on Linux-like operating systems.
If you're using Windows 10/11 with WSL2, create a "disposable" Linux instance using Powershell CLI or Windows Store. This project treats the WSL2 instance as "disposable" meaning it's for development only and can easily be destroyed and recreated whenever necessary. The cost for creation and destruction for a Engineering Sandbox should be so low that it should be treated almost as a container rather than a VM. This means everything done in a sandbox should be scripted, with the scripts stored in GitHub for easy re-running through Fish shell or chezmoi
.
Use Kali Linux
Rolling version as our preferred distribution (see kali-linux for WSL, freely available in the Windows Store for WSL2) or as VMs in Hyper-V. Any Debian-based distro which supports Fish Shell 3.6+ should also work, including Ubuntu 20.04 LTS, Ubuntu 22.04 LTS, Debian 11+ with Fish upgrades, etc.
If you're using Windows WSL, you can use these commands to install/uninstall our preferred distro:
$ wsl --unregister kali-linux
$ wsl --install -d kali-linux
If you're using a Debian-based distro you should be able to run this repo in any Debian user account. It will probably work with any Linux-like OS but has only been tested on Debian-based distros (e.g. Kali Linux and Ubuntu 20.04 LTS).
Bootstrap our preferred Kali environment with required utilities (be sure to use bootstrap-admin-ubuntu.sh
or bootstrap-admin-debian.sh
if you're not using Kali):
cd $HOME && sudo apt-get -qq update && sudo apt-get install curl -y -qq && \
sudo apt-get -qq update && sudo apt-get -qq install -y lsb-release && \
curl -fsSL https://raw.githubusercontent.com/netspective-labs/home-polyglot/master/bootstrap-admin-kali.sh | bash
Once the admin (sudo
) part of the boostrap is complete, continue with non-admin:
curl -fsSL https://raw.githubusercontent.com/netspective-labs/home-polyglot/master/bootstrap-common.sh | bash
We use chezmoi with templates to manage our dotfiles across multiple diverse machines, securely. The bootstrap-*
script has already created the chezmoi
config file which you should personalize before installing chezmoi
. See chezmoi.toml Example to help understand the variables that can be set and used across chezmoi templates.
vim.tiny ~/.config/chezmoi/chezmoi.toml
Install chezmoi
and generate configuration files based on values in Netspective Labs Home chezmoi
templates:
sh -c "$(curl -fsSL git.io/chezmoi)" -- init --apply netspective-labs/home-polyglot
We prefer Fish
as the default shell and Oh My Posh
as the CLI prompts theme manager. These are configured automatically by chezmoi
's first-time configuration. You should switch your user's default shell to Fish
by running:
chsh -s /usr/bin/fish
exit
At this point the default configuration should be complete and you can start using your NLH workspaces.
- Generate GitHub personal access tokens and update
$HOME/.config/chezmoi/chezmoi.toml
file (this file is created at installation and is private to the user). Then, runchez apply
to regenerate all configuration files that use the globalchezmoi.toml
file. $HOME/.pgpass
should follow PostgreSQL .pgpass rules for password management.
Regularly run, or when github.com/netspective-labs/home-polyglot
repo is updated:
chez upgrade
chez update
** DO NOT EDIT ** chezmoi-managed files. To see which files are managed by chezmoi run chezmoi managed and edit those using guidance in the Contributing section below.
There are a few chezmoi-managed scripts that are automatically run when necessary:
run_once_dot_eget.toml.sh.tmpl
run_once_install-packages.sh.tmpl
These and other "managed" scripts show up like this:
$ chezmoi managed | grep '\.sh$'
.eget.toml.sh
install-packages.sh
- If you ever need to run them manually (such as when chezmoi or NL Aide libs are changed or an error occurs and you need to force the execution), as well as forcefully install the utilitize mentioned in this script to latest version. you would use:
$ chezmoi state delete-bucket --bucket=scriptState
$ chezmoi apply
See Clear the state of run_once_ scripts in chezmoi
documentation for more information about how to force execution of scripts instead of using memoized state.
To start Prometheus, prepare the prometheus.yml configuration file at any directory of your choice. Change to the directory containing the Prometheus configuration file and run:
# By default, Prometheus stores its database in ./data (flag --storage.tsdb.path).
prometheus --config.file=prometheus.yml
Prometheus should start up. You should also be able to browse to a status page about itself at localhost:9090
To see which files are managed by chezmoi
run chezmoi managed
. Never edit any managed without using chez edit
or opening the files in the chezmoi
source directory. Use chez edit <managed-file> --apply
like chez edit ~/.config/fish/config.fish --apply
when you want to make quick edits to individual files and apply the changes immediately.
An easier way to modify these file is to use VS Code to edit and manage chezmoi
templates and configurations using the chez-code
alias, which is basically the same as running chezmoi cd
and then opening VS Code.
Be sure to follow the chezmoi workflows for editing configuration files and use chezmoi apply
locally to do your testing.
Whenver possible, create chezmoi
templates that generate configs (especially when secrets are involved, like in .gitconfig
).
PRs are welcome. If you're making changes directly (without a PR), after updating and before pushing code, tag the release:
chezmoi cd
# <git commit ...>
git-semtag final && git push
# or git-semtag final -v "vN.N.N" && git push
Study these chezmoi-tagged repos:
- https://github.com/twpayne/dotfiles
- https://github.com/felipecrs/dotfiles
- https://github.com/renemarc/dotfiles
They will have good ideas about how to properly create fully configurable home
directories across all of our polyglot engineering stations.
A project is only as useful as its documentation so if you contribute to or modify code in this repo be sure to document it using this priority:
- Follow guidance and conventions for Fish (e.g. use
~/.config/fish/*
locations),asdf
,direnv
, etc. so that developers can easily understand your work - Add comments to scripts that explain not just what is being done but, more importantly, why
- Whenever possible, explain concepts through visualizations using Draw.io (or D2/PlantUML/diagram-as-code utilities).
- Instead of Visio or any other desktop-based tools please use the hediet.vscode-drawio VS code extension's
*.drawio.svg
and*.drawio.png
capabilities. This allows you to edit the Draw.io files visually but an.svg
or.png
is automatically created for the repo (which can then be referenced/linked in Markdown like README.me).
- Instead of Visio or any other desktop-based tools please use the hediet.vscode-drawio VS code extension's
We use eget to install prebuilt binaries from GitHub. eget
works great when all we care about is the latest version of a single binary from a particular GitHub repo. In case we care about versioning per-project / per-directory then we should switch to asdf
, nix
, etc.
We use asdf
to manage languages and utilities when deterministic reproducibility is not crucial. asdf
enables tools to be installed and, more importantly, support multiple versions simultaneously. For example, we heavily use Deno
for multiple projects but each project might require a different version. asdf
supports global, per session, and per project (directory) version configuration strategy.
asdf
has centrally managed plugins for many languages and runtimes and there are even more contributed plugins for additional languages and runtimes.
There are good asdf videos worth watching.
In addition to asdf
which supports a flexible version configuration strategy for languages and runtimes, we use direnv to encourage usage of environment variables with per-directory flexibility. Per their documentation:
direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.
We use direnv
and .envrc
files to manage environments on a per-directory (per-project and descendant directories) basis. direnv
can be used to manage secrets as well as non-secret configurations. Many other development automation techniques are possible.
There are some direnv YouTube videos worth watching to get familar with the capabilities.
You can install languages and other packages like this:
asdf plugin add golang
asdf plugin add nodejs
asdf install golang latest
asdf install nodejs latest
asdf global golang latest
asdf global nodejs latest
...
asdf current
Or, use the convenience tasks in ~/bin
:
asdf-setup-plugin java # Install the plugin and its latest stable version but don't set the version
asdf-setup-plugin julia
...
asdf-setup-plugin-global hugo # Install the named plugin, its latest stable release, and then set it as the global version
asdf-setup-plugin-global python
asdf-setup-plugin-global haxe
asdf-setup-plugin-global neko
...
asdf current
Per asdf .tool-versions documentation:
To install all the tools defined in a
.tool-versions
file run asdf install with no other arguments in the directory containing the.tool-versions
file.
To install a single tool defined in a
.tool-versions
file run asdf install in the directory containing the.tool-versions
file. The tool will be installed at the version specified in the.tool-versions
file.
Edit the file directly or use
asdf local
(orasdf global
) which updates it.
This might better than trying to give installation instructions in README.md
and other per-project files.
Basically in each of our Git repos we can give .tool-versions
and then each project user can just run asdf install
.
- We use
$HOME/bin
for binaries whenever possible instead globally installing them usingsudo
. - We use
direnv
and per-directory.envrc
to help manage secrets and env-based configurations per-project rather than globally.
Run nlh-doctor
to get list of useful packages and versions included. Some highlights:
- We use fish shell for our CLI.
- We use
git
andgit-extras
and define manygit-*
individual scripts (e.g.mGit
) because we're a GitOps shop. - We use asdf for basic tools isolation.
- We use deno for custom scripting and
dax
command runner to execute tasks (available in$HOME/bin
). We favordeno
overmake
for new packages butmake
is still a great tool for legacy requirements. If we create complex scripts that need to perform shell manipulation,deno
with dax is preferred over making system calls indeno
. - We use pass the standard unix password manager for managing secrets that should not be in plaintext.
- We use
osQuery
,cnquery
,steampipe
, et. al. system and endpoint observabilty tools for SOC2 and other compliance requirements
Default data tools installed in ~/bin
:
- Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON.
- daff library for comparing tables, producing a summary of their differences.
- csvtk is a cross-platform, efficient and practical CSV/TSV toolkit in Golang.
- xsv is a fast CSV command line toolkit written in Rust.
- OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
- q - Run SQL directly on CSV or TSV files.
- Dasel jq/yq for JSON, YAML, TOML, XML and CSV with zero runtime dependencies.
GitUI provides you with the comfort of a git GUI but right in your terminal. Per their website, "this tool does not fully substitute the git
CLI, however both tools work well in tandem". It can be installed using:
asdf-setup-plugin-global gitui https://github.com/looztra/asdf-gitui
XDG_CACHE_HOME
(defined indot_config/fish/config.fish
)IS_NLH
andIS_NLH_WSL
(defined indot_config/fish/conf.d/netspective-labs-home.fish
)DENO_INSTALL
(defined indot_config/fish/conf.d/deno.fish
)MANAGED_GIT_WORKSPACES_HOME
(defined indirenv
.envrc
formGit
workspaces)NPM_AUTH_TOKEN
set to GitHub token if supplied in.config/chezmoi/chezmoi.toml
(defined indot_config/fish/conf.d/npm.fish.tmpl
)
$HOME/bin
(defined indot_config/fish/config.fish
)$HOME/.deno/bin
(defined indot_config/conf.d/deno.fish
)
chez
: an abbreviation forchezmoi
, use it likechez apply
(defined indot_config/fish/conf.d/chezmoi.fish
)chez-code
: launch Visual Studio Code with $chezmoi source-path
/home.code-workspace (defined indot_config/fish/conf.d/chezmoi.fish
)cdchez
:cd
tochezmoi source-path
without starting a new shell;chez cd
would do the same but start a new shell (defined indot_config/fish/conf.d/chezmoi.fish
)
wscd
: if you're in anmGit
workspace (like$HOME/workspaces
),cd
to the top-level of the workspace (defined indot_config/fish/functions
)
- Why do I see
jq: error (at <stdin>:1): Cannot index object with number
at the CLI sometimes?- We use the GitHub API to get the latest versions of repos. For example
curl -s https://api.github.com/repos/asdf-vm/asdf/tags | jq '.[0].name' -r
returns the latest version of the ASDF library. When GitHub reaches its limit thehttps://api.github.com/repos/asdf-vm/asdf/tags
URL will return503 Forbidden
and thenjq
will fail. The messagejq: error (at <stdin>:1): Cannot index object with number
fools us into thinking the issue is withjq
but the issues is that GitHub reached the API rate limit per hour.
- We use the GitHub API to get the latest versions of repos. For example
Please review the bundled Managed Git and opinionated set of instructions and tools for managing code workspaces that depend on multiple repositories.
We use Semantic Versioning so be sure to learn and regularly use the semtag bash script that is installed as git-semtag
in $HOME/bin
.
For example:
chez cd
# perform regular git commits
git chglog --output CHANGELOG.md && git commit -m "auto-generate CHANGELOG.md" CHANGELOG.md
git semtag final
# or 'git semtag final -v "v0.5.0"' for specific version
git push
We use direnv
to encourage usage of environment variables with per-directory flexibility. Per their documentation:
direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.
We use direnv
and .envrc
files to manage environments on a per-directory (per-project and descendant directories) basis. direnv
can be used to manage secrets as well as non-secret configurations. Many other development automation techniques are possible.
There are some direnv YouTube videos worth watching to get familar with the capabilities.
- Use Single-file scripts that download their dependencies as a guide for learning how to create portable scripts (especially Anything with a Nix package).
Instead of putting passwords directly into .envrc
and other files, use gopass
and summon
(both installed as part of our default packages).
We prefer Deno for scripts (rather than bash
or fish
) because of portability and that Deno scripts are just Typescript. However, we can and should support other languages too:
- rust-script can run Rust files and expressions as scripts without any setup or compilation step
- erning/gorun enables "shebang" in the source code of a Go program to run it
- bitfield/script makes it easy to write shell-like scripts in Go - this can be helpful if we need to customize things like PocketBase.io
The only downside to using Rust, Go, etc. as scripting languages is that we need to have compilers available.
Instead of using simple-http-server
switch to PocketBase.io in case we need a built-in BaaS in home-polyglot.
netspective-labs/sql-aide/lib/postgres/pgpass/pgpass.ts
has a TODO which suggests martin1keogh/zsh_pgpass_completion-like CLI completions. Once that's done incorporate the generated completions into home-polyglot
.
tea claims to be the next generation, cross‐platform package manager replacing brew
, winget
, etc. At our convenience (and as tea
matures) we should evaluate whether it's the right next package manager for us.
We should consider devbox as our nix-based project-specific package manager to install languages and utilities which require more complex package management than what asdf
can handle. jetpack.io devbox should be evaluated after tea
because it uses nix
and supports generating IaC artifacts like Dockerfile.
As we start to use more nix
capabilities, consider installing nix-shell
as part of bootstrap-admin-*.sh
. Since devbox
and other home managers depend on Nix anyway, it probably makes sense to manage nix
core separately from devbox
, et. al.
Define how mGit
and other typical home-polyglot
should work for GitHub Workspaces. For example, the default directory for projects is /workspaces
instead of $HOME/workspaces
. How should we support GitLab IDE?
fish
curl -sL https://git.io/fisher | source && fisher install jorgebucaran/fisher
Install tide
the ultimate Fish prompt, equivalent to Powerlevel10k for ZSH, in case tide
is more favorable to our default Oh My Posh
cross-shell prompt theming engine.
fisher install IlanCosman/tide
See Do you use curl? Stop using -u. Please use curl -n and .netrc. We should update all references to curl
to include curl -n
so that .netrc
is optionally pulled in when we need to use the following configuration:
machine api.github.com
login gitHubUserName
password gh-personal-access-token
When we run into problems of API rate limiting with anonymous use of api.github.com
then users can easily switch to authenticated use of api.github.com
which will increase rate limits.
Use run_once_install-packages.sh.tmpl in case we need to install some defaults. See:
chezmoi execute-template '{{ .chezmoi.osRelease.id }}' # e.g. debian or ubuntu
chezmoi execute-template '{{ .chezmoi.osRelease.idLike }}' # e.g. debian if running ubuntu
If a release is Debian or Debian-like (e.g. Ubuntu and others) we should automatically install some packages through chezmoi
scripts to perform actions. This might be a better way to install postgresql-client
and other database-specific functionality as well as other packages.
- Integrate Wildland, a collection of protocols, conventions, and software, which creates a union file system across S3, WebDAV, K8s, and other storage providers.
Pueue is a command-line task management tool for sequential and parallel execution of long-running tasks. There is also pueued
daemon, with runs processes runs in the background (no need to be logged in).