From a05909c710dea2501d6dbd75282adf7fde6380ab Mon Sep 17 00:00:00 2001 From: Nuru Date: Tue, 5 Dec 2023 18:57:33 -0800 Subject: [PATCH] Update info on Apple Silicon support, fix file layout comments (#601) --- content/docs/fundamentals/geodesic.md | 237 +++++++++++--------------- 1 file changed, 95 insertions(+), 142 deletions(-) diff --git a/content/docs/fundamentals/geodesic.md b/content/docs/fundamentals/geodesic.md index 3f840156a..968d05569 100644 --- a/content/docs/fundamentals/geodesic.md +++ b/content/docs/fundamentals/geodesic.md @@ -13,7 +13,7 @@ import ReactPlayer from 'react-player' In the landscape of developing infrastructure, there are dozens of tools that we all need on our personal machines to do our jobs. In SweetOps, instead of having you install each tool individually, we use Docker to package all of these tools into one convenient image that you can use as your infrastructure automation toolbox. We call it Geodesic and we use it as our DevOps automation shell and as the base Docker image for all of our DevOps tooling. -Geodesic is a DevOps Linux Distribution packaged as a Docker image that provides users the ability to utilize `atmos`, `terraform`, `kubectl`, `helmfile`, the AWS CLI, and many other popular tools that compromise the SweetOps methodology without having to invoke a dozen `install` commands to get started. It’s intended to be used as an interactive cloud automation shell, a base image, or in CI/CD workflows to ensure that all systems are running the same set of versioned, easily accessible tools. +Geodesic is a DevOps Linux Distribution packaged as a Docker image that provides users the ability to utilize `atmos`, `terraform`, `kubectl`, `helmfile`, the AWS CLI, and many other popular tools that compromise the SweetOps methodology without having to invoke a dozen `install` commands to get started. It’s intended to be used as an interactive cloud automation shell, a base image, or in CI/CD workflows to ensure that all systems are running the same set of versioned, easily accessible tools. Most commonly, it is used as a base image for individual user, project, or company Docker images that have additional tools or specific versions of tools pre-installed. For example, Geodesic does not have `atmos` pre-installed, so that you can install the version of `atmos` that you want to use in your project update that version independently of updating Geodesic. These days, the typical software application is distributed as a docker image and run as a container. Why should infrastructure be any different? Since everything we write is "Infrastructure as Code", we believe that it should be treated the same way. This is the "Geodesic Way". Use containers+envs instead of unconventional wrappers, complicated folder structures, and symlink hacks. Geodesic is the container for all your infrastructure automation needs that enables you to truly achieve SweetOps. @@ -25,12 +25,17 @@ An organization may choose to leverage all of these components or just the parts height={"450px"} controls={true} /> -:::caution -**Apple M1 Support** - -TL;DR: Geodesic works on the M1 running as `amd64` (not `arm64`). Docker auto-detects this by default, but otherwise it’s possible to pass `--platform linux/amd64` to `docker` to force the platform. +:::info +**Apple Silicon (M1, M2, etc. Chips) Support** -Geodesic is comprised of a large collection of mostly [third-party open-source tools distributed via our packages repository](https://github.com/cloudposse/packages). As such, **support for the Apple M1 chip is not under Cloud Posse's contro**l, rather it depends on each tool author updating each tool for the M1 chip. All of the compiled tools that Cloud Posse has authored and are included in Geodesic are compiled for M1 (`darwin_arm64`), and of course, all of the scripts work on M1 if the interpreters (e.g. `bash`, `python`) are compiled for M1. Unfortunately, this is only a small portion of the overall toolkit that is assembled in Geodesic. Therefore we do not advise using Geodesic on the M1 at this time and do not anticipate M1 will be well supported before 2022. Historically, widespread support for a new chip takes several years to establish; we hope we will not have to wait that long given the velocity our industry moves. +Geodesic is comprised of a large collection of mostly [third-party open-source tools distributed via our packages repository](https://github.com/cloudposse/packages). +As such, **support for Macs with Apple chips (M1, M2, etc.) is fully not under Cloud Posse's control**, rather it +depends on each tool author updating each tool for the Apple chips, referred to as the `arm64` architecture. +**The Debian-based Geodesic is provided as a multi-architecture Docker image, supporting both Intel chips (`amd64`) and +Apple chips (`arm64`)**, but the (deprecated) Alpine-based Geodesic is only provided for Intel chips (`amd64`). The image +built for `arm64` does not include a few tools that the maintainers have not yet updated for `linux/arm64`, but those +tools are not widely used anymore, so your experience should be good on Apple silicon. (See the current +list of unsupported tools [here](https://github.com/cloudposse/geodesic/blob/master/packages-amd64-only.txt)) ::: @@ -65,11 +70,11 @@ Running `geodesic` as the base image for Spacelift or with GitHub Actions ensure - [How to Customize the Geodesic Shell](/reference-architecture/how-to-guides/tutorials/how-to-customize-the-geodesic-shell) - [How to use Atmos](/reference-architecture/how-to-guides/tutorials/how-to-use-atmos) -## Alpine, Debian, and CentOS Support +## Debian and Alpine Base Images -Starting with Geodesic version 0.138.0, we distribute 2 versions of Geodesic Docker images, one based on Alpine and one based on Debian, tagged `VERSION-BASE_OS`, e.g. `0.138.0-alpine`. +Starting with Geodesic version 0.138.0, we distribute 2 versions of Geodesic Docker images, one based on Debian and one based on Alpine, tagged `VERSION-BASE_OS`, e.g. `0.138.0-alpine`. -Prior to this, all Docker images were based on Alpine only and simply tagged `VERSION`. We encourage people to use the Debian version and report any issues by opening a GitHub issue. We will continue to maintain the `latest-alpine` and `latest-debian` Docker tags for those who want to commit to using one base OS or the other but still want automatic updates. +Prior to this, all Docker images were based on Alpine only and simply tagged `VERSION`. We encourage people to use the Debian version and report any issues by opening a GitHub issue. We will continue to maintain the `latest-alpine` and `latest-debian` Docker tags for those who want to commit to using one base OS or the other but still want automatic updates. However, our primary focus will be on the Debian-based images. The Alpine-based images are deprecated and issues unique to Alpine (i.e. not affecting the Debian version) may not be addressed. ## Packages @@ -77,7 +82,7 @@ Prior to this, all Docker images were based on Alpine only and simply tagged `VE Central to `geodesic` is its rich support for the latest version of [the most popular packages](https://github.com/cloudposse/packages/tree/master/vendor) for DevOps. We maintain hundreds of packages that are graciously hosted by Cloud Smith. Our packages are updated nightly as soon as new releases are made available by vendors. As such, we strongly recommend version pinning packages installed via the `Dockerfile`. -Also unique about our packages is that for `kubectl` and `terraform` we distribute all major versions with `dpkg-alternative` support so they can be concurrently installed without the use of version managers. +Also unique about our packages is that for `kubectl` and `terraform` we distribute all major versions with `dpkg-alternative` support so they can be concurrently installed without the use of version managers. For example, to install the latest version of `kubectl` v1.28 (to match the version of the cluster it is controlling), you can install package `kubectl-1.28`. Package repository hosting is graciously provided by [cloudsmith](https://cloudsmith.io/). Cloudsmith is the only fully hosted, cloud-native, universal package management solution, that enables your organization to create, store and share packages in any format, to any place, with total confidence. We believe there’s a better way to manage software assets and packages, and they’re making it happen! @@ -86,84 +91,82 @@ Package repository hosting is graciously provided by [cloudsmith](https://clouds Here’s a general filesystem layout for an infrastructure repository leveraging `geodesic` with `atmos` together with stacks and components. Note, individual customer repos will resemble this layout but will not be identical. ``` -infrastructure/ -├── Dockerfile -├── Makefile -├── README.md -├── components -│ └── terraform/ -│ └── foobar/ -│ ├── README.md -│ ├── backend.tf.json -│ ├── context.tf -│ ├── default.auto.tfvars -│ ├── main.tf -│ ├── modules/ -│ │ ├── baz/ -│ │ │ ├── context.tf -│ │ │ ├── main.tf -│ │ │ ├── outputs.tf -│ │ │ └── variables.tf -│ │ └── bar/ +infrastructure/ # GitHub Repository +├── Dockerfile # Dockerfile uses `cloudposse/geodesic` as base Image +├── Makefile # Makefile to build custom Geodestic-based Docker image and install wrapper script for `geodesic` +├── README.md +├── components # Location for all re-usable component building blocks +│ └── terraform/ # Location for all terraform (HCL) components +│ └── foobar/ # Example `foobar` component +│ ├── README.md # Every component has a well-maintained `README.md` with usage instructions +│ ├── context.tf # Standard context interface for all cloud posse modules. +│ ├── main.tf # Standard `main.tf` based on HashiCorp best-practices +│ ├── modules/ # Example of submodules within a component (aka child modules) +│ │ ├── baz/ # Submodule named `baz/` +│ │ │ ├── context.tf # Submodules should use the same standard interface with `context.tf` +│ │ │ ├── main.tf # Submodules should also follow HashiCorp best practices for module layout +│ │ │ ├── outputs.tf +│ │ │ └── variables.tf # Submodules should define variables in `variables.tf` and not modify `context.tf` +│ │ └── bar/ # Example of another submodule named `bar/` │ │ ├── context.tf │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf -│ ├── outputs.tf -│ ├── providers.tf -│ ├── remote-state.tf -│ ├── variables.tf -│ └── versions.tf +│ ├── outputs.tf # Outputs exported by this component in the remote state +│ ├── providers.tf # Providers used by this component +│ ├── remote-state.tf # Remote state leveraged by the component using the `remote-state` module +│ ├── variables.tf # Variables used by the component +│ └── versions.tf # Version pinning for providers used by the component │ -├── docs/ -│ ├── adr/ +├── docs/ # Location for documentation specific to this repository +│ ├── adr/ # Home for all Architectural Design Records for your organization │ │ ├── 0001-namespace-abbreviation.md │ │ ├── 0002-infrastructure-repository-name.md │ │ ├── 0003-email-addresses-for-aws-accounts.md │ │ ├── 0004-secure-channel-secrets-sharing.md │ │ ├── 0005-primary-aws-region.md -│ │ ├── README.md -│ │ └── template.md +│ │ ├── README.md # Index of all READMEs +│ │ └── template.md # Markdown template file to create new ADRs │ │ │ └── cold-start.md │ -├── rootfs/ -│ ├── etc/ +├── rootfs/ # The `rootfs` pattern overlays this filesystem on `/` (slash) inside the docker image (e.g. `ADD /rootfs /`) +│ ├── etc/ # The `/etc/` inside the container │ │ ├── aws-config/ -│ │ │ └── aws-config-cicd -│ │ └── motd +│ │ │ └── aws-config-teams # The AWS config used from within Geodesic by setting `AWS_CONFIG_PATH` +│ │ │ └── aws-extend-switch-roles # The config used by browser plugin AWS Extend Switch Roles +│ │ └── motd # Message of the Day (MOTD) displayed to `stdout` on interactive shell logins │ │ -│ └── usr/ +│ └── usr/ # The `usr/` tree inside the docker image │ └── local/ -│ ├── bin/ -│ │ ├── aws-accounts -│ │ ├── eks-update-kubeconfig -│ │ ├── spacelift-git-use-https -│ │ ├── spacelift-tf-workspace -│ │ └── spacelift-write-vars +│ ├── bin/ # Stick all scripts in `/usr/local/bin` +│ │ ├── aws-config # Script used to generate the files in `/rootfs/etc/aws-config` +│ │ ├── eks-update-kubeconfig # Helper script to export the `kubeconfig` for EKS using the `aws` CLI +│ │ ├── spacelift-git-use-https # Helper script to configure git to use HTTPS rather than SSH for Spacelift +│ │ ├── spacelift-tf-workspace # Helper script to configure Spacelift Terraform workspace +│ │ └── spacelift-write-vars # Helper script to write Terraform variables to a file for Spacelift to use │ └── etc/ │ └── atmos/ -│ └── atmos.yaml +│ └── atmos.yaml # Atmos CLI configuration. Instructs where to find stack configs and components. │ -└── stacks/ - ├── catalog/ - │ ├── account-map.yaml - │ ├── account-settings.yaml - │ ├── account.yaml - │ ├── cloudtrail.yaml - │ ├── dns-delegated.yaml - │ ├── dns-primary.yaml - │ ├── ecr.yaml - │ ├── eks +└── stacks/ # Location of all stack configurations + ├── catalog/ # Location where to store catalog imports. See our catalog pattern. + │ ├── account-map.yaml # Catalog entry for [account-map](/components/library/aws/account-map/) + │ ├── account-settings.yaml # Catalog entry for [account-settings](/components/library/aws/account-settings/) + │ ├── account.yaml # ... + │ ├── aws-team-roles.yaml + │ ├── aws-teams.yaml + │ ├── cloudtrail.yaml + │ ├── dns-delegated.yaml + │ ├── dns-primary.yaml + │ ├── ecr.yaml + │ ├── eks │ │ ├── alb-controller.yaml │ │ ├── cert-manager.yaml │ │ ├── eks.yaml │ │ ├── external-dns.yaml │ │ ├── metrics-server.yaml │ │ └── ocean-controller.yaml - │ ├── github-runners.yaml - │ ├── iam-delegated-roles.yaml - │ ├── iam-primary-roles.yaml │ ├── s3 │ │ ├── alb-access-logs.yaml │ │ └── s3-defaults.yaml @@ -171,73 +174,35 @@ infrastructure/ │ ├── tfstate-backend.yaml │ ├── transit-gateway.yaml │ └── vpc.yaml - ├── mgmt-uw2-artifacts.yaml - ├── mgmt-uw2-audit.yaml - ├── mgmt-uw2-automation.yaml - ├── mgmt-uw2-corp.yaml - ├── mgmt-uw2-dns.yaml - ├── mgmt-uw2-globals.yaml - ├── mgmt-uw2-identity.yaml - ├── mgmt-uw2-network.yaml - ├── mgmt-uw2-root.yaml - ├── mgmt-uw2-sandbox.yaml - ├── mgmt-uw2-security.yaml - └── uw2-globals.yaml + └── orgs/ # Environment-specific configuration + └── eg/ # Example organization + ├── corp/ # Example organizaional unit (OU) + │ ├── auto/ # Example environment + │ │ ├── _defaults.yaml # Default settings for all stacks in this environment + │ │ ├── global-region.yaml # Settings for regionless or all-region components (e.g. IAM roles) + │ │ └── us-west-2.yaml # Settings for infrastructure deployed in us-west-2 region + │ ├── identity/ + │ │ ├── _defaults.yaml + │ │ ├── global-region.yaml + │ │ └── us-west-2.yaml + │ ├── root/ + │ │ ├── _defaults.yaml + │ │ ├── global-region.yaml + │ │ └── us-west-2.yaml + └── plat/ + ├── dev/ + │ ├── _defaults.yaml + │ ├── global-region.yaml + │ └── us-west-2.yaml + ├── stage/ + │ ├── _defaults.yaml + │ ├── global-region.yaml + │ └── us-west-2.yaml + └── prod/ + ├── _defaults.yaml + ├── global-region.yaml + └── us-west-2.yaml ``` -GitHub Repository -Dockerfile uses `cloudposse/geodesic` as base Image -Makefile to help build and install wrapper script for `geodesic` -Location for all re-usable component building blocks -Location for all terraform (HCL) components -Example `foobar` component -Every component has a well-maintained `README.md` with usage instructions -Programmatically generated terraform backend created from `atmos terraform backend generate` -Standard context interface for all cloud posse modules. -Terraform defaults in HCL (loaded by `terraform` at run-time). Not deep merged. -Standard `main.tf` based on HashiCorp best-practices -Example of submodules within a component (aka child modules) -Submodule named `baz/` -Submodules should use the same standard interface with `context.tf` -Submodules should also follow HashiCorp best practices for module layout -Submodules should define variables in `variables.tf` and not modify `context.tf` -Example of another submodule named `bar/` - -Outputs exported by this component in the remote state -Remote state leveraged by the component using the `remote-state` module -Variables used by the component -Version pinning for providers used by the component - -Location for documentation specific to this repository -Home for all Architectural Design Records for your organization - -Index of all READMEs -Markdown template file to create new ADRs - -The `rootfs` pattern overlays this filesystem on `/` (slash) inside the docker image (e.g. `ADD /rootfs /`) -The `/etc/` inside the container -The AWS config used by automation by setting `AWS_CONFIG_PATH` -Message of the Day (MOTD) displayed to `stdout` on interactive shell logins - -The `usr/` tree inside the docker image - -Stick all scripts in `/usr/local/bin` -Script used to generate the `~/.aws/config` for SSO profiles. Modify this to suit your needs. -Helper script to export the `kubeconfig` for EKS using the `aws` CLI - -Atmos CLI configuration. Instructs where to find stack configs and components. - -Location of all stack configurations -Location where to store catalog imports. See our catalog pattern. -Catalog entry for [account-map](/components/library/aws/account-map/) -Catalog entry for [account-settings](/components/library/aws/account-settings/) -Catalog entry for [cloudtrail](/components/library/aws/cloudtrail/) -Catalog entry for [dns-delegated](/components/library/aws/dns-delegated/) -Catalog entry for [dns-primary](/components/library/aws/dns-primary/) -Catalog entry for [ecr](/components/library/aws/ecr/) - - - -Global configuration shared by all stacks in `uw2` region (e.g. `import` the `catalog/uw2-globals`) ## Build and Run Geodesic @@ -266,7 +231,7 @@ From here forward, any command-line commands are meant to be run from within the ### Command-line Prompt Ends with a Unicode Placeholder -If your command-line prompt inside of the `geodesic` shell ends with a funky Unicode placeholder, then chances are the default character we use for the end of the command line prompt ([Unicode Z NOTATION SCHEMA PIPING](https://www.compart.com/en/unicode/U+2A20)) is not present in the font library you are using. On the Mac, Terminal (at least) falls back to some other font when the character is missing, so it's not a problem. On other systems, we recommend installing the freely available Noto font from Google, whose mission is to supply workable characters for every defined Unicode code point. On Ubuntu, it is sufficient to install the Noto core fonts, via +If your command-line prompt inside of the `geodesic` shell ends with a funky Unicode placeholder, then chances are the default character we use for the end of the command line prompt ([Unicode Z NOTATION SCHEMA PIPING](https://www.compart.com/en/unicode/U+2A20)) is not present in the font library you are using. On the Mac, Terminal (at least) falls back to some other font when the character is missing, so it's not a problem. On other systems, we recommend installing the freely available Noto font from Google, whose mission is to supply workable characters for every defined Unicode code point. On Ubuntu, it is sufficient to install the Noto core fonts, via ``` apt install fonts-noto-core @@ -280,18 +245,6 @@ export PROMPT_STYLE="fancy" # or "unicode" or "plain" to your Geodesic customizations. See [How to Customize the Geodesic Shell](/reference-architecture/how-to-guides/tutorials/how-to-customize-the-geodesic-shell) for more detail, and also to see how you can completely customize the prompt decorations. -### Geodesic is Slow on the M1 Mac - -The poor performance of Geodesic on M1 is not specific to Geodesic but rather a known issue with Docker for Mac. - -Things to try: - -- Check (enable) Preferences → General → use gRPC FUSE for file sharing - -- Check (enable) Preferences → Experimental Features → Use the new Virtualization Framework - -- Use VMware Fusion to run a Debian VM, and run Geodesic from within the Debian VM - ### Files Written to Mounted Linux Home Directory Owned by Root User