Skip to content

Commit

Permalink
Merge pull request #90 from geneontology/project-management-85-pass-w…
Browse files Browse the repository at this point in the history 
…-ssd

Update PRODUCTION_PROVISION_README.md
  • Loading branch information
@kltm
kltm authored Oct 1, 2024
2 parents 50fe157 + 1a45813 commit e70043c
Showing 1 changed file with 133 additions and 68 deletions.
201 changes: 133 additions & 68 deletions provision/production/PRODUCTION_PROVISION_README.md
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# go-fastapi Deployment

This guide describes the deployment of the `go-fastapi` stack to AWS using Terraform, ansible, and the "go-deploy" Python library.
This guide describes the deployment of the `go-fastapi` stack to AWS using Terraform, ansible, and the "go-deploy" Python library.

## Prerequisites:
**NOTE**: we have a docker-based environment with all these tools installed.
## Prerequisites:
**NOTE**: we have a docker-based environment with all these tools installed.

#### software:

- go-fastapi checkout
- go-fastapi checkout
- Terraform: v1.1.4
- Ansible: 2.10.7
- aws cli
Expand All @@ -30,166 +30,232 @@ This guide describes the deployment of the `go-fastapi` stack to AWS using Terra
- docker-production-compose.yaml
- various configuration files

#### DNS:
#### DNS:

DNS records are used for `go-fastapi`; they are typically the "production" record and the dev/testing record. Yhe go-deploy tool allows for creating DNS records (type A) that would be populated by the public ip addresses of the aws instance. If you don't use this option, you would need to point this record to the elastic IP of the VM. For testing purposes, you can use: `aes-test-go-fastapi.geneontology.org` or any other record that you create in Route 53.

**NOTE**: If using cloudflare, you would need to point the cloudflare dns record to the elastic IP.

#### SSH Keys:
# BREAK FOR NEW DOC #

## Configuring and deploying EC2 _instances_:

This is all completed in a dockerized development environment (all commands take place inside the docker container).

1. Prepare _your_ AWS credentials:

Your (personal developer) AWS credentials are used by Terraform to provision the AWS instance and by the provisioned instance to access the certificate store and the S3 buckets used to store Apache logs. These are your personal AWS credentials and should have been appropriately created to give you these permissions.

**NOTE**: specifically, you will need to supply an `aws_access_key_id` and `aws_secret_access_key`. These will be marked with `REPLACE_ME` in the `go-aws-credentials.sample` file farther down.

2. SSH Keys

The keys we'll be using can be found in the shared SpderOak store. If you don't know what this is, ask @kltm.

For testing purposes you can use your own ssh keys. But for production please ask for the go ssh keys.
/tmp/go-ssh.pub
/tmp/go-ssh

## Configuring and deploying EC2 _instances_:
The names will be:

This is all completed in a dockerized development environment (all commands take place inside the docker container).
```
go-ssh.pub
go-ssh
```

1. Spin up the provided dockerized development environment:
3. Spin up the provided dockerized development environment:

```bash
docker run --name go-dev -it geneontology/go-devops-base:tools-jammy-0.4.1 /bin/bash
docker rm go-dev
docker run --name go-dev -it geneontology/go-devops-base:tools-jammy-0.4.4 /bin/bash
git clone https://github.com/geneontology/go-fastapi.git
cd go-fastapi/provision
```

2. Prepare AWS credentials:
4. Copy in SSH keys

Copy the ssh keys from your docker host into the running docker image, in `/tmp`:

The credentials are used by Terraform to provision the AWS instance and by the provisioned instance to access the certificate store and the s3 buckets used to store Apache logs. Copy and modify the aws credential file to the default location `/tmp/go-aws-credentials`
```
docker cp go-ssh go-dev:/tmp
docker cp go-ssh.pub go-dev:/tmp
```
You should now have the following in your image:
```
/tmp/go-ssh
/tmp/go-ssh.pub
```
Make sure they have the right perms to be used:
```
chmod 600 /tmp/go-ssh*
```

5. Establish the AWS credential files

**NOTE**: you will need to supply an `aws_access_key_id` and `aws_secret_access_key`. These will be marked with `REPLACE_ME` in the `go-aws-credentials.sample` file.
Within the running image, copy and modify the AWS credential file to the default location `/tmp/go-aws-credentials`.

```bash
cp production/go-aws-credentials.sample /tmp/go-aws-credentials
emacs /tmp/go-aws-credentials # update the `aws_access_key_id` and `aws_secret_access_key`
```
Add your personal dev keys into the file; update the `aws_access_key_id` and `aws_secret_access_key`:
```
emacs /tmp/go-aws-credentials
```

6. Initialize the S3 Terraform backend:

3. Prepare and initialize the S3 Terraform backend:
"Initializing" a Terraform backend connects your local Terraform instantiation to a workspace; we are using S3 as the shared workspace medium (Terraform has others as well). This workspace will contain information on EC2 instances, network info, etc.; you (and other developers in the future) can discover and manipulate these states, bringing servers and services up and down in a shared and coordinated way. These Terraform backends are an arbitrary bundle and can be grouped as needed. In general, the production systems should all use the same pre-coordinated workspace, but you may create new ones for experimentation, etc.

"Initializing" a Terraform backend means that you are getting ready to save a bundle of EC2 and networking states to S3, so that you and other developers in the future can discover and manipulate these states in the future, bringing servers and services up and down in a coordinated way. These terraform backends are an arbitrary bundle and can be grouped as needed. In general, the production systems should all use a coordinated set, but you may create new ones for experimentation, etc. If you are trying to work with an already set state, jump to `4`; if you are experimenting, continue here with `3`.
Typically, the name of the workspace is `go-workspace-` + the name of the service; i.e. `go-workspace-api` for the use case here.

```bash

# The S3 backend is used to store the terraform state.
cp ./production/backend.tf.sample ./aws/backend.tf

# replace the REPLACE_ME_GOAPI_S3_STATE_STORE with the appropriate backend
# Replace the REPLACE_ME_GOAPI_S3_STATE_STORE with the appropriate workspace (state store ~= workspace); so `go-workspace-api`.
emacs ./aws/backend.tf

# Use the AWS cli to make sure you have access to the terraform s3 backend bucket
# Use the AWS CLI to make sure you have access to the terraform s3 backend bucket
export AWS_SHARED_CREDENTIALS_FILE=/tmp/go-aws-credentials

# S3 bucket
aws s3 ls s3://REPLACE_ME_GOAPI_S3_STATE_STORE
# Check connection to S3 bucket.
aws s3 ls s3://go-workspace-api

# initialize (if it doesn't work, we fail):
# Initialize (if it doesn't work, we fail):
go-deploy -init --working-directory aws -verbose

# Use these commands to figure out the name of an existing workspace if any. The name should have a pattern `production-YYYY-MM-DD`
go-deploy --working-directory aws -list-workspaces -verbose
# Use these commands to figure out the name of an existing workspace if any. The name should have a pattern `go-api-production-YYYY-MM-DD`
go-deploy --working-directory aws -list-workspaces -verbose
```

4. Provision instance on AWS:
7. Provision new instance on AWS, for potential production use:

If a workspace exists above, then you can skip the provisioning of the AWS instance.
Else, create a workspace using the following namespace pattern `production-YYYY-MM-DD`. e.g.: `production-2023-01-30`
Create a (new) production workspace using the following namespace pattern `go-api-production-YYYY-MM-DD`; e.g.: `go-api-production-2023-01-30`:

```bash
# copy `production/config-instance.yaml.sample` to another location and modify using emacs.

cp ./production/config-instance.yaml.sample config-instance.yaml
emacs config-instance.yaml # verify the location of the ssh keys for your AWS instance in your copy of `config-instance.yaml` under `ssh_keys`.
# verify the location of the public ssh key in `aws/main.tf`
emacs config-instance.yaml # verify the location of the SSH keys for your AWS instance: /tmp/go-ssh
emacs aws/main.tf # technically optional; verify the location of the public ssh key in `aws/main.tf`
```

As well, give a human-readable string for the instance/tags/name (EC2 instance name tag), make it the same at the namespace pattern above; i.e. `go-api-production-2024-01-22`:

```
emacs config-instance.yaml
```

8. Test the deployment

`REPLACE_ME_WITH_S3_WORKSPACE_NAME` would be something like `go-api-production-<TODAYS_DATE>`; i.e. `go-api-production-2024-01-22`

5. test the deployment
```bash
go-deploy --workspace REPLACE_ME_WITH_TERRAFORM_BACKEND --working-directory aws -verbose -dry-run --conf config-instance.yaml
go-deploy --workspace REPLACE_ME_WITH_S3_WORKSPACE_NAME --working-directory aws -verbose -dry-run --conf config-instance.yaml
```

7. deploy if all looks good.
9. Deploy

Deploy command:
```bash
go-deploy --workspace REPLACE_ME_WITH_TERRAFORM_BACKEND --working-directory aws -verbose --conf config-instance.yaml
# display the terraform state. The aws resources that were created.
go-deploy --workspace REPLACE_ME_WITH_TERRAFORM_BACKEND --working-directory aws -verbose -show
# display the public ip address of the aws instance
go-deploy --workspace REPLACE_ME_WITH_TERRAFORM_BACKEND --working-directory aws -verbose -output
go-deploy --workspace REPLACE_ME_WITH_S3_WORKSPACE_NAME --working-directory aws -verbose --conf config-instance.yaml
```

Useful Details for troubleshooting:
This will produce an IP address in the resulting inventory.json file.
The previous command creates a terraform tfvars. These variables override the variables in `aws/main.tf`
10. Checking what we have done

Just to check, ask it to display what it just did (display the Terraform state):
```
go-deploy --workspace REPLACE_ME_WITH_S3_WORKSPACE_NAME --working-directory aws -verbose -show
```

**NOTE**: write down the IP address of the AWS instance that is created.
Finally, just show the IP address of the AWS instance:
```
go-deploy --workspace REPLACE_ME_WITH_S3_WORKSPACE_NAME --working-directory aws -verbose -output
```

**NOTE**: write down the IP address of the AWS instance that is created. This can also be found in `REPLACE_ME_WITH_S3_WORKSPACE_NAME.cfg` (e.g. go-api-production-YYYY-MM-DD.cfg).

Useful details for troubleshooting:
These commands will produce an IP address in the resulting `inventory.json` file.
The previous command creates Terraform "tfvars". These variables override the variables in `aws/main.tf`

This can be found in `REPLACE_ME_WITH_TERRAFORM_BACKEND.cfg` (e.g. production-YYYY-MM-DD.cfg, sm-test-go-fastapi-alias.cfg)
If you need to check what you have just done, here are some helpful Terraform commands:

```bash
cat REPLACE_ME_WITH_TERRAFORM_BACKEND.tfvars.json # e.g, production-YYYY-MM-DD.tfvars.json, sm-test-go-fastapi-alias.tfvars.json
cat REPLACE_ME_WITH_S3_WORKSPACE_NAME.tfvars.json # e.g, go-api-production-YYYY-MM-DD.tfvars.json
```

The previous command creates an ansible inventory file.
```bash
cat REPLACE_ME_WITH_TERRAFORM_BACKEND-inventory.cfg # e.g, production-YYYY-MM-DD-inventory, sm-test-go-fastapi-alias-inventory
cat REPLACE_ME_WITH_S3_WORKSPACE_NAME-inventory.cfg # e.g, go-api-production-YYYY-MM-DD-inventory
```

Useful terraform commands to check what you have just done
Useful Terraform commands to check what you have just done

```bash
terraform -chdir=aws workspace show # current terraform workspace
terraform -chdir=aws show # current state deployed ...
terraform -chdir=aws output # shows public ip of aws instance
terraform -chdir=aws output # shows public ip of aws instance
```

## Configuring and deploying software (go-fastapi) _stack_:

These commands continue to be run in the dockerized development environment.

**POSSIBLE CUT START**
```bash
* replace "REPLACE_ME" values in config-instance.yaml for dns_record_name and dns_zone_id,
dns_zone_id should be "Z04640331A23NHVPCC784" and dns_record_name is the FQDN plus the REPLACE_ME_WITH_TERRAFORM_BACKEND, eg. api-production-2024-08-21.geneontology.org
* Location of SSH keys may need to be replaced after copying config-stack.yaml.sample
* s3 credentials are placed in a file using the format described above
* s3 uri if SSL is enabled. Location of SSL certs/key
* S3 credentials are placed in a file using the format described above
* S3 uri if SSL is enabled. Location of SSL certs/key
* QoS mitigation if QoS is enabled
* Use the same workspace name as in the previous step
**POSSIBLE CUT END**

Let's ready the the instance, starting by editing the config:
```bash
cp ./production/config-stack.yaml.sample ./config-stack.yaml
emacs ./config-stack.yaml # MAKE SURE TO CHANGE THE GO-FASTAPI TAG (strip the v), also replace all the REPLACE_MEs
emacs ./config-stack.yaml
```
Change these in emacs:
* `S3_BUCKET`: "go-workspace-api" (as above)
* `S3_SSL_CERTS_LOCATION`: "s3://go-service-lockbox/geneontology.org.tar.gz"; this is generally of the form: go-service-lockbox/_TLD_.tar.gz";
* `fastapi_host`: "api-test.geneontology.org"; (must be a FQDN)
* `fastapi_tag`: E.g. "0.2.0"; this should be the Dockerhub _tagged_ version of the API (which is how we deploy within the image), which is conincidentally the GitHub version of the API _sans the final "v"_. <- important point!
Finally, get ansible ready:
```
export ANSIBLE_HOST_KEY_CHECKING=False
````
**NOTE**: change the command below to point to the terraform workspace you use above.
go-deploy --workspace REPLACE_ME_WITH_TERRAFORM_BACKEND --working-directory aws -verbose --conf config-stack.yaml
Run the deployment of the stack within the instance:
```bash
go-deploy --workspace REPLACE_ME_WITH_TERRAFORM_BACKEND --working-directory aws -verbose --conf config-stack.yaml
go-deploy --workspace REPLACE_ME_WITH_S3_WORKSPACE_NAME --working-directory aws -verbose --conf config-stack.yaml
```
## Testing deployment (within the dev image):
## Testing deployment:
1. Access go-fastapi from the command line by ssh'ing into the newly provisioned EC2 instance (this too is run via the dockerized dev environment):
1. Access go-fastapi instance from the CLI by ssh'ing into the newly provisioned EC2 instance:
```
ssh -i /tmp/go-ssh ubuntu@IP_ADDRESS
```

2. Access go-fastapi from a browser:
3. Access go-fastapi from a browser:

We use health checks in the `docker-compose` file.
Use go-fastapi DNS name. http://{go-fastapi_host}/docs
+We use health checks in the `docker-compose` file.+ (where to put this?)

3. Debugging:
Use the go-fastapi CNAME name. https://{fastapi_host}/docs

3. Debugging (in the AWS instance):

* Use -dry-run and copy and paste the command and execute it manually
* ssh to the machine; the username is ubuntu. Try using DNS names to make sure they are fine.

```bash
docker-compose -f stage_dir/docker-compose.yaml ps
docker-compose -f stage_dir/docker-compose.yaml down # whenever you make any changes
docker-compose -f stage_dir/docker-compose.yaml down # whenever you make any changes
docker-compose -f stage_dir/docker-compose.yaml up -d
docker-compose -f stage_dir/docker-compose.yaml logs -f
docker-compose -f stage_dir/docker-compose.yaml logs -f
```

4. Testing LogRotate:
Expand All @@ -211,12 +277,12 @@ docker inspect --format "{{json .State.Health }}" go-fastapi
```


### Destroy Instance and Delete Workspace:
## Destroy Instance and other destructive things:

```bash
# Destroy Using Tool.
# Make sure you point to the correct workspace before destroying the stack by using the -show command or the -output command
go-deploy --workspace REPLACE_ME_WITH_TERRAFORM_BACKEND --working-directory aws -verbose -destroy
go-deploy --workspace REPLACE_ME_WITH_S3_WORKSPACE_NAME --working-directory aws -verbose -destroy
```

```bash
Expand All @@ -239,14 +305,14 @@ terraform -chdir=aws workspace delete <NAME_OF_WORKSPACE_THAT_IS_NOT_DEFAULT> #
1. start the docker container `go-dev` in interactive mode.

```bash
docker run --rm --name go-dev -it geneontology/go-devops-base:tools-jammy-0.4.2 /bin/bash
docker run --rm --name go-dev -it geneontology/go-devops-base:tools-jammy-0.4.4 /bin/bash
```

In the command above we used the `--rm` option which means the container will be deleted when you exit.
If that is not the intent and you want to delete it later at your own convenience. Use the following `docker run` command.

```bash
docker run --name go-dev -it geneontology/go-devops-base:tools-jammy-0.4.2 /bin/bash
docker run --name go-dev -it geneontology/go-devops-base:tools-jammy-0.4.4 /bin/bash
```

2. To exit or stop the container:
Expand All @@ -272,4 +338,3 @@ chown root /tmp/go-*
chgrp root /tmp/go-*
chmod 400 /tmp/go-ssh
```

0 comments on commit e70043c

Please sign in to comment.