Add Documents sync to S3

.gitignore

@@ -1,3 +1,5 @@
 ec2gaming.ovpn
 ec2gaming.rdp
 ec2gaming.auth
+ec2gaming.bat
+ec2gaming-permissionpolicy.json

README.md

@@ -61,7 +61,9 @@ Use `ec2gaming.cfg` to configure options such as the bid price over the minimum
 
 # First-time configuration
 
-The goal is to take the base [ec2gaming AMI](http://lg.io/2015/07/05/revised-and-much-faster-run-your-own-highend-cloud-gaming-service-on-ec2.html) and create a reusable image. These steps assume that you already have:
+The goal is to take the base [ec2gaming AMI](http://lg.io/2015/07/05/revised-and-much-faster-run-your-own-highend-cloud-gaming-service-on-ec2.html) and create a reusable image. This configuration differs from the original blog post, in that the goal is to keep the AMI immutable from session to session (call it an [institituional bias](http://techblog.netflix.com/2016/03/how-we-build-code-at-netflix.html) ;)
+
+These steps assume that you already have:
 
 - Homebrew and Homebrew Cask (but feel free to install the required software any way you like)
 - An Amazon AWS account, and have generated credentials from the AWS Console
@@ -80,6 +82,13 @@ From a terminal:
 
 - Install Microsoft Remote Desktop from the App Store
 - Run `aws configure` to configure your AWS credentials and region
+- Create a `ec2gaming.auth` file in the `ec2gaming` location (it's `.gitignored`) with two lines, the `administrator` username and the password you plan to use to login to the instance. It's used for later configuration and needs to be available before you proceed to the next step:
+
+    ```
+    administrator
+    <new password>
+    ```
+
 - Run `ec2gaming start` to bootstrap an instance
 
 ## EC2 Spot Instances
@@ -111,13 +120,17 @@ Edit `C:\Program Files\OpenVPN\config\server.ovpn` and add `cipher none` to the
 - Install and run several of the games you intend to play to `Z:\`. This performs first-time installation, avoiding the redistributable installation overhead. Delete local files once you're done
 - Exit Steam from the system tray, otherwise Steam will forget your password when the instance next starts
 
-## Steam remote install to ephemeral storage
+## Ephemeral storage
+
+The `Z:\` is SSD instance storage, that's destroyed when the instance is stopped, but is the ideal place for installing games and data for your gaming session thanks to it's high performance.
 
-The Steam remote install feature assumes the default Stream library, even if a second library is available and set to default. So, to install games to the emphemeral storage on `Z:\` remotely, we create a junction on instance startup:
+The Steam remote install feature assumes the default Stream library, even if a second library is available and set to default. So, to install games to the emphemeral storage on `Z:\` remotely, we create a junction on instance startup. Unfortunately, you can't install games larger than the freespace on `C:` with this approach, but in that case you can use RDP/VNC to install the game.
 
-- Copy `bootstrap/steamapps-junction.bat` to `C:\Users\Administrator\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup`
+Steam Cloud will also do a decent job of saving, but it's good to have coverage for games that don't cloud save, or if your instance terminates and Steam doesn't have a chance to perform the cloud sync. So, we also want to configure a periodic sync to S3 to save Documents from `Z:\Documents`.
 
-Unfortunately, you can't install games larger than the freespace on `C:` with this approach, but in that case you can use RDP and this avoids having to RDP for every installation.
+- Install the 64-bit AWS CLI from https://aws.amazon.com/cli/
+- Copy `ec2gaming.bat` to `C:\Users\Administrator\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup`. This file creates the junction, performs an initial one-way S3 sync, and schedules a task that runs every minute to keep the directory in sync
+- Right-click the Documents folder from the file explorer, and set the location to `Z:\Documents`
 
 ## Windows automatic login
 
@@ -158,22 +171,10 @@ Driver 373.06 is the latest driver version that has the K520 included in the dev
 - Restart Windows and attach a VNC session (`ec2gaming vnc`)
 - Right-click the desktop, select Screen resolution and select the highest available resolution
 
-## Cloud sync My Documents
-
-WIP - need to figure out how to allow sync that doesn't delete files if they're missing on the host.
-
-Steam Cloud will do a decent job, but it's good to have coverage for games that don't cloud save, or if your instance terminates and Steam doesn't have a chance to perform the cloud sync.
-
 ## Final steps
 
 - Run `ec2gaming snapshot` to snapshot the EBS volume and create your AMI
 - Run `ec2gaming terminate` to terminate the instance
-- Create a `ec2gaming.auth` file in the `ec2gaming` location (it's `.gitignored`) with two lines, it'll be used to authenticate the VPN for gaming:
-
-    ```
-    administrator
-    <new password>
-    ```
 
 ## Steam client configuration
 
@@ -194,7 +195,7 @@ On your Mac, go to Steam Home-Streaming settings and:
 
 # Periodic maintenance
 
-This configuration differs from the original blog post, in that the goal is to keep the AMI immutable from session to session (call it an [institituional bias](http://techblog.netflix.com/2016/03/how-we-build-code-at-netflix.html) ;). You'll want to periodically update Steam, run Windows Update etc. and re-snapshot and replace your AMI using the `ec2gaming snapshot` command.
+Because we treat the AMI as immutable, you'll want to periodically update Steam, run Windows Update etc. and re-snapshot and replace your AMI using the `ec2gaming snapshot` command.
 
 # Help
 

ec2gaming-permissionpolicy.json.template

@@ -0,0 +1,19 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["s3:ListBucket"],
+      "Resource": ["arn:aws:s3:::BUCKET"]
+    },
+    {
+      "Effect": "Allow",
+      "Action": [
+        "s3:PutObject",
+        "s3:GetObject",
+        "s3:DeleteObject"
+      ],
+      "Resource": ["arn:aws:s3:::BUCKET/*"]
+    }
+  ]
+}

ec2gaming-start.sh

@@ -36,9 +36,34 @@ if [ -z "$EC2_SECURITY_GROUP_ID" ]; then
   aws ec2 authorize-security-group-ingress --group-name ec2gaming --protocol udp --port 1194 --cidr "0.0.0.0/0"
   EC2_SECURITY_GROUP_ID=$(describe_security_group ec2gaming)
 fi
-
 echo "$EC2_SECURITY_GROUP_ID"
 
+echo -n "Looking for S3 bucket... "
+ACCOUNT_ID=$(aws iam get-user | jq '.User.Arn' | cut -d ':' -f 5)
+BUCKET="ec2gaming-$ACCOUNT_ID"
+if ! aws s3api head-bucket --bucket "$BUCKET" &> /dev/null; then
+  echo -n "not found. Creating... "
+  REGION=$(aws configure get region)
+  aws s3api create-bucket --bucket "$BUCKET" --region "$REGION" --create-bucket-configuration LocationConstraint="$REGION" > /dev/null
+fi
+USERNAME=$(head -1 ec2gaming.auth)
+PASSWORD=$(tail -1 ec2gaming.auth)
+sed "s/BUCKET/$BUCKET/g;s/USERNAME/$USERNAME/g;s/PASSWORD/$PASSWORD/g" ec2gaming.bat.template > ec2gaming.bat
+echo "$BUCKET"
+
+PROFILE_NAME="ec2gaming"
+echo -n "Looking for instance profile... "
+if ! aws iam get-instance-profile --instance-profile-name ec2gaming &> /dev/null; then
+  echo -n "not found. Creating... "
+  aws iam create-role --role-name "$PROFILE_NAME" --assume-role-policy-document file://ec2gaming-trustpolicy.json > /dev/null
+  sed "s/BUCKET/$BUCKET/g" ec2gaming-permissionpolicy.json.template > ec2gaming-permissionpolicy.json > /dev/null
+  aws iam put-role-policy --role-name "$PROFILE_NAME" --policy-name "$PROFILE_NAME" --policy-document file://ec2gaming-permissionpolicy.json > /dev/null
+  aws iam create-instance-profile --instance-profile-name "$PROFILE_NAME"
+  aws iam add-role-to-instance-profile --instance-profile-name "$PROFILE_NAME" --role-name "$PROFILE_NAME"
+fi
+INSTANCE_PROFILE_ARN=$(aws iam get-instance-profile --instance-profile-name ec2gaming | jq -r '.InstanceProfile.Arn')
+echo "$INSTANCE_PROFILE_ARN"
+
 echo -n "Creating spot instance request... "
 SPOT_INSTANCE_ID=$(aws ec2 request-spot-instances --spot-price "$FINAL_SPOT_PRICE" --launch-specification "
   {
@@ -47,6 +72,9 @@ SPOT_INSTANCE_ID=$(aws ec2 request-spot-instances --spot-price "$FINAL_SPOT_PRIC
     \"InstanceType\": \"$INSTANCE_TYPE\",
     \"Placement\": {
       \"AvailabilityZone\": \"$ZONE\"
+    },
+    \"IamInstanceProfile\": {
+      \"Arn\": \"$INSTANCE_PROFILE_ARN\"
     }
   }" | jq --raw-output '.SpotInstanceRequests[0].SpotInstanceRequestId')
 echo "$SPOT_INSTANCE_ID"

ec2gaming-trustpolicy.json

@@ -0,0 +1,8 @@
+{
+  "Version": "2012-10-17",
+  "Statement": {
+    "Effect": "Allow",
+    "Principal": {"Service": "ec2.amazonaws.com"},
+    "Action": "sts:AssumeRole"
+  }
+}

ec2gaming-vnc.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 source "$(dirname "$0")/ec2gaming.header"
 
-USER=$(head -1 ec2gaming.auth)
-PASSWD=$(tail -1 ec2gaming.auth)
-open "vnc://$USER:$PASSWD@10.8.0.1"
+USERNAME=$(head -1 ec2gaming.auth)
+PASSWORD=$(tail -1 ec2gaming.auth)
+open "vnc://$USERNAME:$PASSWORD@10.8.0.1"

ec2gaming.bat.template

@@ -0,0 +1,7 @@
+@echo off
+rmdir /Q /S  "C:\Program Files (x86)\Steam\steamapps"
+md Z:\SteamLibrary\steamapps
+cmd /c mklink /j "C:\Program Files (x86)\Steam\steamapps" Z:\SteamLibrary\steamapps
+md Z:\Documents
+aws s3 sync Z:\Documents s3://BUCKET/Documents
+schtasks /Create /RU USERNAME /RP PASSWORD /SC MINUTE /MO 1 /TN "Sync Documents with S3" /TR "aws s3 sync Z:\Documents s3://ec2gaming-639801188054/Documents --delete"