automated builds publishing to GCS bucket for sandbox, staging (#449)

* updated url

* testing updates to travis build

* testing updates to travis build

* testing updates to travis build

* testing updates to travis build

* testing build without notebooks support

* testing build without notebooks support

* testing build with travischecks disabled

* remove observatory from travis-checks, uncommented script

* testing with latest ruby

* testing script deploy to GCE VM since dpl SSL digest issue is unresolved upstream, blocking deployment to the 'gcs' provider

* removed because we don't need this

* testing script inline versus external file

* removed scripts dir, added ssh key to specific location & added it using ssh-add

* wrong path for input file

* correct spelling of deploy key filename

* correct path to deploy key

* GCE VMs don't allow root login

* testing deploy key

* use travis user for deploy

* adjusted remote command to copy new site files

* added staging deployment to travis, renamed deploy keys

* added mlab travis submodule to reuse common functions like gsutil, service accounts, etc.

* use travis to deploy sandbox build to GCS

* added skip_cleanup

* use recursive flag

* remove some extraneous comments no longer needed

* updated path for travis checkout

* changed repo name, removed link to openwrt since it was down

* test fix for build error

* test fix for build error

* possible fix for unbound variable error

* debugging deploy script

* debugging deploy script

* debugging deploy script

* debugging deploy script

* (3) debugging deploy script

* corrected travis linting errors

* (4) debugging deploy script

* (5) debugging deploy script

* activate saging service account if statement

* corrected issue with if statement syntax

* service account credentials don't seem to be available in the deploy: section. Move here to test

* update script to deploy, add build site command, add staging deploy script

* testing gsutil without if logic

* revereted to retest with deploy plugin.

* only include sandbox service account activation for testing

* service account is not recognized in deploy section?

* testing deploy without if statement, added gcloud config list to confirm what account is active

* specifying service account names explicitly

* test without if logic

* enclosing build command in quotes

* testing download of service account key file

* testing download of service account key file

* debugging travis scripts and interaction with GCP

* added if logic back, reset service accounts manually, regenerated

* testing changes to deployment script

* moved staging build commands to the deploy area

* test

* updated link

* exclude travis, ndt from build

* remove AWS related items, added stub for production deploy

* added notebooks support back

* added missing noteboooks

* updates based on review

Author: critzo

.gitignore

@@ -16,8 +16,9 @@ coverage
 _config-deploy.yml
 _config-deploy.ymle
 js/app.js
-_cf_s3_invalidator.yml
 .vagrant
 ubuntu-xenial-16.04-cloudimg-console.log
 .jekyll-metadata
-
+_cf_s3_invalidator.yml
+_staging_deploy_key
+_sandbox_deploy_key

.gitmodules

@@ -4,3 +4,6 @@
 [submodule "ndt"]
 	path = ndt
 	url = https://github.com/m-lab/ndt
+[submodule "travis"]
+	path = travis
+	url = https://github.com/m-lab/travis.git

.travis.yml

@@ -1,56 +1,48 @@
+dist: xenial
 language: ruby
-rvm:
-  - 2.4.1
-script:
-  - ./_tests/travis-checks --quick
+
 env:
   global:
-    - NOKOGIRI_USE_SYSTEM_LIBRARIES=true # speeds up installation of html-proofer
+  - NOKOGIRI_USE_SYSTEM_LIBRARIES=true
+
+script:
+- set -e
+- $TRAVIS_BUILD_DIR/travis/install_gcloud.sh
+- sudo apt-get install -y python3-pip
+- sudo python3 -m pip install --upgrade setuptools pip
+- sudo python3 -m pip install jupyter nbconvert
+- "./_tests/travis-checks --quick"
+
 deploy:
-  - provider: s3
-    access_key_id: AKIAIRWAFHZDDWJ37X5A
-    secret_access_key:
-      secure: BlXQIyq/lHCHPGX5YUasTGWHkoL32lZHYG/n1yLBjniht2AKPWBnbQ+oWLJt2d28xZksXaDYQ02rh8lWted0SKhqYUN6dcmrXu8rVwcqpqgihUZsC1yqO4qGZ2qGkeQ2X0+HSKIBJ5VUKrMJiS3YK+N/W00/f9FatrSGvr8HBKSWpKpDiHq7z5kx/ab5Aae0hGyN1Ux6I7jTkYDI6oi9gQFJgv4oMKUt11SVMqnBNXoeOvDnIIkJqSf0hKuSZpeEDoX1FDlEaC/NR11ASlPj1ev+mxuNi1S1WTYFka7tB7piD9TQTv0PxjU+t4+TaH9jkd35eGUW1VeKPrdBtsK2TnCM6zYuM7/4U2i/7ZHf4bRVG8UkL+h32K25b24SQIyJddAzvrz8cM2/TLpqid+xdvay2Ax5T3acsxQhBq+vXVwohhl2q3fu9o6o6Bh8SGnn7XSK3JxE9Z3e8Uqg9JV3Pmgz8VyeYCwjuaV5lyabujQAn6sX8rw0RQ2A35bvQjAkNDiLUsmmOQBuRnnmS6fxeYDWf1D/h+6V0TWqPFQLTNBCjYt7WBDyTAU6i71c8ZDSay2CqVca0djxzXJHuqys5q4gRqXpLkuOiJUJMR8s5NhgQAI6bSCkgsJ/EcJqpSIwI72W+NgBb0I31KeREJXO5UlNmTa7RBFSX02KE23ut/E=
-    bucket: www-measurementlab-net
-    skip_cleanup: true
-    local_dir: _site
-    on:
-      repo: m-lab/m-lab.github.io
-      tags: true
-  - provider: gcs
-    access_key_id: GOOGWS5RSVP6RJ3HGCQWGFZG
-    secret_access_key:
-      secure: pdKctNlM2bfEzyFBi9Rr6BYua8hWXsyzUDm5WI4Alr06qLmp1zFNcPymKcihgoE8k2vr85B4BN8HYpUPvPPpzb/KlAZAuUjbKGH0hIklBN4+G+ldt2IBN+2YxYYKYu3bXjKJ5yQOHKCBVU/CR3O6UB+Llp3Ty42OCa71WDfsG2aW6EHGkWV1TljXl3fGVerPfcNeyigIFZ6Qz2Vy0Ay+hzXEZBxjLjUsrWbK5aM6PX7OuErBzKk/kFTlQNkuRQdJs7nS4Y+Pxjyjr6NVd0wjFGqtP+sLjC9hvNaaCHdQ46kA24BqNWwNq1c3++9/0NrEj2MbRrAWNrEcSDskX+XJdV+wantzJ0xQAdqgYUugLj+TSYTKTxrPnwy9WGcWxsfdlrrlnysWVDc1OpQgtXNtU66qwgsjflsOijNeV/XBhR4bOifhRJ80VcQ6KyDWwIAKcJbU8prkMAqr1SqyVbw35gH6ZfHxBcf1CYCEoCGlVn00QPyEQDBkSCFzQC8coSZStGGnxGaebHomxMiOsowM4JIZqjB3RWg+Gf5S6dAZZUQTHoYHAU02Kil+wHmYWEcJ+3eb/FWtNokeZRX3zNCKncBAaZWef+JzMuEtcq67cio/io3z9yglVcWpZz/xGSMOTOVIbHwFIjyzO8cSvp6IWLrlq8MJOge5u0ADgi0rMAE=
-    bucket: website.mlab-staging.measurementlab.net
-    skip_cleanup: true
-    local_dir: _site
-    on:
-      repo: m-lab/m-lab.github.io
-      all_branches: true
-      condition: $TRAVIS_BRANCH == master && $TRAVIS_EVENT_TYPE == push
-  - provider: gcs
-    access_key_id: GOOGTVZEKZMTGZSV27YKVWP3
-    secret_access_key: 
-      secure: "YjvtwOWEVSJ1qC7gx9ABM0z0BcHf2IVpeSTzLzykZ5y6s2nTXBNOccC2b2dri7zRqen5sLxsB6cXqC5+X764NE1H/pG37EyU27AS3sRzJzdl+NAYfXujd77bByhS+PY2Rk2IndhGNv2b6xFGwqYUprqhEEit+mBLuHWDRkK3eVVDkNd8r0AsML/imT3UNYwXXThgKr9jAGDPrxUX39KNWWtFn0goU96qfEh7jncChu9/JteDxcBn18AFJE/4pRSPibBFNZoels5TKG5ISyyff6N0s+zVtbqXcDnCY2yMl4LL66z8L9x7c7loXz1txoz38jvxW3bvQM/asrWPwMpXYfw/MJ1F1pNP0aqKcaLDlZ5Ow88wJpGxY6C3laBWUBKXp6u+sew1uLtybfClgGc/Ma3s3LjwzdfJjvmzWPbQT9tTKiVtJ+17VSmxMRMWpksYWEuCy+iySLLmwMaK7A9gfVYP6yqIm5Z0S6oBQqCk0y20TGkY9IRJ/HvXt4JbZi+rgLAWP8lKFt7V2ATrbCzjcllp9VzgsYNqxeqsuGFLYv5CChUMcC+lZe8O6YTzKi272DAGb8DItJDXcHu1pGHMx1nI8ihECWXMnEri0EpsKGvi4kT9BM++23j0ZlZ+F6Mr7vdmXKUL5YrxNIWmvXc+Xu7pVF1+31pkteyORVKKBKI="
-    bucket: website.mlab-sandbox.measurementlab.net
-    skip_cleanup: true
-    local_dir: _site
-    on:
-      repo: m-lab/m-lab.github.io
-      all_branches: true
-      condition: $TRAVIS_BRANCH == sandbox-* && $TRAVIS_EVENT_TYPE == push
-before_install:
-  # The Amazon AWS credentials needed by the cf-s3-inv utility in after_success
-  # are stored in a file encrypted with the Travis keys associated with this
-  # repository. The openssl command descrypts this file, using some secure
-  # environment variables associated with the Travis account/repo.  See
-  # README.md for information on generating this encrypted file.
-  - if [ "$TRAVIS_BRANCH" == "master" ] && [ "$TRAVIS_PULL_REQUEST" == "false" ] && [ "$TRAVIS_EVENT_TYPE" != "cron" ]; then openssl aes-256-cbc -K $encrypted_69b240b11ca3_key -iv $encrypted_69b240b11ca3_iv -in _cf_s3_invalidator.yml.enc -out _cf_s3_invalidator.yml -d; fi
-  - sudo apt-get install -y python3-pip
-  - sudo python3 -m pip install --upgrade setuptools pip
-  - sudo python3 -m pip install jupyter nbconvert
-after_deploy:
-  # Only invalidate CloudFront cache if this on the master branch and is not a PR
-  if [ "$TRAVIS_BRANCH" == "master" ] && [ "$TRAVIS_PULL_REQUEST" == "false" ] && [ "$TRAVIS_EVENT_TYPE" != "cron" ]; then
-    cf-s3-inv || travis_terminate 1;
-  fi
+# Separate deployments to various places based on branch and push type
+## - sandbox
+- provider: script
+  skip_cleanup: true
+  script: 
+    $TRAVIS_BUILD_DIR/travis/activate_service_account.sh SERVICE_ACCOUNT_mlab_sandbox
+    && gsutil cp -r $TRAVIS_BUILD_DIR/_site/* gs://website.mlab-sandbox.measurementlab.net/
+  on:
+    repo: m-lab/website
+    all_branches: true
+    condition: "$TRAVIS_BRANCH == sandbox-* && $TRAVIS_EVENT_TYPE == push"
+
+## - staging
+- provider: script
+  skip_cleanup: true
+  script: 
+    $TRAVIS_BUILD_DIR/travis/activate_service_account.sh SERVICE_ACCOUNT_mlab_staging
+    && gsutil cp -r $TRAVIS_BUILD_DIR/_site/* gs://website.mlab-staging.measurementlab.net/
+  on:
+    repo: m-lab/website
+    all_branches: true
+    condition: "$TRAVIS_BRANCH == master && $TRAVIS_EVENT_TYPE == push"
+
+## - production
+#- provider: script
+#  script:
+#    $TRAVIS_BUILD_DIR/travis/activate_service_account.sh SERVICE_ACCOUNT_mlab_oti
+#    && gsutil cp -r $TRAVIS_BUILD_DIR/_site/* gs://website.mlab-oit.measurementlab.net/
+#  skip_cleanup: true
+#  on:
+#    repo: m-lab/website
+#    tags: true

Gemfile

@@ -11,3 +11,5 @@ gem 'html-proofer', '3.0.5'
 gem 'mdl', '0.3.1'
 gem 'cf-s3-invalidator'
 gem 'jekyll-jupyter-notebook'
+gem 'openssl'
+gem 'jekyll-jupyter-notebook'

_cf_s3_invalidator.yml.enc

@@ -26,7 +26,10 @@ exclude:
   - README.md
   - Gemfile
   - Gemfile.lock
+  - _scripts
   - vendor
+  - travis
+  - ndt
 
 # Markdown settings
 markdown: kramdown

_config.yml

@@ -7,7 +7,7 @@ breadcrumb: tests
 
 # BISmark
 
-BISmark is an [OpenWRT](http://openwrt.org){:target="_blank"}-based platform that measures Internet service provider (ISP) performance and traffic inside home networks. BISmark takes the place of your home router, performing all of your router's normal functions in addition to periodic network performance measurements. BISmark users gain access to a personalized dashboard showing performance over time and comparing performance across ISPs and regions.
+BISmark is an OpenWRT-based (http://openwrt.org) platform that measures Internet service provider (ISP) performance and traffic inside home networks. BISmark takes the place of your home router, performing all of your router's normal functions in addition to periodic network performance measurements. BISmark users gain access to a personalized dashboard showing performance over time and comparing performance across ISPs and regions.
 
 [Apply to host a BISmark gateway](http://projectbismark.net/){:target="_blank"}
 

_pages/tests/bismark.md

@@ -8,14 +8,14 @@ grid-section:
     group-list-class: "l-grid l-3col"
     grid-img-class: " framed"
     grid-items:
-      - link: "http://www.netflix.com/"
+      - link: "https://www.netflix.com/"
         image: "jeff-w-boote.png"
         heading: "Jeff W. Boote"
         desc: "Netflix"
-      - link: "https://www.newamerica.org/our-people/georgia-bullen/"
+      - link: "https://simplysecure.org/who-we-are/georgia.html"
         image: "georgia-bullen.png"
         heading: "Georgia Bullen"
-        desc: "New America / Open Technology Institute"
+        desc: "Simply Secure"
       - link: "http://research.google.com/pubs/author32412.html"
         image: "vint-cerf.png"
         heading: "Vint Cerf"

_pages/who.md

@@ -28,4 +28,4 @@ fi
 
 # Run HTMLProofer
 bundle exec jekyll build --config _config-deploy.yml
-bundle exec htmlproofer ./_site --only-4xx --check-html --file-ignore "/mlab_observatory/","/ndt/" "$htmlproofer_args_extra"
+bundle exec htmlproofer ./_site --only-4xx --check-html --file-ignore "/ndt/" "$htmlproofer_args_extra"