Add subcommand for GCP

[scottyhq]

Currently things only work for AWS as noted in the readme

[consideRatio]

I did some brief exploration without arriving at the goal of extracting the credentials from a jupyter server started in a 2i2c k8s based server to use from my laptop.

First here is a basic Python command to trial access.

from google.cloud import storage
blobs = storage.Client().list_blobs("<some globally unique google storage bucket name>")
print(blobs.__next__().name)

$ gcloud auth list

                 Credentialed Accounts
ACTIVE  ACCOUNT
*       redacted-gcp-sa-name@redacted-project-name.iam.gserviceaccount.com

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

$ gcloud auth print-access-token
ya29.cREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTEDREDACTED.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

$ gcloud --help

OTHER FLAGS
     --access-token-file=ACCESS_TOKEN_FILE
        A file path to read the access token. Use this flag to authenticate
        gcloud with an access token. The credentials of the active account (if
        exists) will be ignored. The file should only contain an access token
        with no other information. Overrides the default auth/access_token_file
        property value for this command invocation.

     --impersonate-service-account=SERVICE_ACCOUNT_EMAILS
        For this gcloud invocation, all API requests will be made as the given
        service account or target service account in an impersonation
        delegation chain instead of the currently selected account. You can
        specify either a single service account as the impersonator, or a
        comma-separated list of service accounts to create an impersonation
        delegation chain. The impersonation is done without needing to create,
        download, and activate a key for the service account or accounts.

        In order to make API requests as a service account, your currently
        selected account must have an IAM role that includes the
        iam.serviceAccounts.getAccessToken permission for the service account
        or accounts.

        The roles/iam.serviceAccountTokenCreator role has the
        iam.serviceAccounts.getAccessToken permission. You can also create a
        custom role.

        You can specify a list of service accounts, separated with commas. This
        creates an impersonation delegation chain in which each service account
        delegates its permissions to the next service account in the chain.
        Each service account in the list must have the
        roles/iam.serviceAccountTokenCreator role on the next service account
        in the list. For example, when --impersonate-service-account=
        SERVICE_ACCOUNT_1,SERVICE_ACCOUNT_2, the active account must have the
        roles/iam.serviceAccountTokenCreator role on SERVICE_ACCOUNT_1, which
        must have the roles/iam.serviceAccountTokenCreator role on
        SERVICE_ACCOUNT_2. SERVICE_ACCOUNT_1 is the impersonated service
        account and SERVICE_ACCOUNT_2 is the delegate.

        Overrides the default auth/impersonate_service_account property value
        for this command invocation.

[consideRatio]

Note to myself: if this feature is implemented, Arpita would be happy to help trial it out, and I can connect back to Arpita via https://2i2c.freshdesk.com/a/tickets/322.

[consideRatio]

To use an access token directly is possible in three ways as outlined here, and summarized by me below:

Declaring the CLOUDSDK_AUTH_ACCESS_TOKEN environment variable, see https://cloud.google.com/sdk/docs/authorizing

--access_token_file flag, see https://cloud.google.com/sdk/gcloud/reference#--access-token-file.

Configuration of access_token_file, see https://cloud.google.com/sdk/gcloud/reference/config/set and search for access_token_file

---

I think what we should do is to trial and document the procedure of using an access token directly via CLOUDSDK_AUTH_ACCESS_TOKEN, if it works out well, we work on this project to acquire a access token by gcloud auth print-access-token.

[consideRatio]

I tried some things using Google Cloud SDK 413.0.0 and the Python client google-storage-client and learned somethings of relevance.

1. Options work with gcloud, but not all clients

All of the options below worked to setup the gcloud CLI with credentials.

• export CLOUDSDK_AUTH_ACCESS_TOKEN=<access token>
• gcloud config set auth/access_token_file $(pwd)/my-access-token.txt
• gcloud storage ls <bucket> --access_token_file=$(pwd)/my-access-token.txt

2. Python clients don't respect options listed above

In this github issue, support for CLOUDSDK_AUTH_ACCESS_TOKEN by Google's Python libraries is requested.

However, they can consume access tokens directly like this:

# Example on using a temporary GCP access token.
# 
# To acquire a token from some location, run
# 
#     gcloud auth print-access-token
#
# To use it with python libraries like google-cloud-storage, first create a
# credentials object to pass to the client.
#
import getpass
from google.cloud import storage
from google.oauth2.credentials import Credentials

# import an access token
# - option 1: read an access token from a file
with open("my-access-token.txt") as f:
    access_token = f.read().strip()
# - option 2: read an access token from user input
access_token = getpass.getpass("Enter access token: ")

# setup a storage client using credentials
credentials = Credentials(access_token)
storage_client = storage.Client(credentials=credentials)

# test the storage client by trying to list content in a google storage bucket
bucket_name = "something"  # don't include gs:// here
blobs = list(storage_client.list_blobs(bucket_name))
print(len(blobs))

[consideRatio]

Docs on how to generate
a token: https://cloud.google.com/iam/docs/create-short-lived-credentials-direct#sa-credentials-oauth

Access tokens are valid for one hour by default as described in the example on requesting an access token using the REST api directly, see https://cloud.google.com/iam/docs/create-short-lived-credentials-direct#rest_2.

The lifetime can be specified only when using the REST API directly (not using gcloud as of google cloud sdk 417.0.0). But no more than 3600 can be granted unless elevanted permissions have been configured ahead of time.

It seems that gcloud auth print-access-token will keep printing the same token for a long time, so it probably has a cache or similar.

[scottyhq]

Thanks for digging and documenting @consideRatio I can confirm it all seems to be working on the pangeo GCP jupyterhub:

Can confirm this works on pangeo-cloud GCP Hub :

1. create short-lived token on jupyterhub (https://us-central1-b.gcp.pangeo.io/)

mamba create -n gcloud google-cloud-sdk
conda activate gcloud
gcloud auth print-access-token
#ya29.c.b0Aaekm1I4h6J_I4lTX…….

2. use the credentials from a laptop:

export CLOUDSDK_AUTH_ACCESS_TOKEN=ya29.c.b0Aaekm1I4h6J_I4lTX…….

gcloud storage ls gs://pangeo-integration-te-3eea-prod-scratch-bucket/scottyhq/
# gs://pangeo-integration-te-3eea-prod-scratch-bucket/scottyhq/hi-from-the-hub.txt

gcloud storage cp hi-from-scotts-laptop.txt gs://pangeo-integration-te-3eea-prod-scratch-bucket/scottyhq/hi-from-scotts-laptop.txt
# Copying file://hi-from-scotts-laptop.txt to gs://pangeo-integration-te-3eea-prod-scratch-bucket/scottyhq/hi-from-scotts-laptop.txt
#  Completed files 1/1 | 15.0B/15.0B

gcloud storage cp -n -r /Users/scott/Data/tutorial-data gs://pangeo-integration-te-3eea-prod-scratch-bucket/scottyhq/

#  Completed files 17/17 | 175.4MiB/175.4MiB                                                                                                           
 # Average throughput: 97.5MiB/s

The last command is really handy (-r for recursive copy and -n for “no clobber” if the path already exists, and it was quite fast!). cc @jbusecke