Merge branch 'feature/restructure-aws-signing-round-tripper' into main

awsauth/README.md

@@ -0,0 +1,132 @@
+# awsauth package
+
+## Round Tripper
+
+The `NewAWSSignerRoundTripper` creates a http.Transport that can then be used instead of the default transport used in this libraries http package to autosign http requests to AWS services, such as AWS Elasticsearch.
+
+See [Signer](#signer) for details on various signing mechanisms.
+
+Below is an example of how to setup the aws signer round tripper and attach to a new client using the `NewClientWithTransport` method in http package. Example uses elasticsearch as the aws service denoted by `es` and `eu-west-1` for region.
+
+    ```go
+    import (
+        "github.com/ONSdigital/dp-net/v2/awsauth"
+        dphttp "github.com/ONSdigital/dp-net/v2/http"
+    )
+    ...
+    awsSignerRT, err := awsauth.NewAWSSignerRoundTripper("", "", "eu-west-1", "es")
+	if err != nil {
+		...
+	}
+
+	httpClient := dphttp.NewClientWithTransport(awsSignerRT)
+    ```
+
+If you are looking to connect a local instance of application to managed AWS Elasticsearch,
+then you will need to create a tunnel onto the VPC that Elasticsearch is running on, and
+implement the following in your application:
+
+    ```go
+    import (
+        "github.com/ONSdigital/dp-net/v2/awsauth"
+        dphttp "github.com/ONSdigital/dp-net/v2/http"
+    )
+    ...
+
+    awsSignerRT, err := awsauth.NewAWSSignerRoundTripper("~/.aws/credentials", "default", "eu-west-1", "es",
+        awsauth.Options{TlsInsecureSkipVerify: true})
+	if err != nil {
+		...
+	}
+
+	httpClient := dphttp.NewClientWithTransport(awsSignerRT)
+    ```
+
+The file location and profile need to be set as the first two variables in method signature
+respectively; in the above example these are the default values expected across the industry.
+
+:warning: setting `TlsInsecureSkipVerify` to `true` should only be used for developer testing. If used in an application use a new environment variable to control whether this is on/off, e.g. `awsauth.Options{TlsInsecureSkipVerify: cfg.TlsInsecureSkipVerify}` :warning:
+
+## Signer
+
+Using AWS SDK library to create a signer function to successfully sign elasticsearch requests hosted in AWS.
+The function adds multiple providers to the credentials chain that is used by the [AWS SDK V4 signer method `Sign`](https://docs.aws.amazon.com/sdk-for-go/api/aws/signer/v4/#Signer.Sign).
+
+1) **Environment Provider** will attempt to retrieve credentials from `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` set on the environment.
+
+    Requires `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` variables to be set/exported onto the environment.
+
+    ```go
+    import awsauth "github.com/ONSdigital/dp-net/v2/awsauth"
+    ...
+        signer, err := esauth.NewAwsSigner("", "", "eu-west-1", "es")
+        if err != nil {
+            ... // Handle error
+        }
+    ...
+    ```
+
+2) **Shared Credentials Provider** will attempt to retrieve credentials from the absolute path to credentials file, the default value if the filename is set to empty string `""` will be `~/.aws/credentials` and the default profile will be `default` if set to empty string.
+
+    Requires credentials file to exist in the location specified in NewAwsSigner func.
+    File must contain the keys necessary under the matching Profile heading, see example below:
+
+    ```
+        [development]
+        aws_access_key_id=<access key id>
+        aws_secret_access_key=<secret access key>
+        region=<region>
+    ```
+
+    ```go
+    import esauth "github.com/ONSdigital/dp-net/v2/awsauth"
+    ...
+        signer, err := esauth.NewAwsSigner("~/.aws/credentials", "development", "eu-west-1", "es")
+        if err != nil {
+            ...
+        }
+    ...
+    ```
+
+3) **EC2 Role Provider** will attempt to retrieve credentials using an EC2 metadata client (this is created using an AWS SDK session).
+
+    Requires Code is run on EC2 instance.
+
+    ```go
+    import esauth "github.com/ONSdigital/dp-net/v2/awsauth"
+    ...
+        signer, err := esauth.NewAwsSigner("", "", "eu-west-1", "es")
+        if err != nil {
+            ...
+        }
+    ...
+    ```
+
+For more information on Providers for obtaining credentials, [see AWS documentation](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials).
+
+The signer object should be created once on startup of an application and reused for each request, otherwise you will experience performance issues due to creating a session for every request.
+
+To sign elasticsearch requests, one can use the signer like so:
+
+```go
+    ...
+
+    var req *http.Request
+    
+    // TODO set request
+
+    var bodyReader io.ReadSeeker
+
+    if payload != <zero value of type> { // Check for a payload
+        bodyReader = bytes.NewReader(<payload in []byte>)
+        req, err = http.NewRequest(<method>, <path>, bodyReader)
+    } else { // No payload (request body is empty)
+        req, err = http.NewRequest(<method>, <path>, nil)
+    }
+    
+    if err = signer.Sign(req, bodyReader, time.Now()); err != nil {
+        ... // handle error
+    }
+    
+    ...
+```

awsauth/roundtripper.go

@@ -0,0 +1,67 @@
+package awsauth
+
+import (
+	"bytes"
+	"crypto/tls"
+	"fmt"
+	"io"
+	"net/http"
+	"time"
+
+	dphttp "github.com/ONSdigital/dp-net/v2/http"
+)
+
+type AwsSignerRoundTripper struct {
+	signer       *Signer
+	roundTripper http.RoundTripper
+}
+
+type Options struct {
+	// InsecureSkipVerify controls whether a client verifies the server's certificate
+	// chain and host name. If InsecureSkipVerify is true, crypto/tls accepts any
+	// certificate presented by the server and any host name in that certificate.
+	// In this mode, TLS is susceptible to machine-in-the-middle attacks unless custom
+	// verification is used. This should be used only for testing or in combination
+	// with VerifyConnection or VerifyPeerCertificate.
+	TlsInsecureSkipVerify bool
+}
+
+var defaultAWSTransport = dphttp.DefaultTransport
+
+func NewAWSSignerRoundTripper(awsFilename, awsProfile, awsRegion, awsService string, options ...Options) (*AwsSignerRoundTripper, error) {
+
+	if awsRegion == "" || awsService == "" {
+		return nil, fmt.Errorf("aws region and service should be valid options")
+	}
+
+	awsSigner, err := NewAwsSigner(awsFilename, awsProfile, awsRegion, awsService)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create aws v4 signer: %w", err)
+	}
+
+	if len(options) > 0 && options[0].TlsInsecureSkipVerify {
+		defaultAWSTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
+	}
+
+	return &AwsSignerRoundTripper{
+		signer:       awsSigner,
+		roundTripper: defaultAWSTransport,
+	}, nil
+}
+
+func (srt *AwsSignerRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+	var body []byte
+	var err error
+	if req.Body != nil {
+		body, err = io.ReadAll(req.Body)
+		if err != nil {
+			return nil, fmt.Errorf("failed to read request body: %w", err)
+		}
+	}
+
+	if err := srt.signer.Sign(req, bytes.NewReader(body), time.Now()); err != nil {
+		return nil, fmt.Errorf("failed to sign the request: %w", err)
+	}
+
+	return srt.roundTripper.RoundTrip(req)
+}

awsauth/roundtripper_test.go

@@ -0,0 +1,91 @@
+package awsauth
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"testing"
+
+	dphttp "github.com/ONSdigital/dp-net/v2/http"
+	"github.com/ONSdigital/dp-net/v2/http/httptest"
+	. "github.com/smartystreets/goconvey/convey"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestNewAWSSignerRoundTripper(t *testing.T) {
+	t.Parallel()
+
+	awsSignerRT, err := NewAWSSignerRoundTripper("some_filename", "some_profile", "some_region", "some_service")
+
+	assert.Nil(t, err, "error should be nil")
+	assert.NotNilf(t, awsSignerRT, "aws signer roundtripper should  not return nil")
+}
+
+func TestNewAWSSignerRoundTripper_WhenAWSRegionIsEmpty_Returns(t *testing.T) {
+	t.Parallel()
+
+	awsSignerRT, err := NewAWSSignerRoundTripper("some_filename", "some_profile", "", "some_service")
+
+	assert.NotNil(t, err, "error should not be nil")
+	assert.Nil(t, awsSignerRT, "aws signer roundtripper should return nil")
+}
+
+func TestNewAWSSignerRoundTripper_WhenAWSServiceIsEmpty_Returns(t *testing.T) {
+	t.Parallel()
+
+	awsSignerRT, err := NewAWSSignerRoundTripper("some_filename", "", "some_region", "")
+
+	assert.NotNil(t, err, "error should not be nil")
+	assert.Nil(t, awsSignerRT, "aws signer roundtripper should return nil")
+}
+
+func TestNewClientWithTransport(t *testing.T) {
+	t.Parallel()
+
+	Convey("Given a access and secret key are set in the environement", t, func() {
+		accessKeyID, secretAccessKey := setEnvironmentVars()
+
+		Convey("When a new client is created with aws signer round tripper", func() {
+
+			awsSignerRT, err := NewAWSSignerRoundTripper("", "", "eu-west-1", "es")
+			if err != nil {
+				t.Errorf(fmt.Sprintf("unable to implement roundtripper for test, error: %v", err))
+			}
+
+			httpClient := dphttp.NewClientWithTransport(awsSignerRT)
+
+			ts := httptest.NewTestServer(200)
+			expectedCallCount := 0
+			Convey("When Get() is called on a URL", func() {
+				expectedCallCount++
+				resp, err := httpClient.Get(context.Background(), ts.URL)
+				So(err, ShouldBeNil)
+				So(resp, ShouldNotBeNil)
+
+				call, err := unmarshallResp(resp)
+				So(err, ShouldBeNil)
+
+				Convey("Then the server sees a GET with no body", func() {
+					So(call.CallCount, ShouldEqual, expectedCallCount)
+					So(call.Method, ShouldEqual, "GET")
+					So(call.Body, ShouldEqual, "")
+					So(call.Error, ShouldEqual, "")
+					So(resp.Header.Get("Content-Type"), ShouldContainSubstring, "text/plain")
+				})
+			})
+		})
+
+		removeTestEnvironmentVariables(accessKeyID, secretAccessKey)
+	})
+}
+
+func unmarshallResp(resp *http.Response) (*httptest.Responder, error) {
+	responder := &httptest.Responder{}
+	body := httptest.GetBody(resp.Body)
+	err := json.Unmarshal(body, responder)
+	if err != nil {
+		panic(err.Error() + string(body))
+	}
+	return responder, err
+}

awsauth/sign_test.go

@@ -16,6 +16,8 @@ const (
 
 	testAccessKey       = "TEST_ACCESS_KEY"
 	testSecretAccessKey = "TEST_SECRET_KEY"
+	testCredFile        = "testdata/.aws/credentials"
+	testProfile         = "default"
 )
 
 func TestCreateNewSigner(t *testing.T) {
@@ -100,6 +102,21 @@ func TestSignFunc(t *testing.T) {
 
 			removeTestEnvironmentVariables(accessKeyID, secretAccessKey)
 		})
+
+		Convey("When the signer.v4 is a valid aws file and profile", func() {
+			signer, err := NewAwsSigner(testCredFile, testProfile, "eu-west-1", "es")
+			So(err, ShouldBeNil)
+			So(signer, ShouldNotBeNil)
+			So(signer.v4, ShouldNotBeNil)
+
+			Convey("Then the request successfully signs and does not return an error", func() {
+
+				req := httptest.NewRequest("GET", "http://test-url", nil)
+
+				err = signer.Sign(req, nil, time.Now())
+				So(err, ShouldBeNil)
+			})
+		})
 	})
 }
 

awsauth/testdata/.aws/credentials

@@ -0,0 +1,4 @@
+[default]
+aws_access_key_id=testaccesskey
+aws_secret_access_key=testsecretaccesskey
+region=eu-west-1

ci/lint.yml

@@ -6,7 +6,7 @@ image_resource:
   type: docker-image
   source:
     repository: golang
-    tag: 1.16
+    tag: 1.17
 
 inputs:
   - name: dp-net

go.mod

@@ -1,16 +1,36 @@
 module github.com/ONSdigital/dp-net/v2
 
-go 1.16
+go 1.17
 
 require (
-	github.com/ONSdigital/dp-api-clients-go/v2 v2.1.7-beta
-	github.com/ONSdigital/dp-net v1.0.12
-	github.com/ONSdigital/log.go/v2 v2.0.9
-	github.com/aws/aws-sdk-go v1.38.15
+	github.com/ONSdigital/dp-api-clients-go/v2 v2.92.2
+	github.com/ONSdigital/dp-net v1.2.0
+	github.com/ONSdigital/log.go/v2 v2.1.0
+	github.com/aws/aws-sdk-go v1.42.47
 	github.com/gorilla/mux v1.8.0
 	github.com/justinas/alice v1.2.0
 	github.com/pkg/errors v0.9.1
 	github.com/smartystreets/goconvey v1.7.2
 	github.com/stretchr/testify v1.7.0
-	golang.org/x/net v0.0.0-20201110031124-69a78807bb2b
+	golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
+)
+
+require (
+	github.com/ONSdigital/dp-api-clients-go v1.43.0 // indirect
+	github.com/ONSdigital/dp-healthcheck v1.2.3 // indirect
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/fatih/color v1.13.0 // indirect
+	github.com/gopherjs/gopherjs v0.0.0-20220104163920-15ed2e8cf2bd // indirect
+	github.com/hokaccha/go-prettyjson v0.0.0-20211117102719-0474bc63780f // indirect
+	github.com/jmespath/go-jmespath v0.4.0 // indirect
+	github.com/jtolds/gls v4.20.0+incompatible // indirect
+	github.com/kr/pretty v0.2.0 // indirect
+	github.com/mattn/go-colorable v0.1.12 // indirect
+	github.com/mattn/go-isatty v0.0.14 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/smartystreets/assertions v1.2.1 // indirect
+	golang.org/x/sys v0.0.0-20220204135822-1c1b9b1eba6a // indirect
+	gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
+	gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
 )