Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed error invalid URL escape "% ^" Closes #736 #735

Closed
wants to merge 1 commit into from

Conversation

ParthaI
Copy link

@ParthaI ParthaI commented Jan 23, 2024

@kai, I've created a draft PR to address issue #1954 and would appreciate your review.

I used the package k8s.io/apimachinery/pkg/util/yaml, which seemed effective in resolving the issue, though I'm open to suggestions if there's a more commonly used package for this purpose.

During my attempts to fix the issue, I explored several other packages but they didn't yield the desired results:

To detail the error:

  • The UnmarshalYAML() function used yaml.Unmarshal() from github.com/ghodss/yaml and url.QueryUnescape from net/url for decoding and parsing YAML to JSON.
  • An error occurred with QueryUnescape when a '%' wasn't followed by two hexadecimal digits.
  • Additionally, UnmarshalYAML encountered an error with a specific code block in the template.
CopyZipsFunction:
    Type: AWS::Lambda::Function
    Properties:
      Description: Copies objects from a source S3 bucket to a destination
      Handler: index.handler
      Runtime: python3.8
      Role: !GetAtt "CopyZipsRole.Arn"
      Timeout: 240
      Code:
        ZipFile: |
          import json
          import logging
          import threading
          import boto3
          import cfnresponse
          def copy_objects(source_bucket, dest_bucket, prefix, objects):
              s3 = boto3.client('s3')
              for o in objects:
                  key = prefix + o
                  copy_source = {
                      'Bucket': source_bucket,
                      'Key': key
                  }
                  print('copy_source: %s' % copy_source)
                  print('dest_bucket = %s'%dest_bucket)
                  print('key = %s' %key)
                  s3.copy_object(CopySource=copy_source, Bucket=dest_bucket,
                        Key=key)
          def delete_objects(bucket, prefix, objects):
              s3 = boto3.client('s3')
              objects = {'Objects': [{'Key': prefix + o} for o in objects]}
              s3.delete_objects(Bucket=bucket, Delete=objects)
          def timeout(event, context):
              logging.error('Execution is about to time out, sending failure response to CloudFormation')
              cfnresponse.send(event, context, cfnresponse.FAILED, {}, None)
          def handler(event, context):
              # make sure we send a failure to CloudFormation if the function
              # is going to timeout
              timer = threading.Timer((context.get_remaining_time_in_millis()
                        / 1000.00) - 0.5, timeout, args=[event, context])
              timer.start()
              print('Received event: %s' % json.dumps(event))
              status = cfnresponse.SUCCESS
              try:
                  source_bucket = event['ResourceProperties']['SourceBucket']
                  dest_bucket = event['ResourceProperties']['DestBucket']
                  prefix = event['ResourceProperties']['Prefix']
                  objects = event['ResourceProperties']['Objects']
                  if event['RequestType'] == 'Delete':
                      delete_objects(dest_bucket, prefix, objects)
                  else:
                      copy_objects(source_bucket, dest_bucket, prefix, objects)
              except Exception as e:
                  logging.error('Exception: %s' % e, exc_info=True)
                  status = cfnresponse.FAILED
              finally:
                  timer.cancel()
                  cfnresponse.send(event, context, status, {}, None)

Please feel free to suggest any modifications or alternative packages.

Thanks!

@ParthaI ParthaI requested a review from kaidaguerre January 23, 2024 15:26
@ParthaI ParthaI self-assigned this Jan 23, 2024
@ParthaI ParthaI changed the title Fixed error invalid URL escape "% ^" Closes #2060 Fixed error invalid URL escape "% ^" Closes #736 Jan 23, 2024
@ParthaI ParthaI linked an issue Jan 23, 2024 that may be closed by this pull request
@ParthaI ParthaI marked this pull request as draft January 23, 2024 15:29
"strings"
"time"

"github.com/turbot/steampipe-plugin-sdk/v5/grpc"

"github.com/ghodss/yaml"
"k8s.io/apimachinery/pkg/util/yaml"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need the kubernetes reference here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flowpipe uses "github.com/goccy/go-yaml"

if err != nil {
return nil, err

// Escape only URLs instead of checking if any % is not followed by two hexadecimal digits.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add some tests

@kaidaguerre kaidaguerre closed this Feb 1, 2024
@kaidaguerre kaidaguerre deleted the fix-yaml-unmarsal-func branch February 1, 2024 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix error invalid URL escape "% ^"
2 participants