Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: adding encoding. It is required by S3Boto3Storage. #31742

Closed
wants to merge 2 commits into from

Conversation

awais786
Copy link
Contributor

@awais786 awais786 commented Feb 11, 2023

Upgrade PR from boto to boto3 gives error on production.

Run this command python manage.py cms export_course_metadata_for_all_courses.

Faced errors on edxapp-cms-worker related to Course Metadata failure with following stack trace

[2023-02-09 13:29:44,028: ERROR/ForkPoolWorker-15] Task cms.djangoapps.export_course_metadata.tasks.export_course_metadata_task[1978fb43-09a0-4973-9cd4-8635aca89aec] raised unexpected: TypeError('Unicode-objects must be encoded before hashing')
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/newrelic/hooks/application_celery.py", line 99, in wrapper
    return wrapped(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/export_course_metadata/tasks.py", line 31, in export_course_metadata_task
    course_metadata_export_storage.save(f'course_metadata_export/{course_key}.json', highlights_content)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
    name = self._save(name, content)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/boto3/s3/inject.py", line 511, in object_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/client.py", line 591, in _make_api_call
    handler, event_response = self.meta.events.emit_until_response(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/handlers.py", line 209, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/handlers.py", line 187, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/handlers.py", line 201, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing

Boto3 PR #31282

We are testing this change on boto.

@awais786 awais786 marked this pull request as ready for review February 11, 2023 06:53
@awais786
Copy link
Contributor Author

awais786 commented Feb 11, 2023

down the stream it tried to encoded here also. So eventually encoding is happening

Copy link
Contributor

@iamsobanjaved iamsobanjaved left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified that json.loads works fine with encoded value.

@awais786 awais786 closed this Feb 14, 2023
@nedbat nedbat deleted the encoding-data branch January 8, 2024 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants