Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exclude non-personal, non-globus endpoints from local file record delete #812

Merged
merged 1 commit into from
Oct 13, 2023

Conversation

k1o0
Copy link
Collaborator

@k1o0 k1o0 commented Oct 12, 2023

Deleting large local files on the Steinmetz server failed with the following error:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/ubuntu/alyxvenv/lib/python3.8/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/home/ubuntu/alyxvenv/lib/python3.8/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/ubuntu/alyxvenv/lib/python3.8/site-packages/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/ubuntu/alyxvenv/lib/python3.8/site-packages/django/core/management/base.py", line 448, in execute
    output = self.handle(*args, **options)
  File "/var/www/alyx-main/alyx/data/management/commands/files.py", line 151, in handle
    transfers.globus_delete_local_datasets(dsets, dry=dry)
  File "/var/www/alyx-main/alyx/data/transfers.py", line 676, in globus_delete_local_datasets
    delete_clients.append(globus_sdk.DeleteData(gtc, ge, label=label))
  File "/home/ubuntu/alyxvenv/lib/python3.8/site-packages/globus_sdk/services/transfer/data/delete_data.py", line 118, in __init__
    raise exc.GlobusSDKUsageError("endpoint is required")
globus_sdk.exc.base.GlobusSDKUsageError: endpoint is required

This issue is that the file records associated with the input datasets include those on AWS where the data repository Globus endpoint ID is null. The key change here was adding a filter for globus_is_personal=True, and globus_endpoint_id__isnull=True.

I am surprised this error has not been reported before and hence would like a review before merging. I couldn't locate a recent change that would have caused a regression.

Copy link
Collaborator

@oliche oliche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per our discussion.
Thanks !

@k1o0 k1o0 merged commit 13d608a into dev Oct 13, 2023
1 check passed
@k1o0 k1o0 deleted the globus_delete_local_datasets_hotfix branch October 13, 2023 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants