Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resubmission with OOM doesn't seem to be kicked off #65

Open
pcm32 opened this issue Dec 13, 2022 · 7 comments
Open

Resubmission with OOM doesn't seem to be kicked off #65

pcm32 opened this issue Dec 13, 2022 · 7 comments

Comments

@pcm32
Copy link
Member

pcm32 commented Dec 13, 2022

On my k8s setup, having fixed the OOM labeling, I use the following tpv setup:

  /galaxy/server/config/persist-seq-extra-tpv.yaml:
    applyToJob: true
    applyToWeb: true
    applyToSetupJob: false
    applyToWorkflow: true
    applyToNginx: false
    tpl: false
    content: |
      global:
        default_inherits: default
      tools:
        default:
          mem: 6
          env:
            OOM_TOOL_MEMORY: "{mem * int(job.destination_params.get('SCALING_FACTOR', 1)) if job.destination_params else 1}"
          params:
            SCALING_FACTOR: "{2 * int(job.destination_params.get('SCALING_FACTOR', 2)) if job.destination_params else 2}"
        resubmit:
          with_more_mem_on_failure:
            condition: memory_limit_reached and attempt <= 3
            destination: tpv_dispatcher

following I bit one of the test cases here.

on top of the existing one by default on the Helm chart:

  job_conf.yml:
    runners:
      k8s:
        k8s_extra_job_envs:
          HDF5_USE_FILE_LOCKING: 'FALSE'
    execution:
      environments:
        tpv_dispatcher:
          tpv_config_files:
            - https://raw.githubusercontent.com/galaxyproject/tpv-shared-database/main/tools.yml
            - lib/galaxy/jobs/rules/tpv_rules_local.yml
            - /galaxy/server/config/persist-seq-extra-tpv.yaml

on run I see the following DEBUG outputs from tpv:

tpv.core.entities DEBUG 2022-12-13 12:56:45,941 [pN:job_handler_0,p:8,tN:JobHandlerQueue.monitor_thread] Ranking destinations: [<class 'tpv.core.entities.Destination'> id=k8s, cores=None, mem=None, gpus=None, env=None, params={'limits_cpu': '{cores}', 'limits_memory': '{mem}Gi', 'requests_cpu': '{cores}', 'requests_memory': '{mem}Gi'}, resubmit=None, tags=<class 'tpv.core.entities.TagSetManager'> tags=[<Tag: name=scheduling, value=docker, type=TagType.ACCEPT>], rank=, inherits=None, context=None, rules={}] for entity: <class 'tpv.core.entities.Tool'> id=default, cores=1, mem=6, gpus=None, env={'OOM_TOOL_MEMORY': '1'}, params={'container_monitor': False, 'docker_default_container_id': 'quay.io/galaxyproject/galaxy-min:22.05', 'docker_enabled': 'true', 'tmp_dir': 'true', 'SCALING_FACTOR': '2'}, resubmit={}, tags=<class 'tpv.core.entities.TagSetManager'> tags=[<Tag: name=scheduling, value=local, type=TagType.REJECT>, <Tag: name=scheduling, value=offline, type=TagType.REJECT>], rank=helpers.we, inherits=None, context={}, rules={} using custom function

which looks fine I guess besides the fact that resubmit is empty or "{}" depending on where you look. But then I see a more worrying:

galaxy.security.object_wrapper WARNING 2022-12-13 12:56:46,202 [pN:job_handler_0,p:8,tN:KubernetesRunner.work_thread-0] Unable to create dynamic subclass SafeStringWrapper(galaxy.model.none_like.None:<class 'NoneType'>,<class 'NotImplementedType'>,<class 'bool'>,<class 'bytearray'>,<class 'ellipsis'>,<class 'galaxy.security.object_wrapper.SafeStringWrapper'>,<class 'galaxy.tools.wrappers.ToolParameterValueWrapper'>,<class 'numbers.Number'>) for <class 'galaxy.model.none_like.NoneDataset'>, None: type() doesn't support MRO entry resolution; use types.new_class()

which I suspect is related. On Galaxy, the jobs shows the OOM message on the UI ("Tool failed due to insufficient memory. Try with more memory."). Any idea of what might be going wrong? Is the DEBUG showing what one would expect? Thanks.

I also tried the setup in the readthedocs docs, it didn't work for me either, will post the results here as well.

@pcm32
Copy link
Member Author

pcm32 commented Dec 13, 2022

When I stick to the example in readthedocs (maybe just changing some of the default integers):

  /galaxy/server/config/persist-seq-extra-tpv.yaml:
    applyToJob: true
    applyToWeb: true
    applyToSetupJob: false
    applyToWorkflow: true
    applyToNginx: false
    tpl: false
    content: |
      global:
        default_inherits: default
      tools:
        default:
          mem: 5 * int(job.destination_params.get('SCALING_FACTOR', 1)) if job.destination_params else 5
          params:
            SCALING_FACTOR: "{2 * int(job.destination_params.get('SCALING_FACTOR', 2)) if job.destination_params else 2}"
        resubmit:
          with_more_mem_on_failure:
            condition: memory_limit_reached and attempt <= 3
            destination: tpv_dispatcher

I get in the logs more or less the same messages with slight variations

tpv.core.entities DEBUG 2022-12-13 13:21:32,151 [pN:job_handler_0,p:8,tN:JobHandlerQueue.monitor_thread] Ranking destinations: [<class 'tpv.core.entities.Destination'> id=k8s, cores=None, mem=None, gpus=None, env=None, params={'limits_cpu': '{cores}', 'limits_memory': '{mem}Gi', 'requests_cpu': '{cores}', 'requests_memory': '{mem}Gi'}, resubmit=None, tags=<class 'tpv.core.entities.TagSetManager'> tags=[<Tag: name=scheduling, value=docker, type=TagType.ACCEPT>], rank=, inherits=None, context=None, rules={}] for entity: <class 'tpv.core.entities.Tool'> id=default, cores=1, mem=5, gpus=None, env={}, params={'container_monitor': False, 'docker_default_container_id': 'quay.io/galaxyproject/galaxy-min:22.05', 'docker_enabled': 'true', 'tmp_dir': 'true', 'SCALING_FACTOR': '2'}, resubmit={}, tags=<class 'tpv.core.entities.TagSetManager'> tags=[<Tag: name=scheduling, value=local, type=TagType.REJECT>, <Tag: name=scheduling, value=offline, type=TagType.REJECT>], rank=helpers.we, inherits=None, context={}, rules={} using custom function

again with resubmit either None or {} depending on where you look, and

galaxy.security.object_wrapper WARNING 2022-12-13 13:21:32,445 [pN:job_handler_0,p:8,tN:KubernetesRunner.work_thread-1] Unable to create dynamic subclass SafeStringWrapper(galaxy.model.none_like.None:<class 'NoneType'>,<class 'NotImplementedType'>,<class 'bool'>,<class 'bytearray'>,<class 'ellipsis'>,<class 'galaxy.security.object_wrapper.SafeStringWrapper'>,<class 'galaxy.tools.wrappers.ToolParameterValueWrapper'>,<class 'numbers.Number'>) for <class 'galaxy.model.none_like.NoneDataset'>, None: type() doesn't support MRO entry resolution; use types.new_class()

@pcm32
Copy link
Member Author

pcm32 commented Dec 13, 2022

After hammering this a bit more I managed to get the resubmission working using:

      global:
        default_inherits: default
      tools:
        default:
          mem: 5 * int(job.destination_params.get('SCALING_FACTOR', 1)) if job.destination_params else 5
          params:
            SCALING_FACTOR: "{2 * int(job.destination_params.get('SCALING_FACTOR', 2)) if job.destination_params else 2}"
          rules: []
          resubmit:
            with_more_mem_on_failure:
              condition: memory_limit_reached and attempt <= 3
              destination: tpv_dispatcher
            on_failure:
              condition: any_failure and attempt <= 2
              destination: tpv_dispatcher

filled the resubmit in the logs:

tpv.core.entities DEBUG 2022-12-13 13:55:25,490 [pN:job_handler_0,p:8,tN:JobHandlerQueue.monitor_thread] Ranking destinations: ...

resubmit={'with_more_mem_on_failure': {'condition': 'memory_limit_reached and attempt <= 3', 'destination': 'tpv_dispatcher'}, 'on_failure': {'condition': 'any_failure and attempt <= 2', 'destination': 'tpv_dispatcher'}}, tags=<class 'tpv.core.entities.TagSetManager'> 

Might be good to update the readthedocs slightly to reflect this difference.

@pcm32
Copy link
Member Author

pcm32 commented Dec 13, 2022

Ok, this is resubmitting, but only once....

@pcm32
Copy link
Member Author

pcm32 commented Dec 13, 2022

I remember someone once having issues when trying to do multiple resubmissions to the same destination all the time. Does this work for you guys? In the past all my resubmissions would go from destination A to destination B to destination C, never to the same one again... I think.

@pcm32
Copy link
Member Author

pcm32 commented Dec 13, 2022

It seems to be that this is more of a galaxy problem than TVP. After the first resubmission, the resubmit_definition is empty: https://github.com/galaxyproject/galaxy/blob/release_22.05/lib/galaxy/jobs/runners/state_handlers/resubmit.py#L22

@pcm32
Copy link
Member Author

pcm32 commented Dec 13, 2022

So, on the first failure, the k8s destination does have a job_state.job_destination.get("resubmit"), however, on the second go, the same destination no longer has that resubmit destination filled. I suspect this must be happening at the same resubmit.py code....

@nuwang
Copy link
Member

nuwang commented Dec 13, 2022

I'm trying to get the resubmit integration tests working again: https://github.com/galaxyproject/total-perspective-vortex/blob/main/tests/test_mapper_resubmit.py, and see whether we can test this exact scenario. However, it sure does look like this line in the block you highlighted could cause issues in particular https://github.com/galaxyproject/galaxy/blob/e67cb9b615d3f373fdf3d8534a6e5208f20e94b9/lib/galaxy/jobs/runners/state_handlers/resubmit.py#L87-L89

If I understood that comment correctly, it won't re-evaluate the dynamic code, and instead uses the cached destination, which would explain this behaviour.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants