[Bug] Duplicate tasks in experiment description file

[k1o0]

There are potentially >100 sessions where the same task has been erroneously extracted as multiple chained protocols. This appears to happen because the experiment description file contains duplicate tasks. For example the session 13565016-ac41-4b32-b71d-474fd60a5052 KM_023/2024-08-21/001 contains the following experiment description:

{'devices': {'cameras': {'left': {'collection': 'raw_video_data',
    'sync_label': 'audio'}},
  'microphone': {'microphone': {'collection': 'raw_task_data_00',
    'sync_label': 'audio'}}},
 'procedures': ['Behavior training/tasks'],
 'projects': ['u19_proj1_multiareacom'],
 'sync': {'bpod': {'acquisition_software': 'pybpod',
   'collection': 'raw_task_data_00',
   'extension': '.jsonable'}},
 'tasks': [{'_iblrig_tasks_biasedChoiceWorld': {'collection': 'raw_task_data_00'}},
  {'_iblrig_tasks_biasedChoiceWorld': {'collection': 'raw_task_data_00'}},
  {'_iblrig_tasks_biasedChoiceWorld': {'collection': 'raw_task_data_00'}},
  {'_iblrig_tasks_biasedChoiceWorld': {'collection': 'raw_task_data_00'}},
  {'_iblrig_tasks_biasedChoiceWorld': {'collection': 'raw_task_data_00'}}],
 'version': '1.0.0'}

The following Django query uncovers 118 potential sessions with this same issue:

from actions.models import Session
from django.db.models import Q, Count, F
n_raw_task_files = Count('data_dataset_session_related', distinct=True, filter=Q(data_dataset_session_related__name='_iblrig_taskData.raw.jsonable'))
n_task_tables = Count('data_dataset_session_related', distinct=True, filter=Q(data_dataset_session_related__name='_ibl_trials.table.pqt'))
ses = (Session
       .objects
       .prefetch_related('data_dataset_session_related')
       .annotate(n_raw_task_files=n_raw_task_files)
       .annotate(n_task_tables=n_task_tables)
       .filter(n_raw_task_files__lt=F('n_task_tables')))

This issue may be related to the aggregation of experiment description stubs, especially if behaviour data is somehow copied more than once.

1. This copy and merge of a sub should only be happening once per session (for a given acquisition PC)
2. Given that the collection is always provided, the tasks key should never contain duplicates.

[oliche]

For the UCL in London, we have 2 instances of duplicate tasks in the description file:

{'id': 'd0e1b460-9439-4aa5-8cf5-bd8f127a6154', 'subject': 'CQ001', 'start_time': '2024-10-07T15:24:19.975000', 'number': 1, 'lab': 'cortexlab', 'projects': ['ibl_fibrephotometry'], 'url': 'https://alyx.internationalbrainlab.org/sessions/d0e1b460-9439-4aa5-8cf5-bd8f127a6154', 'task_protocol': '_iblrig_tasks_trainingChoiceWorld8.19.6'}
(S3) /mnt/s1/spikesorting/raw_data/cortexlab/Subjects/CQ001/2024-10-07/001/_ibl_experiment.description.yaml: 100%|██████████| 581/581 [00:00<00:00, 1.83kB/s]
Multiple tasks: [{'_iblrig_tasks_trainingChoiceWorld': {'collection': 'raw_task_data_00', 'sync_label': 'bpod'}}, {'_iblrig_tasks_trainingChoiceWorld': {'collection': 'raw_task_data_00', 'sync_label': 'bpod'}}]

{'id': 'd3ebb4bb-2790-421d-bd71-1a062f2c4a6e', 'subject': 'CQ001', 'start_time': '2024-11-01T14:34:40.200000', 'number': 1, 'lab': 'cortexlab', 'projects': ['ibl_fibrephotometry'], 'url': 'https://alyx.internationalbrainlab.org/sessions/d3ebb4bb-2790-421d-bd71-1a062f2c4a6e', 'task_protocol': '_iblrig_tasks_trainingChoiceWorld8.24.7'}
(S3) /mnt/s1/spikesorting/raw_data/cortexlab/Subjects/CQ001/2024-11-01/001/_ibl_experiment.description.yaml: 100%|██████████| 537/537 [00:00<00:00, 1.60kB/s]
Multiple tasks: [{'_iblrig_tasks_trainingChoiceWorld': {'collection': 'raw_task_data_00'}}, {'_iblrig_tasks_trainingChoiceWorld': {'collection': 'raw_task_data_00'}}]

Many other similar experiments over one month do not exhibit the behaviour, so it is not a systematic bug

[mayofaulkner]

• Patch duplicate description files on flatiron
• Patch duplicate description files on local servers
• Remove excess task protocols from sign off checklist (ibl and cortexlab alyx)
• Remove excess task protocols from session extended qc (ibl and cortexlab alyx)
• Patch and reregister description files on cortexlab server
• Fix cortexlab alyx import fix import of datasets from cortex lab cortex-lab/alyx#899
• Check files have been updated