Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ALMA: Unique ASDMs associated with a MOUS #3148

Open
keflavich opened this issue Dec 11, 2024 · 8 comments
Open

ALMA: Unique ASDMs associated with a MOUS #3148

keflavich opened this issue Dec 11, 2024 · 8 comments
Labels

Comments

@keflavich
Copy link
Contributor

@andamian How can we get the list of ASDMs associated with a given MOUS? The online archive shows it:

image image

but I can't figure out how to get at the ASDM list, even using query_tap:

rslt = Alma.query_tap("select * from ivoa.obscore where member_ous_uid like 'uid://A001/X3577/X6c5'")
rslt.table['asdm_uid']
WARNING: AstropyDeprecationWarning: Using the table property is deprecated. Please use the to_table() instead. [pyvo.dal.query]
Out[102]:
<MaskedColumn name='asdm_uid' dtype='str32' description='UID of the ASDM containing this Field.' length=12>
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c
uid://A002/X1044380/X10d6c

(there should be 10 unique ASDM UIDs)

The use case here is that I want to determine how many executions were performed, how long they were, and the weather conditions (PWV) for each. I'm using this to assemble the metadata for a table publication. This can clearly be done with the online archive, but it's at best non-obvious and at worst not possible using TAP queries.

@keflavich keflavich added the alma label Dec 11, 2024
@keflavich
Copy link
Contributor Author

The datalink query has the information, but not in a usable way:

In [104]: Alma.get_data_info('uid://A001/X3577/X6c5')
Out[104]:
<Table length=15>
          ID                                                    access_url                                                                   service_def                          error_message ... content_qualifier link_auth link_authorized
                                                                                                                                                                                                ...
        object                                                    object                                                                        object                                object    ...       object        object        bool
--------------------- ---------------------------------------------------------------------------------------------- ------------------------------------------------------------ ------------- ... ----------------- --------- ---------------
uid://A001/X3577/X6c5                https://almascience.nrao.edu/dataPortal/member.uid___A001_X3577_X6c5.README.txt                                                                            ...                       false            True
uid://A001/X3577/X6c5    https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A001_X3577_X6c5_001_of_001.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5                                                                                                DataLink.2021.1.00172.L_uid___A001_X3577_X6c5_001_of_001.tar               ...                                          --
uid://A001/X3577/X6c5     https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A001_X3577_X6c5_auxiliary.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5                                                                                                 DataLink.2021.1.00172.L_uid___A001_X3577_X6c5_auxiliary.tar               ...                                          --
uid://A001/X3577/X6c5 https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A002_X1044380_X10d6c.asdm.sdm.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5  https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A002_X1044380_Xe910.asdm.sdm.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5  https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A002_X1044380_Xf1ec.asdm.sdm.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5  https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A002_X1044380_Xfd16.asdm.sdm.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5  https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A002_X1048ed8_X40d6.asdm.sdm.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5  https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A002_X1048ed8_X4814.asdm.sdm.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5  https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A002_X1048ed8_X4fcf.asdm.sdm.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5  https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A002_X1048ed8_X5b0d.asdm.sdm.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5  https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A002_X1048ed8_X66c7.asdm.sdm.tar                                                                            ...                       false            True
uid://A001/X3577/X6c5  https://almascience.nrao.edu/dataPortal/2021.1.00172.L_uid___A002_X1048ed8_Xccce.asdm.sdm.tar                                                                            ...                       false            True

(i.e., the asdm names are all encoded into tarball names)

@keflavich
Copy link
Contributor Author

The data seem deeply inaccessible.

Alma.query_tap("select * from ivoa.obscore where asdm_uid = 'uid://A002/X1048ed8/X5b0d'")

gives zero hits, while

Alma.query_tap("select * from ivoa.obscore where asdm_uid = 'uid://A002/X1044380/X10d6c'")

gives 10. The latter corresponds to the first ASDM in the above list. There must be some other database or UID list that contains these...

@andamian
Copy link

Looking into it @keflavich

@keflavich
Copy link
Contributor Author

@andamian Is there any temporary workaround for this I could try?

@andamian
Copy link

Unfortunately the info doesn't seem to be available in the database that astroquery (through the TAP service) hits. I think that the Web site uses a different DB and we don't know much about it. Anyway, the issue is now with ALMA and I'll let you know when they get back to us.
The only workaround that I can think of is the one that you discovered yourself through (name of the files) but that's obviously not suitable for use in direct queries.

@keflavich
Copy link
Contributor Author

Alright, thanks. I may be back to reverse engineering the web interface, then.

@keflavich
Copy link
Contributor Author

Easy enough (...ish) to retrieve the ASDMs:

>>> resp = requests.post('https://almascience.nrao.edu/aq/service/api/search/observations/_search', data='{"_source":["asdms"],"query":{"terms":{"_id":["uid://A001/X3577/X6c5.source.Sgr_A_star_OFF_0"]}}}', headers={'Content-type':'application/json'})
>>> resp.json()
{'took': 1,
 'timed_out': False,
 '_shards': {'total': 5, 'successful': 5, 'skipped': 0, 'failed': 0},
 'hits': {'total': 1,
  'max_score': 1.0,
  'hits': [{'_index': 'observations',
    '_type': 'observation',
    '_id': 'uid://A001/X3577/X6c5.source.Sgr_A_star_OFF_0',
    '_score': 1.0,
    '_source': {'asdms': ['uid://A002/X1044380/X10d6c',
      'uid://A002/X1044380/Xe910',
      'uid://A002/X1044380/Xf1ec',
      'uid://A002/X1044380/Xfd16',
      'uid://A002/X1048ed8/X40d6',
      'uid://A002/X1048ed8/X4814',
      'uid://A002/X1048ed8/X4fcf',
      'uid://A002/X1048ed8/X5b0d',
      'uid://A002/X1048ed8/X66c7',
      'uid://A002/X1048ed8/Xccce']}}]}}

next question (for my own notes): how do I find any useful metadata about these ASDMs? Stay tuned

@keflavich
Copy link
Contributor Author

The metadata I want are available on SNOOPI, but I have to login for that. Hacky.

from astroquery.alma import Alma
Alma.login('keflavich')
self = Alma._auth
login_url = f'https://{self.get_valid_host()}{self._LOGIN_ENDPOINT}'

password=Alma._get_auth_info('keflavich')[1]
username='keflavich'
login_response = self._request('POST', login_url, data=data, cache=False)


data = {'username': username,      'password': password,    'grant_type': self._GRANT_TYPE,        'client_id': self._CLIENT_ID}

resp2 = requests.post('https://asa.alma.cl/snoopi/restapi/schedblock/uid:%7C%7CA001%7CX15b4%7CXc3?authormode=pi&', data={'authormode': 'pi'}, headers={'Authorization': f'Bearer {login_response.json()["access_token"]}'})
resp2.raise_for_status()

resp2.json()

result:

{'notes': '',
 'dec': '-29d  3m  39.7s',
 'offdec': '',
 'standardMode': 'true',
 'sensitivityUnit': 'mJy',
 'unitDependencyDelayUnit': 's',
 'frequency': '100.50343083087068',
 'offra': '',
 'expexecutions': '8',
 'band': 'ALMA_RB_03',
 'minAcceptableAngResolutionUnit': 'arcsec',
 'parentObsUnitSetStatusId': 'uid://A001/X15b4/Xc5',
 'singlePoints': True,
 'rectanglesCount': 0.0,
 'temporal': False,
 'ousState': 'Delivered',
 'qa0PassCount': 7,
 'isMosaic': 'true',
 'ra': '17h 45m  47.7s',
 'receiverBandName': 'ALMA_RB_03',
 'totaltime': 30182,
 'obsprojectuid': 'uid://A001/X1525/X290',
 'name': 'Sgr_A_st_i_updated_03_7M',
 'boolean_totalPowerRectangle': False,
 'rectangle': False,
 'sensitivity': '0.5877559508105298',
 'maxAcceptableAngResolution': '0.0',
 'sourcesCount': 1,
 'temporalConstraints': [],
 'frequencyUnit': 'GHz',
 'pointsCount': 89.0,
 'modeName': 'Standard Interferometry',
 'flag': None,
 'schedblockuid': 'uid://A001/X15b4/Xc3',
 'executionFraction': 8.641,
 'domainEntityState': 'ObservingTimedOut',
 'targetCount': 52,
 'nominalconfiguration': '7M',
 'array': 'SEVEN-M',
 'qa2status': 'SemiPass',
 'lines': 'H13CN/others\tH13CO/SiO\tHCO+_v=0_1-0\tHNCO_v=0_4(0,4)-3(0,3)\tCont1\tCont2',
 'isWeblogAvailable': True,
 'targetType': 'F_MultiplePoints',
 'sourceEphemeris': '',
 'unitDependencyExecCount': '1',
 'unitDependencyList': '',
 'number_pointings': 52.0,
 'executions': [{'date': '2022-08-08 05:18:31.485',
   'execblockuid': 'uid://A002/Xfc8dc8/X96ae',
   'hasObserveTargetIntent': 'true',
   'time': '72.50951666666667',
   'status': 'Pass'},
  {'date': '2022-08-08 04:05:56.718',
   'execblockuid': 'uid://A002/Xfc8dc8/X8c1e',
   'hasObserveTargetIntent': 'true',
   'time': '74.75793333333333',
   'status': 'Pass'},
  {'date': '2022-08-08 01:46:47.338',
   'execblockuid': 'uid://A002/Xfc8dc8/X885c',
   'hasObserveTargetIntent': 'true',
   'time': '24.5578',
   'status': 'Fail'},
  {'date': '2022-08-05 01:51:25.21',
   'execblockuid': 'uid://A002/Xfc69ac/X60f0',
   'hasObserveTargetIntent': 'true',
   'time': '74.87191666666666',
   'status': 'Pass'},
  {'date': '2022-08-04 02:41:08.277',
   'execblockuid': 'uid://A002/Xfc69ac/X281',
   'hasObserveTargetIntent': 'true',
   'time': '43.07449999999999',
   'status': 'SemiPass'},
  {'date': '2022-08-03 03:38:48.358',
   'execblockuid': 'uid://A002/Xfc5b59/X1921',
   'hasObserveTargetIntent': 'true',
   'time': '74.8749',
   'status': 'Pass'},
  {'date': '2022-08-01 05:18:54.294',
   'execblockuid': 'uid://A002/Xfc434c/Xfe0',
   'hasObserveTargetIntent': 'true',
   'time': '56.15788333333333',
   'status': 'Pass'},
  {'date': '2022-08-01 04:22:40.848',
   'execblockuid': 'uid://A002/Xfc434c/X39a',
   'hasObserveTargetIntent': 'true',
   'time': '74.4294',
   'status': 'Pass'},
  {'date': '2022-07-31 04:45:18.226',
   'execblockuid': 'uid://A002/Xfc3353/X25ce',
   'hasObserveTargetIntent': 'true',
   'time': '75.49146666666667',
   'status': 'Pass'},
  {'date': '2022-07-31 03:28:47.219',
   'execblockuid': 'uid://A002/Xfc3353/X25b5',
   'hasObserveTargetIntent': 'true',
   'time': '1.4182166666666667',
   'status': 'Fail'}],
 'scienceGoalName': 'SG OUS (cmz01)',
 'visit': False,
 'minAcceptableAngResolution': '0.0',
 'maxAcceptableAngResolutionUnit': 'arcsec',
 'unitDependencyDelay': '0.0'}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants