Add MA table reference file

[tddesjardins]

To better keep in sync the versions of the multi-accumulation (MA) tables used by various simulation tools at the SOC as well as by the romancal pipeline, we request a new type of reference file to capture this information. I welcome comments on this issue for how best to format the file, but I'm going to sketch out an initial suggestion here. Note that ref_common includes ROMAN.META.INSTRUMENT.DETECTOR, but this would apply to all detectors...I guess we can just always put WFI01? We're not going to match on it, so we don't need to put a p_ keyword for this, just a placeholder.

CRDS Selectors:

• ROMAN.META.EXPOSURE.START_TIME

Schema:

title: WFI Multi-Accumulation (MA) Tables

type: object
properties:
     meta:
          allOf:
               - $ref_common
               - type: object
                     properties:
                          type: string
                         enum: [MATAB]
     ma_table:
          title: Multi-Accumulation (MA) Table Readout Patterns
          description: Readout specifications of the multi-accumulation (MA) tables.
          type: object
          patternProperties:
               ??? - Need to put the MA table ID numbers here...can we leave this general to any integer 1 – 99?
               type: object
               properties:
                    read_pattern:
                         title: Readout Pattern
                         description: The enumeration of detector reads to resultants making up the Level 1 ramp data cube. The read pattern is nested such that each grouping of read numbers represents a resultant. For example, a readout pattern of [[1], [2, 3], [4]] consists of four reads that were downlinked as three resultants with both the first and last resultant each containing a single read.
                         type: array
                         items:
                             type: array
                             items:
                                 type: integer
     timing:
          title: Detector Readout Time (s)
         description: The readout time per observing mode in seconds of the Wide Field Instrument (WFI) detectors. This represents the time between the start and end of the readout. The difference in readout times is governed by the guide window mode in use, which is tied to the observing mode.
         patternProperties: 
                "^(WFI_IMAGE|WFI_GRISM|WFI_PRISM|WFI_WFSC)$":
                       frame_time:
                            title: Detector Readout Time (s)
                            description: The readout time in seconds of the Wide Field Instrument (WFI) detectors. This represents the time between the start and end of the readout.

Questions:

1. What does exposure type WFI_WFSC represent? There are differences in the frame times between WFI imaging, WFI spectroscopy, large defocus, and moderate defocus modes. Expect these to change, but in last PRD this was 3.04, 4.03, 4.41, and 6.40 seconds, respectively). Does WFI_WFSC correspond to one of the defocus modes? Both? We may need to think about this...

[tddesjardins]

cc from RTB @rcosenti-stsci and @wcschultz in addition to the usual RCAL folks @schlafly , @PaulHuwe , @ddavis-stsci , @nden .

[stscijgbot-rstdms]

This issue is tracked on JIRA as RAD-180.

[schlafly]

I guess the other option would be to add a new possible WFI detector, ALL. But I agree that doing a nominal WFI01 is easier.

I haven't thought hard enough about the timings. Do the grism / prism / imaging modes use the same MA tables, albeit with different SCA readout parameters (and therefore frame times)? It seems to me like it must be possible mechanically to observe with an imaging filter but a spectroscopic frame time (i.e., imaging filters with spectroscopic guide windows)? I don't think this is a good idea, but I'm trying to understand what connections are guaranteed and what connections are conventions.

Can we add something about "for a single read of the array" to the description of "frame_time"? For me "start and end of a readout" isn't immediately obviously "one readout" vs. "all of the readouts in the exposure."

[wcschultz]

@schlafly The MA tables inherently specify the observing mode so if you want the same readout pattern for an image and a spectroscopic image, you would need two different MA tables. Theoretically this does not need to be the case, but in practice it is. I agree with your other comments.

I am a little concerned that the frame times might change over the nominal mission while the MA tables will not. In the current schema, we cannot change the frame times to be MA table specific. Could we just add the read and reset time for each table to the first dictionary alongside the read pattern? We could also add the observing mode if that is also needed. That would at least safeguard for possible future changes, even if it does increase the file size a little.

Also I thought the MA table ID's need to be between 1-9999 as we increased the maximum to be sure we would never run out of unique IDs. This is not super important, I just wanted to clarify.

[tddesjardins]

@wcschultz beat me to the explanation. :) Everything he said is correct about how the MA tables operate. Practically you will not be able to combine an imaging mode table with spectroscopic guide windows. Those two things are tied together in PSS.

For which detector to use, we can use the p_ keyword. It's how we do this sort of thing in CRDS to ensure any match to any detector works (it's how we did the F158 flat applying to all filters as a placeholder, for example).

For the question of the description, I think it's actually more accurate to say that frame_time is the time between the start of one readout of the WFI detectors and the start of the next readout. We can finesse that, but that's what it means.

Re: @wcschultz 's point about evolution, that's a good idea. I'd preferred to not have to replicate that information a bunch, but it could change and older observations could have different frame_time values, so I think we can make that part of each table definition.

And 1-9999 is fine, I just was making a note to myself that that needs to be an incrementing field in the schema.

[schlafly]

Okay. If there is a 1:1 mapping between MA tables and frame times, then I agree with Will that it is more natural to move frame_time under ma_table parallel to read_pattern.

For the question of the description, I think it's actually more accurate to say that frame_time is the time between the start of one readout of the WFI detectors and the start of the next readout. We can finesse that, but that's what it means.

The distinction I want to make clear is that a "readout" here is for one frame. The device is continually reading out for the whole exposure, etc., so there's another notion of readout time that is more like the total exposure time---that's how long the device is reading out. I know you know this, and maybe I'm being dumb coming from CCD-land, but my two cents are that it's beneficial to have some language that "start and end of a readout" is for a single non-destructive read of the whole array.

[tddesjardins]

Not being dumb, just being precise and clear. Feedback received! So if we're ~happy with this kind of setup, then @wcschultz and @rcosenti-stsci can finesse this a bit and come back with something implementable?

[wcschultz]

Sure! @rcosenti-stsci and I can coordinate this later this week and I can try to have an example by next Thursday if that is a reasonable timeline for everyone.

[rcosenti-stsci]

This all sounds good to me. We can be explicit in our definitions and change as needed for clarity.