Spec3 runtime for NIRSpec MSA

[stscijgbot-jp]

Issue JP-3849 was created on JIRA by David Law:

As part of testing https://jira.stsci.edu/browse/JP-3762 I tried processing a subset of data from PID 3215 Obs 901 through calwebb_spec3 and was surprised just how long this step took.  Nearly 15 minutes from starting the step to get past the 'Starting calwebb_spec3' step, and then roughly 1 min per source thereafter for what looks like about 600 sources (so estimate about 10 hours).

Filing this ticket to flag this case for attention in future runtime and efficiency improvements.

Also tagging Christian Hayes ; does this seem about normal for MSA data in your experience?

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

Odd about taking 15 minutes to get past the starting calwebb_spec3, I'll try running this and see if there's an obvious reason why (perhaps because many files with lots of extensions for each MOS slitlet are being opened?).  

I see that there are some 24-30 exposures in the association files that are being combined (depending on the filter/grating).  That seems a bit of an extreme case for a single observation, but Dan Coe might be able to say more about how typical the configurations are (often I'm working with single MOS configurations using a 3-pt or 5-pt dither pattern, so the asn's only have 6-10 exposures).

I know that one of the MOS bottlenecks in spec2/3 is the resampling and if you are trying to resample 24-30 spectra together during both outlier detection (and blotting that back 24-30 times) and resample_spec, I'm not surprised that the runtimes are long.  

That being said, the MOS runtimes are a bit on the long side (I think it might be our tallest runtime pole now that BOTS and IFU have seen some significant speed ups), so if there are ways to speed this up that would be great.

[stscijgbot-jp]

Comment by David Law on JIRA:

Just to clarify, in this test I restricted the association file to just the first 3 exposures (times 2 detectors, for 6 cal files total).  asn attached, called via strun calwebb_spec3 on the association direct from command line.