[PIPE2D-1189] fitFluxCal fails on making flux table Created: 20/Mar/23  Updated: 21/Mar/23  Resolved: 21/Mar/23

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: Takuji Yamashita Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description    
$ fitFluxCal.py /work/drp --calib=/work/drp/CALIB --rerun=test01 --no-versions --clobber-config --id visit=82905 'spectrograph=1^3'
[...]
fitFluxCal FATAL: Failed on dataId={'visit': 82905, 'spectrograph': 3, 'dateObs': '2022-11-18'}: KeyError: 'NO_DATA'
Traceback (most recent call last):
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 436, in __call__
    result = self.runTask(task, dataRef, kwargs)
  File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 504, in runTask
    return task.runDataRef(dataRef, **kwargs)
  File "/work/takuji/pipeline/drp_stella/python/pfs/drp/stella/fitFluxCal.py", line 206, in runDataRef
    outputs = self.run(pfsMerged, pfsMergedLsf, references, pfsConfig, pfsArmList, sky1dList)
  File "/work/takuji/pipeline/drp_stella/python/pfs/drp/stella/fitFluxCal.py", line 130, in run
    extracted.fluxTable = self.fluxTable.run(
  File "/work/takuji/pipeline/drp_stella/python/pfs/drp/stella/FluxTableTask.py", line 37, in run
    return makeFluxTable(identities, spectra, self.config.ignoreFlags,
  File "/work/takuji/pipeline/drp_stella/python/pfs/drp/stella/makeFluxTable.py", line 161, in makeFluxTable
    maskVal = flags.get("NO_DATA", *ignoreFlags)
  File "/work/takuji/pipeline/datamodel/python/pfs/datamodel/masks.py", line 47, in get
    return sum(self[name] for name in args)
  File "/work/takuji/pipeline/datamodel/python/pfs/datamodel/masks.py", line 47, in <genexpr>
    return sum(self[name] for name in args)
  File "/work/takuji/pipeline/datamodel/python/pfs/datamodel/masks.py", line 35, in __getitem__
    return 2**self.flags[name]
KeyError: 'NO_DATA'

This looks like a bug from a corner case: it's iterating over a fiber that doesn't have any data associated with it.

 Comments   
Comment by price [ 21/Mar/23 ]

I've fixed the error by reducing the selection of fibers to those that are available, but the root of the problem is the use of spectrograph=1^3 in the --id specification. Since the input to the task is a visit, this gets parsed as one visit with spectrograph=1 and one visit with spectrograph=3. There is a warning in the log:

root WARN: Unexpected ID spectrograph; guessing type is "str"

That's not exactly clear, and this behaviour is not really what we want, but since we're going to be dumping the Gen2 middleware soon-ish, I'm not going to go to the trouble of discovering what the proper fix should be. Users should take note that sub-components of a visit should not be specified for fitFluxCal.py.

Comment by price [ 21/Mar/23 ]

Merged.

Generated at Sat Feb 10 16:04:01 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.