[PIPE2D-945] mergeArms fails on arc Created: 12/Nov/21  Updated: 12/Nov/21  Resolved: 12/Nov/21

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Reviewers: hassan

 Description   

ncaplar reports the following failure when running mergeArms.py on visit=67720, an arc:

(lsst-scipipe) [ncaplar@tiger2-sumire ~]$ mergeArms.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun ncaplar/PIPE2D-918_ApCorFalse_Nov11 --id visit=67720 -j 12
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/Subaru/registry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
root INFO: Running: /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45a/bin/mergeArms.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun ncaplar/PIPE2D-918_ApCorFalse_Nov11 --id visit=67720 -j 12
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/Subaru/registry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
mergeArms WARN: Set of fiberIds of LSFs != fiberIds for spectra:  Only in fiberId: {1, 515, 229, 137, 426, 651, 45, 336, 273, 560, 316, 471, 184, 92, 382, 607} (fixing)
mergeArms WARN: Set of fiberIds of LSFs != fiberIds for spectra:  Only in fiberId: {1, 515, 229, 137, 426, 651, 45, 336, 273, 560, 316, 471, 184, 92, 382, 607} (fixing)
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
    result = self.runTask(task, dataRef, kwargs)
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
    return task.runDataRef(dataRef, **kwargs)
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45a/python/pfs/drp/stella/mergeArms.py", line 186, in runDataRef
    results = self.run(spectra, pfsConfig, lsfList)
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45a/python/pfs/drp/stella/mergeArms.py", line 138, in run
    sky1d.append(self.skySubtraction(ss, pfsConfig))
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45a/python/pfs/drp/stella/mergeArms.py", line 409, in skySubtraction
    sky1d = self.fitSkyModel.run(skySpectra, skyConfig)
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45a/python/pfs/drp/stella/fitFocalPlane.py", line 96, in run
    **fitParams)
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45a/python/pfs/drp/stella/focalPlaneFunction.py", line 67, in fit
    raise RuntimeError("fiberId mismatch between spectra and pfsConfig")
RuntimeError: fiberId mismatch between spectra and pfsConfig

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 400, in __call__
    ", ".join(str(ref.dataId) for ref in dataRef), eName, e)
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 400, in <genexpr>
    ", ".join(str(ref.dataId) for ref in dataRef), eName, e)
AttributeError: 'list' object has no attribute 'dataId'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45a/bin/mergeArms.py", line 3, in <module>
    MergeArmsTask.parseAndRun()
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 603, in parseAndRun
    resultList = taskRunner.run(parsedCmd)
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 221, in run
    resultList = list(mapFunc(self, targetList))
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 45, in _runPool
    return pool.map_async(function, iterable).get(timeout)
  File "/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AttributeError: 'list' object has no attribute 'dataId'

The code is attempting to do sky subtraction, but there are no sky fibers. Fix the obscure error message.

 Comments   
Comment by price [ 12/Nov/21 ]

Now there's a better error message:

$ mergeArms.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun price/test --id visit=67720 --no-versions
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/Subaru/registry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
root INFO: Running: /home/pprice/pfs/drp_stella/bin/mergeArms.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun price/test --id visit=67720 --no-versions
mergeArms WARN: Set of fiberIds of LSFs != fiberIds for spectra:  Only in fiberId: {1, 515, 229, 137, 426, 651, 45, 336, 273, 560, 316, 471, 184, 92, 382, 607} (fixing)
mergeArms WARN: Set of fiberIds of LSFs != fiberIds for spectra:  Only in fiberId: {1, 515, 229, 137, 426, 651, 45, 336, 273, 560, 316, 471, 184, 92, 382, 607} (fixing)
Traceback (most recent call last):
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
    result = self.runTask(task, dataRef, kwargs)
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
    return task.runDataRef(dataRef, **kwargs)
  File "/home/pprice/pfs/drp_stella/python/pfs/drp/stella/mergeArms.py", line 186, in runDataRef
    results = self.run(spectra, pfsConfig, lsfList)
  File "/home/pprice/pfs/drp_stella/python/pfs/drp/stella/mergeArms.py", line 138, in run
    sky1d.append(self.skySubtraction(ss, pfsConfig))
  File "/home/pprice/pfs/drp_stella/python/pfs/drp/stella/mergeArms.py", line 409, in skySubtraction
    raise RuntimeError("No sky spectra to use for sky subtraction")
RuntimeError: No sky spectra to use for sky subtraction

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pprice/pfs/drp_stella/bin/mergeArms.py", line 3, in <module>
    MergeArmsTask.parseAndRun()
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 603, in parseAndRun
    resultList = taskRunner.run(parsedCmd)
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 221, in run
    resultList = list(mapFunc(self, targetList))
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 400, in __call__
    ", ".join(str(ref.dataId) for ref in dataRef), eName, e)
  File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 400, in <genexpr>
    ", ".join(str(ref.dataId) for ref in dataRef), eName, e)
AttributeError: 'list' object has no attribute 'dataId'
Comment by hassan [ 12/Nov/21 ]

No problems with suggested changes.

Comment by price [ 12/Nov/21 ]

Merged.

Generated at Sat Feb 10 16:00:11 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.