[PIPE2D-653] Fix failing weekly Created: 17/Nov/20  Updated: 05/Jan/21  Resolved: 08/Dec/20

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: 2DDRP-2021 A
Reviewers: hassan

 Description   

After re-implementing the weekly after PIPE2D-641, I found that the weekly is failing:

17286 INFO  2020-11-16T16:17:29.137-0500 fiberFlat: Dither values: [-4.0, -2.0, 0.0, 2.0, 4.0]
17287 INFO  2020-11-16T16:17:29.137-0500 fiberFlat: Dither values: [-4.0, -2.0, 0.0, 2.0, 4.0]
17285 INFO  2020-11-16T16:17:29.137-0500 fiberFlat: Dither values: [-4.0, -2.0, 0.0, 2.0, 4.0]
17287 INFO  2020-11-16T16:17:39.310-0500 fiberFlat: Combined 3 images for dither 2.0
17286 INFO  2020-11-16T16:17:39.477-0500 fiberFlat: Combined 3 images for dither -4.0
17285 INFO  2020-11-16T16:17:39.486-0500 fiberFlat: Combined 3 images for dither 0.0
RuntimeError on tiger2-sumire.princeton.edu:17287 in run: Matched fiber to a used fiberId
Traceback (most recent call last):
  File "/scratch/pprice/jenkins/weekly/2020-11-16/build/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 113, in wrapper
    return func(*args, **kwargs)
  File "/scratch/pprice/jenkins/weekly/2020-11-16/build/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 1070, in run
    while not menu[command]():
  File "/scratch/pprice/jenkins/weekly/2020-11-16/build/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 240, in wrapper
    return func(*args, **kwargs)
  File "/scratch/pprice/jenkins/weekly/2020-11-16/build/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 1088, in reduce
    result = self._processQueue(context, func, [(index, data)], *args, **kwargs)[0]
  File "/scratch/pprice/jenkins/weekly/2020-11-16/build/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 547, in _processQueue
    return self._reduceQueue(context, None, func, queue, *args, **kwargs)
  File "/scratch/pprice/jenkins/weekly/2020-11-16/build/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 573, in _reduceQueue
    resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
  File "/scratch/pprice/jenkins/weekly/2020-11-16/build/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/18.1.0/python/lsst/ctrl/pool/pool.py", line 573, in <listcomp>
    resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
  File "/scratch/pprice/jenkins/weekly/2020-11-16/build/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2020.46/python/pfs/drp/stella/constructFiberFlatTask.py", line 103, in combine
    profileData = self.profiles.run(afwImage.makeExposure(image), detMap)
  File "/scratch/pprice/jenkins/weekly/2020-11-16/build/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2020.46/python/pfs/drp/stella/buildFiberProfiles.py", line 102, in run
    identifications = self.identifyFibers(profiles, centers, detectorMap, pfsConfig)
  File "/scratch/pprice/jenkins/weekly/2020-11-16/build/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2020.46/python/pfs/drp/stella/buildFiberProfiles.py", line 466, in identifyFibers
    raise RuntimeError("Matched fiber to a used fiberId")
RuntimeError: Matched fiber to a used fiberId
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3

To reproduce:

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-641 $ $PFS_PIPE2D_DIR/weekly/process_weekly.sh -d /projects/HSC/PFS/weekly -r pipe2d-641 -c 10 /scratch/pprice/pipe2d-641
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-641 $ constructFiberFlat.py /scratch/pprice/pipe2d-641 --calib=/scratch/pprice/pipe2d-641/CALIB --rerun=pipe2d-641/calib/brn/flat --doraise --batch-type=smp --cores=10 --id field=FLAT 'arm=b^r^n'


 Comments   
Comment by price [ 04/Dec/20 ]

This needed more fixes than just constructFiberFlat.py, due to other features that have landed while the weekly was broken, but it works now.

Comment by price [ 08/Dec/20 ]

Merged to master. I'll now try a weekly run through Jenkins...

Comment by price [ 08/Dec/20 ]

The weekly is working now.

Generated at Sat Feb 10 15:56:04 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.