[PIPE2D-830] Fix breakage in mergeArms Created: 05/May/21  Updated: 07/May/21  Resolved: 07/May/21

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: 2DDRP-2021 A 4, 2DDRP-2021 A5
Reviewers: hassan

 Description   

rhl writes:

I'm having trouble reducing SuNSS data with master. I built new fiberTraces using constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS --id visit=46629..46649 --rerun rhl/sunss --batch-type none -c isr.doFlat=False isr.doDefect=False isr.doDark=False isr.doBrokenRedShutter=True isr.brokenRedShutter.useAnalytic=True isr.brokenRedShutter.checkParallelOverscan=True psfFwhm=3 forceFiberIds=True profiles.doBlindFind=False profiles.centroidRadius=3 profiles.profileRadius=3 rerunISR=False -j 20 (followed by ingest), but when I run reduceExposure and mergeArms I get this error. Looking at the pfsArm files in the r only shows around 5 spectra, possibly those which happen to have signal down the fibres.

 
$ reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS --rerun rhl/sunss -c doWriteCalexp=True isr.doFlat=False isr.doDefect=False isr.doDark=False repair.interp.modelPsf.defaultFwhm=2.25
$ mergeArms.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS --rerun rhl/sunss --id visit=46032 -c doBarycentricCorr=False doSubtractSky1d=False 
mergeArms WARN: Set of fiberIds of LSFs != fiberIds for spectra:  Only in fiberId: {512, 514, 3, 6, 518, 520, 9, 522, 12, 15, 527, 529, 18, 532, 21, 535, 24, 537, 27, 540, 30, 543, 32, 546, 35, 549, 38, 552, 41, 555, 46, 558, 49, 563, 52, 565, 55, 567, 58, 570, 572, 61, 574, 576, 66, 579, 69, 582, 72, 585, 75, 589, 78, 592, 81, 595, 84, 598, 87, 601, 90, 604, 95, 609, 98, 612, 101, 615, 104, 618, 107, 621, 110, 623, 113, 625, 627, 116, 630, 119, 632, 122, 634, 124, 636, 127, 639, 130, 642, 133, 645, 647, 649, 138, 141, 144, 147, 150, 153, 155, 156, 157, 158, 159, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 187, 189, 191, 193, 195, 197, 199, 201, 203, 205, 208, 210, 212, 215, 217, 219, 221, 223, 225, 227, 231, 233, 235, 238, 240, 242, 245, 247, 249, 251, 253, 257, 259, 261, 263, 266, 268, 270, 275, 278, 281, 284, 287, 290, 293, 296, 299, 302, 304, 306, 308, 311, 313, 315, 338, 341, 344, 347, 349, 351, 354, 357, 360, 363, 366, 369, 372, 375, 378, 381, 385, 387, 389, 391, 393, 395, 397, 399, 403, 406, 408, 410, 413, 415, 417, 419, 421, 423, 425, 429, 431, 433, 436, 438, 440, 443, 445, 447, 449, 451, 453, 455, 457, 459, 461, 466, 468, 474, 477, 479, 482, 485, 488, 491, 494, 497, 500, 502, 504, 506, 508, 510} (fixing) Only in armPsf: {192, 2, 339, 650}
mergeArms WARN: Set of fiberIds of LSFs != fiberIds for spectra:  Only in fiberId: {512, 514, 3, 6, 518, 520, 9, 522, 12, 15, 527, 529, 18, 532, 21, 535, 24, 537, 27, 540, 30, 543, 32, 546, 35, 549, 38, 552, 41, 555, 46, 558, 49, 563, 52, 565, 55, 567, 58, 570, 572, 61, 574, 576, 66, 579, 69, 582, 72, 585, 75, 589, 78, 592, 81, 595, 84, 598, 87, 601, 90, 604, 95, 609, 98, 612, 101, 615, 104, 618, 107, 621, 110, 623, 113, 625, 627, 116, 630, 119, 632, 122, 634, 124, 636, 127, 639, 130, 642, 133, 645, 647, 649, 138, 141, 144, 147, 150, 153, 155, 156, 157, 158, 159, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 187, 189, 191, 193, 195, 197, 199, 201, 203, 205, 208, 210, 212, 215, 217, 219, 221, 223, 225, 227, 231, 233, 235, 238, 240, 242, 245, 247, 249, 251, 253, 257, 259, 261, 263, 266, 268, 270, 275, 278, 281, 284, 287, 290, 293, 296, 299, 302, 304, 306, 308, 311, 313, 315, 338, 341, 344, 347, 349, 351, 354, 357, 360, 363, 366, 369, 372, 375, 378, 381, 385, 387, 389, 391, 393, 395, 397, 399, 403, 406, 408, 410, 413, 415, 417, 419, 421, 423, 425, 429, 431, 433, 436, 438, 440, 443, 445, 447, 449, 451, 453, 455, 457, 459, 461, 466, 468, 474, 477, 479, 482, 485, 488, 491, 494, 497, 500, 502, 504, 506, 508, 510} (fixing) Only in armPsf: {192, 2, 339, 650}
/home/rhl/PFS/drp/stella/python/pfs/drp/stella/datamodel/interpolate.py:61: RuntimeWarning: invalid value encountered in greater
  intIndex[(intIndex == index) & (index > 0)] -= 1  # Linear interpolation takes the index before
Traceback (most recent call last):
  File "/tigress/HSC/PFS/stack/current/stack/current/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
    result = self.runTask(task, dataRef, kwargs)
  File "/tigress/HSC/PFS/stack/current/stack/current/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
    return task.runDataRef(dataRef, **kwargs)
  File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/mergeArms.py", line 175, in runDataRef
    results = self.run(spectra, pfsConfig, lsfList)
  File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/mergeArms.py", line 125, in run
    lsfList = [self.mergeLsfs(ll, ss) for ll, ss in zip(lsfList, spectra)]
  File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/mergeArms.py", line 125, in <listcomp>
    lsfList = [self.mergeLsfs(ll, ss) for ll, ss in zip(lsfList, spectra)]
  File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/mergeArms.py", line 299, in mergeLsfs
    return {ff: coaddLsf([ww[ff] for ww in warpedLsfList]) for ff in fiberId}
  File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/mergeArms.py", line 299, in <dictcomp>
    return {ff: coaddLsf([ww[ff] for ww in warpedLsfList]) for ff in fiberId}
  File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/mergeArms.py", line 299, in <listcomp>
    return {ff: coaddLsf([ww[ff] for ww in warpedLsfList]) for ff in fiberId}
KeyError: 2


 Comments   
Comment by price [ 06/May/21 ]

The problem is that the fiberProfiles used for visit=46032 is /projects/HSC/PFS/Subaru/CALIB-SuNSS/FIBERPROFILES/pfsFiberProfiles-2021-01-16-041373-b1.fits, which was constructed with visit=41373^41380^41417^41424^41431^41438^41445^41452^41459^41466^41473. These visits are DCB data, and not SuNSS:

sqlite> SELECT DISTINCT visit, pfsDesignId FROM raw WHERE visit IN (41373,41380,41417,41424,41431,41438,41445,41452,41459,41466,41473);
41373|1099528409104
41380|1099528409104
41417|1099528409104
41424|1099528409104
41431|1099528409104
41438|1099528409104
41445|1099528409104
41452|1099528409104
41459|1099528409104
41466|1099528409104
41473|1099528409104

>>> DummyCableBDatabase().interpret(1099528409104)
['green', 'red1', 'red4', 'red8']

How do we prevent this from happening? We would need to include some state in the dataId for the calib. At the present time, perhaps the best choice is the pfsDesignId (which presently tells us which fibers are illuminated), but that may or may not be something we want for operations.

Comment by rhl [ 07/May/21 ]

We discussed this on slack, and realised that we can detect this piece of user error in reduceExposure. As soon as that ticket is filed, we can close this one.

Comment by price [ 07/May/21 ]

I think I've already implemented the changes on this ticket branch. Hope to submit for review soon.

Comment by price [ 07/May/21 ]

Passed integration test and ready for review.

Comment by hassan [ 07/May/21 ]

Review done. Changes look fine.

Comment by price [ 07/May/21 ]

Merged.

Generated at Sat Feb 10 15:58:34 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.