[PIPE2D-916] Debug 2021-10-17 weekly Created: 20/Oct/21 Updated: 05/Nov/21 Resolved: 04/Nov/21 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | price | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Story Points: | 1 |
| Sprint: | 2DDRP-2021 A 10, 2DDRP-2021 A11 |
| Reviewers: | hassan |
| Description |
|
The weekly failed due to wavelength residuals being high: ====================================================================== FAIL: testResiduals (__main__.ArcTestCase_brn_39) (arm='r', fiberId=165) Test that wavelength fit residuals are reasonable ---------------------------------------------------------------------- Traceback (most recent call last): File "/scratch/pprice/jenkins/weekly/2021-10-17/build/stack/miniconda3-4.5.12-1172c30/Linux64/pfs_pipe2d/w.2021.42/python/pfs/pipe2d/weekly/test_weekly.py", line 136, in testResiduals self.assertFloatsAlmostEqual(median, 0.0, atol=atol) File "/scratch/pprice/jenkins/weekly/2021-10-17/build/stack/miniconda3-4.5.12-1172c30/Linux64/utils/18.1.0/python/lsst/utils/tests.py", line 735, in assertFloatsAlmostEqual testCase.assertFalse(failed, msg="\n".join(errMsg)) AssertionError: True is not false : -0.0212058729346154 != 0.0; diff=0.0212058729346154/0.0212058729346154=1.0 with rtol=2.220446049250313e-16, atol=0.02 I also noticed that the sky model fit has a non-finite chi^2:
+ mergeArms.py /scratch/pprice/jenkins/weekly/2021-10-17/process --calib=/scratch/pprice/jenkins/weekly/2021-10-17/process/CALIB --rerun=weekly/pipeline/bmn/pipeline --doraise -j10 --id 'visit=50^51^52^55^56^58'
[...]
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=76238.614984 length=4172/4176 numSamples=120 numGood=494172 numBad=6583 numRejected=386
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=154280.445883 length=4172/4176 numSamples=120 numGood=494228 numBad=6239 numRejected=726
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=154219.982931 length=4172/4176 numSamples=120 numGood=494184 numBad=6317 numRejected=721
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4172/4176 numSamples=120 numGood=494141 numBad=6274 numRejected=740
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4172/4176 numSamples=120 numGood=494431 numBad=5957 numRejected=768
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4173/4176 numSamples=120 numGood=494298 numBad=6149 numRejected=754
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4095/4176 numSamples=120 numGood=477835 numBad=18264 numRejected=5624
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4105/4176 numSamples=120 numGood=459494 numBad=24426 numRejected=21165
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4104/4176 numSamples=120 numGood=421857 numBad=64460 numRejected=20467
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4096/4176 numSamples=120 numGood=459512 numBad=23583 numRejected=21346
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4104/4176 numSamples=120 numGood=374855 numBad=112364 numRejected=21396
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4104/4176 numSamples=120 numGood=425090 numBad=60247 numRejected=21474
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4149/4176 numSamples=120 numGood=487595 numBad=13488 numRejected=37
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4150/4176 numSamples=120 numGood=487177 numBad=12690 numRejected=1284
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4149/4176 numSamples=120 numGood=486650 numBad=13298 numRejected=1224
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4148/4176 numSamples=120 numGood=486955 numBad=13019 numRejected=1185
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4152/4176 numSamples=120 numGood=487169 numBad=12833 numRejected=1187
mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=nan length=4152/4176 numSamples=120 numGood=487049 numBad=12926 numRejected=1249
|
| Comments |
| Comment by price [ 22/Oct/21 ] |
|
I believe I've made the residual check much more robust. |
| Comment by rhl [ 22/Oct/21 ] |
|
What changed that the check needed to be more robust? And are those NaNs new; if so, we need a ticket to investigate (actually, we probably need one anyway) |
| Comment by price [ 22/Oct/21 ] |
|
That check has long been overly-sensitive. It's calculating the RMS of the residual from the wavelength solution. The problem is that it's using all lines, including low S/N lines, which artificially inflates the RMS. By restricting the calculation to high-S/N lines, the RMS calculation is much more robust. The {{NAN}}s are new (not present in the weekly of 2021-10-10). I'm not sure how that's possible, since I traced them back to the spectral extraction (bad variance though good flux). But they are not appearing now when I run the weekly again with the fixed code. |
| Comment by rhl [ 26/Oct/21 ] |
|
Well, in general I think that making the test more robust is a good thing, but I worry about what changed to trigger the failures. What did we change that could have changed the wavelength residuals? |
| Comment by hassan [ 04/Nov/21 ] |
|
All the code changes look reasonable. The questions rhl above would need to be addressed in a separate ticket. |
| Comment by price [ 04/Nov/21 ] |
|
Merged. |