https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-669 DRP 2-D Pipeline <p>As part of addressing <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-653" title="Fix failing weekly" class="issue-link" data-issue-key="PIPE2D-653"><del>PIPE2D-653</del></a>, a temporary fix has been made in the <tt>testResiduals()</tt> function:</p> <p><a href="https://github.com/Subaru-PFS/pfs_pipe2d/pull/53/commits/d1348f2d129351c7c11d400fcb7159d17b4aaba7" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/pfs_pipe2d/pull/53/commits/d1348f2d129351c7c11d400fcb7159d17b4aaba7</a></p> <blockquote><p>Getting larger wavelength residuals than the test allows. Not<br/> clear whether this is due to line mis-identification or<br/> genuinely bad fits at the edges of the detector, but not going<br/> to put effort into tracking this down right now; so increasing<br/> the threshold to allow the test to pass until we can take a<br/> good look at this.</p></blockquote> <p>This ticket is to track that a future review of the cause of the larger wavelength residuals.</p> PIPE2D-669

Larger than expected wavelength residuals encountered during weekly test

Task Normal Done Done price hassan Fri, 4 Dec 2020 21:27:11 +0000 Wed, 10 Feb 2021 16:33:35 +0000 Thu, 4 Feb 2021 21:20:10 +0000 0 2 <p>In investigating here, discovered that master is broken. Including all the fiberIds in the pfsConfig (<a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-96" title="Index fibers from multiple spectrographs in PfsDesign" class="issue-link" data-issue-key="DAMD-96"><del>DAMD-96</del></a>) broke a few things in the science pipeline. The integration test didn’t catch this because there was a bug in the integration test (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-708" title="Jenkins integration test not using drp_stella_data branch" class="issue-link" data-issue-key="PIPE2D-708"><del>PIPE2D-708</del></a>) where it didn’t use the appropriate branch for the raw data. I fixed that as part of this work.</p> <p>Notes on looking at the wavelength residuals:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">import</span> os <span class="code-keyword">import</span> numpy as np from lsst.daf.persistence <span class="code-keyword">import</span> Butler from pfs.drp.stella <span class="code-keyword">import</span> DetectorMap root = <span class="code-quote">"/scratch/pprice/pipe2d-669"</span> butler = Butler(os.path.join(root, <span class="code-quote">"rerun/pipe2d-660/calib/brn/arc_brn/detectorMap"</span>)) visit = 39 <span class="code-keyword">for</span> arm in <span class="code-quote">"brn"</span>: detMap = DetectorMap.readFits(os.path.join(root, f<span class="code-quote">"rerun/pipe2d-660/calib/brn/arc_brn/detectorMap/DETECTORMAP/pfsDetectorMap-000039-{arm}1.fits"</span>)) #detMap = butler.get(<span class="code-quote">"detectorMap"</span>, visit=visit, arm=arm) lines = butler.get(<span class="code-quote">"arcLines"</span>, visit=visit, arm=arm) <span class="code-keyword">for</span> fiberId in set(lines.fiberId): select = lines.fiberId == fiberId num = select.sum() fitWavelength = detMap.findWavelength(fiberId, lines.y[select]) residual = lines.wavelength[select] - fitWavelength lq, median, uq = np.percentile(residual, (25.0, 50.0, 75.0)) rms = 0.741*(uq - lq) print(arm, fiberId, num, median, rms) Even with PIPE2D-660 (DifferentialDetectorMap), still getting median ~ -0.03 and rms ~ 0.01. The detectorMap here is a SplinedDetectorMap, apparently coming from the calibs. Using the DifferentialDetectorMap, the median goes to ~1e-3, but still have rms ~ 0.01. We know the SplinedDetectorMap from the simulator is off by 0.5 pixel. For a dispersion of 0.07 nm/pixel, that corresponds to the observed 0.03 nm median offset. The RMS of 0.01 nm appears to be constant, and probably not bad: it<span class="code-quote">'s 0.14 pixels. For arm=b, that'</span>s decent; we could probably set the limit at 0.02. For arm=r (0.09 nm/pixel), we<span class="code-quote">'re getting rms ~ 0.007 nm ~ 0.08 pixels, and <span class="code-keyword">for</span> arm=n (0.08 nm/pixel), rms ~ 0.01 nm ~ 0.13 pixels. So the numbers look reasonable. Not sure how we got better than 0.01 nm previously, but that'</span>s our current state of the art. But, why are we getting the SplinedDetectorMap, when we want the DifferentialDetectorMap? It's probably a butler thing, or maybe a simulator thing (unexpected times in the header). &gt;&gt;&gt; butler.get(<span class="code-quote">"detectorMap_filename"</span>, visit=57, arm=<span class="code-quote">"r"</span>) [<span class="code-quote">'/scratch/pprice/pipe2d-669/CALIB/DETECTORMAP/pfsDetectorMap-000000-r1.fits'</span>] sqlite&gt; select * from detectorMap; id|arm|ccd|spectrograph|calibDate|calibTime|visit0|validStart|validEnd 1|b|0|1|2020-11-26|2020-11-26T00:10:52|0|2020-11-25T12:05:26.000001|2025-10-31T00:10:52 2|r|1|1|2020-11-26|2020-11-26T00:08:25|0|2020-11-25T12:04:12.500001|2025-10-31T00:08:25 3|n|1|1|2020-11-26|2020-11-26T00:09:41|0|2020-11-25T12:04:50.500001|2025-10-31T00:09:41 4|m|2|1|2020-11-26|2020-11-26T00:12:08|0|2020-11-25T12:06:04.000001|2025-10-31T00:12:08 5|b|0|1|2020-11-25T23:27:34|2020-11-25|39|2015-12-22T00:00:00|2020-11-25T12:05:26 6|n|1|1|2020-11-25T23:26:26|2020-11-25|39|2015-12-22T00:00:00|2020-11-25T12:04:50.500000 7|r|1|1|2020-11-25T23:26:57|2020-11-25|39|2015-12-22T00:00:00|2020-11-25T12:04:12.500000 8|m|2|1|2020-11-25T23:26:38|2020-11-25|40|2015-12-22T00:00:00|2020-11-25T12:06:04 Strange: the SplinedDetectorMap is dated after the DifferentialDetectorMap, and the latter has calibDate and calibTime switched. This is probably because the simulator times can be a bit random (different images done in parallel, so times don't sort the same as visits). Fixed the dates on the simulated data (both images and detectorMaps), and am now getting the right detectorMap in the test. Reverted the commit that loosened the precision, and actually tightened it a bit. </pre> </div></div> <p>The new weekly dataset is available: <a href="http://tigress-web.princeton.edu/~HSC/pfs-drp-2d/weekly-20200204/" class="external-link" rel="nofollow">http://tigress-web.princeton.edu/~HSC/pfs-drp-2d/weekly-20200204/</a></p> <p>I've got a final integration test and weekly running, but I expect them to go through.</p> <p>Merged to master.</p> Development Rank 0|02qpt6:508yl0chzzy0000p920zv Reviewers hassan Sprint 2DDRP-2021 A 2 Story Points 2.0