https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1329 DRP 2-D Pipeline <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=shamano" class="user-hover" rel="shamano">Satoshi Hamano</a> reports that some individual fibers (e.g., the right-most few on r3) have a consistent x offset w.r.t. the <tt>detectorMap&#95;used</tt>, when looking at visits from the July run (in particular, I think, quartz exposures from 2023-07-26). This would indicate that the fix from <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1324" title="Support fitting x/y offsets when updating the detectorMap" class="issue-link" data-issue-key="PIPE2D-1324"><del>PIPE2D-1324</del></a> was unsuccessful.</p> PIPE2D-1329

Individual fibers have significant offsets

Story Normal Won't Fix Won't Fix Satoshi Hamano price EDR Tue, 14 Nov 2023 01:13:05 +0000 Wed, 31 Jan 2024 20:06:37 +0000 Wed, 31 Jan 2024 20:06:37 +0000 0 6 <p>I am unable to reproduce the plots that <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=shamano" class="user-hover" rel="shamano">Satoshi Hamano</a> showed last night. His plots had a solid red band extending all the way down the right-hand side, indicating that fibers were not being corrected. The following plot looks very different: no fiber is all red or all blue (in line with our expectations after applying slit offsets in the detectorMap), and we're seeing features that we know are present in the data (like the epoxy bumps on the edges of each CCD) and revealing large-scale tree rings clearly.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=shamano" class="user-hover" rel="shamano">Satoshi Hamano</a>, could you please provide examples that demonstrate the problems you've discovered, including the <tt>dataId</tt> and plots for a small number of egregious cases?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">import</span> matplotlib.pyplot as plt <span class="code-keyword">import</span> matplotlib.cm from matplotlib.colors <span class="code-keyword">import</span> Normalize from pfs.drp.stella.fitDistortedDetectorMap <span class="code-keyword">import</span> addColorbar dataId = dict(visit=98626, arm=<span class="code-quote">"r"</span>, spectrograph=3) lines = butler.get(<span class="code-quote">"arcLines"</span>, dataId) detectorMap = butler.get(<span class="code-quote">"detectorMap_used"</span>, dataId) dx = lines.x - detectorMap.getXCenter(lines.fiberId, lines.y) norm = Normalize(vmin=-0.1, vmax=0.1) cmap = matplotlib.cm.get_cmap(<span class="code-quote">"coolwarm"</span>) fig, axes = plt.subplots() axes.hexbin(lines.x, lines.y, C=dx, gridsize=100, cmap=cmap, norm=norm) addColorbar(fig, axes, cmap, norm, <span class="code-quote">"dx (pixels)"</span>) axes.set_aspect(<span class="code-quote">"equal"</span>) axes.set_xlabel(<span class="code-quote">"x (pixels)"</span>) axes.set_ylabel(<span class="code-quote">"y (pixels)"</span>) axes.set_title(f<span class="code-quote">"{dataId}"</span>) </pre> </div></div> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/16700/16700_98626r3-dx.png" width="900" style="border: 0px solid black" /></span></p> <p>Have you chosen a mod4 visit intentionally ? Same plot with full density quartz, it certainly feels different : <br/> <span class="image-wrap" style=""><a id="16702_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/16702/16702_%E3%82%B9%E3%82%AF%E3%83%AA%E3%83%BC%E3%83%B3%E3%82%B7%E3%83%A7%E3%83%83%E3%83%88+2023-11-15+9.38.11.png" title="スクリーンショット 2023-11-15 9.38.11.png" file-preview-type="image" file-preview-id="16702" file-preview-title="スクリーンショット 2023-11-15 9.38.11.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/16701/_thumb_16701.png" style="border: 0px solid black" /></a></span> </p> <p>As an example, dataId={visit=98627, spectrograph=3, arm='r'}.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/16702/_thumb_16702.png" style="border: 0px solid black" /></span></p> <p>I make an image from the pfsArm and traces as follows. I measure the profiles center and widths from the calexp.image.array and image.array.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>from pfs.drp.stella import SpectrumSet calexp = butler.get('calexp', dataId) detMap = butler.get('detectorMap_used', dataId) profiles = butler.get("fiberProfiles", dataId) pfsArm = butler.get("pfsArm", dataId) spectra = SpectrumSet.fromPfsArm(pfsArm) traces = profiles.makeFiberTracesFromDetectorMap(detMap) image = spectra.makeImage(calexp.getDimensions(), traces)</pre> </div></div> <p>98627 is a different visit than what I've posted, but I don't think it can look so very different. So I believe your results are different from what both <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=wtgee" class="user-hover" rel="wtgee">Wilfred Gee</a> and I get (although both of these are based on the arcLines, but I think those are generally regarded as reliable), which I think means you need to validate your work before we can proceed on this.</p> <p>This is the same request as the one on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1330" title="Investigate the large scale feature of detectorMap offsets on detector" class="issue-link" data-issue-key="PIPE2D-1330"><del>PIPE2D-1330</del></a>. Please provide the code as .py files on GitHub, and also details of how your centroids compare with the ones in the arcLines files. It's perfectly possible that your ones are correct, but we need to sort out the differences (and potentially correct the centroids that the pipeline generates).</p> <p>I checked my code step by step, but couldn't identify the cause of the large X offset seen in my r3 plots. I attached a notebook that contains the part of calculating residual image in my extraction QA script. I run the notebook with w.2023.45 version, and confirmed that the same large X offset could be seen in the residual image. Could you check if there is a problem with my code?<br/> <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/16712/16712_largeXOffset.ipynb" title="largeXOffset.ipynb attached to PIPE2D-1329">largeXOffset.ipynb^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>What in the notebook looks reasonable, but it's far from enough to be able to reproduce your results.</p> <p>I attached the reproduced result. <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/16800/16800_largeXOffset-2.ipynb" title="largeXOffset-2.ipynb attached to PIPE2D-1329">largeXOffset-2.ipynb^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>The python code used for generating the QA plots is also attached, but the readability is poor. <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/16801/16801_extractionQa.py" title="extractionQa.py attached to PIPE2D-1329">extractionQa.py^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=shamano" class="user-hover" rel="shamano">Satoshi Hamano</a> is measuring the x position of the trace in the <tt>calexp</tt> and in the model image (constructed from the <tt>pfsArm</tt>, <tt>profiles</tt> and <tt>detectorMap&#95;used</tt>), and his plots show that there is a difference in these x positions for those fibers on the far right. This is also revealed in a straight subtraction of the <tt>calexp</tt> and the model image. These differences are not present in the <tt>arcLines</tt> measurements written by the pipeline (which measures the position of the trace in the <tt>calexp</tt>) because the problem is in the model image: the profile is off-center (by about a third of a pixel) for those far-right fibers. This is further fallout from the inaccurate detectorMaps caused by the flat-field screen not being used on the first night of the July run. Below I show the profile for <tt>fiberId=1307</tt>. The results of <tt>profiles<span class="error">&#91;1307&#93;</span>.calculateStatistics()</tt> demonstrate the problem also (note the <tt>centroid</tt> values):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Struct(centroid=[-0.3664167395365968 -0.33592957113018446 -0.2319049640704939]; width=[2.0435098090603625 2.0388422530999777 2.073611949172245]; min=[-0.006259117354348317 -0.002724690172294439 -0.0007977413487588208]; max=[2.3501915455243494 2.3594814341754806 2.32601427057499]) </pre> </div></div> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/16802/16802_profile-r3-1307.png" width="900" style="border: 0px solid black" /></span></p> <p>Regenerating profiles with slit offset option resolved this problem, so I think we can close this ticket.</p> Relates PIPE2D-1330 Development Rank 0|02qph7:06r091 Sprint EDR+AIT