https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1313 DRP 2-D Pipeline <p>The extraction QA for engineering data release 2 is summarized here.</p> PIPE2D-1313

extraction QA for engineering data release 2

Task Normal Done Done Satoshi Hamano Satoshi Hamano EDR QualityAssurance Tue, 10 Oct 2023 12:38:55 +0000 Wed, 31 Jan 2024 23:39:08 +0000 Wed, 31 Jan 2024 23:39:08 +0000 0 2 <p>Summary of extraction QA for edr2-20231203.</p> <p><span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/16958/16958_extractionQA-20231203-b.pdf" title="extractionQA-20231203-b.pdf attached to PIPE2D-1313">extractionQA-20231203-b.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p><span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/16957/16957_extractionQA-20231203-r.pdf" title="extractionQA-20231203-r.pdf attached to PIPE2D-1313">extractionQA-20231203-r.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p><span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/16956/16956_extractionQA-20231203-nm.pdf" title="extractionQA-20231203-nm.pdf attached to PIPE2D-1313">extractionQA-20231203-nm.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p> </p> <p>Points:</p> <ul> <li>For each arm, the rms of chi is plotted for all fibers and visits in a plot (the colors of points show the chi’s rms values). The problems of extraction are listed for all arms in the slides (and in this post).</li> <li>As known, it can be seen from the comparison between the chi rms maps and the mean pfsArm flux maps that most large chi value spots be related to the bright fibers. A few fibers around very bright fibers also show large chi values. The effects of the bright fibers on the next fibers should be investigated more quantitatively (see P.3 of the b-arm slide for plots).</li> <li>All arm data of the quartz visits (<tt>97036-97047, 97050-97060, 97063. 97064</tt>) have large chi because of the scattered light problem. The scattered light affects the measurement of the profile widths in extractionQA and makes dσ/σ negative (meaning the profiles of calExp image is broader than those of model image).</li> <li>Two fibers, ID=<tt>1373</tt> and <tt>1415</tt>, are found to have a problem in their profiles in <tt>b3</tt> data (P.12-16 of b-arm slide).</li> <li>The extraction quality of the <tt>r3</tt> data of <tt>82595-83249</tt> visits (November run) is found to be poor, because the fiberProfiles of April run, in which <tt>r3</tt> arm data was out of focus, are used as the calibration data. (P.10-12 of r-arm slide)</li> <li>As known, the m-arm data of visits (<tt>92536, 92537, 92733, 92734, 92892, 92893 93301, 93302</tt>) have large residuals because of the poor fiberProfile quality of m-arm data.</li> <li>The results of the detectorMap QA and extraction QA should be compared numerically as commented for weeks. I tried to show some comparison in the drp telecon, but there is not enough time to prepare plots.<br/>  </li> </ul> <p>extractionQA script is put at /work/shamano/fiberProfile/extractionQa.py on the pfsa server.</p> <p>This script requires following arguments.</p> <ul> <li>The rerun name (positional argument)</li> <li>The output directory name (positional argument. If the directory does not exist, the script makes it.)</li> <li>Visit number(s) (optional argument with -v or --visit)</li> </ul> <p>The default paths of the data repository and calib are set as follows.</p> <ul> <li>data repository: /work/drp (the path can be changed with --repository option)</li> <li>calib: /work/drp/CALIB (the path can be changed with --calib option)</li> </ul> <p>Example command (extraction QA for all arms for the visits 98042 and 98043):</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> python extractionQa.py pfs/internal/edr2-20231203 extractionQa-edr2-20231203/ -v 98042 98043  </pre> </div></div> <p> </p> <p>The arms and spectrographs can be limited with the optional arguments <del>a (</del>-arm) and -s (–specrograph).</p> <p>Example command (extraction QA for r1 and r3 data for visit=98042):</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> python extractionQa.py pfs/internal/edr2-20231203 extractionQa-edr2-20231203/ -v 98042 -a r -s 1 3  </pre> </div></div> <p> </p> <p>Output files are as follows:</p> <ul> <li>{visit}_{arm}{spectrograph}_chistats.pdf: the plots of some statistics from extraction QA (e.g., the standard deviation of chi for each fiber, the maps of X offsets and width differences).<br/> * {visit} <p>_{arm}</p> {spectrograph}_chifiber.pdf: the detailed plots of the fiber regions for the fibers with relatively bad extraction quality.<br/> * {visit}_{arm}{spectrograph} <p>_chifiber.pickle: the numerical output from extraction QA is stored. dictionary object.</p></li> </ul> <p> </p> <p>Plotter using the pickle files is in the notebook at /work/shamano/fiberProfile/extractionQA_summary.ipynb.</p> <p> </p> <p> </p> <p>The extractionQA results have been delivered, so can we close this ticket? <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=msyktnk" class="user-hover" rel="msyktnk">Masayuki Tanaka</a></p> <p>Yes, we can close this ticket.</p> Relates PIPE2D-1249 Development Rank 0|02qph7:06r0903g Sprint EDR+AIT Story Points 2.0