https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-864 DRP 2-D Pipeline <p>As briefly mentioned during the last DRP telecon 2021-06-18, I am having difficulty generating a bootstrapped detectormap for medium-resolution data.</p> <p>running <tt>bootstrapDetectorMap</tt> with arc visit=63106 (room_lights, no_screen) and flat visit=63050 with debug on:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> bootstrapDetectorMap.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-2021-06-16 --rerun hassan/bstrapMed --flatId visit=63050 arm=m --arcId visit=63106 arm=m --debug </pre> </div></div> <p>We see the line identification shown in figure 'bootstrap medRes.png'. The arc lines are clearly apparent, but the crosses do not match any of the lines.</p> <p>This is in contrast to running the same task against DCB <tt>r</tt> data:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> bootstrapDetectorMap.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-2021-06-16/ --rerun hassan/bstrapRed --flatId visit=30044 arm=r --arcId visit=30045 arm=r --debug </pre> </div></div> <p>Where the lines are clearly identified. See figure 'bootstrap-red-DCB.png' .</p> <p>Also no problems with identification noticed when using simulated arcs from the integration test.</p> PIPE2D-864

bootstrap does not identify lines correctly for med-res data

Bug Normal Done Done price hassan Mon, 28 Jun 2021 13:24:09 +0000 Sat, 10 Jul 2021 02:18:50 +0000 Sat, 10 Jul 2021 02:18:50 +0000 0 3 <p>There are astropy fit warnings and abysmal RMS values coming from <tt>bootstrapDetectorMap.py</tt> when running on <tt>m</tt> SuNSS data.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-2021-06-16 --rerun price/pipe2d-864 --flatId visit=63050 arm=m --arcId visit=63106 arm=m [...] WARNING: The fit may be unsuccessful; check fit_info[<span class="code-quote">'message'</span>] <span class="code-keyword">for</span> more information. [astropy.modeling.fitting] astropy WARN: The fit may be unsuccessful; check fit_info[<span class="code-quote">'message'</span>] <span class="code-keyword">for</span> more information. WARNING: The fit may be unsuccessful; check fit_info[<span class="code-quote">'message'</span>] <span class="code-keyword">for</span> more information. [astropy.modeling.fitting] astropy WARN: The fit may be unsuccessful; check fit_info[<span class="code-quote">'message'</span>] <span class="code-keyword">for</span> more information. bootstrap INFO: Found 6586 lines in 254 traces bootstrap INFO: Matched 5617 lines bootstrap INFO: Median difference from detectorMap: -5.012633,0.618370 pixels bootstrap INFO: Fit 1340/2859 points, rms: x=4.320284 y=6.319983 total=3.232250 pixels bootstrap INFO: Updating detectorMap... bootstrap INFO: Median difference from detectorMap: 5.852661,0.959997 pixels bootstrap INFO: Fit 888/2758 points, rms: x=3.860911 y=4.141929 total=1.879898 pixels bootstrap INFO: Updating detectorMap... </pre> </div></div> <p>Yes, I think the fit is failing due to the poor identification of lines prior to that.</p> <p>One problem is that no lamps are indicated as being on in the header, and so the software assumes it's looking at the sky. Looking at "dome arcs" is a special case, so we need to configure it specially:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> config.readLineList.lampList = [<span class="code-quote">"HgI"</span>, <span class="code-quote">"KrI"</span>, <span class="code-quote">"ArI"</span>] </pre> </div></div> <p>I had ran <tt>config.readLineList.lampList = <span class="error">&#91;&quot;Ar&quot;&#93;</span>'</tt> a few days ago with the same problem. Will check that again.</p> <p>For future acquisitions, ICS <em>could</em> either adjust some of the lamp cards, or add some other PFS-specific cards. </p> <p>Note that <tt>DATA-TYP</tt> cannot be sensible. The only available choices are: ACQUISITION, BIAS, COMPARISON, DARK, DOMEFLAT, DOMEFLAT_ON, DOMEFLAT_OFF, FLAT, FOCUSING, INSTLAT, OBJECT, SKYFLAT, STANDARD, STANDARD_STAR, TEST</p> <p>Actually, we are planning to add a room light card, based on the parasite data we are getting from the telescope while running SuNSS; I'll let you know what that is when we get there. </p> <p>The main problems were identifying the lamps being used (the code defaults to assuming sky if no lamps are set in the FITS header) and identifying a good set of fibers.</p> <p>With the following configuration overrides:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> config.readLineList.lampList = [<span class="code-quote">"HgI"</span>, <span class="code-quote">"KrI"</span>, <span class="code-quote">"ArI"</span>] # otherwise we get sky config.spectralOffset = -20 # by-eye estimate config.profiles.profileRadius = 3 # because we have some close-packed fibers # The following give us only the good fibers, which give us a reliable fiber identification from pfs.datamodel <span class="code-keyword">import</span> FiberStatus config.fiberStatus = [FiberStatus.GOOD] config.profiles.findThreshold = 2000 # convolution is probably picking up a lot of scattered light </pre> </div></div> <p>I get a pretty good bootstrap result:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> bootstrap.readLineList INFO: Filtered line lists <span class="code-keyword">for</span> lamps: ArI,HgI,KrI bootstrap INFO: Found 6748 lines in 251 traces bootstrap INFO: Matched 6741 lines bootstrap INFO: Median difference from detectorMap: 6.389995,4.018591 pixels bootstrap INFO: Fit 2983/3309 points, rms: x=0.056048 y=0.173571 total=0.126654 pixels bootstrap INFO: Updating detectorMap... bootstrap INFO: Median difference from detectorMap: 5.759607,5.856603 pixels bootstrap INFO: Fit 2969/3432 points, rms: x=0.067750 y=0.108061 total=0.075888 pixels bootstrap INFO: Updating detectorMap... </pre> </div></div> <p>I am able to reproduce Paul's findings above, and from eye-balling the lines predicted by the resultant detectormap, the results are much improved (sub-pixel error in both x and y). </p> <p>Minor comment added pull request.</p> <p>Command line executed:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> bootstrapDetectorMap.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-2021-06-16 --rerun hassan/bstrap-pipe2d-864 --flatId visit=63050 arm=m --arcId visit=63106 arm=m --debug --configfile pipe2d-864.config </pre> </div></div> <p>with the file <tt>pipe2d-864.config</tt> containing the contents Paul has already listed above.</p> <p>Merged.</p> Blocks PIPE2D-869 Development Rank 0|02qpt6:508yl0chzzy0000026k Reviewers hassan Sprint 2DDRP-2021 A 6 Story Points 2.0