https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-999 DRP 2-D Pipeline <p>When using the weekly build <tt>w.2022.10</tt> and running <tt>reduceExposure</tt> with 2D sky subtraction enabled:</p> <blockquote> <p>reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS --rerun=hassans/sunss/v46588_calibsunss_20220307b --doraise --id<br/> visit=46588 arm=b^r -c measurePsf.oversampleFactor=11 measurePsf.version=Aug0121_v1 measurePsf.targetSize=21 doWriteCalexp=True doSubtractSky2d=True doMeasurePsf=True isr.doFlat=False </p></blockquote> <p>Results in over-subtracted lines (see <span class="image-wrap" style=""><a id="15118_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15118/15118_v46588_r1_sunsscalib_A.png" title="v46588_r1_sunsscalib_A.png" file-preview-type="image" file-preview-id="15118" file-preview-title="v46588_r1_sunsscalib_A.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15118/_thumb_15118.png" style="border: 0px solid black" /></a></span> , <span class="image-wrap" style=""><a id="15117_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15117/15117_v46588_r1_sunsscalib_B.png" title="v46588_r1_sunsscalib_B.png" file-preview-type="image" file-preview-id="15117" file-preview-title="v46588_r1_sunsscalib_B.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15117/_thumb_15117.png" style="border: 0px solid black" /></a></span> and <span class="image-wrap" style=""><a id="15116_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15116/15116_v46588_r1_sunsscalib_C.png" title="v46588_r1_sunsscalib_C.png" file-preview-type="image" file-preview-id="15116" file-preview-title="v46588_r1_sunsscalib_C.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15116/_thumb_15116.png" style="border: 0px solid black" /></a></span> ). </p> <p>Note that some of those images indicate misalignment issues indicative of a poor detectormap. The above run makes use of the calib <tt>CALIB-SuNSS-2021-06-16</tt>. Using calib <tt>CALIB-PFI-20211220</tt> requires a fix to DRP (see <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-993" title="reduceExposure fails when using PFI calib and 2d sky subtraction" class="issue-link" data-issue-key="PIPE2D-993"><del>PIPE2D-993</del></a>). Using a local fix and running the above against the PFI calib results in an improved image <span class="image-wrap" style=""><a id="15115_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15115/15115_v46588_r1_pficalib_D.png" title="v46588_r1_pficalib_D.png" file-preview-type="image" file-preview-id="15115" file-preview-title="v46588_r1_pficalib_D.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15115/_thumb_15115.png" style="border: 0px solid black" /></a></span> . In any case the main purpose of this ticket is to address the over subtraction, not the detectormap issues.</p> PIPE2D-999

2D sky subtraction results in over subtracted images

Bug Normal Done Done price hassan Mon, 7 Mar 2022 21:21:40 +0000 Wed, 30 Mar 2022 16:39:35 +0000 Wed, 30 Mar 2022 16:39:35 +0000 0 3 <p>I think the main problem is not setting <tt>isr.doApplyGains=True</tt> (required when <tt>isr.doFlat=False</tt> in order to get the different amplifiers on something approaching a common system). I built some new calibs (<tt>/projects/HSC/PFS/Subaru/CALIB-SuNSS-20220314</tt>), and the adjusted detectorMap fit now is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceExposure.adjustDetectorMap INFO: Final fit: chi2=78975.644377 dof=49380 xRMS=0.040120 yRMS=0.061799 (0.005336 nm) xSoften=0.005633 ySoften=0.033382 from 24702/30239 lines </pre> </div></div> <p>Below is an image showing the 2D sky subtraction I get at 876 nm. On the left is the vanilla pipeline (same command as above, plus <tt>isr.doApplyGains=True</tt>, and using the updated SuNSS calibs). The line complex at 876 nm is causing problems (you can see the awful horizontal artifacts). There are 10 lines in a 1 nm range, which can be resolved into two complexes of 5 lines each, each spanning about 0.05 nm. That range is comparable to the precision of our wavelength solution, so we're not getting good flux measurements. If I merge each of these two line complexes (using intensity-weighted mean wavelengths), I get the subtraction on the right. Clearly we're not getting all the flux, but I don't think we can do a better job until we get a better sky line flux model that doesn't treat all lines independently.</p> <p>The other configuration change that could help this is modifying <tt>subtractSky2d.selectFibers.targetType</tt> to include only one of <tt>SUNSS_DIFFUSE</tt> and <tt>SUNSS_IMAGING</tt>. Oh, and we're obviously missing a lot of lines, including the O~2~ complex around 864 nm, but I don't think this is the place to fix that.</p> <p><span class="image-wrap" style=""><a id="15132_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15132/15132_pipe2d-999-merge876.jpg" title="pipe2d-999-merge876.jpg" file-preview-type="image" file-preview-id="15132" file-preview-title="pipe2d-999-merge876.jpg"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15132/_thumb_15132.png" style="border: 0px solid black" /></a></span></p> <p>I'm a bit unclear about what the 2-D subtraction is doing. I thought that the line estimation code (resulting in arcLineSets) fits all the lines in a given fibre simultaneously, so I'd expect 2D sky subtraction to do the same thing, for the sky fibres, then use the resulting model to subtract sky from the other fibres. Apparently it's doing something different. Can you clarify for me?</p> <p>The fluxes are measured by simultaneously fitting all lines that are blended. From the fluxes we derive a sky line flux model, which is currently done by averaging across fibers the fluxes of each line. It is the flux of each line from the sky line flux model that is used to subtract the sky lines from all fibers (that have a PSF model defined).</p> <p>I believe that we are not measuring the line fluxes well because there are five lines separated by less than a pixel, so that small errors in the wavelength solution lead to large differences in the distribution of the measured fluxes of the blended lines for different fibers, which means that the sky line flux model is inaccurate. Maybe we could determine a reasonable sky line flux model if we knew the covariances between all the fluxes, but we do not record them.</p> <p>Ah, OK, got it. </p> <p>We may always have some (covariant) wavelength errors, and need to think about how to include them in the fitted sky model. Of course, if the wavelength errors are large enough to show up in the residuals we should be able to tweak the wavelength solutions, hence the use of the phrase "may always".</p> <p>I think the best way to deal with this pair of line complexes at 876 nm would be to derive the sky line flux model from other lines: the OH lines should go up and down in concert, so we should be able to get the flux of each of the blended lines from the fluxes of the host of unblended lines.</p> <p>In the long run, where we model the sky intensities from the sky fibres, that's definitely the thing to do (there are 8 vibrational occupations + probably a single rotational temperature, so a finite set of parameters).</p> <p>However, we do still need to think about wavelength errors in the subtractions.</p> <p>Following review: </p> <p>Changes to <tt>drp_stella</tt> (changing default value for <tt>subtractSky2d.selectFibers.targetType</tt> to <tt>SUNSS_DIFFUSE</tt>) accepted. </p> <p>Changes to <tt>obs_pfs</tt> (modifying the linelist) raised joint concerns, and as discussed in the comments above. Leave those as-is till a solution involving deriving the flux from lines outside the problematic area has been implemented.</p> <p>From our weekly meeting: I will demonstrate that the line fluxes are being measured accurately in each fiber, proving that the problem lies in the interpolation of fluxes across fibers when there are huge degeneracies.</p> <p>This image demonstrates that the line fluxes and aperture corrections are being measured correctly. The continuum is not being measured very well on both sides, around the 864nm bands, but the lines are being measured nicely.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15138/15138_pipe2d-999-phot.jpg" style="border: 0px solid black" /></span></p> <p>I added some extra commits to clean up problems discovered while subtracting the lines. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a>, would you please check those too?</p> <p>Agree with the additional commits described in <a href="https://github.com/Subaru-PFS/drp_stella/pull/255" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/pull/255</a> . Only one very minor question on one of the commits, although this does not affect the code changes.</p> <p>Addressed review comment and merged.</p> Development Rank 0|02qpq3:r3s00000000008i Reviewers hassan Sprint 2DDRP-2022 B 2DDRP-2022 C Story Points 2.0