https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-77 DRP 2-D Pipeline <p>1. Understand why the normalized flat is only five pixels wide;<br/> 2. Understand why the variation is at ~3% rather than &lt;1%.</p> PIPE2D-77

Make sure there are no systematics in the normalized flat

Sub-task PIPE2D-10 Major Done Done aritter swinbank Fri, 9 Sep 2016 17:31:02 +0000 Fri, 10 Feb 2017 18:27:05 +0000 Thu, 9 Feb 2017 05:41:37 +0000 0 3 <p>This ticket is based on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-80" title="Task-ify code for creating normalized flats" class="issue-link" data-issue-key="PIPE2D-80"><del>PIPE2D-80</del></a> which is based on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90" title="Create Calib Task to create and write FiberTraces" class="issue-link" data-issue-key="PIPE2D-90"><del>PIPE2D-90</del></a> as it needs to read the pfsFiberTrace file. Once <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-90" title="Create Calib Task to create and write FiberTraces" class="issue-link" data-issue-key="PIPE2D-90"><del>PIPE2D-90</del></a> and <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-80" title="Task-ify code for creating normalized flats" class="issue-link" data-issue-key="PIPE2D-80"><del>PIPE2D-80</del></a> are reviewed and merged into master, I will rebase this ticket on master and submit for review</p> <p>It appears that we looked at the wrong file when we created this ticket. The width of the normalized Flat (point 1.) was actually correct, and so was the variation (point 2.). However concerning the width we decided to not use the proposed SNR cut nor cut the normalized Flat FiberTrace to the width of the spectral FiberTraces for diagnostic reasons. We might need to rethink not using the SNR cut though as it will probably add noise to the adjacent FiberTraces.<br/> We found that in the center of the FiberTraces the spatial profile is slightly too low (of the order of 0.03%). The explanation for this seems to be in the way the spatial profiles are calculated. To avoid negative values for the wings of the spatial profiles aritter set all negative pixel values in the original FiberFlat exposures to zero, what causes the wings to be slightly too high and therefore the centers of the spatial profiles (normalized to a sum of 1.0) slightly too low. We decided that this systematic is within the acceptable errors and that no fix (code changes) is needed at least at this stage.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a>, are you happy with the text above? If so, we'll close this.</p> <blockquote> <p>The explanation for this seems to be in the way the spatial profiles are calculated</p></blockquote> <p>Please check/demonstrate this by not setting all negative pixels to zero and making sure that the profile is no longer too low. Should be quick to do this: just hack away at the source.</p> <p>Turns out that I don't set negative pixel values to zero (anymore), only negative profile values. I'm going to check the weights for the most outer pixels next.</p> <p>Per our meeting of 2016-12-19, please add:</p> <ul> <li>A description of what's going on here;</li> <li>How we can be sure it's not going to be a problem in dithered flats.</li> </ul> <p>Per our meeting of 2017-01-09, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=aritter" class="user-hover" rel="aritter">aritter</a> believes that this problem may have vanished (or be much reduced?) with newer simulations and/or the current version of his code. He will check this, and update this ticket with material to convince us that that is the case. We can then close this (a post-mortem of bugs in old code or simulations is not necessary).</p> <p>I have produced plots comparing the original and reconstructed (from the spatial profile and the extracted spectrum) pixel values in all FiberTraces in one Flat, sorted by their distance to the center of the FiberTrace. The profile was calculated with 10 times oversampling (bin width 1/10th pixel). The bin width for which the mean and standard deviation of the difference (original pixel value) / (reconstructed pixel value) were calculated is 1/100th pixel. We originally saw strong systematics in the most outer 1/20th (half a bin width of the profile calculation) of a pixel which seems resolved now:<br/> <span class="image-wrap" style=""><a id="10419_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10419/10419_curvedFiberTraceNormalizedFlatFromDRP_STELLA_DATA_profile_wider.png" title="curvedFiberTraceNormalizedFlatFromDRP_STELLA_DATA_profile_wider.png" file-preview-type="image" file-preview-id="10419" file-preview-title="curvedFiberTraceNormalizedFlatFromDRP_STELLA_DATA_profile_wider.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10412/_thumb_10412.png" style="border: 0px solid black" /></a></span> <br/> <span class="image-wrap" style=""><a id="10414_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10414/10414_origMinusRec_SPLINE3_center_new.png" title="origMinusRec_SPLINE3_center_new.png" file-preview-type="image" file-preview-id="10414" file-preview-title="origMinusRec_SPLINE3_center_new.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10413/_thumb_10413.png" style="border: 0px solid black" /></a></span> <br/> <span class="image-wrap" style=""><a id="10415_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10415/10415_origMinusRec_SPLINE3_right_new.png" title="origMinusRec_SPLINE3_right_new.png" file-preview-type="image" file-preview-id="10415" file-preview-title="origMinusRec_SPLINE3_right_new.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10414/_thumb_10414.png" style="border: 0px solid black" /></a></span> <br/> <span class="image-wrap" style=""><a id="10423_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10423/10423_straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_new_pixel_distribution.png" title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_new_pixel_distribution.png" file-preview-type="image" file-preview-id="10423" file-preview-title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_new_pixel_distribution.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10415/_thumb_10415.png" style="border: 0px solid black" /></a></span> <br/> This looks reasonable to me.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> — in case it's not obvious, I've just been talking to <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=aritter" class="user-hover" rel="aritter">aritter</a> and he points out that the review of this ticket involves not just understanding his figures and agreeing that they indicate that there's nothing to worry about here, but also reviewing the outstanding PR on GitHub.</p> <p>Please ignore the ticket branch for obs_pfs as this is superseeded by <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-80" title="Task-ify code for creating normalized flats" class="issue-link" data-issue-key="PIPE2D-80"><del>PIPE2D-80</del></a>. I have filed a ticket <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-151" title="Check obs_pfs tickets/PIPE2D-77.bak for changes worth keeping" class="issue-link" data-issue-key="PIPE2D-151"><del>PIPE2D-151</del></a> to go over this branch and look for code changes worth keeping.</p> <p>As for what is going with this ticket in general: I used the Flat with visit number 104 to generate a normalized Flat. This is done by calculating the spatial profile, optimally extract the spectrum by scaling the spatial profile to each row in the FiberTrace, reconstruct the (smooth) FiberTrace image by multiplying the spatial profile with the spectrum, and dividing the original FiberTrace image by the reconstructed one. The spatial profile is calculated from the Flat rows in a swath containing 500 rows. The swaths overlap (swath 1: rows 0-499, swath 2: rows 255-754,...). Within each swath the rows are (noisy) dithered representations of the spatial profile, scaled with the flux in the dispersion element. These dithered representations are normalized to a sum of 1 and binned with a bin width of <tt>1/oversample</tt> (with <tt>oversample=10</tt>). For each bin center, the mean value (with sigma rejection iterations) is calculated, and fitted with a cubic spline.<br/> The profile calculated for each swath is assumed true for the center of the swath. In between the swath's centers the profile is interpolated from the 2 adjacent profiles.<br/> More comments to follow...</p> <p>I've made some comments about the code on the GitHub PR. I still need to grok what it is you're doing because I'm still getting my head around the algorithms. It might be helpful if you could make some "before" plots to match the "after" ones you've already posted, so we can see the problem along with its solution.</p> <p>Here are some images which illustrate the problems we saw:</p> <ul> <li>systematics in the pixels which are the farthest away from the center of the FiberTrace, just before a curved FiberTrace crosses over to another column:<br/> before: <span class="image-wrap" style=""><a id="10419_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10419/10419_curvedFiberTraceNormalizedFlatFromDRP_STELLA_DATA_profile_wider.png" title="curvedFiberTraceNormalizedFlatFromDRP_STELLA_DATA_profile_wider.png" file-preview-type="image" file-preview-id="10419" file-preview-title="curvedFiberTraceNormalizedFlatFromDRP_STELLA_DATA_profile_wider.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10418/_thumb_10418.png" style="border: 0px solid black" /></a></span> <br/> after: <span class="image-wrap" style=""><a id="10420_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10420/10420_straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_old.png" title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_old.png" file-preview-type="image" file-preview-id="10420" file-preview-title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_old.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10419/_thumb_10419.png" style="border: 0px solid black" /></a></span><br/> The solution was to make the FiberTrace wider. I also improved the profile determination for the most outer pixels by extrapolating the cubic spline outside the limits where the spline was defined. These limits are set by the bin centers of the oversampled profile. Originally I just used the values of the most outer bins.</li> <li>systematics in the outer pixels of the normalized Flat, especially in the straight FiberTraces. I only just discovered this when I prepared the images. I will move <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-77" title="Make sure there are no systematics in the normalized flat" class="issue-link" data-issue-key="PIPE2D-77"><del>PIPE2D-77</del></a> back into progress and put the fix into a new commit:<br/> before: <span class="image-wrap" style=""><a id="10421_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10421/10421_straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_old_pixel_distribution.png" title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_old_pixel_distribution.png" file-preview-type="image" file-preview-id="10421" file-preview-title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_old_pixel_distribution.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10420/_thumb_10420.png" style="border: 0px solid black" /></a></span> <span class="image-wrap" style=""><a id="10422_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10422/10422_straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_new.png" title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_new.png" file-preview-type="image" file-preview-id="10422" file-preview-title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_new.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10421/_thumb_10421.png" style="border: 0px solid black" /></a></span> <br/> after: <span class="image-wrap" style=""><a id="10423_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10423/10423_straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_new_pixel_distribution.png" title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_new_pixel_distribution.png" file-preview-type="image" file-preview-id="10423" file-preview-title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_new_pixel_distribution.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10422/_thumb_10422.png" style="border: 0px solid black" /></a></span> <span class="image-wrap" style=""><a id="10425_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10425/10425_straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_overSample10_minSNR10.png" title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_overSample10_minSNR10.png" file-preview-type="image" file-preview-id="10425" file-preview-title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_overSample10_minSNR10.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10423/_thumb_10423.png" style="border: 0px solid black" /></a></span> <br/> The problem was that the dithered FiberTraces were extracted using the <tt>pfsFiberTrace</tt> from the non-dithered Flat. The fix was to trace the FiberTraces for all (dithered) Flats individually.</li> </ul> <p>RHL suggests:<br/> Add 2 more pixels far out on both sides and set them to 0 to constrain the spline extrapolation.<br/> Calculate ChiSquared, show residual divided by variance ("Chi")</p> <p>Some more comments about the strategy for the construction of the master Flat from the dithered Flats:</p> <ul> <li>Only <tt>FiberTrace</tt> s with zero shift in x are extracted using the <tt>pfsFiberTrace</tt>, all others are traced before the extraction.</li> <li>For each <tt>FiberTrace</tt>, all dithered realizations are added into an empty image to construct a single, wide FiberTrace representation.</li> <li>The same is done for their reconstructions, meaning that each dithered realization is reconstructed individually.</li> <li>The sum of the original dithered <tt>FiberTrace</tt> is divided by the sum of the reconstructed ones to create a normalized wide <tt>FiberTrace</tt> Flat.</li> <li>Pixels with a SNR less than <tt>minSNR</tt> (config parameter) are set to Unity</li> <li>The normalized <tt>FiberTrace</tt> is extracted from the image using the <tt>pfsFiberTrace</tt>, and put into the final master Flat</li> </ul> <p>I was thinking that maybe it's better/faster, to trace, extract, and reconstruct the wide sum of the dithered <tt>FiberTrace</tt> representations, instead of reconstructing each dithered <tt>FiberTrace</tt> individually. This version is implemented in a different branch now (obs_pfs <a href="https://github.com/Subaru-PFS/obs_pfs/tree/tickets/PIPE2D-77_1profile" class="external-link" rel="nofollow">tickets/PIPE2D-77_1profile</a>). Both versions are working fine, but I didn't exactly compare and profile them yet. I filed another ticket to compare both versions for quality and speed (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-154" title="Compare normalized FiberFlats using many profiles to one profile" class="issue-link" data-issue-key="PIPE2D-154"><del>PIPE2D-154</del></a>).</p> <p>A last (hopefully) systematic I forgot to mention earlier in the wings of the spatial profile, where the model was ~10% too low. The fix was to turn up the oversampling factor (config parameter <tt>config.profile.overSample</tt>) from 10 to 50-100, reducing the bin size for the profile calculation to 1/50-1/100 of a pixel:</p> <ul> <li>before:<br/> <span class="image-wrap" style=""><a id="10426_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10426/10426_straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_overSample100_minSNR10.png" title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_overSample100_minSNR10.png" file-preview-type="image" file-preview-id="10426" file-preview-title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_overSample100_minSNR10.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10425/_thumb_10425.png" style="border: 0px solid black" /></a></span> </li> <li>after (with 100 times oversampling):<br/> <span class="image-wrap" style=""><a id="10426_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/10426/10426_straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_overSample100_minSNR10.png" title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_overSample100_minSNR10.png" file-preview-type="image" file-preview-id="10426" file-preview-title="straightFiberTraceNormalizedFlatFromDRP_STELLA_DATA_overSample100_minSNR10.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/10426/_thumb_10426.png" style="border: 0px solid black" /></a></span> </li> </ul> <p>Please review the latest commits in <tt>drp_stella</tt> and the branch <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-77" title="Make sure there are no systematics in the normalized flat" class="issue-link" data-issue-key="PIPE2D-77"><del>PIPE2D-77</del></a></tt> in <tt>obs_pfs</tt></p> <p>I added review comments to the PR on obs_pfs. Nothing further to add to drp_stella.</p> <p>Merged into master</p> Relates PIPE2D-155 PIPE2D-212 PIPE2D-151 PIPE2D-154 PIPE2D-160 Development Rank 0|ii00p3: Reviewers price Sprint 2014-17