https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE1D-63 DRP 1D pipeline <p>The latest weekly test shows there seems to be the redshift measurement failure. The <em>Z</em> feature looks like what was seen in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE1D-51" title="Bad redshift determination with version 0.22.0?" class="issue-link" data-issue-key="PIPE1D-51"><del>PIPE1D-51</del></a>. So, this may be due to something in the mask treatment. I'll take a look into this in more details but please investigate the issue in your side.</p> <p><a href="https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e_extended/check_results/check_redshift_determination_w.2022.29_0.32.1+_0.32.1.pdf" class="external-link" rel="nofollow">https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e_extended/check_results/check_redshift_determination_w.2022.29_0.32.1+_0.32.1.pdf</a></p> <p> </p> PIPE1D-63

Bad redshift measurements again?

Story Normal Done Done Ali Allaoui Kiyoto Yabe Wed, 20 Jul 2022 10:59:06 +0000 Thu, 1 Dec 2022 01:43:44 +0000 Thu, 1 Dec 2022 01:43:44 +0000 0 6 <p>The problem seems to arise from a more noisy red part of the spectrum (see <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/15236/15236_weekly_20220718.png" title="weekly_20220718.png attached to PIPE1D-63">weekly_20220718.png^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> as compared to <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/15237/15237_weekly_20220708.png" title="weekly_20220708.png attached to PIPE1D-63">weekly_20220708.png^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>), and the noise seems to be underestimated (see <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/15235/15235_weekly_20220708_zoom.png" title="weekly_20220708_zoom.png attached to PIPE1D-63">weekly_20220708_zoom.png^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>) , hence it modifies the balance between solutions and produces errors</p> <p>There was a fix to the variances applied during sky subtraction made as part of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1055" title="Fix weekly failure 2022-06-12" class="issue-link" data-issue-key="PIPE2D-1055"><del>PIPE2D-1055</del></a> that merged on 2022-07-12. I was fairly sure that fix was an improvement, but I'll take another look.</p> <p>The attachment <tt>pfsObject-00006-00000-0,0-00000000000103dd-002-0x11fcdfd13c364618.jpg</tt> shows the pfsObject spectrum of interest from the 2022-07-17 weekly that ran on Tiger. The black line is the flux, while the two dotted red lines are flux +/- sqrt(variance). The divot around 680 nm is a 3-sigma deviation for a single pixel, but that can't be statistically significant as a line detection. I failed to find any problems in the <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1055" title="Fix weekly failure 2022-06-12" class="issue-link" data-issue-key="PIPE2D-1055"><del>PIPE2D-1055</del></a> fixes. I think I need a more detailed description about the problem before I can track it down on the 2D side.</p> <p>I have put another zoomed view, where there is a huge spike (about 1000 times above mean value) which is as well around 3sigma, and which might an effect on the continuum.</p> <p>I have also identified a star spectrum where the variance is obviously wrong especially around 10000A :  <span class="image-wrap" style=""><a id="15241_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15241/15241_Significant_spikes.png" title="Significant_spikes.png" file-preview-type="image" file-preview-id="15241" file-preview-title="Significant_spikes.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15240/_thumb_15240.png" style="border: 0px solid black" /></a></span></p> <p>yet another example of significant features that are mistaken for lines :  <span class="image-wrap" style=""><a id="15242_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15242/15242_Mask_10000A.png" title="Mask_10000A.png" file-preview-type="image" file-preview-id="15242" file-preview-title="Mask_10000A.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15241/_thumb_15241.png" style="border: 0px solid black" /></a></span></p> <p>There are enough pixels in the spectra that 3 and 4 sigma single-pixel spikes will be plentiful. I don't think there's anything we can do about that. That star spectrum looks problematic though.</p> <p>The problems with the star spectrum are due to overzealous CR masking, which is a known problem. The simulator produces sharper features than the actual instrument does, so fiber traces often get mis-identified as CRs. This is especially bad in the NIR, where there are lots of bright sky lines, and when working with bright targets. Filed <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1063" title="Tune CR parameters for simulator data" class="issue-link" data-issue-key="PIPE2D-1063"><del>PIPE2D-1063</del></a> to tune the CR parameters.</p> <p>Oh, I should point out that most of the pixels in the drop-out region between 1000 and 1020 nm are masked, but that is not apparent from your plots. It would be helpful if masked pixels were readily distinguishable (or not plotted at all) in your plots, if we are going to use them to find problems.</p> <p>As for the mask, we take them into account, the point is that the valid pixels are related by lines in out plots. That explains these straight lines over large intervals on this plot  <span class="image-wrap" style=""><a id="15243_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15243/15243_IsolatedPixelsGalaxies.png" title="IsolatedPixelsGalaxies.png" file-preview-type="image" file-preview-id="15243" file-preview-title="IsolatedPixelsGalaxies.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15242/_thumb_15242.png" style="border: 0px solid black" /></a></span></p> <p>but there are valid pixels with underestimated variance. If this is due to the specific star spectra problem then fine, but indeed we use the masks. More generally, I wonder if this is worth keeping single isolated pixels between large masked intervals, how sure can we be they are really valid ?</p> <p> </p> <p>It appears those isolated unmasked pixels with huge flux and underestimated variance also occur in galaxy spectra  <span class="image-wrap" style=""><a id="15245_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15245/15245_mask_w.2022.28.png" title="mask_w.2022.28.png" file-preview-type="image" file-preview-id="15245" file-preview-title="mask_w.2022.28.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15243/_thumb_15243.png" style="border: 0px solid black" /></a></span></p> <p>and then are mistaken for emission lines. </p> <p>Sorry for this basic question, but there are two mask columns (one in HDU2 and the other in HDU8). Which mask is <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=vlebrun" class="user-hover" rel="vlebrun">vlebrun</a> using after all discussion in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE1D-53" title="[0.24.0] Mask values no properly used" class="issue-link" data-issue-key="PIPE1D-53"><del>PIPE1D-53</del></a>? I found that HDU2 was very different between w.2022.28 and w.2022.29 (and after). No, we are just seeing there are almost no mask in blue and red? Anyway, I still don't quite understand the difference between HDU2 and HDU8 masks. </p> <p><span class="image-wrap" style=""><a id="15244_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15244/15244_mask_w.2022.29.png" title="mask_w.2022.29.png" file-preview-type="image" file-preview-id="15244" file-preview-title="mask_w.2022.29.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15245/_thumb_15245.png" style="border: 0px solid black" /></a></span> <span class="image-wrap" style=""><a id="15244_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15244/15244_mask_w.2022.29.png" title="mask_w.2022.29.png" file-preview-type="image" file-preview-id="15244" file-preview-title="mask_w.2022.29.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15244/_thumb_15244.png" style="border: 0px solid black" /></a></span></p> <p>drp1d_pipe pfsobject reader uses PfsObject class to read fits ( <a href="https://github.com/Subaru-PFS/drp_1dpipe/blob/master/drp_1dpipe/io/reader.py" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_1dpipe/blob/master/drp_1dpipe/io/reader.py</a>) and we use this reader to convert pfs objects files to our format in the weekly run.</p> <p>OK, thank you. So, you use basically pfsObject.mask which corresponds to HDU2 on the plots above, I guess. Then, I'm not sure if spectra are masked properly or not.</p> Relates PIPE2D-1063 PIPE1D-64 PIPE1D-65 Development Rank 0|zzsuc7: