https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE1D-53 DRP 1D pipeline <p>The DRP-1D should use only 0 values of the mask provided by the DRP2D</p> PIPE1D-53

[0.24.0] Mask values no properly used

Bug Normal Done Done Ali Allaoui vlebrun Wed, 8 Sep 2021 15:20:39 +0000 Mon, 18 Oct 2021 02:05:55 +0000 Mon, 18 Oct 2021 02:05:55 +0000 0 4 <p>We have investigated further, and it happens that the 1D Pipeline actually uses the mask values provided by the conversion parser in the pipeline. We have actually discovered three issues from the file provided by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE1D-51" title="Bad redshift determination with version 0.22.0?" class="issue-link" data-issue-key="PIPE1D-51"><del>PIPE1D-51</del></a>Bad redshift determination with version 0.22.0? :</p> <ul> <li>A mask value is provided in two extensions : the #2 and the 4th column of the table in extension #8 (which is the resampled table), but the two values are not compatible (see attached pictures).</li> <li>In the pipeline, the #2 extension is used, but in this one, the pixels around 1229 nm (where a fake emission line is used to give a redshift measure) are not masked </li> <li>In the table of extension #8, the mask is &gt;0 above 600 nm, which would indicate that only the blue arm spectrum is useful.</li> </ul> <p>These problems explain the bad redshift measurement, and mask values have to be corrected</p> <p> </p> <p>Thank you for the investigation. What about this one (<span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/13979/13979_pfsObject-00001-00000-0%2C0-0000000000000318-006-0x7ace412dc05971ff.fits" title="pfsObject-00001-00000-0,0-0000000000000318-006-0x7ace412dc05971ff.fits attached to PIPE1D-53">pfsObject-00001-00000-0,0-0000000000000318-006-0x7ace412dc05971ff.fits^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>)? This is a product of the latest pipeline. The mask values look very different from the previous one.</p> <p>Indeed the values are different for the extension  #2 (#8 seems to be identical), did you try DRP-1D on this one? we'll try our version on our side</p> <p> </p> <p>The mask seems to be correct with respect to strong features around 1220-1230 nm</p> <p>and now we measure the correct redshift, which shows that everything seems OK. On the other side, the mask in the resampled data table are still weird. Should we raise a PIPE-2D ticket ?</p> <p>Yes, the recent comparison (<span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/13980/13980_check_redshift_determination.pdf" title="check_redshift_determination.pdf attached to PIPE1D-53">check_redshift_determination.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>) shows better results. The redshifts are consistent for that object. I still see some discrepancy. I'll take a look into which objects are so.</p> <p>Also here I invite <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> for discussion on mask values in pfsObject.</p> <p>the value of the std deviation of 0.2 indicates that there are still a large number of catastrophic failures. Let's check that ...</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=ali.allaoui" class="user-hover" rel="ali.allaoui">Ali Allaoui</a></p> <p>The latest weekly products (at IPMU) are here:</p> <p><a href="https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/check_redshift_determination.csv" class="external-link" rel="nofollow">https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/</a></p> <p>I also added reference redshift and <span class="error">&#91;OII&#93;</span> luminosity in csv file:</p> <p><a href="https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/check_redshift_determination.csv" class="external-link" rel="nofollow">https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/check_redshift_determination.csv</a></p> <p>Original input spectra are not in the weekly products but I can send you specific objects if you need them.</p> <p>I have recently discovered and fixed some bugs in the generation of the mask. I hope these fixes will make it to a weekly release soon.</p> <p>With the latest run (2D=w.2021.40 and 1D=0.24.0), the comparison between input and output redshifts is like this:</p> <p><span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/14126/14126_check_redshift_determination_20211006.pdf" title="check_redshift_determination_20211006.pdf attached to PIPE1D-53">check_redshift_determination_20211006.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>The result looks obviously strange. Could you check this in your side? pfsObjects are here:</p> <p><a href="https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/INTEGRATION/rerun/integration/pipeline/pfsObject/00001/00000/0,0/" class="external-link" rel="nofollow">https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/INTEGRATION/rerun/integration/pipeline/pfsObject/00001/00000/0,0/</a></p> <p>Input redshifts are here:</p> <p><a href="https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/check_redshift_determination.csv" class="external-link" rel="nofollow">https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/check_redshift_determination.csv</a></p> <p> </p> <p>For the spectrum I looked, the noise level does not make sense; it is about 10^12 while the spectrum has a (normal) 10^16 average value, hence the noise weighted fit cannot work properly. On this plot <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/14128/14128_Weekly_run_noise_level.png" title="Weekly_run_noise_level.png attached to PIPE1D-53">Weekly_run_noise_level.png^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> the good solution is found but only on the 3rd rank.</p> <p>The weekly tests in this morning showed that the comparisons</p> <p><a href="https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/check_redshift_determination.pdf" class="external-link" rel="nofollow">https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e/check_redshift_determination.pdf</a></p> <p><a href="https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e_extended/check_redshift_determination.pdf" class="external-link" rel="nofollow">https://pfs.ipmu.jp/internal/devarch/ipmu/pipe_e2e_extended/check_redshift_determination.pdf</a></p> <p>were fair enough. So I think we can close this ticket and post any further issues in different ticket (maybe <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE1D-38" title="Weekly run at LAM" class="issue-link" data-issue-key="PIPE1D-38">PIPE1D-38</a>?).</p> Blocks PIPE2D-913 Development Rank 0|zzs91c: