https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1008 DRP 2-D Pipeline <p>The expected units for the <tt>pfsArm</tt> is electrons per pixel and <tt>pfsMerged</tt> files are electrons per nanometre.</p> <p>Please go step by step through the 2D DRP code and products, from the raw exposures to the <tt>pfsArm</tt> and <tt>pfsMerged</tt> files to confirm that the units of the latter two products are as expected.</p> <p>Please document the details of each step in this ticket.</p> <p>This is important in general <del>, and to help determine the cause of the b/r flux ratio discrepancy</del>.</p> PIPE2D-1008

Confirm units of pfsArm flux are electrons/pixel and pfsMerged electrons/nm

Task Normal Done Done arnaud.lefur hassan Tue, 22 Mar 2022 16:46:53 +0000 Fri, 19 May 2023 15:01:06 +0000 Fri, 19 May 2023 15:01:06 +0000 0 7 <p>Units will not be electrons per nm with the code as it currently is.<br/> Converting ADU to electrons requires specifying the parameter <tt>isr.doApplyGains=True</tt>.<br/> The code does not convert to flux density, due to the desire to that the extracted spectra have units matching that of the image.</p> <p>Note that "The expected units for the pfsArm and pfsMerged files are electrons per nanometre." is not specified in <tt>datamodel.txt</tt>. Rather, the flux units of <tt>pfsArm</tt> explicitly called out to be "counts".</p> <p>This is about the overall throughput, not the b/r discrepancy. </p> <p>Paul's comments are correct, but the comparison is still valid. We should be able to interpret the numbers in the pfsArm/pfsMerged files in terms of electrons/nm (and how to do so should be in the datamodel).</p> <p>So I spend a bit of time looking into this.<br/> starting from the raw image to the pfsMerged data.<br/> I've used my pseudo ISR, pseudo extraction code to compare with.<br/> The plots are for visit 72034 which should be moon with offset(d_ra=1 deg)<br/> I've selected a subset of 10 fibers from SUNSS_DIFFUSE.</p> <p>on the red arm: <br/> spectra : <span class="image-wrap" style=""><a id="15146_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15146/15146_blue-arm+gain-corrected.png" title="blue-arm gain-corrected.png" file-preview-type="image" file-preview-id="15146" file-preview-title="blue-arm gain-corrected.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15144/_thumb_15144.png" style="border: 0px solid black" /></a></span> <br/> ratio of extraction: <span class="image-wrap" style=""><a id="15147_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15147/15147_ratio+between+blue+extraction.png" title="ratio between blue extraction.png" file-preview-type="image" file-preview-id="15147" file-preview-title="ratio between blue extraction.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15145/_thumb_15145.png" style="border: 0px solid black" /></a></span><br/> the ratio is close enough to 1, so I think I understand well enough what's being done.</p> <p>on the blue arm:<br/> spectra: <span class="image-wrap" style=""><a id="15147_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15147/15147_ratio+between+blue+extraction.png" title="ratio between blue extraction.png" file-preview-type="image" file-preview-id="15147" file-preview-title="ratio between blue extraction.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15146/_thumb_15146.png" style="border: 0px solid black" /></a></span> <br/> ratio of extraction : <span class="image-wrap" style=""><a id="15149_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15149/15149_pfsArms_vs_pfsMerged.png" title="pfsArms_vs_pfsMerged.png" file-preview-type="image" file-preview-id="15149" file-preview-title="pfsArms_vs_pfsMerged.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15147/_thumb_15147.png" style="border: 0px solid black" /></a></span> <br/> here I'm being penalized by the a bad detectorMap on the blue end, which is useful to know, but not really the point here.</p> <p>Anyhow, from what I'm seeing, I confirm Paul statement, the pfsArm fluxes are in counts (or electrons if doApplyGains).</p> <p>Regarding, pfsMerged :<br/> <span class="image-wrap" style=""><a id="15152_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15152/15152_pfsArm_extraction_compared.png" title="pfsArm_extraction_compared.png" file-preview-type="image" file-preview-id="15152" file-preview-title="pfsArm_extraction_compared.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15149/_thumb_15149.png" style="border: 0px solid black" /></a></span> <br/> I end up with the same numbers if I divide the flux by the spectral resolution (nm per pixel) which gives me flux in electrons/nm <br/> <span class="image-wrap" style=""><a id="15153_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15153/15153_pixel_per_nm.png" title="pixel_per_nm.png" file-preview-type="image" file-preview-id="15153" file-preview-title="pixel_per_nm.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15150/_thumb_15150.png" style="border: 0px solid black" /></a></span><br/> This balance slightly the flux between the two arms, but the absolute value is now ~12 times higher.<br/> To get back on my feet, I just need to correct so that the totalFlux of the mergedArm stay constant.</p> <p>Looks good, but I'm still slightly confused.</p> <p>You're plotting the pfsArm files against wavelength, but not applying a Jacobian, so are you really plotting per row, but labelling the rows with the corresponding wavelength?</p> <p>What happened with 477 in the blue? A bad detector map everywhere? Which rerun/calib are you using for this work?</p> <p>In the pfsMerged files, the units should be electrons/nm which is easy to think about away from the dichroics. Is this what you observe? What is the factor of c. 12 coming from? pixels to nm?</p> <p><cite>You're plotting the pfsArm files against wavelength, but not applying a Jacobian, so are you really plotting per row, but labelling the rows with the corresponding wavelength</cite>?<br/> Right exactly, for the pfsArm, the x-axis can be either <em>range(4176)</em> (row number) or the matching wavelength for each row. </p> <p><cite>What happened with 477 in the blue? A bad detector map everywhere? Which rerun/calib are you using for this work</cite>?<br/> Do you mean fiber 6 ? I checked a bit closely and I don't understand what's happening with the measured flux, the detectorMap is actually pretty good for that fiber pretty much all the way. If you compare fiber 6 and 296 which appear to have a very similar throughput :<br/> <span class="image-wrap" style=""><a id="15152_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15152/15152_pfsArm_extraction_compared.png" title="pfsArm_extraction_compared.png" file-preview-type="image" file-preview-id="15152" file-preview-title="pfsArm_extraction_compared.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15152/_thumb_15152.png" style="border: 0px solid black" /></a></span> <br/> you can see that the dotted line are pretty much consistent, from the pipeline, 296 is bad on the blue end which I blame the detectorMap for but 6 is bad at the red end and the detectorMap is not that bad there.<br/> Also we can see small bands where the flux drops to 0, might be fiberProfiles ? <br/> I've used hassan's: <em>CALIB-SuNSS-20220322-dev</em> , rerun: <em>alefur/pipe2d-1008-new</em></p> <p><cite>In the pfsMerged files, the units should be electrons/nm which is easy to think about away from the dichroics. Is this what you observe? What is the factor of c. 12 coming from? pixels to nm</cite>?</p> <p>right, I just divided the pfsArm flux (e-/row) by the spectral resolution (nm/row) (or multiplied by this curve <span class="image-wrap" style=""><a id="15153_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15153/15153_pixel_per_nm.png" title="pixel_per_nm.png" file-preview-type="image" file-preview-id="15153" file-preview-title="pixel_per_nm.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15153/_thumb_15153.png" style="border: 0px solid black" /></a></span> )<br/> that should be e-/nm strictly speaking so I did not apply any normalization after that.</p> <p>We should get to the bottom of fibre 6 (sorry, I mis-read the legend). It could be a problem in the fiberTrace or the normalisation or ...</p> <p>The dropouts matter too. This is not unrelated to <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1015" title="Mask bit flags not propagated to pfsArm correctly" class="issue-link" data-issue-key="PIPE2D-1015"><del>PIPE2D-1015</del></a> &#8211; there should be a bit set wherever we don't accurately extract the flux.</p> <p>I still don't understand the pfsMerged case. The fluxes should be in e/nm and your values should agree with the pfsMerged everywhere except where the dichroic matters. So the blue and red side should agree, and not have any structure with lambda or arm. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a>: what am I missing?</p> Relates PIPE2D-1024 PIPE2D-799 DAMD-113 Development Rank 0|02qpjd:00r20060i2001800009 Sprint 2DDRP-2022 C 2DDRP-2022 D 2DDRP-2022 E 2DDRP-2022 F Story Points 4.0