https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-278 DRP 2-D Pipeline <p>The noise in the reduced images is currently not correctly captured by the variance map. </p> <p>For an example, looking at the image 8603 and taking random patch of the image without sources gives variance of around 20. On the other hand, mean value in the variance image is around 36, 37. These results seem to be consistent across the whole image. </p> <p> </p> <p>Possible causes are 1. wrong gains, 2. given that there seems to be structure in the noise, wrong estimation of the readout, 3. other options.</p> <p> </p> PIPE2D-278

Variance maps should accurately estimate noise in the images

Bug Normal Done Done price ncaplar Fri, 20 Apr 2018 14:01:09 +0000 Tue, 11 Sep 2018 01:39:20 +0000 Tue, 11 Sep 2018 01:39:20 +0000 0 3 <p>Work is being done on <a href="https://jira.lsstcorp.org/browse/DM-14182" class="external-link" rel="nofollow">DM-14182</a>.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pprice@tiger-sumire:~/LSST/ip/isr (tickets/DM-14182=) $ detrend.py /tigress/HSC/PFS/LAM/ --calib /tigress/HSC/PFS/LAM/CALIB-PAP --rerun price/pipe2d-278-fix --id visit=8603 -c isr.doEmpiricalReadNoise=True ... detrend.isr INFO: Calculated empirical read noise <span class="code-keyword">for</span> amp 0: 4.423752 detrend.isr INFO: Calculated empirical read noise <span class="code-keyword">for</span> amp 1: 4.266983 detrend.isr INFO: Calculated empirical read noise <span class="code-keyword">for</span> amp 2: 4.403465 detrend.isr INFO: Calculated empirical read noise <span class="code-keyword">for</span> amp 3: 4.178715 detrend.isr INFO: Calculated empirical read noise <span class="code-keyword">for</span> amp 4: 3.960904 detrend.isr INFO: Calculated empirical read noise <span class="code-keyword">for</span> amp 5: 4.411193 detrend.isr INFO: Calculated empirical read noise <span class="code-keyword">for</span> amp 6: 5.042430 detrend.isr INFO: Calculated empirical read noise <span class="code-keyword">for</span> amp 7: 4.181415 </pre> </div></div> <p>Then, in python:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> &gt;&gt;&gt; <span class="code-keyword">import</span> lsst.afw.image &gt;&gt;&gt; exposure = lsst.afw.image.ExposureF(<span class="code-quote">"/tigress/HSC/PFS/LAM/rerun/price/pipe2d-278-fix/calExp/2017-12-15/v0008603/PFLAr1.fits"</span>) &gt;&gt;&gt; <span class="code-keyword">import</span> numpy as np &gt;&gt;&gt; np.<span class="code-keyword">var</span>(exposure.image.array[760:2960,640:1140]) 21.075794 &gt;&gt;&gt; np.mean(exposure.variance.array[760:2000,640:1140]) 19.159437 </pre> </div></div> <p>Should I also update the read noise in the detector? They don't seem far off, but it might be good to update them with the hardware.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> &gt;&gt;&gt; ff = pyfits.open(<span class="code-quote">"/home/lsst/pfs/obs_pfs/pfs/camera/r_1.fits"</span>) &gt;&gt;&gt; ff[1].data[<span class="code-quote">"readnoise"</span>] array([ 3.61, 3.78, 3.18, 2.95, 3.19, 3.8 , 4.51, 3.18]) </pre> </div></div> <p>Probably. With luck we will have time to figure out why the system is that much noisier in the real cryostat at LAM than in the lab dewar JHU.</p> <p>I've updated the hard-wired read noise values. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=ncaplar" class="user-hover" rel="ncaplar">ncaplar</a>, could you please verify this helps?</p> <p>The original values were:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> &gt;&gt;&gt; <span class="code-keyword">import</span> pyfits &gt;&gt;&gt; ff = pyfits.open(<span class="code-quote">"/home/lsst/pfs/obs_pfs/pfs/camera/r_1.fits"</span>) &gt;&gt;&gt; ff[1].data[<span class="code-quote">"readnoise"</span>] array([ 3.61, 3.78, 3.18, 2.95, 3.19, 3.8 , 4.51, 3.18]) </pre> </div></div> <p>They are now:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> array([ 4.423752, 4.266983, 4.403465, 4.178715, 3.960904, 4.411193, 5.04243 , 4.181415]) </pre> </div></div> <p>Just to have it here for documentation purposes, this fix did help and variance is now consistent with what we observe in the data.</p> <p>Merged to master.</p> Development Rank 0|ii04af: Reviewers ncaplar