https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/SIM2D-119 DRP 2-D Simulator <p>Upper panel shows cutout from the science image from calExp-FA000031r1.fits and lower panel shows cutout from the variance image. As we can see, variance image is very different within fiber traces and outside fibertrace. I doubt that this is wanted behavior? It is creating problems for <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naka.keigo" class="user-hover" rel="naka.keigo">Keigo Nakamura</a> when he wants to analyze PSF images in 2d, as the variance image outside fiber traces is very unrealistics.</p> SIM2D-119

Variance map outside fiber traces is very suspicious

Task Normal Open Unresolved rhl ncaplar Wed, 7 Aug 2019 21:48:41 +0000 Fri, 13 Jan 2023 12:52:05 +0000 0 3 <p>The variance is discontinuous because the flat-field is discontinuous: there is no flux outside the area of the trace.</p> <p>What specific problems is this causing?</p> <p>Ok, I suspected something similar. Thanks for clarifying.</p> <p>We encountered this problem when he was modeling PSF with PSFx (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-355" title="Model the 2D PSF using PSFEx" class="issue-link" data-issue-key="PIPE2D-355"><del>PIPE2D-355</del></a>), i.e., when he tried to work with 2D type algorithm on the simulated data and was cutting 40x40 and 20x20 post-stamp images. I never encountered this problem as I never look at the simulated data and only worked with ``real data''.</p> <p>But what is the problem?<br/> This effect will be present on both simulated and real data.</p> <p>This effect is not present in the real data. The flux outside traces is not zero in the real data and variance map behaves as expected. If this was present in the real data 2D modeling would be seriously hampered. You might remember, we were even tweaking numbers so that mean of the variance map is the equal to mean of the variance in the science image (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-278" title="Variance maps should accurately estimate noise in the images" class="issue-link" data-issue-key="PIPE2D-278"><del>PIPE2D-278</del></a>)? I think we are misunderstanding each other here?</p> <p>Here's the image (left), mask (center) and variance (right) planes for a part of a <tt>calexp</tt> produced with the command <tt>detrend.py DATA --calib DATA/CALIB --rerun sim2d-119 --id visit=21460 arm=r</tt>. I think this shows the same features you're pointing out in the simulated data. If I'm mistaken, could you please be more specific in identifying the problem?</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/12531/12531_sim2d-119.png" style="border: 0px solid black" /></span></p> <p>I am very confused. I have never seen this feature when running <tt>detrend.py</tt> on my side. </p> <p>But the variance value of a pixel should not change depending on the fact if the pixel is within a trace or outside a trace. Right? You are showing there is a clear difference in variance value for pixels within trace and outside trace and that <em>has</em> to be unwanted behaviour unless I am misunderstanding something.</p> <p>All the pixels outside the trace are masked out, and shouldn't be used. We have no variance estimate for them in the flat, but they get set to 1. I could set them to a variance of zero, but the drp_stella code we've inherited wouldn't like that. Alternatively, I could set them to a variance of infinity, but I think <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> has argued against that in the past. What value do you think they should be set to?</p> <p>Note also that the flat-field outside the traces is set to 1 everywhere, so if you're flat-fielding donut images with the fiber flats, you're probably messing up the images.</p> <p>How is the computation of variance of pixels outside of trace different than the computation of variance of pixels inside of trace?</p> <p>Pixels outside of trace are still needed for estimates of scattering etc... </p> <p>It is not possible to flat field 2d images for me as we do not have traces (as you correctly pointed out)</p> <p>I guess why I never noticed this is because I turned off flatfielding when doing <tt>detrend</tt> ?</p> <p>That's possible.</p> <p>I suggest we close this ticket.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> Is the data that you generate above with command <tt>detrend.py DATA --calib DATA/CALIB --rerun sim2d-119 --id visit=21460 arm=r</tt> avaliable on Tiger somewhere?</p> <p>My data repo is <tt>/scratch/pprice/sim2d-119/DATA</tt>.</p> <p>Could this have been a rerun from the "official" data repo?</p> <p>I am not aware of an "official" data repo.</p> <p>We are supposed to have all the data on tiger, so that would be the official one.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a>, did you decide how you want to handle this? The problem is that pixels outside the trace have zero weight when normalising by the trace.</p> Development Rank 0|02qpjd:00r20060i2001800i Sprint 2DDRP-2019 H 2DDRP-2021 A 2 Story Points 2.0