https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1221 DRP 2-D Pipeline <p>During the 2023 Apr/May engineering run, we obtained data from which to generate fiber flats for n1. Build fiber flats and install in a new calib repository (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1216" title="Create calibs in support of 2023 Apr/May engineering run" class="issue-link" data-issue-key="PIPE2D-1216"><del>PIPE2D-1216</del></a>).</p> PIPE2D-1221

Create fiber flats from 2023 Apr/May engineering data

Story Normal Done Done price price Fri, 12 May 2023 17:52:45 +0000 Fri, 26 May 2023 15:51:50 +0000 Fri, 26 May 2023 15:51:50 +0000 0 2 <p>How does this relate to <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1194" title="Generate flat fields for n1" class="issue-link" data-issue-key="PIPE2D-1194">PIPE2D-1194</a>?</p> <p>That one says "based on lab flats". It may not be necessary once these are available, but I don't know the true goal of that ticket (looks like something to do with persistence).</p> <p>From my notes:</p> <blockquote> <p>Running the following with <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1205" title="Improve the ISR for the HgCdTe detectors on the 2023-04 run" class="issue-link" data-issue-key="PIPE2D-1205"><del>PIPE2D-1205</del></a>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> constructFiberFlat.py /work/drp --calib=/work/price/CALIB-202304-eng --rerun=price/pipe2d-1221 --id visit=91203..91235:2^91292..91339:2 arm=n spectrograph=1 --cores 50 -c profiles.profileSwath=2000 profiles.profileOversample=5 minSNR=20 profiles.pruneMinLength=3000 profiles.associationDepth=100 </pre> </div></div> <p>Having problems with transactions, and then it can't find a dark.<br/> There's a dark with a bad name: <tt>pfsDark-2023-04-19T16:32:16.617-000000-n1.fits</tt></p> <p>Let's build a new dark.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lsst-scipipe-3.0.0) price@pfsa-usr02-gb:/work/price/CALIB-202304-eng/DARK $ rm pfsDark-2023-04-19T16\:32\:16.617-000000-n1.fits sqlite&gt; DELETE FROM dark WHERE arm = <span class="code-quote">"n"</span>; sqlite&gt; DELETE FROM dark_visit WHERE arm = <span class="code-quote">"n"</span>; constructPfsDark.py /work/drp --calib /work/price/CALIB-202304-eng --rerun price/pipe2d-1221 --id visit=90792..90797 arm=n --batch-type=none </pre> </div></div> <p>That worked, and looks like the name is correct.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ingestPfsCalibs.py /work/drp --calib /work/price/CALIB-202304-eng /work/drp/rerun/price/pipe2d-1221/DARK/pfsDark-2023-04-19-090792-n1.fits --validity 1000 --mode=copy -c clobber=True </pre> </div></div> <p>Back to the flat.<br/> Final result looks good.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> sqlite&gt; DELETE FROM flat WHERE arm = <span class="code-quote">"n"</span>; sqlite&gt; DELETE FROM flat_visit WHERE arm = <span class="code-quote">"n"</span>; ingestPfsCalibs.py /work/drp --calib /work/price/CALIB-202304-eng /work/drp/rerun/price/pipe2d-1221/FLAT/pfsFlat-2023-04-23-091203-n1.fits --validity 1000 --mode=copy -c clobber=True </pre> </div></div></blockquote> <p>The new n1 flat doesn’t serve to flatten the data: the “flattened” data still has strong detector features visible. I’m thinking that this must be the result of persistence, but perhaps I’ve missed something?<br/> In the below plot, I’ve taken two points on the detector (one is deliberately selected to be on a strong positive feature in the flat), separated by 6 pixels (about one fiber) horizontally. I’ve found fiber traces on the image, and then used the traces and extracted spectra to create a model for the image. I’ve divided the image by the model to give a normalised image, and then plotted the mean and ratio of the values on the normalised image at the two points of interest. We have multiple images as the slit is dithered, giving many points, but there is not a consistent ratio of flux between the two points (despite the fixed distance between them in the spatial dimension). The black cross is the corresponding point for the flat-field image.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15632/15632_image+%288%29.png" width="900" style="border: 0px solid black" /></span></p> <p>Can we see some images too, please?</p> <p>In the below image, the unflattened image is shown on the left, and the flattened image is on the right.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15633/15633_91594n1_flat_noFlat.jpg" width="900" style="border: 0px solid black" /></span></p> <p>That's depressing. What does the flat look like?</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15634/15634_91594n1_noFlat_flattened_flat.jpeg" width="900" style="border: 0px solid black" /></span></p> <p>What's the ratio of the pre- and post-flattened images? The flat looks fine, so I don't understand why it didn't work better. With proper handling of 0/0</p> <p>The ratio of the pre- and post-flattened images is the flat.</p> <p>If you look carefully, the flat has removed high-frequency structure. The bright features are smeared out in the post-flattened image.</p> <p>Yes, the ratio should be the flat modulo bugs. However, something is wrong. If you take two quartz exposures and divide one by the other you should get something close to unity; right? Your findings suggest that the dithered flat code isn't equivalent to that procedure.</p> <p>The problem was due to the normalisation of the individual exposures, wherein we measure the fiber profile, extract spectra, and produce a model image which we use to normalise the exposure. This process naturally samples things nicely in the spatial dimension (through the combination of reducing every row of every fiber to a single number, and then dithering). The problem is that because we have been dividing by the actual spectra, the normalised image didn't contain any structure in the spectral dimension. I've been trying to deal with that by applying a median filter to each spectrum, and I like the results I get with a median filter of half-width 400 pixels. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> has suggested we normalise by an average spectrum, which will require future developement (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1226" title="Use average spectrum to normalise traces for flat fields" class="issue-link" data-issue-key="PIPE2D-1226">PIPE2D-1226</a>).</p> <p>I have installed the flats in <tt>CALIB-2023-04-v2</tt>.</p> Development Rank 0|zzsx6f: