https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1242 DRP 2-D Pipeline <p>Looking at the reduced spectrum of a bright calspec target (the processing tag is `edr2-20230621` with CALIB-2023-04-v3+w.2023.25), it was found that sky-subtraction in n-band was bad showing a sawtooth-like pattern. Investigate the reason of this.</p> <p>  <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15877/15877_pfsArm_v92557_f9.png" style="border: 0px solid black" /></span></p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15876/15876_pfsMerged_v92557_f9.png" style="border: 0px solid black" /></span></p> PIPE2D-1242

Investigate significant `sawtooth` in sky-subtracted spectra in n-band

Task Normal Done Done price Kiyoto Yabe Mon, 26 Jun 2023 21:02:43 +0000 Fri, 7 Jul 2023 18:31:11 +0000 Fri, 7 Jul 2023 18:31:11 +0000 0 5 <p>So I looked at this today, I've looked at the flat used to build fiberProfiles. <br/> As Paul pointed out on slack, you can also see those sawtooth features in the extracted spectra :<br/> <span class="image-wrap" style=""><a id="15882_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15882/15882_90845_n1_pfsArm-pfsMerged-boxCar.png" title="90845_n1_pfsArm-pfsMerged-boxCar.png" file-preview-type="image" file-preview-id="15882" file-preview-title="90845_n1_pfsArm-pfsMerged-boxCar.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15881/_thumb_15881.png" style="border: 0px solid black" /></a></span> <br/> When I do the extraction using the same detectorMap and my naive box-car code, I don't see those features, (my profiles are 5 pixels wide).<br/> <span class="image-wrap" style=""><a id="15884_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15884/15884_90845_n1_pfsArm%28flux%2Bnorm%29.png" title="90845_n1_pfsArm(flux+norm).png" file-preview-type="image" file-preview-id="15884" file-preview-title="90845_n1_pfsArm(flux+norm).png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15880/_thumb_15880.png" style="border: 0px solid black" /></a></span> <br/> So yes I think there is an issue in the fiberProfiles, I would try to regenerate wider profiles first. </p> <p>I don't understand your conclusion. The normalisation should recover the input spectra used to estimate it in the first place, so if it doesn't I'd have thought that that meant that there was a bug in the code.</p> <p>The previous plot just show that we're missing some flux and that's coming from the fiberProfiles.<br/> But actually in the range(1000, 1050 nm) that yabe-san were showing the extraction of the quartz isn't terrible.<br/> <span class="image-wrap" style=""><a id="15883_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15883/15883_90845_n1_pfsMerged%28flux%2Bnorm%29.png" title="90845_n1_pfsMerged(flux+norm).png" file-preview-type="image" file-preview-id="15883" file-preview-title="90845_n1_pfsMerged(flux+norm).png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15882/_thumb_15882.png" style="border: 0px solid black" /></a></span> </p> <p>Looking at the pfsArm file, if you compare pfsArm.flux and pfsArm.norm <br/> <span class="image-wrap" style=""><a id="15885_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15885/15885_90845_n1_pfsArm%28flux%2Bnorm%29_newProfiles.png" title="90845_n1_pfsArm(flux+norm)_newProfiles.png" file-preview-type="image" file-preview-id="15885" file-preview-title="90845_n1_pfsArm(flux+norm)_newProfiles.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15884/_thumb_15884.png" style="border: 0px solid black" /></a></span> <br/> Those two should basically match so I don't understand from where the normalisations come, I mean they are embedded in the profiles, so I know their provenance but I don't understand the numbers inside, maybe the profiles were generated using some old code ? It's puzzling.</p> <p> <span class="image-wrap" style=""><a id="15886_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15886/15886_90845_n1_pfsMerged%28flux%2Bnorm%29_newProfiles.png" title="90845_n1_pfsMerged(flux+norm)_newProfiles.png" file-preview-type="image" file-preview-id="15886" file-preview-title="90845_n1_pfsMerged(flux+norm)_newProfiles.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15883/_thumb_15883.png" style="border: 0px solid black" /></a></span> <br/> Same plot with the merged file, where we can see the smooth normalisation.</p> <p>So I've regenerated the profiles, and re-reduced the data, now the normalisation in the pfsArm makes sense :<br/> <span class="image-wrap" style=""><a id="15892_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15892/15892_92557_n1_pfsArm-pfsMerged_newProfiles.png" title="92557_n1_pfsArm-pfsMerged_newProfiles.png" file-preview-type="image" file-preview-id="15892" file-preview-title="92557_n1_pfsArm-pfsMerged_newProfiles.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15885/_thumb_15885.png" style="border: 0px solid black" /></a></span> </p> <p><del>But the merged spectra is wrong</del> (I withdraw that statement)<br/> <span class="image-wrap" style=""><a id="15891_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15891/15891_92557_n1_pfsArm-pfsMerged_newProfiles2.png" title="92557_n1_pfsArm-pfsMerged_newProfiles2.png" file-preview-type="image" file-preview-id="15891" file-preview-title="92557_n1_pfsArm-pfsMerged_newProfiles2.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15886/_thumb_15886.png" style="border: 0px solid black" /></a></span> </p> <p>With the new profile the sawtooth feature did disappear <br/> <span class="image-wrap" style=""><a id="15894_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15894/15894_91401_pfsMerged%28flux%2Bnorm%29.png" title="91401_pfsMerged(flux+norm).png" file-preview-type="image" file-preview-id="15894" file-preview-title="91401_pfsMerged(flux+norm).png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15892/_thumb_15892.png" style="border: 0px solid black" /></a></span> </p> <p>Full wavelength range : <br/> <span class="image-wrap" style=""><a id="16137_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/16137/16137_pfsMerged_v92557_f9_20230707.png" title="pfsMerged_v92557_f9_20230707.png" file-preview-type="image" file-preview-id="16137" file-preview-title="pfsMerged_v92557_f9_20230707.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15891/_thumb_15891.png" style="border: 0px solid black" /></a></span> </p> <p>I've also looked at 91401, which is another quartz taken on all 3 arms, (90845, used for the profiles generation is n-arm only) <br/> <span class="image-wrap" style=""><a id="15894_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15894/15894_91401_pfsMerged%28flux%2Bnorm%29.png" title="91401_pfsMerged(flux+norm).png" file-preview-type="image" file-preview-id="15894" file-preview-title="91401_pfsMerged(flux+norm).png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15894/_thumb_15894.png" style="border: 0px solid black" /></a></span> <br/> We expect ~=unity, it looks actually worse in the n band.</p> <p>Looking a bit closely at the mergeArms.py code, it seems to be there is a bug indeed, in normalizeSpectra function.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>for ss in spectra: badBitmask = ss.flags.get(*self.config.mask) for ii in range(len(ss)): dispersion = calculateDispersion(ss.wavelength[ii]) nn = interpolateFlux(ss.wavelength[ii], ss.norm[ii]/dispersion, wavelength, fill=0.0) </pre> </div></div> <p>interpolateFlux already applies the jacobian by default, so it looks like it's applied twice.</p> <p>The <tt>n1</tt> fiber profiles were created with the following command:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> constructFiberProfiles.py /work/drp --calib=/work/price/CALIB-2023-04-v1 --rerun=price/pipe2d-1221 --id visit=91014 arm=n spectrograph=1 --config isr.doFlat=True profiles.profileRadius=2 profiles.profileSwath=2000 profiles.profileOversample=5 --batch-type=none --no-versions </pre> </div></div> <p>It turns out that the profiles for n1 are just plain awful:<br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/16117/16117_badProfiles.png" width="700" style="border: 0px solid black" /></span></p> <p>I've no idea how that happened, but re-running the <tt>constructFiberProfiles</tt> command, I get much better profiles:<br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/16118/16118_goodProfiles.png" width="700" style="border: 0px solid black" /></span></p> <p>And the normalisation is much better too:<br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/16119/16119_goodNorm.png" width="700" style="border: 0px solid black" /></span></p> <p>I'm going to create a new <tt>CALIB</tt> directory with these profiles.</p> <p>Built a new calib directory:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> price@pfsa-usr02-gb:/work/drp $ cp -r CALIB-2023-04-v2 CALIB-2023-04-v3 price@pfsa-usr02-gb:/work/drp/CALIB-2023-04-v3/FIBERPROFILES $ rm pfsFiberProfiles-2023-04-20-090845-n1.fits price@pfsa-usr02-gb:/work/drp/CALIB-2023-04-v3 $ sqlite3 calibRegistry.sqlite3 <span class="code-quote">'DELETE FROM fiberProfiles WHERE arm = <span class="code-quote">"n"</span>; DELETE FROM fiberProfiles_visit WHERE arm = <span class="code-quote">"n"</span>;'</span> ingestPfsCalibs.py /work/drp --calib /work/drp/CALIB-2023-04-v3 /work/drp/rerun/price/pipe2d-1242/FIBERPROFILES/pfsFiberProfiles-2023-04-21-091014-n1.fits --validity 10000 -c clobber=True --mode=copy price@pfsa-usr02-gb:/work/drp $ ln -s CALIB-2023-04-v3 CALIB </pre> </div></div> <p>Looks like another place where even a simple QA process would be very helpful.</p> <p>The conversation about the code snippet <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a> pointed out has been moved to <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1248" title="Reconsider the scaling of fluxes in pfsMerged, and the use of Jacobians in interpolation" class="issue-link" data-issue-key="PIPE2D-1248"><del>PIPE2D-1248</del></a>.</p> <p>The entire process is still running but I checked that visit, which has been finished, that particular feature is gone now.</p> <p><span class="image-wrap" style=""><a id="16137_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/16137/16137_pfsMerged_v92557_f9_20230707.png" title="pfsMerged_v92557_f9_20230707.png" file-preview-type="image" file-preview-id="16137" file-preview-title="pfsMerged_v92557_f9_20230707.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/16137/_thumb_16137.png" style="border: 0px solid black" /></a></span></p> <p>Merged to master a small fix to <tt>FiberProfileSet.plot</tt>.</p> Development Rank 0|zzsx8g: Sprint Eng12July