https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-693 DRP 2-D Pipeline <p>The extracted spectra in the pfsArm file can be very unstable near the dichroic transitions; this is not surprising, but the errors are underestimated.  Please fix.</p> <p>This could be due to inaccurate fibre traces or to some other problem.</p> <p> </p> PIPE2D-693

Flux errors near the dichroic are underestimated

Story Normal Done Done price rhl Fri, 8 Jan 2021 17:38:27 +0000 Fri, 19 Feb 2021 21:18:14 +0000 Fri, 19 Feb 2021 21:18:14 +0000 0 2 <p>See e.g. <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/13313/_thumb_13313.png" style="border: 0px solid black" /></span> from the 2020-12-14 weekly.  The fits are a 5th-order spline but that is irrelevant here.</p> <p>That data is the <tt>b</tt> arm of visit 47, spectrograph 1</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> Could you please attach the script used to create this plot?</p> <p>I found a sky fiber (<tt>fiberId=350</tt>) with some crazy fluxes and a bad chi value (&gt; 100) between 643 and 645 nm (fairly clean region of sky, without strong lines). I tracked this down to a bad fiberProfile, due to not neglecting the <tt>BAD_FLAT</tt> mask plane.</p> <p>Bad chi: <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/13342/13342_pipe2d-693-chi-bad.png" style="border: 0px solid black" /></span></p> <p>Bad profile: <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/13345/13345_pipe2d-693.png" style="border: 0px solid black" /></span></p> <p>Fixed profile: <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/13344/13344_pipe2d-693-fixed.png" style="border: 0px solid black" /></span></p> <p>New chi: <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/13343/13343_pipe2d-693-chi.png" style="border: 0px solid black" /></span></p> <p>Notes:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> PIPE2D-693: Flux errors near the dichroic are underestimated (Had to hack repositoryCfg.yaml to replace /scratch/pprice/jenkins/weekly/2020-12-14 with /projects/HSC/PFS/saved/weekly-2020-12-14) def fitLine(xx, yy): xMean = xx.mean() yMean = yy.mean() dx = xx - xMean dy = yy - yMean xySum = np.sum(dx*dy) xxSum = np.sum(dx**2) slope = xySum/xxSum intercept = yMean - slope*xMean from types <span class="code-keyword">import</span> SimpleNamespace <span class="code-keyword">return</span> SimpleNamespace(slope=slope, intercept=intercept, xMean=xMean, yMean=yMean) <span class="code-keyword">import</span> numpy as np <span class="code-keyword">import</span> matplotlib.pyplot as plt from lsst.daf.persistence <span class="code-keyword">import</span> Butler from pfs.datamodel <span class="code-keyword">import</span> TargetType butler = Butler(<span class="code-quote">"/projects/HSC/PFS/saved/weekly-2020-12-14/process/rerun/weekly/pipeline/brn/pipeline"</span>) dataId = dict(visit=47, arm=<span class="code-quote">"b"</span>) pfsArm = butler.get(<span class="code-quote">"pfsArm"</span>, dataId) pfsConfig = butler.get(<span class="code-quote">"pfsConfig"</span>, dataId) skyIndices = pfsConfig.selectByTargetType(TargetType.SKY, pfsArm.fiberId) wlMin, wlMax = 641, 646 chiRms = [] <span class="code-keyword">for</span> ii in skyIndices: wlSelect = (pfsArm.wavelength[ii] &gt; wlMin) &amp; (pfsArm.wavelength[ii] &lt; wlMax) wl = pfsArm.wavelength[ii][wlSelect] flux = pfsArm.flux[ii][wlSelect] error = np.sqrt(pfsArm.variance[ii][wlSelect]) line = fitLine(wl, flux) fit = line.slope*wl + line.intercept chi = (flux - fit)/error lq, uq = np.percentile(chi, (25.0, 75.0)) rms = 0.741*(uq - lq) chiRms.append(rms) print(pfsArm.fiberId[ii], rms) _ = plt.plot(wl, chi, <span class="code-quote">"k-"</span>) plt.xlabel(<span class="code-quote">"Wavelength (nm)"</span>) plt.ylabel(r<span class="code-quote">"\chi"</span>) plt.show() fiberId=350 has a bad chi in the 641-646 region. pfsArm.plot([350]) shows a strange feature in that region. detrend.py /projects/HSC/PFS/saved/weekly-2020-12-14/process --calib /projects/HSC/PFS/saved/weekly-2020-12-14/process/CALIB --rerun price/pipe2d-693 --id visit=47 arm=b fiberId = 350 from lsst.afw.display <span class="code-keyword">import</span> Display ds9 = Display(backend=<span class="code-quote">"ds9"</span>, frame=1) butler = Butler(<span class="code-quote">"/projects/HSC/PFS/saved/weekly-2020-12-14/process/rerun/price/pipe2d-693"</span>) calexp = butler.get(<span class="code-quote">"calexp"</span>, dataId) detMap = butler.get(<span class="code-quote">"detectorMap"</span>, dataId) ds9.mtv(calexp) detMap.display(ds9, [fiberId], [wlMin, wlMax]) Looks clean. The peak in the spectrum is at y=3966, but there's nothing there in the image. Maybe the problem is in the fiberTrace? profiles = butler.get(<span class="code-quote">"fiberProfiles"</span>, dataId) ft = profiles[fiberId].makeFiberTraceFromDetectorMap(detMap, fiberId) ds9.mtv(ft.trace) Yes, there<span class="code-quote">'s a strange discontinuity in the fiberTrace image at y=3966, so that'</span>s the source of the problem. detrend.py /projects/HSC/PFS/saved/weekly-2020-12-14/process --calib /projects/HSC/PFS/saved/weekly-2020-12-14/process/CALIB --rerun price/pipe2d-693 --id visit=36^38 arm=b quartz = butler.get(<span class="code-quote">"calexp"</span>, visit=36, arm=<span class="code-quote">"b"</span>) ds9.mtv(quartz) detMap.display(ds9, [fiberId], [wlMin, wlMax]) Confirmed that the fiber of interest is on visit=36. No obvious problems apparent in the image. prof = profiles[fiberId] plt.plot(prof.norm, <span class="code-quote">"k-"</span>) plt.xlim(3966, 3982) plt.show() The normalisation is negative...? prof.plot() The <span class="code-keyword">final</span> sub-profile (y=4016) looks awful: no clear locus of points. It seems likely that <span class="code-keyword">this</span> is the cause. The calexp has a lot of BAD_FLAT masked pixels at the end, but that's not in the list of mask planes <span class="code-keyword">for</span> buildFiberProfiles... fixed. constructFiberProfiles.py /projects/HSC/PFS/saved/weekly-2020-12-14/process --calib=/projects/HSC/PFS/saved/weekly-2020-12-14/process/CALIB --rerun=price/pipe2d-693 --doraise --cores 1 --id visit=35^37 arm=b from pfs.drp.stella <span class="code-keyword">import</span> FiberProfileSet profiles = FiberProfileSet.readFits(<span class="code-quote">"/projects/HSC/PFS/saved/weekly-2020-12-14/process/rerun/price/pipe2d-693/FIBERPROFILES/pfsFiberProfiles-2020-11-26-000036-b1.fits"</span>) The <span class="code-keyword">final</span> sub-profile (now at y=3950) now looks MUCH nicer! There's still a discontinuity in the fiberTrace, now at y=4048. The normalisation is now noise around zero rather than crazy negative. Let<span class="code-quote">'s extract spectra with <span class="code-keyword">this</span>, and see <span class="code-keyword">if</span> it'</span>s improved... (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-693 $ combineFiberProfiles.py pfsFiberProfiles-2020-11-26-000035-b1.fits /projects/HSC/PFS/saved/weekly-2020-12-14/process/rerun/price/pipe2d-693/FIBERPROFILES/pfsFiberProfiles-2020-11-26-00003?-b1.fits profiles = FiberProfileSet(<span class="code-quote">"pfsFiberProfiles-2020-11-26-000035-b1.fits"</span>) traces = profiles.makeFiberTracesFromDetectorMap(detMap) spectra = traces.extractSpectra(calexp.maskedImage) <span class="code-keyword">for</span> ss in spectra: ss.setWavelength(detMap.getWavelength(ss.fiberId)) pfsArm = spectra.toPfsArm(dataId) Now when I plot chi between 643 and 645 nm, all values are less than 3, and generally less than 2: there are no wild outliers. </pre> </div></div> <p>More notes:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> RHL asks: &gt; Have you checked that the quoted errors across the dichroic are correct, in the sense that the red-side and blue-side measurements of the flux at a given wavelength are statistically consistent? (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-693 $ $PFS_PIPE2D_DIR/weekly/process_weekly.sh -r pipe2d-693 -D /scratch/pprice/pipe2d-693/fixed/ detrend.py /scratch/pprice/pipe2d-693/fixed --calib /scratch/pprice/pipe2d-693/fixed/CALIB --rerun pipe2d-693/pipeline/brn/pipeline --id visit=47 arm=b^r -j 2 <span class="code-keyword">import</span> numpy as np from lsst.daf.persistence <span class="code-keyword">import</span> Butler butler = Butler(<span class="code-quote">"/scratch/pprice/pipe2d-693/fixed/rerun/pipe2d-693/pipeline/brn/pipeline"</span>) blueId = dict(visit=47, arm=<span class="code-quote">"b"</span>) redId = dict(visit=47, arm=<span class="code-quote">"r"</span>) blue = butler.get(<span class="code-quote">"pfsArm"</span>, blueId) red = butler.get(<span class="code-quote">"pfsArm"</span>, redId) wlMin, wlMax = 643, 645 <span class="code-keyword">for</span> ii in range(len(red)): redSelect = (red.wavelength[ii] &gt; wlMin) &amp; (red.wavelength[ii] &lt; wlMax) redFlux = red.flux[ii][redSelect].mean() redErr = np.sqrt(red.variance[ii][redSelect]).mean() blueSelect = (blue.wavelength[ii] &gt; wlMin) &amp; (blue.wavelength[ii] &lt; wlMax) blueFlux = blue.flux[ii][blueSelect].mean() blueErr = np.sqrt(blue.variance[ii][blueSelect]).mean() diff = np.abs(redFlux - blueFlux) err = np.hypot(redErr, blueErr) nSigma = diff/err <span class="code-keyword">if</span> nSigma &gt; 3: print(ii, red.fiberId[ii], blue.fiberId[ii], redFlux, redErr, redSelect.sum(), blueFlux, blueErr, blueSelect.sum()) 151 160 160 0.004514443263991073 0.000411609902932711 23 0.04298878209533732 0.0013043212679074675 30 162 171 171 0.12044178672043664 0.0016886640203687363 23 0.13277716863815206 0.00340707000100194 31 163 172 172 0.13493079559900753 0.001793808020948718 23 0.156978592248888 0.0037948187510930113 31 327 364 364 0.1681084379102511 0.0022442062516512913 23 0.13056916431689652 0.0031925625966626276 30 337 374 374 0.12048133641788118 0.001687048880942079 23 0.16057112050003808 0.004201387857562435 31 392 433 433 0.12349895673745459 0.001663794545175998 23 0.18600130173594448 0.004604363788815017 31 396 437 437 0.0038277092440687304 0.0003792050218488275 23 0.1664399803512527 0.04771144933588916 30 419 460 460 0.12122264921626778 0.0016422057235593282 23 0.14605515479435868 0.0036308343087460503 31 index = 392 fiberId = blue.fiberId[index] <span class="code-keyword">import</span> matplotlib.pyplot as plt plt.plot(blue.wavelength[index], blue.flux[index], <span class="code-quote">"b-"</span>) plt.plot(red.wavelength[index], red.flux[index], <span class="code-quote">"r-"</span>) plt.xlabel(<span class="code-quote">"Wavelength (nm)"</span>) plt.ylabel(<span class="code-quote">"Flux (uncalibrated)"</span>) plt.show() There<span class="code-quote">'s a large bump in the blue that isn'</span>t there in the red. blueDetMap = butler.get(<span class="code-quote">"detectorMap"</span>, blueId) redDetMap = butler.get(<span class="code-quote">"detectorMap"</span>, redId) blueImage = butler.get(<span class="code-quote">"calexp"</span>, blueId) redImage = butler.get(<span class="code-quote">"calexp"</span>, redId) from lsst.afw.display <span class="code-keyword">import</span> Display blueDisplay = Display(backend=<span class="code-quote">"ds9"</span>, frame=1) blueDisplay.mtv(blueImage) blueDetMap.display(blueDisplay, [blue.fiberId[index]], [wlMin, wlMax]) redDisplay = Display(backend=<span class="code-quote">"ds9"</span>, frame=2) redDisplay.mtv(redImage) redDetMap.displindeay(redDisplay, [red.fiberId[index]], [wlMin, wlMax]) It<span class="code-quote">'s not visible in the image. It must be in the fiberTrace. It can'</span>t be in the flat, since I'm looking at the flattened image. profiles = butler.get(<span class="code-quote">"fiberProfiles"</span>, blueId) prof = profiles[fiberId] Display(backend=<span class="code-quote">"ds9"</span>, frame=2).mtv(prof.makeFiberTraceFromDetectorMap(blueDetMap, fiberId).trace) There's a negative feature that peaks around y=4004, x=-4 relative to the trace --&gt; 647.1 nm. The bump in the spectrum is peaked at 643.2 nm --&gt; y=3945. There<span class="code-quote">'s a smaller negative feature in the trace there too. But that more extreme negative feature doesn'</span>t appear to have done anything awful to the spectrum. Could it be due to some coupling to the neighbouring fibers? Nothing in their fiberTraces around the row of interest, and no flux stolen from their spectra. plt.plot(profiles[432].norm, <span class="code-quote">"r-"</span>) plt.plot(profiles[433].norm, <span class="code-quote">"k-"</span>) plt.plot(profiles[434].norm, <span class="code-quote">"b-"</span>) plt.show() The black line, which is <span class="code-keyword">for</span> the troublesome fiber, undercuts the line <span class="code-keyword">for</span> both the other fibers, at around the right place. Dividing by <span class="code-keyword">this</span> deficit is causing an excess in the extracted spectrum. plt.plot(profiles[434].norm/profiles[433].norm, <span class="code-quote">"k-"</span>) plt.show() Yes, there's the bump, peaking at 3945. constructFiberProfiles.py /scratch/pprice/pipe2d-693/fixed --calib=/scratch/pprice/pipe2d-693/fixed/CALIB --rerun=fiberProfiles --doraise --batch-type=smp --cores=10 --no-versions --clobber-config --id visit=37 butler = Butler(<span class="code-quote">"/scratch/pprice/pipe2d-693/fixed/rerun/fiberProfiles"</span>) exposure = butler.get(<span class="code-quote">"postISRCCD"</span>, visit=37, arm=<span class="code-quote">"b"</span>) profiles = FiberProfileSet.readFits(<span class="code-quote">"/scratch/pprice/pipe2d-693/fixed/rerun/fiberProfiles/FIBERPROFILES/pfsFiberProfiles-2020-01-01-000037-b1.fits"</span>) plt.plot(profiles[435].norm/profiles[433].norm, <span class="code-quote">"k-"</span>) plt.show() The bump is gone now. Not sure why. Maybe I got some versions wrong? Let's re-run everything after checking the versions. (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-693 $ $PFS_PIPE2D_DIR/weekly/process_weekly.sh -r pipe2d-693 -D /scratch/pprice/pipe2d-693/revised The bump is still there. constructFiberProfiles.py /scratch/pprice/pipe2d-693/revised --calib=/scratch/pprice/pipe2d-693/revised/CALIB --rerun=fiberProfiles --doraise --batch-type=smp --cores=10 --no-versions --clobber-config --id visit=35^37 arm=b combineFiberProfiles.py pfsFiberProfiles-2020-01-01-000000-b1.fits revised/rerun/fiberProfiles/FIBERPROFILES/pfsFiberProfiles-2020-01-01-00003*.fits Looks fine. Must be something to <span class="code-keyword">do</span> with the timing of the construction of the profiles. Maybe the detectorMap? reduceArc.py is run after constructFiberProfiles.py, so the official profiles must be created using the original detectorMap, which is slightly off. Let<span class="code-quote">'s turn on doBlindFind. In fact, it'</span>s on by <span class="code-keyword">default</span>. But the extraction of the spectra uses the old detectorMap, rather than the results of the blind find! Fix that. (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-693 $ $PFS_PIPE2D_DIR/weekly/process_weekly.sh -r pipe2d-693 -D /scratch/pprice/pipe2d-693/fix2 The bump is gone! merged = butler.get(<span class="code-quote">"pfsMerged"</span>, blueId) merged.plot([433]) Good! There's something at y=209, at wavelength=389nm, which might be a bit bluer than we care about (&lt;10% throughput). Increased profiles.centerFit.order=9 to get better fits to the traces and ran again as fix3. Ideally, we should fit a detectorMap and measure the profile at the same time (the detectorMap gives the best model of the traces, but the traces are used to extract the spectra, which is (currently) required to <span class="code-keyword">do</span> line identification. Looking <span class="code-keyword">for</span> red-blue differences &gt; 3 sigma on <span class="code-keyword">this</span> <span class="code-keyword">new</span> dataset, we find: 151 160 160 0.004270502681566373 0.0003893999866585177 23 0.03895871326810748 0.0012311061697475381 30 There's a huge spike in the blue, which is due to a CR visible in the image. Ready to call <span class="code-keyword">this</span> done. </pre> </div></div> <p>Not a lot of changes, but enough to make the fluxes in the dichroic consistent.</p> <p>Merged to master.</p> Development Rank 0|02qpt6:508yl0chzzy00006 Reviewers hassan Sprint 2DDRP-2021 A 2 Story Points 2.0