PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 [PIPE2D-933] Fix normalisation of extracted spectra https://pfspipe.ipmu.jp/jira/browse/PIPE2D-933 DRP 2-D Pipeline <p>Extracted spectra (<tt>pfsArm</tt> files) have normalisation of unity. They should have the normalisation of the fiberTrace.</p> PIPE2D-933 Fix normalisation of extracted spectra Story Normal Done Done price price Thu, 4 Nov 2021 19:41:37 +0000 Mon, 8 Nov 2021 19:47:09 +0000 Mon, 8 Nov 2021 19:47:09 +0000 0 3 <p>Tracked this to the normalisation of the fiberProfiles, which fell out by accident in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-904" title="Add aperture corrections" class="issue-link" data-issue-key="PIPE2D-904"><del>PIPE2D-904</del></a>. Made a bunch of other small fixes in the process of regenerating the calibs.</p> <p>New calibs are in <tt>/projects/HSC/PFS/Subaru/CALIB-PFI-20211105</tt>. I haven't had time to regenerate the SuNSS calibs yet, but the code needs to merge.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> PIPE2D-933: Fix normalisation of extracted spectra mergeArms.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/sky --id visit=68878^68879^68881 -j 20 -c doSubtractSky1d=False --no-versions reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/sky -c isr.doFlat=False --id visit=68878 arm=r --no-versions &gt;&gt;&gt; from lsst.daf.persistence <span class="code-keyword">import</span> Butler &gt;&gt;&gt; butler = Butler(<span class="code-quote">"/projects/HSC/PFS/Subaru"</span>, calibRoot=<span class="code-quote">"CALIB-PFI-20211026"</span>) &gt;&gt;&gt; profiles = butler.get(<span class="code-quote">"fiberProfiles"</span>, visit=68878, arm=<span class="code-quote">"r"</span>) &gt;&gt;&gt; profiles[7].norm &gt;&gt;&gt; Profile normalisations aren't set!? They fell out by accident in PIPE2D-904. Gonna need some <span class="code-keyword">new</span> calibs. (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ mkdir CALIB-PFI-20211105 (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 /projects/HSC/PFS/Subaru/CALIB-SuNSS/{BIAS,DARK}/*-[brn]?.fits --mode=copy --validity 3600 (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 ~/pfs/drp_pfs_data/detectorMap/detectorMap-sim-??.fits (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/bootstrap --flatId visit=68351 arm=r --arcId visit=68100 arm=r -C bootstrap-pfi.py -c spectralOffset=-5 bootstrap INFO: Found 23615 lines in 596 traces bootstrap INFO: Matched 20916 lines bootstrap INFO: Median difference from detectorMap: -3.129519,-0.945394 pixels bootstrap INFO: Fit 8893/10451 points, rms: x=0.049871 y=0.068376 total=0.037381 pixels bootstrap INFO: Updating detectorMap... bootstrap INFO: Median difference from detectorMap: -3.785663,0.140286 pixels bootstrap INFO: Fit 8556/10465 points, rms: x=0.093251 y=0.074023 total=0.051825 pixels bootstrap INFO: Updating detectorMap... (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/bootstrap --flatId visit=68351 arm=b --arcId visit=68100 arm=b -C bootstrap-pfi.py -c spectralOffset=-5 --clobber-config --no-versions bootstrap INFO: Found 10657 lines in 596 traces bootstrap INFO: Matched 10629 lines bootstrap INFO: Median difference from detectorMap: 12.159949,-1.086394 pixels bootstrap INFO: Fit 4431/5298 points, rms: x=0.043972 y=0.056688 total=0.028492 pixels bootstrap INFO: Updating detectorMap... bootstrap INFO: Median difference from detectorMap: -0.923853,-4.523398 pixels bootstrap INFO: Fit 4588/5331 points, rms: x=0.052255 y=0.058974 total=0.031261 pixels bootstrap INFO: Updating detectorMap... (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-933/pfi/bootstrap/DETECTORMAP/pfsDetectorMap-0*.fits constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/calibs --id visit=68345^68350^68351 --cores 6 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3 repair.doCosmicRay=False ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-933/pfi/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits --config clobber=True reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/calibs --id visit=68096..68100 -j 10 -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False b: reduceArc.fitDetectorMap INFO: Final fit: chi2=91276.262056 dof=28574 xRMS=0.030331 yRMS=0.030932 (0.002109 nm) from 14317/21351 lines reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=65748.464411 xRMS=0.052390 yRMS=0.051339 (0.003500 nm) from 2372 lines (10.0%) reduceArc.fitDetectorMap INFO: Softening errors by x=0.028431, y=0.030154 pixels (0.002056 nm) to yield chi^2/dof=1 reduceArc.fitDetectorMap INFO: Softened fit: chi2=28261.310354 dof=28574 xRMS=0.035210 yRMS=0.037056 (0.002526 nm) from 14317 lines reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=17768.151862 xRMS=0.052306 yRMS=0.050377 (0.003434 nm) from 2372 lines r: reduceArc.fitDetectorMap INFO: Final fit: chi2=262729.727681 dof=114342 xRMS=0.037429 yRMS=0.036734 (0.003172 nm) from 57201/68817 lines reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=162889.828353 xRMS=0.058592 yRMS=0.056103 (0.004844 nm) from 7646 lines (10.0%) reduceArc.fitDetectorMap INFO: Softening errors by x=0.034156, y=0.033141 pixels (0.002861 nm) to yield chi^2/dof=1 reduceArc.fitDetectorMap INFO: Softened fit: chi2=113618.567835 dof=114342 xRMS=0.049241 yRMS=0.047806 (0.004128 nm) from 57201 lines reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=36502.919065 xRMS=0.058728 yRMS=0.056225 (0.004854 nm) from 7646 lines ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-933/pfi/calibs/DETECTORMAP/pfsDetectorMap-068096-*.fits --config clobber=True constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/calibs --id visit=68345^68350^68351 --cores 6 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3 repair.doCosmicRay=False ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-933/pfi/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits --config clobber=True reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/calibs --id visit=68096..68100 -j 10 -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False b: reduceArc.fitDetectorMap INFO: Final fit: chi2=90290.016120 dof=28424 xRMS=0.030334 yRMS=0.030778 (0.002098 nm) from 14242/21348 lines reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=64151.511007 xRMS=0.050466 yRMS=0.051126 (0.003486 nm) from 2372 lines (10.0%) reduceArc.fitDetectorMap INFO: Softening errors by x=0.028370, y=0.029792 pixels (0.002031 nm) to yield chi^2/dof=1 reduceArc.fitDetectorMap INFO: Softened fit: chi2=28128.336779 dof=28424 xRMS=0.035163 yRMS=0.036727 (0.002504 nm) from 14242 lines reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=17812.028153 xRMS=0.050841 yRMS=0.050537 (0.003445 nm) from 2372 lines r: reduceArc.fitDetectorMap INFO: Final fit: chi2=262415.216414 dof=114292 xRMS=0.037372 yRMS=0.036730 (0.003171 nm) from 57176/68829 lines reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=101206.611769 xRMS=0.059672 yRMS=0.055131 (0.004760 nm) from 7648 lines (10.0%) reduceArc.fitDetectorMap INFO: Softening errors by x=0.034177, y=0.033090 pixels (0.002857 nm) to yield chi^2/dof=1 reduceArc.fitDetectorMap INFO: Softened fit: chi2=113585.647857 dof=114292 xRMS=0.049257 yRMS=0.047749 (0.004123 nm) from 57176 lines reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=22745.784108 xRMS=0.059856 yRMS=0.055236 (0.004769 nm) from 7648 lines ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-933/pfi/calibs/DETECTORMAP/pfsDetectorMap-068096-*.fits --config clobber=True cp -r CALIB-PFI-20211105 /projects/HSC/PFS/Subaru reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/sky -c isr.doFlat=False --id visit=68878^68879^68881 -j 20 --longlog | tee pfi.log mergeArms.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/sky --id visit=68878^68879^68881 -j 20 -c doSubtractSky1d=False --no-versions </pre> </div></div> <p>Proposed changes look fine overall, but unit tests currently fail. Provided suggestions in <a href="https://github.com/Subaru-PFS/drp_stella/pull/228" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/pull/228</a>. Once those are fixed, feel free to merge/close ticket at your convenience.</p> <blockquote><p>y = 2945<br/> for x in <span class="error">&#91;97, 692, 711, 2602&#93;</span>:<br/> print(f"(<br/> Unknown macro: {x}<br/> ,<br/> Unknown macro: {y}<br/> )<br/> Unknown macro: {detMap.findFiberId(geom.PointD(x, y))}<br/> ")</p></blockquote> <p>returns</p> <blockquote><p>(97, 2945) 315<br/> (692, 2945) 315<br/> (711, 2945) 315<br/> (2602, 2945) 231</p></blockquote> <p>so the fiberIds are wrong somewhere – those are not all fibre 315(the right side of the chip seems fine)</p> <p>The extraction seems to somehow be inserting NaN into the calexp. Note how the white lines of NaN only appear where there are fibres, so I don't think it's a problem in the calibs (and it isn't in the raw).</p> <blockquote> <p>reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211105 --rerun rhl/pfi -c doWriteCalexp=True isr.doFlat=False isr.doDefect=False isr.doDark=False repair.cosmicray.keepCRs=False repair.interp.modelPsf.defaultFwhm=2.25 --id visit=68878</p></blockquote> <p> <span class="image-wrap" style=""><a id="14443_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/14443/14443_calExp-SA068878r1.png" title="calExp-SA068878r1.png" file-preview-type="image" file-preview-id="14443" file-preview-title="calExp-SA068878r1.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/14443/_thumb_14443.png" style="border: 0px solid black" /></a></span> </p> <p>The <tt>findFiberId</tt> bug appears to be a problem with <tt>SplinedDetectorMap::findFiberId</tt>, because <tt>butler.get("detectorMap", visit=68878, arm="r")</tt> gets the bootstrapped detectorMap rather than the proper fitted <tt>DoubleDetectorMap</tt>.</p> <p>I'm going to create some new calibs for PFI and SuNSS, and then merge this branch. I've created new tickets to deal with the two issues <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> has identified: <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-936" title="SplinedDetectorMap::findFiberId is broken" class="issue-link" data-issue-key="PIPE2D-936"><del>PIPE2D-936</del></a> (<tt>findFiberId</tt>) and <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-937" title="Extraction is producing NaNs in the calexp" class="issue-link" data-issue-key="PIPE2D-937"><del>PIPE2D-937</del></a> (<tt>NaN</tt> values in the image).</p> <p>Changed my mind: merged this work, and will create new calibs under <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-939" title="Build new calibs" class="issue-link" data-issue-key="PIPE2D-939"><del>PIPE2D-939</del></a>.</p> Blocks PIPE2D-773 PIPE2D-918 Development Rank 0|zzs9ul:x Reviewers hassan Sprint 2DDRP-2021 A11 Story Points 1.0