[PIPE2D-933] Fix normalisation of extracted spectra Created: 05/Nov/21  Updated: 09/Nov/21  Resolved: 09/Nov/21

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File calExp-SA068878r1.png    
Issue Links:
Blocks
blocks PIPE2D-773 Make small improvements to utilities Done
blocks PIPE2D-918 Provide flux data for calibration lam... Done
Story Points: 1
Sprint: 2DDRP-2021 A11
Reviewers: hassan

 Description   

Extracted spectra (pfsArm files) have normalisation of unity. They should have the normalisation of the fiberTrace.

 Comments   
Comment by price [ 06/Nov/21 ]

Tracked this to the normalisation of the fiberProfiles, which fell out by accident in PIPE2D-904. Made a bunch of other small fixes in the process of regenerating the calibs.

New calibs are in /projects/HSC/PFS/Subaru/CALIB-PFI-20211105. I haven't had time to regenerate the SuNSS calibs yet, but the code needs to merge.

PIPE2D-933: Fix normalisation of extracted spectra

mergeArms.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/sky --id visit=68878^68879^68881 -j 20 -c doSubtractSky1d=False --no-versions

reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/sky -c isr.doFlat=False --id visit=68878 arm=r --no-versions


>>> from lsst.daf.persistence import Butler
>>> butler = Butler("/projects/HSC/PFS/Subaru", calibRoot="CALIB-PFI-20211026")
>>> profiles = butler.get("fiberProfiles", visit=68878, arm="r")
>>> profiles[7].norm
>>> 

Profile normalisations aren't set!? They fell out by accident in PIPE2D-904.

Gonna need some new calibs.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ mkdir CALIB-PFI-20211105
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 /projects/HSC/PFS/Subaru/CALIB-SuNSS/{BIAS,DARK}/*-[brn]?.fits --mode=copy --validity 3600
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 ~/pfs/drp_pfs_data/detectorMap/detectorMap-sim-??.fits
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/bootstrap --flatId visit=68351 arm=r --arcId visit=68100 arm=r -C bootstrap-pfi.py -c spectralOffset=-5

bootstrap INFO: Found 23615 lines in 596 traces
bootstrap INFO: Matched 20916 lines
bootstrap INFO: Median difference from detectorMap: -3.129519,-0.945394 pixels
bootstrap INFO: Fit 8893/10451 points, rms: x=0.049871 y=0.068376 total=0.037381 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: -3.785663,0.140286 pixels
bootstrap INFO: Fit 8556/10465 points, rms: x=0.093251 y=0.074023 total=0.051825 pixels
bootstrap INFO: Updating detectorMap...

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/bootstrap --flatId visit=68351 arm=b --arcId visit=68100 arm=b -C bootstrap-pfi.py -c spectralOffset=-5 --clobber-config --no-versions

bootstrap INFO: Found 10657 lines in 596 traces
bootstrap INFO: Matched 10629 lines
bootstrap INFO: Median difference from detectorMap: 12.159949,-1.086394 pixels
bootstrap INFO: Fit 4431/5298 points, rms: x=0.043972 y=0.056688 total=0.028492 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: -0.923853,-4.523398 pixels
bootstrap INFO: Fit 4588/5331 points, rms: x=0.052255 y=0.058974 total=0.031261 pixels
bootstrap INFO: Updating detectorMap...

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-933/pfi/bootstrap/DETECTORMAP/pfsDetectorMap-0*.fits

constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/calibs --id visit=68345^68350^68351 --cores 6 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3 repair.doCosmicRay=False
ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-933/pfi/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits --config clobber=True
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/calibs --id visit=68096..68100 -j 10 -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=91276.262056 dof=28574 xRMS=0.030331 yRMS=0.030932 (0.002109 nm) from 14317/21351 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=65748.464411 xRMS=0.052390 yRMS=0.051339 (0.003500 nm) from 2372 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.028431, y=0.030154 pixels (0.002056 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=28261.310354 dof=28574 xRMS=0.035210 yRMS=0.037056 (0.002526 nm) from 14317 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=17768.151862 xRMS=0.052306 yRMS=0.050377 (0.003434 nm) from 2372 lines

r:
reduceArc.fitDetectorMap INFO: Final fit: chi2=262729.727681 dof=114342 xRMS=0.037429 yRMS=0.036734 (0.003172 nm) from 57201/68817 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=162889.828353 xRMS=0.058592 yRMS=0.056103 (0.004844 nm) from 7646 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.034156, y=0.033141 pixels (0.002861 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=113618.567835 dof=114342 xRMS=0.049241 yRMS=0.047806 (0.004128 nm) from 57201 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=36502.919065 xRMS=0.058728 yRMS=0.056225 (0.004854 nm) from 7646 lines

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-933/pfi/calibs/DETECTORMAP/pfsDetectorMap-068096-*.fits --config clobber=True

constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/calibs --id visit=68345^68350^68351 --cores 6 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3 repair.doCosmicRay=False
ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-933/pfi/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits --config clobber=True

reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/calibs --id visit=68096..68100 -j 10 -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=90290.016120 dof=28424 xRMS=0.030334 yRMS=0.030778 (0.002098 nm) from 14242/21348 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=64151.511007 xRMS=0.050466 yRMS=0.051126 (0.003486 nm) from 2372 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.028370, y=0.029792 pixels (0.002031 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=28128.336779 dof=28424 xRMS=0.035163 yRMS=0.036727 (0.002504 nm) from 14242 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=17812.028153 xRMS=0.050841 yRMS=0.050537 (0.003445 nm) from 2372 lines

r:
reduceArc.fitDetectorMap INFO: Final fit: chi2=262415.216414 dof=114292 xRMS=0.037372 yRMS=0.036730 (0.003171 nm) from 57176/68829 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=101206.611769 xRMS=0.059672 yRMS=0.055131 (0.004760 nm) from 7648 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.034177, y=0.033090 pixels (0.002857 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=113585.647857 dof=114292 xRMS=0.049257 yRMS=0.047749 (0.004123 nm) from 57176 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=22745.784108 xRMS=0.059856 yRMS=0.055236 (0.004769 nm) from 7648 lines

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211105 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-933/pfi/calibs/DETECTORMAP/pfsDetectorMap-068096-*.fits --config clobber=True

cp -r CALIB-PFI-20211105 /projects/HSC/PFS/Subaru

reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/sky -c isr.doFlat=False --id visit=68878^68879^68881 -j 20 --longlog | tee pfi.log
mergeArms.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211105 --rerun price/pipe2d-933/pfi/sky --id visit=68878^68879^68881 -j 20 -c doSubtractSky1d=False --no-versions
Comment by hassan [ 06/Nov/21 ]

Proposed changes look fine overall, but unit tests currently fail. Provided suggestions in https://github.com/Subaru-PFS/drp_stella/pull/228. Once those are fixed, feel free to merge/close ticket at your convenience.

Comment by rhl [ 06/Nov/21 ]

y = 2945
for x in [97, 692, 711, 2602]:
print(f"(
Unknown macro: {x}
,
Unknown macro: {y}
)
Unknown macro: {detMap.findFiberId(geom.PointD(x, y))}
")

returns

(97, 2945) 315
(692, 2945) 315
(711, 2945) 315
(2602, 2945) 231

so the fiberIds are wrong somewhere – those are not all fibre 315(the right side of the chip seems fine)

Comment by rhl [ 06/Nov/21 ]

The extraction seems to somehow be inserting NaN into the calexp. Note how the white lines of NaN only appear where there are fibres, so I don't think it's a problem in the calibs (and it isn't in the raw).

reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211105 --rerun rhl/pfi -c doWriteCalexp=True isr.doFlat=False isr.doDefect=False isr.doDark=False repair.cosmicray.keepCRs=False repair.interp.modelPsf.defaultFwhm=2.25 --id visit=68878

Comment by price [ 07/Nov/21 ]

The findFiberId bug appears to be a problem with SplinedDetectorMap::findFiberId, because butler.get("detectorMap", visit=68878, arm="r") gets the bootstrapped detectorMap rather than the proper fitted DoubleDetectorMap.

Comment by price [ 09/Nov/21 ]

I'm going to create some new calibs for PFI and SuNSS, and then merge this branch. I've created new tickets to deal with the two issues rhl has identified: PIPE2D-936 (findFiberId) and PIPE2D-937 (NaN values in the image).

Comment by price [ 09/Nov/21 ]

Changed my mind: merged this work, and will create new calibs under PIPE2D-939.

Generated at Sat Feb 10 16:00:01 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.