[PIPE2D-939] Build new calibs Created: 09/Nov/21  Updated: 10/Nov/21  Resolved: 10/Nov/21

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: 2DDRP-2021 A11
Reviewers: hassan

 Description   

Following PIPE2D-924 and PIPE2D-933, build new SuNSS and PFI calibs. Take care that the bootstrapped detectorMap isn't accessible once a regular detectorMap has been created.

 Comments   
Comment by price [ 10/Nov/21 ]

Created new calibs (CALIB-PFI-20211108 and CALIB-SuNSS-20211108) in /projects/HSC/PFS/Subaru. There are a couple of code changes which improve the quality of the calibs.

Comment by price [ 10/Nov/21 ] 
PIPE2D-939: Build new calibs

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ setup pfs_pipe2d
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ eups list -s pfs_pipe2d
   w.2021.45    current setup
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ setup -jr ~/pfs/drp_stella  # Get recent merges

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ mkdir CALIB-PFI-20211108
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 /projects/HSC/PFS/Subaru/CALIB-SuNSS/{BIAS,DARK}/*-[brn]?.fits --mode=copy --validity 3600

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --mode=copy --validity 3600 ~/pfs/drp_pfs_data/detectorMap/detectorMap-sim-??.fits
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ cat bootstrap-pfi.py 
config.isr.doFlat = False
config.isr.doSaturation = False
config.profiles.profileRadius = 3  # because we have some close-packed fibers
config.profiles.centroidRadius = 3
# The following give us only the good fibers, which give us a reliable fiber identification
config.fiberStatus = ["GOOD"]
config.profiles.findThreshold = 1000  # convolution is probably picking up a lot of scattered light
config.profiles.columnFwhm = 0.1
config.profiles.associationDepth = 50
config.badFibers = [114]
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --rerun price/pipe2d-939/pfi/bootstrap --flatId visit=68351 arm=r --arcId visit=68100 arm=r -C bootstrap-pfi.py -c spectralOffset=-5

bootstrap INFO: Found 23615 lines in 596 traces
bootstrap INFO: Matched 20916 lines
bootstrap INFO: Median difference from detectorMap: -3.129519,-0.945394 pixels
bootstrap INFO: Fit 8893/10451 points, rms: x=0.049871 y=0.068376 total=0.037381 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: -3.785663,0.140286 pixels
bootstrap INFO: Fit 8556/10465 points, rms: x=0.093251 y=0.074023 total=0.051825 pixels
bootstrap INFO: Updating detectorMap...

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --rerun price/pipe2d-939/pfi/bootstrap --flatId visit=68351 arm=b --arcId visit=68100 arm=b -C bootstrap-pfi.py -c spectralOffset=-5

bootstrap INFO: Found 10657 lines in 596 traces
bootstrap INFO: Matched 10629 lines
bootstrap INFO: Median difference from detectorMap: 12.159949,-1.086394 pixels
bootstrap INFO: Fit 4431/5298 points, rms: x=0.043972 y=0.056688 total=0.028492 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: -0.923853,-4.523398 pixels
bootstrap INFO: Fit 4588/5331 points, rms: x=0.052255 y=0.058974 total=0.031261 pixels
bootstrap INFO: Updating detectorMap...

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/pfi/bootstrap/DETECTORMAP/pfsDetectorMap-0*.fits

constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --rerun price/pipe2d-939/pfi/calibs --id visit=68345^68350^68351 --cores 6 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3 repair.doCosmicRay=False doAdjustDetectorMap=False
ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/pfi/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits --config clobber=True
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --rerun price/pipe2d-939/pfi/calibs --id visit=68096..68100 -j 10 -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=91254.797114 dof=28598 xRMS=0.030326 yRMS=0.030900 (0.002107 nm) from 14329/21351 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=76545.884191 xRMS=0.052300 yRMS=0.052151 (0.003555 nm) from 2372 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.028397, y=0.030124 pixels (0.002054 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=28301.087923 dof=28598 xRMS=0.035185 yRMS=0.037028 (0.002524 nm) from 14329 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=23918.160717 xRMS=0.052528 yRMS=0.050385 (0.003435 nm) from 2372 lines

r:
reduceArc.fitDetectorMap INFO: Final fit: chi2=261865.094122 dof=114346 xRMS=0.037219 yRMS=0.036750 (0.003173 nm) from 57203/68808 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=214315.135739 xRMS=0.059309 yRMS=0.055855 (0.004823 nm) from 7645 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.033653, y=0.033173 pixels (0.002864 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=113704.644969 dof=114346 xRMS=0.048785 yRMS=0.047811 (0.004128 nm) from 57203 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=49771.406983 xRMS=0.059597 yRMS=0.056254 (0.004857 nm) from 7645 lines

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ sqlite3 CALIB-PFI-20211108/calibRegistry.sqlite3 'DELETE FROM detectorMap'

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/pfi/calibs/DETECTORMAP/pfsDetectorMap-068096-*.fits --config clobber=True

constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --rerun price/pipe2d-939/pfi/calibs --id visit=68345^68350^68351 --cores 6 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3 repair.doCosmicRay=False doAdjustDetectorMap=False
ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/pfi/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits --config clobber=True
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --rerun price/pipe2d-939/pfi/calibs --id visit=68096..68100 -j 10 -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=90658.400414 dof=28508 xRMS=0.030286 yRMS=0.030844 (0.002103 nm) from 14284/21343 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=53519.632833 xRMS=0.052036 yRMS=0.053859 (0.003672 nm) from 2372 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.028234, y=0.029964 pixels (0.002043 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=28221.911930 dof=28508 xRMS=0.035048 yRMS=0.036892 (0.002515 nm) from 14284 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=11704.806638 xRMS=0.052306 yRMS=0.053595 (0.003654 nm) from 2372 lines

r:
reduceArc.fitDetectorMap INFO: Final fit: chi2=261693.155032 dof=114398 xRMS=0.037275 yRMS=0.036682 (0.003167 nm) from 57229/68826 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=174519.820787 xRMS=0.059343 yRMS=0.055293 (0.004774 nm) from 7647 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.033762, y=0.033065 pixels (0.002855 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=113753.640053 dof=114398 xRMS=0.048903 yRMS=0.047727 (0.004121 nm) from 57229 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=45092.343331 xRMS=0.059299 yRMS=0.055443 (0.004787 nm) from 7647 lines

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211108 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/pfi/calibs/DETECTORMAP/pfsDetectorMap-068096-*.fits --config clobber=True

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ cp -r CALIB-PFI-20211108 /projects/HSC/PFS/Subaru

reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108 --rerun price/pipe2d-939/pfi/sky -c isr.doFlat=False --id visit=68878^68879^68881 -j 20 --longlog | tee pfi.log
mergeArms.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108 --rerun price/pipe2d-939/pfi/sky --id visit=68878^68879^68881 -j 20 -c doSubtractSky1d=False --no-versions


Now for SuNSS:

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ mkdir CALIB-SuNSS-20211108
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 /projects/HSC/PFS/Subaru/CALIB-SuNSS/{BIAS,DARK}/*-[brn]?.fits --mode=copy --validity 3600
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --mode=copy --validity 3600 ~/pfs/drp_pfs_data/detectorMap/detectorMap-sim-??.fits

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ cat bootstrap-sunss.py 
config.isr.doFlat = False
config.readLineList.lampList = ["HgI", "KrI", "ArI"]  # otherwise we get sky
config.readLineList.minIntensity = 100
config.profiles.profileRadius = 3  # because we have some close-packed fibers
# The following give us only the good fibers, which give us a reliable fiber identification
from pfs.datamodel import FiberStatus
config.fiberStatus = ["GOOD"]
config.profiles.findThreshold = 1000  # convolution is probably picking up a lot of scattered light

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/bootstrap --flatId visit=63050 arm=m --arcId visit=63102 arm=m -C bootstrap-sunss.py -c spectralOffset=-15

bootstrap INFO: Found 5288 lines in 251 traces
bootstrap INFO: Matched 5004 lines
bootstrap INFO: Median difference from detectorMap: 6.534795,-1.027642 pixels
bootstrap INFO: Fit 2054/2439 points, rms: x=0.053748 y=0.135395 total=0.086560 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: 5.841463,0.837069 pixels
bootstrap INFO: Fit 2109/2565 points, rms: x=0.065804 y=0.087715 total=0.055430 pixels
bootstrap INFO: Updating detectorMap...

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/bootstrap --flatId visit=63050 arm=b --arcId visit=63091 arm=b -C bootstrap-sunss.py -c spectralOffset=-5 profiles.findThreshold=500 --clobber-config

bootstrap INFO: Found 1858 lines in 251 traces
bootstrap INFO: Matched 1557 lines
bootstrap INFO: Median difference from detectorMap: 10.204748,2.796891 pixels
bootstrap INFO: Fit 623/760 points, rms: x=0.044142 y=0.058955 total=0.030086 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: 4.173050,-1.861993 pixels
bootstrap INFO: Fit 506/797 points, rms: x=0.042830 y=0.045330 total=0.023350 pixels
bootstrap INFO: Updating detectorMap...

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/bootstrap --flatId visit=46670 arm=r --arcId visit=46671 arm=r -C bootstrap-sunss.py -c spectralOffset=-5 profiles.findThreshold=500

bootstrap INFO: Found 8652 lines in 251 traces
bootstrap INFO: Matched 8284 lines
bootstrap INFO: Median difference from detectorMap: -3.533983,-1.197599 pixels
bootstrap INFO: Fit 3204/3986 points, rms: x=0.040485 y=0.051986 total=0.026731 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: -4.143563,-0.079184 pixels
bootstrap INFO: Fit 2898/4298 points, rms: x=0.053554 y=0.046953 total=0.027826 pixels
bootstrap INFO: Updating detectorMap...

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/sunss/bootstrap/DETECTORMAP/pfsDetectorMap-0*.fits

constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/calibs --id visit=63050..63061 --cores 12 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3
constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/calibs --id visit=46628..46670 arm=r --cores 20 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/sunss/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits

reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/calibs --id visit=63102..63106 arm=m -j 10 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/calibs --id visit=63091..63100 arm=b -j 10 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/calibs --id visit=46671..46688 arm=r -j 20 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False

m:
reduceArc.fitDetectorMap INFO: Final fit: chi2=46975.946655 dof=20244 xRMS=0.042644 yRMS=0.040764 (0.001923 nm) from 10152/11148 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=8944.874029 xRMS=0.054261 yRMS=0.052452 (0.002475 nm) from 1239 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.036306, y=0.031782 pixels (0.001499 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=20139.775086 dof=20244 xRMS=0.048784 yRMS=0.045171 (0.002131 nm) from 10152 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=3256.569812 xRMS=0.054001 yRMS=0.052570 (0.002480 nm) from 1239 lines

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=45894.399193 dof=17700 xRMS=0.036179 yRMS=0.033662 (0.002295 nm) from 8880/12104 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=213429.940510 xRMS=0.065514 yRMS=0.071927 (0.004904 nm) from 1345 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.041879, y=0.039927 pixels (0.002722 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=17336.250450 dof=17700 xRMS=0.053772 yRMS=0.057247 (0.003903 nm) from 8880 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=90685.402412 xRMS=0.067142 yRMS=0.071322 (0.004862 nm) from 1345 lines

r:
reduceArc.fitDetectorMap INFO: Final fit: chi2=57219.826334 dof=22760 xRMS=0.045080 yRMS=0.041015 (0.003541 nm) from 11410/49088 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=75083778.704599 xRMS=0.073016 yRMS=1.786299 (0.154229 nm) from 5454 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.038248, y=0.032963 pixels (0.002846 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=22705.004118 dof=22760 xRMS=0.047883 yRMS=0.043496 (0.003755 nm) from 11410 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=14211380.354101 xRMS=0.084680 yRMS=1.716056 (0.148164 nm) from 5454 lines

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ sqlite3 CALIB-SuNSS-20211108/calibRegistry.sqlite3 'DELETE FROM detectorMap'

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/sunss/calibs/DETECTORMAP/pfsDetectorMap-063102-m1.fits /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/sunss/calibs/DETECTORMAP/pfsDetectorMap-063091-b1.fits /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/sunss/calibs/DETECTORMAP/pfsDetectorMap-046671-r1.fits --config clobber=True

constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/calibs --id visit=63050..63061 --cores 12 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3
constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/calibs --id visit=46628..46670 arm=r --cores 20 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/sunss/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits -c clobber=True

reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/calibs --id visit=63102..63106 arm=m -j 10 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/calibs --id visit=63091..63100 arm=b -j 10 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --rerun price/pipe2d-939/sunss/calibs --id visit=46671..46688 arm=r -j 20 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False

m:

reduceArc.fitDetectorMap INFO: Final fit: chi2=46361.376663 dof=20264 xRMS=0.042472 yRMS=0.040698 (0.001920 nm) from 10162/11164 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=10148.136464 xRMS=0.055002 yRMS=0.053319 (0.002515 nm) from 1241 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.035820, y=0.031616 pixels (0.001492 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=20164.162942 dof=20264 xRMS=0.048481 yRMS=0.045132 (0.002129 nm) from 10162 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=4290.119765 xRMS=0.055182 yRMS=0.053179 (0.002509 nm) from 1241 lines

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=43627.868472 dof=16602 xRMS=0.035138 yRMS=0.033968 (0.002316 nm) from 8331/12232 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=307923.793044 xRMS=0.072413 yRMS=0.069183 (0.004717 nm) from 1359 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.041310, y=0.042387 pixels (0.002890 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=15991.981724 dof=16602 xRMS=0.052024 yRMS=0.058249 (0.003971 nm) from 8331 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=120809.726451 xRMS=0.070005 yRMS=0.067611 (0.004609 nm) from 1359 lines

r:
reduceArc.fitDetectorMap INFO: Final fit: chi2=69903.183317 dof=24610 xRMS=0.041204 yRMS=0.040308 (0.003480 nm) from 12335/49155 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=99007643.389318 xRMS=0.195780 yRMS=2.097342 (0.181084 nm) from 5462 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.036880, y=0.038115 pixels (0.003291 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=24372.413861 dof=24610 xRMS=0.045284 yRMS=0.046209 (0.003990 nm) from 12335 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=15100321.994442 xRMS=0.205532 yRMS=2.054140 (0.177354 nm) from 5462 lines

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211108 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/sunss/calibs/DETECTORMAP/pfsDetectorMap-063102-m1.fits /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/sunss/calibs/DETECTORMAP/pfsDetectorMap-063091-b1.fits /projects/HSC/PFS/Subaru/rerun/price/pipe2d-939/sunss/calibs/DETECTORMAP/pfsDetectorMap-046671-r1.fits --config clobber=True

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ cp -r CALIB-SuNSS-20211108/ /projects/HSC/PFS/Subaru/


Looking at PFI, visit=68878 arm=b, the detectorMap trace goes off the stars on the blue end (450 nm). It looks like this is coming from reduceArc, since there are no arc lines bluer than about 540 nm. Using a different set of arcs (e.g., 68088..68100 which includes Ar as well as Ne; and 69008..69011) doesn't help.

Adding in eigenvalue threshold in solveLeastSquares, which seems to suppress large high-order terms in the distortion.

Original:
reduceArc.fitDetectorMap INFO: Final fit: chi2=90658.400414 dof=28508 xRMS=0.030286 yRMS=0.030844 (0.002103 nm) from 14284/21343 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=53519.632833 xRMS=0.052036 yRMS=0.053859 (0.003672 nm) from 2372 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.028234, y=0.029964 pixels (0.002043 nm) to yield chi^2/dof=1

DoubleDistortion(order=4, range=Box2D(Point2D(-0.5, -0.5), Extent2D(4096, 4176)), xLeft=[ 11.8968, -1.278, -2.57784, -0.128741, 0.328256, 11.1863, 0.0837936, -0.0150401, -0.768916, -14.1884, 0.0353176, -0.107074, 0.0842868, 0.389752, 6.14712], yLeft=[0.913304, 0.53975, -3.21696, 0.661937, -0.552806, 1.10907, 0.0442087, -0.337012, 1.13838, -1.78072, -0.565028, 0.0658796, 0.228855, -0.641401, 1.0891], xRight=[ 2.17889, -1.15704, -2.82388, -0.119154, 0.578412, -6.14942, 0.0264814, 0.175, -1.10116, 7.93606, 0.170401, -0.00205453, -0.0903007, 0.637259, -3.43464], yRight=[  -2.844, 2.2928, -1.6785, 0.236936, 0.616165, -3.2297, -0.00857055, 0.112032, -1.1441, 3.52594, -0.20297, 0.157601, 0.0279908, 0.780751, -1.12307])


threshold=1.0e-6:
reduceArc.fitDetectorMap INFO: Final fit: chi2=90087.297341 dof=28336 xRMS=0.030590 yRMS=0.030860 (0.002104 nm) from 14198/21344 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=55444.486703 xRMS=0.050485 yRMS=0.054259 (0.003699 nm) from 2372 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.028681, y=0.029762 pixels (0.002029 nm) to yield chi^2/dof=1

DoubleDistortion(order=4, range=Box2D(Point2D(-0.5, -0.5), Extent2D(4096, 4176)), xLeft=[ 11.6484, -1.2145, 1.2082, -0.187126, -0.0136972, -0.441182, 0.0265069, 0.10072, -0.19156, -0.221711, 0.0403013, -0.0366034, 0.0263717, 0.0439511, 0.350097], yLeft=[0.727964, 0.585672, -2.70092, 0.740837, -0.63489, -0.57557, -0.0200698, -0.774085, 1.15691, 0.311932, -0.474958, 0.157663, 0.572809, -0.648648, 0.230541], xRight=[ 2.42268, -1.075, -4.72807, -0.124606, 0.0141471, -0.293503, -0.0345848, 0.151413, -0.132083, 0.874576, 0.152107, 0.0751538, -0.0314495, 0.100432, -0.508766], yRight=[-2.85058, 2.34202, -2.68187, 0.261091, 0.559378, -0.453514, -0.0800166, -0.0216966, -1.21107, 0.318053, -0.217069, 0.219499, 0.12738, 0.882388, 0.234554])

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ mv /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ /projects/HSC/PFS/Subaru/CALIB-PFI-20211108-bad
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ cp -r CALIB-PFI-20211108/ /projects/HSC/PFS/Subaru/
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-939 $ reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108 --rerun price/pipe2d-939/pfi/sky -c isr.doFlat=False --id visit=68878^68879^68881 -j 20 --longlog | tee pfi.log

Repeat PFI calib construction from above. And that's fixed the problem, the spectra look normal now.
Repeat SuNSS calib construction from above.

Increased the 'soften' parameter in fitDistortedDetectorMap, so that now running adjustDetectorMap on quartzes doesn't end up rejecting 90% of the data. Not rebuilding the calibs with this change, as I hope it's not too important.
Comment by hassan [ 10/Nov/21 ]

Software changes (in pull request) approved. If there are any issues with the regenerated calibs, a new ticket can be filed.

Comment by price [ 10/Nov/21 ]

Merged.

Generated at Sat Feb 10 16:00:04 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.