[PIPE2D-924] Create calibs and process recent data Created: 26/Oct/21  Updated: 05/Nov/21  Resolved: 05/Nov/21

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Blocks
blocks PIPE2D-918 Provide flux data for calibration lam... Done
Duplicate
is duplicated by PIPE2D-909 Regenerate detector maps including en... Won't Fix
Story Points: 3
Sprint: 2DDRP-2021 A11
Reviewers: hassan

 Description   

Following the addition of engineering fibers (SIM2D-144) and various improvements to detectorMap construction (e.g., PIPE2D-798, PIPE2D-891), create new calibs for SuNSS runs and the recent engineering PFI run, and process sky observations.

hassan identified the following data:

For SuNSS exposures, as a start, look at 63350..63369 (29-30 June 2021) and 69039..69177(21-25 Oct 2021). There are no data in between those two dates, unfortunately. I can certainly give you the visits for more data in June later if you need that right now. (edited)

and

SuNSS
m-band: flats=63050..63061, arcs=63102..63106 (roomlights).
b-band: flats=63050..63061, arcs=63091..63100 (roomlights)
r-band: flats=46628..46670, arcs=46723..46748 (skylines; supplied by Arnaud)
PFI
r-band: flats=68351, arcs=68100
(that was a limited set for testing. You could possibly could use flats=68345^68350^68351 and for arcs 68096..68100)
I did not look at b-band PFI data.

 Comments   
Comment by price [ 27/Oct/21 ] 
Create new calibs that include the engineering fibers, and apply these calibs to recent observation runs.

Hassan writes:
> For SuNSS exposures, as a start, look at 63350..63369 (29-30 June 2021) and 69039..69177(21-25 Oct 2021).  There are no data in between those two dates, unfortunately. I can certainly give you the visits for more data in June later if you need that right now. (edited) 
> SuNSS
>     m-band: flats=63050..63061,  arcs=63102..63106 (roomlights).
>     b-band: flats=63050..63061, arcs=63091..63100 (roomlights)
>     r-band: flats=46628..46670, arcs=46723..46748 (skylines; supplied by Arnaud)
> PFI
>    r-band:  flats=68351, arcs=68100
>   (that was a limited set for testing. You could possibly could use flats=68345^68350^68351 and for arcs 68096..68100)
> I did not look at b-band PFI data.


(lsst-scipipe) pprice@tiger2-sumire:~ $ setup pfs_pipe2d
(lsst-scipipe) pprice@tiger2-sumire:~ $ eups list -s pfs_pipe2d
   w.2021.43    current setup

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ mkdir CALIB-SuNSS-20211026
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ mkdir CALIB-PFI-20211026


We've not been provided new biases or darks. We'll assume those haven't changed.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 /projects/HSC/PFS/Subaru/CALIB-SuNSS/{BIAS,DARK}/*-[brn]?.fits --mode=copy --validity 3600
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 /projects/HSC/PFS/Subaru/CALIB-SuNSS/{BIAS,DARK}/*-[brn]?.fits --mode=copy --validity 3600

We're skipping flat creation completely, and disabling flats in all subsequent processing due to lack of dithered flats.

Initial detectorMaps, now with enginerring fibers (SIM2D-144):

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --mode=copy --validity 3600 ~/pfs/drp_pfs_data/detectorMap/detectorMap-sim-??.fits
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --mode=copy --validity 3600 ~/pfs/drp_pfs_data/detectorMap/detectorMap-sim-??.fits

Bootstrapping a detectorMap

m-band first:

Determine rough offset by eye:

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ cat > display.config 
config.readLineList.lampList = ["HgI", "KrI", "ArI"]  # otherwise we get sky
config.readLineList.minIntensity = 100
config.backend = "ds9"

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ detrend.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/bootstrap --id visit=63102 arm=m -c isr.doFlat=False
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ displayDetectorMap.py /projects/HSC/PFS/Subaru/ --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/bootstrap --id visit=63102 arm=m -C display.config 

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ cat > bootstrap.py
config.isr.doFlat = False
config.readLineList.lampList = ["HgI", "KrI", "ArI"]  # otherwise we get sky
config.readLineList.minIntensity = 100
config.profiles.profileRadius = 3  # because we have some close-packed fibers
# The following give us only the good fibers, which give us a reliable fiber identification
from pfs.datamodel import FiberStatus
config.fiberStatus = ["GOOD"]
config.profiles.findThreshold = 1000  # convolution is probably picking up a lot of scattered light

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/bootstrap --flatId visit=63050 arm=m --arcId visit=63102 arm=m -C bootstrap.py -c spectralOffset=-15

bootstrap INFO: Found 5288 lines in 251 traces
bootstrap INFO: Matched 5004 lines
bootstrap INFO: Median difference from detectorMap: 6.534795,-1.027642 pixels
bootstrap INFO: Fit 2054/2439 points, rms: x=0.053748 y=0.135395 total=0.086560 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: 5.841463,0.837069 pixels
bootstrap INFO: Fit 2109/2565 points, rms: x=0.065804 y=0.087715 total=0.055430 pixels
bootstrap INFO: Updating detectorMap...

b-band next:

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ detrend.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/bootstrap --id visit=63091 arm=b -c isr.doFlat=False
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ displayDetectorMap.py /projects/HSC/PFS/Subaru/ --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/bootstrap --id visit=63091 arm=b -C display.config 
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/bootstrap --flatId visit=63050 arm=b --arcId visit=63091 arm=b -C bootstrap.py -c spectralOffset=-5 profiles.findThreshold=500 --clobber-config

bootstrap INFO: Found 1858 lines in 251 traces
bootstrap INFO: Matched 1557 lines
bootstrap INFO: Median difference from detectorMap: 10.204748,2.796891 pixels
bootstrap INFO: Fit 623/760 points, rms: x=0.044142 y=0.058955 total=0.030086 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: 4.173050,-1.861993 pixels
bootstrap INFO: Fit 506/797 points, rms: x=0.042830 y=0.045330 total=0.023350 pixels
bootstrap INFO: Updating detectorMap...

And finally r-band:

Above, I've been given sky spectra, but I want to bootstrap and calibrate off of room arcs (for consistency with b,m; and reduced blending/confusion for a better quality result). Going to use a set I've used for bootstrap before.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/bootstrap --flatId visit=46670 arm=r --arcId visit=46671 arm=r -C bootstrap.py -c spectralOffset=-5 profiles.findThreshold=500 --clobber-config

bootstrap INFO: Found 8652 lines in 251 traces
bootstrap INFO: Matched 8284 lines
bootstrap INFO: Median difference from detectorMap: -3.533983,-1.197599 pixels
bootstrap INFO: Fit 3204/3986 points, rms: x=0.040485 y=0.051986 total=0.026731 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: -4.143563,-0.079184 pixels
bootstrap INFO: Fit 2898/4298 points, rms: x=0.053554 y=0.046953 total=0.027826 pixels
bootstrap INFO: Updating detectorMap...


(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/sunss/bootstrap/DETECTORMAP/pfsDetectorMap-0*.fits

Repeat for PFI r-band:

This is a Ne arc, rather than room lights (looks nice and clean with no continuum!).

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ detrend.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/bootstrap --id visit=68100 arm=r -c isr.doFlat=False
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ displayDetectorMap.py /projects/HSC/PFS/Subaru/ --calib CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/bootstrap --id visit=68100 arm=r -c backend=ds9

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ cat > bootstrap-pfi.py 
config.isr.doFlat = False
config.profiles.profileRadius = 3  # because we have some close-packed fibers
# The following give us only the good fibers, which give us a reliable fiber identification
from pfs.datamodel import FiberStatus
config.fiberStatus = ["GOOD"]
config.profiles.findThreshold = 500  # convolution is probably picking up a lot of scattered light

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/bootstrap --flatId visit=68351 arm=r --arcId visit=68100 arm=r -C bootstrap-pfi.py -c spectralOffset=-5

Not finding the right number of fibers.

config.isr.doFlat = False
config.profiles.profileRadius = 3  # because we have some close-packed fibers
config.profiles.centroidRadius = 3
# The following give us only the good fibers, which give us a reliable fiber identification
from pfs.datamodel import FiberStatus
config.fiberStatus = ["GOOD"]
config.profiles.findThreshold = 500  # convolution is probably picking up a lot of scattered light
config.profiles.columnFwhm = 1.0
config.profiles.associationDepth = 25

yields:

bootstrap INFO: Found 23616 lines in 597 traces
bootstrap INFO: Matched 20917 lines
bootstrap INFO: Median difference from detectorMap: -3.128776,-0.944901 pixels
bootstrap INFO: Fit 8908/10451 points, rms: x=0.050031 y=0.068727 total=0.037702 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: -9.662171,0.093222 pixels
bootstrap INFO: Fit 8088/10466 points, rms: x=1.206367 y=0.160703 total=0.655779 pixels
bootstrap INFO: Updating detectorMap...

But it must be misidentifying fibers to get x rms ~ 1.2 pixels.

There are some saturated pixels in the flat, but that doesn't appear to be causing trouble.
There are two pfsConfigs associated with visit=68351:
pfsConfig-0x071c4c79bd4b5d30-068351.fits
pfsConfig-0x1f8dc068ce7f1647-068351.fits

The active one according to the registry is 0x071c4c79bd4b5d30. It has 2458 fibers, while the other has 2394: must include the engineering fibers.


from lsst.afw.image import ExposureF
from lsst.afw.display import Display
from pfs.drp.stella.buildFiberProfiles import BuildFiberProfilesTask
from pfs.drp.stella.datamodel import PfsConfig
exposure = ExposureF("calExp-SA068351r1.fits")
pfsConfig = PfsConfig.readFits("pfsConfig-0x071c4c79bd4b5d30-068351.fits")
config = BuildFiberProfilesTask.ConfigClass()
config.profileRadius = 3
config.centroidRadius = 3
config.fiberStatus = ["GOOD"]
config.findThreshold = 1000
config.associationDepth = 50
results = BuildFiberProfilesTask(config=config).run(exposure, pfsConfig=pfsConfig)
display = Display(frame=1, backend="ds9")
display.mtv(exposure)
yy = 2048
with display.Buffering():
    for fiberId in results.centers:
        xx = results.centers[fiberId](yy)
        display.dot(str(fiberId), xx, yy)

They look fine. No text where there aren't fibers, no double text on a single fiber.
Check the detectorMap

from pfs.datamodel import TargetType
from pfs.drp.stella import DetectorMap
select = pfsConfig.getSelection(targetType=TargetType.SCIENCE, spectrograph=1)
detMap = DetectorMap.readFits("../drp_pfs_data/detectorMap/detectorMap-sim-r1.fits")
detMap.applySlitOffsets(-4, 0)  # Guess
with display.Buffering():
    for fiberId in pfsConfig.fiberId[select]:
        display.dot(str(fiberId), detMap.getXCenter(fiberId, yy), yy)

Found fiberId=114 is absent.

Added facility to bootstrap so we can ignore it for now and move on.

config.badFibers = [114]

And now we totally nail it:

bootstrap INFO: Found 23615 lines in 596 traces
bootstrap INFO: Matched 20916 lines
bootstrap INFO: Median difference from detectorMap: -3.129519,-0.945394 pixels
bootstrap INFO: Fit 8893/10451 points, rms: x=0.049871 y=0.068376 total=0.037381 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: -3.785663,0.140286 pixels
bootstrap INFO: Fit 8556/10465 points, rms: x=0.093251 y=0.074023 total=0.051825 pixels
bootstrap INFO: Updating detectorMap...

Can we just do the same with b?

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/bootstrap --flatId visit=68351 arm=b --arcId visit=68100 arm=b -C bootstrap-pfi.py -c spectralOffset=-5 --clobber-config --no-versions

bootstrap INFO: Found 10657 lines in 596 traces
bootstrap INFO: Matched 10629 lines
bootstrap INFO: Median difference from detectorMap: 12.159949,-1.086394 pixels
bootstrap INFO: Fit 4431/5298 points, rms: x=0.043972 y=0.056688 total=0.028492 pixels
bootstrap INFO: Updating detectorMap...
bootstrap INFO: Median difference from detectorMap: -0.923853,-4.523398 pixels
bootstrap INFO: Fit 4588/5331 points, rms: x=0.052255 y=0.058974 total=0.031261 pixels
bootstrap INFO: Updating detectorMap...

Hooray, that's a win.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/pfi/bootstrap/DETECTORMAP/pfsDetectorMap-0*.fits


Now it's time for the fiberProfiles-detectorMap iteration loop.

pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ cat > arc.py
config.reduceExposure.readLineList.lampList = ["HgI", "KrI", "ArI"]  # otherwise we get sky


constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=63050..63061 --cores 12 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3
constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=46628..46670 arm=r --cores 20 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/sunss/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits

Using PIPE2D-916 for reduceArc changes.

reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=63102..63106 arm=m -j 10 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=True reduceExposure.readLineList.exclusionRadius=0.3 fitDetectorMap.minSignalToNoise=20 --clobber-config
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=63091..63100 arm=b -j 10 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=True reduceExposure.readLineList.exclusionRadius=0.3 fitDetectorMap.minSignalToNoise=20 fitDetectorMap.soften=0.05 --clobber-config
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=46671..46688 arm=r -j 20 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=True reduceExposure.readLineList.exclusionRadius=0.3 fitDetectorMap.minSignalToNoise=20 --clobber-config

m:
reduceArc.fitDetectorMap INFO: Final fit: chi2=39129.251223 dof=20814 xRMS=0.041243 yRMS=0.031340 (0.001479 nm) from 10437/11157 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=7905.757254 xRMS=0.054841 yRMS=0.039475 (0.001863 nm) from 1240 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.036057, y=0.018555 pixels (0.000875 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=20641.648934 dof=20814 xRMS=0.047938 yRMS=0.033468 (0.001579 nm) from 10437 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=3670.131031 xRMS=0.054729 yRMS=0.039717 (0.001874 nm) from 1240 lines

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=13945.349999 dof=5906 xRMS=0.027563 yRMS=0.027930 (0.001904 nm) from 2983/13736 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=1199281.970033 xRMS=0.207412 yRMS=0.570632 (0.038905 nm) from 1526 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.047864, y=0.056510 pixels (0.003853 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=5517.549472 dof=5906 xRMS=0.057132 yRMS=0.068534 (0.004673 nm) from 2983 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=335186.248979 xRMS=0.215483 yRMS=0.552325 (0.037657 nm) from 1526 lines

That's bad. I think the slitOffsets fit is adding noise.

Without slitOffsets:
reduceArc.fitDetectorMap INFO: Final fit: chi2=14915.021963 dof=15262 xRMS=0.060196 yRMS=0.062138 (0.004236 nm) from 7661/13734 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=324509.576129 xRMS=0.075656 yRMS=0.165612 (0.011291 nm) from 1526 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.048625, y=0.049557 pixels (0.003379 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=15261.883569 dof=15262 xRMS=0.059991 yRMS=0.061999 (0.004227 nm) from 7661 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=326835.879890 xRMS=0.075534 yRMS=0.165190 (0.011262 nm) from 1526 lines

Filed PIPE2D-926 ("Make slit offset measurement more robust"), and will deal with it later. In the mean time, proceed without slit offsets.
Increased exclusionRadius to 0.5 for r-band, as that's what we used successfully in PIPE2D-819. It's less important in m (more resolution) and b (fewer lines).

reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=63102..63106 arm=m -j 10 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False reduceExposure.readLineList.exclusionRadius=0.3 fitDetectorMap.minSignalToNoise=20 --clobber-config
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=63091..63100 arm=b -j 10 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False reduceExposure.readLineList.exclusionRadius=0.3 fitDetectorMap.minSignalToNoise=20 fitDetectorMap.soften=0.05 --clobber-config
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=46671..46688 arm=r -j 20 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False reduceExposure.readLineList.exclusionRadius=0.5 fitDetectorMap.minSignalToNoise=20 --clobber-config

m:
reduceArc.fitDetectorMap INFO: Final fit: chi2=46433.143287 dof=20228 xRMS=0.042602 yRMS=0.040534 (0.001912 nm) from 10144/11157 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=9560.668107 xRMS=0.056843 yRMS=0.055275 (0.002608 nm) from 1240 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.036279, y=0.031439 pixels (0.001483 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=20112.870290 dof=20228 xRMS=0.048762 yRMS=0.044888 (0.002118 nm) from 10144 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=3576.816951 xRMS=0.055812 yRMS=0.055764 (0.002631 nm) from 1240 lines

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=14915.021963 dof=15262 xRMS=0.060196 yRMS=0.062138 (0.004236 nm) from 7661/13734 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=324509.576129 xRMS=0.075656 yRMS=0.165612 (0.011291 nm) from 1526 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.048625, y=0.049557 pixels (0.003379 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=15261.883569 dof=15262 xRMS=0.059991 yRMS=0.061999 (0.004227 nm) from 7661 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=326835.879890 xRMS=0.075534 yRMS=0.165190 (0.011262 nm) from 1526 lines

r:
reduceArc.fitDetectorMap INFO: Final fit: chi2=108948.187060 dof=35300 xRMS=0.043590 yRMS=0.046888 (0.004048 nm) from 17680/29330 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=480680.984483 xRMS=0.063229 yRMS=0.105138 (0.009078 nm) from 3259 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.039237, y=0.052318 pixels (0.004517 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=34821.119324 dof=35300 xRMS=0.048903 yRMS=0.060405 (0.005215 nm) from 17680 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=142673.417027 xRMS=0.063769 yRMS=0.106804 (0.009221 nm) from 3259 lines

Happy enough with those. Let's ingest and iterate.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/sunss/calibs/DETECTORMAP/pfsDetectorMap-063102-m1.fits /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/sunss/calibs/DETECTORMAP/pfsDetectorMap-063091-b1.fits /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/sunss/calibs/DETECTORMAP/pfsDetectorMap-046671-r1.fits --config clobber=True

constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=63050..63061 --cores 12 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3
constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=46628..46670 arm=r --cores 20 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/sunss/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits --config clobber=True

reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=63102..63106 arm=m -j 10 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False reduceExposure.readLineList.exclusionRadius=0.3 fitDetectorMap.minSignalToNoise=20 --clobber-config
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=63091..63100 arm=b -j 10 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False reduceExposure.readLineList.exclusionRadius=0.3 fitDetectorMap.minSignalToNoise=20 --clobber-config
reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/calibs --id visit=46671..46688 arm=r -j 20 -C arc.py -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False reduceExposure.readLineList.exclusionRadius=0.5 fitDetectorMap.minSignalToNoise=20 --clobber-config

m:
reduceArc.fitDetectorMap INFO: Final fit: chi2=47103.915166 dof=20316 xRMS=0.042856 yRMS=0.040797 (0.001925 nm) from 10188/11181 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=28194.231981 xRMS=0.058023 yRMS=0.053154 (0.002508 nm) from 1242 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.036630, y=0.031693 pixels (0.001495 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=20188.744891 dof=20316 xRMS=0.049097 yRMS=0.045154 (0.002130 nm) from 10188 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=11384.467776 xRMS=0.057817 yRMS=0.053210 (0.002510 nm) from 1242 lines

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=29123.389612 dof=10624 xRMS=0.030600 yRMS=0.030020 (0.002047 nm) from 5342/13629 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=1068071.010487 xRMS=0.096184 yRMS=0.524766 (0.035776 nm) from 1514 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.031041, y=0.032236 pixels (0.002198 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=10514.009480 dof=10624 xRMS=0.040747 yRMS=0.042334 (0.002886 nm) from 5342 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=790270.163381 xRMS=0.097301 yRMS=0.581868 (0.039669 nm) from 1514 lines

r:
reduceArc.fitDetectorMap INFO: Final fit: chi2=97593.825251 dof=31946 xRMS=0.044248 yRMS=0.044796 (0.003868 nm) from 16003/29171 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=493261.191267 xRMS=0.068806 yRMS=0.110828 (0.009569 nm) from 3241 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.041561, y=0.047369 pixels (0.004090 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=31679.681214 dof=31946 xRMS=0.050547 yRMS=0.056043 (0.004839 nm) from 16003 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=139187.471703 xRMS=0.068508 yRMS=0.111483 (0.009625 nm) from 3241 lines

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-SuNSS-20211026 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/sunss/calibs/DETECTORMAP/pfsDetectorMap-063102-m1.fits /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/sunss/calibs/DETECTORMAP/pfsDetectorMap-063091-b1.fits /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/sunss/calibs/DETECTORMAP/pfsDetectorMap-046671-r1.fits --config clobber=True


Now for PFI:

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ cat > profiles-pfi.py
config.profiles.badFibers = [114]

constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/calibs --id visit=68345^68350^68351 --cores 6 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3 -C profiles-pfi.py

Getting lots of traces flagged as CRs, which breaks the profiles. Disable CR masking. Then it's not the lack of fiberId=114 that's breaking things, so may as well stick it in (though zero).

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-924 $ constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/calibs --id visit=68345^68350^68351 --cores 6 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3 repair.doCosmicRay=False

root WARN: No data found for dataId=OrderedDict([('visit', 68350)])

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/pfi/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits --config clobber=True

reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/calibs --id visit=68096..68100 -j 10 -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False reduceExposure.readLineList.exclusionRadius=0.5 fitDetectorMap.minSignalToNoise=20

root WARN: No data found for dataId=OrderedDict([('visit', 68097)])
root WARN: No data found for dataId=OrderedDict([('visit', 68098)])

And we need something in reduceExposure to toss out engineering fibers (or we get exceptions when correcting photometry for fiber normalisations). Not sure this is the right thing to do, but we're short on time.

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=88372.639423 dof=25766 xRMS=0.029970 yRMS=0.030553 (0.002083 nm) from 12913/19960 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=55725.529939 xRMS=0.052059 yRMS=0.049976 (0.003407 nm) from 2218 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.028975, y=0.030570 pixels (0.002084 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=25511.450308 dof=25766 xRMS=0.034504 yRMS=0.036200 (0.002468 nm) from 12913 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=10997.161273 xRMS=0.052304 yRMS=0.050502 (0.003443 nm) from 2218 lines

r:
reduceArc.fitDetectorMap INFO: Final fit: chi2=234248.828253 dof=103316 xRMS=0.038139 yRMS=0.037513 (0.003239 nm) from 51688/61477 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=128624.098191 xRMS=0.059242 yRMS=0.054859 (0.004737 nm) from 6831 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.034171, y=0.033497 pixels (0.002892 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=102621.517264 dof=103316 xRMS=0.049630 yRMS=0.048476 (0.004185 nm) from 51688 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=36462.850466 xRMS=0.059231 yRMS=0.055602 (0.004801 nm) from 6831 lines

That looks acceptable. Ingest and iterate once more.

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/pfi/calibs/DETECTORMAP/pfsDetectorMap-068096-*.fits --config clobber=True

constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/calibs --id visit=68345^68350^68351 --cores 6 -c isr.doFlat=False profiles.profileRadius=3 profiles.centroidRadius=3 repair.doCosmicRay=False --no-versions
ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/pfi/calibs/FIBERPROFILES/pfsFiberProfiles-2021-0*.fits --config clobber=True

reduceArc.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --rerun price/pipe2d-924/pfi/calibs --id visit=68096..68100 -j 10 -c reduceExposure.isr.doFlat=False fitDetectorMap.doSlitOffsets=False reduceExposure.readLineList.exclusionRadius=0.5 fitDetectorMap.minSignalToNoise=20 --no-versions

b:
reduceArc.fitDetectorMap INFO: Final fit: chi2=87954.111692 dof=25692 xRMS=0.029958 yRMS=0.030410 (0.002073 nm) from 12876/19957 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=52803.221844 xRMS=0.049379 yRMS=0.051474 (0.003509 nm) from 2218 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.029038, y=0.030344 pixels (0.002069 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=25441.142990 dof=25692 xRMS=0.034538 yRMS=0.035971 (0.002452 nm) from 12876 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=10840.152654 xRMS=0.049403 yRMS=0.050957 (0.003474 nm) from 2218 lines

r:
reduceArc.fitDetectorMap INFO: Final fit: chi2=234494.355564 dof=103340 xRMS=0.038207 yRMS=0.037544 (0.003242 nm) from 51700/61490 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=123054.662331 xRMS=0.058418 yRMS=0.054653 (0.004719 nm) from 6832 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by x=0.034324, y=0.033570 pixels (0.002898 nm) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=102618.227541 dof=103340 xRMS=0.049793 yRMS=0.048571 (0.004194 nm) from 51700 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=35028.833185 xRMS=0.058576 yRMS=0.054801 (0.004732 nm) from 6832 lines

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --calib CALIB-PFI-20211026 --mode=copy --validity 3600 /projects/HSC/PFS/Subaru/rerun/price/pipe2d-924/pfi/calibs/DETECTORMAP/pfsDetectorMap-068096-*.fits --config clobber=True

And we're done.
Comment by price [ 27/Oct/21 ]

I've put my new calibs in /projects/HSC/PFS/Subaru: CALIB-SuNSS-20211026 and CALIB-PFI-20211026.

Comment by price [ 27/Oct/21 ]

The ticket branch includes a few fixes (including working around engineering fibers) and a merge from tickets/PIPE2D-916.

Comment by price [ 03/Nov/21 ]

I have reduced the visits I was told of, and am soliciting further visits to reduce.

reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211026 --rerun price/pipe2d-924/sunss/sky -c isr.doFlat=False --id visit=63350..63369^69039..69177 -j 20
Comment by rhl [ 03/Nov/21 ]

Why are there two sets of calibrations? Can't we handle this via date ranges? Separately, can we put them in the default place (or make a link). Users should not have to think about which set of calibrations to use to process PFS data; it's our job to provide them.

Comment by rhl [ 03/Nov/21 ]

Is there a reason not to reduce all the SuNSS data we've ever taken?

Comment by price [ 03/Nov/21 ]

rhl wrote:

Why are there two sets of calibrations? Can't we handle this via date ranges?

The date ranges used by the calib system are crude: the calib ingestion simply takes the central date between two calibs as the boundary. We could manually adjust the calib registry to know the date where we switched, which would allow us to use a single calib repo. Do you know the relevant dates?

rhl also wrote:

Is there a reason not to reduce all the SuNSS data we've ever taken?

I know we had a ticket to identify the good exposures worth processing. I don't think these are easily identifiable from the butler registry or opdb, but require special knowledge.

Comment by price [ 04/Nov/21 ]

I'm going to punt on the multiple calib problem, due to the desire to merge quickly.
Note that this is built on top of PIPE2D-916, which is blocked on my researching the answer to rhl's question.

Comment by rhl [ 04/Nov/21 ]

Merging first is fine, esp. as this work is more than the resulting calibs. We can and should merge them on another ticket

Comment by price [ 04/Nov/21 ]

The calibs have been created, data has been processed, but more important are the changes to the pipeline to support processing PFI data (generally due to inclusion of engineering fibers).

Comment by hassan [ 05/Nov/21 ]

Changes look fine. Minor comment on one commit in https://github.com/Subaru-PFS/drp_stella/pull/227.

Comment by price [ 05/Nov/21 ]

Reworded the commit that didn't match its message, and merged.

Generated at Sat Feb 10 15:59:53 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.