PIPE2D-1036: Set up shared Gen3 datastore with Postgresql registry Going to do an initial test with the test registry database and a reduced dataset. Choosing the most recent run, Run 17: 2024-05-28 through 2024-06-03 (UT: 2024-05-29 to 2024-06-04). The following changes are needed to butler.yaml, to use a Postgresql registry. Stole and adapted this snippet from /projects/MERIAN/repo/butler.yaml on tiger. registry: db: postgresql+psycopg2://pfsa-db:5435/drp_test managers: attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager datasets: lsst.daf.butler.registry.datasets.byDimensions.ByDimensionsDatasetRecordStorageManagerUUID datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager Need to add in ~/.lsst/db-auth.yaml: - url: postgresql://pfsa-db:5435/drp username: pfs password: - url: postgresql://pfsa-db:5435/drp_test username: pfs password: db-auth.yaml must be readable only by the user: chmod 600 ~/.lsst/db-auth.yaml Setting up the datastore: export DATASTORE=/work/price/pipe2d-1036/datastore butler create $DATASTORE --seed-config /work/price/pipe2d-1036/butler.yaml --dimension-config $OBS_PFS_DIR/gen3/dimensions.yaml --override butler register-instrument $DATASTORE lsst.obs.pfs.PrimeFocusSpectrograph Ingesting the raw images: Let's try with a single day first: butler ingest-raws $DATASTORE /data/raw/2024-05-29/{sps,ramps}/PFS[AB]*.fits --ingest-task lsst.obs.pfs.gen3.PfsRawIngestTask --transfer link --fail-fast And we failed pretty quickly: lsst.ingest INFO: Successfully extracted metadata from 675 files with 0 failures lsst.ingest WARNING: Exposure PFS:111009 could not be registered: Conflict ({'tracking_ra': 227.298840317845, 'tracking_dec': 67.2128393185726}) in sync after successful insert; this is possible if the same table is being updated by a concurrent process that isn't using sync, but it may also be a bug in daf_butler. Tim Jenness points out that our Postgresql database version (10.6) is too old ("we have had problems with old postgres versions having floating point problems"). He recommends upgrading to version 16. In the mean time, I can try to do some things with a sqlite3 registry. butler create $DATASTORE --seed-config $OBS_PFS_DIR/gen3/butler.yaml --dimension-config $OBS_PFS_DIR/gen3/dimensions.yaml --override butler register-instrument $DATASTORE lsst.obs.pfs.PrimeFocusSpectrograph Repeating the ingest, it goes very smoothly: lsst.ingest INFO: Ingested 675 distinct Butler datasets Then let's do the full ingest (after restarting the datastore): butler ingest-raws $DATASTORE /data/raw/2024-{05-28,05-29,05-30,06-01,06-02,06-03,06-04}/{sps,ramps}/PFS[AB]*.fits --ingest-task lsst.obs.pfs.gen3.PfsRawIngestTask --transfer link --fail-fast lsst.ingest INFO: Successfully processed data from 795 exposures with 0 failures from exposure registration and 0 failures from file ingest. lsst.ingest INFO: Ingested 6166 distinct Butler datasets ingestPfsConfig.py $DATASTORE PFS PFS/pfsConfig skymap /data/raw/2024-{05-28,05-29,05-30,06-01,06-02,06-03,06-04}/pfsConfig/pfsConfig*.fits --transfer link There are a bunch of pfsConfig files without corresponding PFS[AB] files. No exposure records found for instrument=PFS, exposure=111013 ... No exposure records found for instrument=PFS, exposure=111979 Presumably, these are pseudo-exposures, where a pfsConfig was created for a fiber move, but the spectrographs weren't employed. makePfsDefects --mko butler write-curated-calibrations $DATASTORE PFS All right, then the next thing to do is to ingest the Gen2 calibs that we can't build using the pipeline yet: price@pfsa-usr02-gb:/work/drp/CALIB $ cp BIAS/pfsBias-2024-05-02-109212-* /work/price/pipe2d-1036/calibs/ price@pfsa-usr02-gb:/work/drp/CALIB $ cp DARK/pfsDark-2024-0* /work/price/pipe2d-1036/calibs/ price@pfsa-usr02-gb:/work/drp/CALIB $ cp DETECTORMAP/pfsDetectorMap-110921-* DETECTORMAP/pfsDetectorMap-110946-m* /work/price/pipe2d-1036/calibs/ price@pfsa-usr02-gb:/work/drp/CALIB $ cp FIBERPROFILES/pfsFiberProfiles-2024-05-05-109* /work/price/pipe2d-1036/calibs/ butler register-dataset-type $DATASTORE bias ExposureF instrument arm spectrograph --is-calibration butler register-dataset-type $DATASTORE dark ExposureF instrument arm spectrograph --is-calibration butler register-dataset-type $DATASTORE fiberFlat ExposureF instrument arm spectrograph --is-calibration butler register-dataset-type $DATASTORE detectorMap_calib DetectorMap instrument arm spectrograph --is-calibration butler register-dataset-type $DATASTORE fiberProfiles FiberProfileSet instrument arm spectrograph --is-calibration butler ingest-files $DATASTORE bias PFS/calib/ingest --prefix calibs calibs/bias.ecsv --transfer copy butler ingest-files $DATASTORE dark PFS/calib/ingest --prefix calibs calibs/dark.ecsv --transfer copy butler ingest-files $DATASTORE fiberFlat PFS/calib/ingest --prefix calibs calibs/flat.ecsv --transfer copy butler ingest-files $DATASTORE detectorMap_calib PFS/calib/ingest --prefix calibs calibs/detectorMap.ecsv --transfer copy butler ingest-files $DATASTORE fiberProfiles PFS/calib/ingest --prefix calibs calibs/fiberProfiles.ecsv --transfer copy OK, let's try it out! Not using fiberNorms, because we don't have some that we're very happy with yet. pipetask run --register-dataset-types -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib,PFS/calib/ingest -o firstTry -p '$DRP_STELLA_DIR/pipelines/reduceExposure.yaml' -d "instrument='PFS' AND exposure = 111169" --fail-fast -c mergeArms:doApplyFiberNorms=False -j 20 lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 24 quanta successfully, 0 failed and 0 remain out of total 24 quanta. Success! pipetask run --register-dataset-types -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib,PFS/calib/ingest -o ge-faint -p '$DRP_STELLA_DIR/pipelines/science.yaml' -d "instrument='PFS' AND exposure IN (110995, 110996, 111169, 111170, 111172, 111172, 111321, 111322, 111347, 111348, 111486, 111488, 111489, 111490, 111493, 111628, 111629, 111630, 111631, 111803, 111804, 111805)" --fail-fast -c mergeArms:doApplyFiberNorms=False -c preCoadd:doApplyFiberNorms=False -c coaddSpectra:doApplyFiberNorms=False -j 20 Taking a very long time to build the pipeline for just these 22 exposures... (45 minutes+). It doesn't help that we're working with sqlite rather than postgresql, but I wouldn't have thought that would be such a huge difference (it's only 270 MB). Maybe we need to rethink the idea of separating the objects (PIPE2D-1525)? Using only 111486, 111488, 111489, 111490, 111493 (i.e., 5 exposures), it's still taking over 15 minutes. Reverting the change to coaddSpectra+fluxCalibrate to write individual object spectra, the pipeline building goes MUCH faster. Can read pfsObject from a pfsCoadd (inefficient!): (lsst-scipipe-7.0.1) price@pfsa-usr02-gb:/work/price/pipe2d-1036 $ python -c 'from lsst.daf.butler import Butler; print(Butler("INTEGRATION", collections="integration/science").get("pfsCoadd.single", cat_id=1, combination="object", parameters=dict(objId=55)).target)' catId=1 tract=0 patch=0,0 objId=55 Trying with the new 'combination' dimension: defineCombination.py $DATASTORE PFS ge-faint/20240812 110995 110996 111169 111170 111172 111172 111321 111322 111347 111348 111486 111488 111489 111490 111493 111628 111629 111630 111631 111803 111804 111805 pipetask run --register-dataset-types -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib,PFS/calib/ingest -o ge-faint -p '$DRP_STELLA_DIR/pipelines/science.yaml' -d "combination = 'ge-faint/20240812'" --fail-fast -c reduceExposure:doApplyFiberNorms=False -j 20 (That built the pipeline very quickly!) Quanta Tasks ------ ------------------- 170 isr 170 reduceExposure 17 mergeArms 51 fiberNorms 17 fitPfsFluxReference 17 fitFluxCal 5 coaddSpectra lsst.ctrl.mpexec.singleQuantumExecutor INFO: Constructing task and executing quantum for label=coaddSpectra dataId={instrument: 'PFS', cat_id: 3006, combination: 'ge-faint/20240812'}. lsst.ctrl.mpexec.singleQuantumExecutor INFO: Constructing task and executing quantum for label=coaddSpectra dataId={instrument: 'PFS', cat_id: 10011, combination: 'ge-faint/20240812'}. lsst.ctrl.mpexec.singleQuantumExecutor INFO: Constructing task and executing quantum for label=coaddSpectra dataId={instrument: 'PFS', cat_id: -1, combination: 'ge-faint/20240812'}. lsst.ctrl.mpexec.singleQuantumExecutor INFO: Constructing task and executing quantum for label=coaddSpectra dataId={instrument: 'PFS', cat_id: 1002, combination: 'ge-faint/20240812'}. lsst.ctrl.mpexec.singleQuantumExecutor INFO: Constructing task and executing quantum for label=coaddSpectra dataId={instrument: 'PFS', cat_id: 1004, combination: 'ge-faint/20240812'}. Yabe-san has provided this list of calibration exposures: https://sumire.pbworks.com/w/page/157722183/Data%20Processing%20of%20Engineering%20Run%20Data Let's set up the full datastore. Setting in /work/price/pipe2d-1036/butler.yaml: registry: db: postgresql+psycopg2://pfsa-db:5435/drp Created /work/datastore-20240913 and a symlink to that as /work/datastore (I think the Gen3 datastore is relocatable, so we should easily be able to change the pointer to the active one). export DATASTORE=/work/datastore butler create $DATASTORE --seed-config /work/price/pipe2d-1036/butler.yaml --dimension-config $OBS_PFS_DIR/gen3/dimensions.yaml --override butler register-instrument $DATASTORE lsst.obs.pfs.PrimeFocusSpectrograph makePfsDefects --mko butler write-curated-calibrations $DATASTORE PFS Now, ingest the data day by day. for dd in /data/raw/{2023-12-??,2024-??-??} ; do dateobs=$(basename $dd) butler ingest-raws $DATASTORE $dd/{sps,ramps}/PFS[AB]*.fits --ingest-task lsst.obs.pfs.gen3.PfsRawIngestTask --transfer link 2>&1 | tee ingest/raws-$dateobs.log ingestPfsConfig.py $DATASTORE PFS PFS/pfsConfig $dd/pfsConfig/pfsConfig*.fits --transfer link 2>&1 | tee ingest/pfsConfig-$dateobs.log done drp=> SELECT COUNT(*) FROM exposure; count ------- 10316 (1 row) Here's how to wipe the registry database (to start over): DO $$ DECLARE r RECORD; BEGIN FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = current_schema()) LOOP EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(r.tablename) || ' CASCADE'; END LOOP; END $$; OK, now that data we care about is ingested let's build some calibs. Dates (based on range of dates for calibs taken during the run): Run 14: 2023-12-15 through 2023-12-29 Run 15: 2024-03-06 through 2024-03-18 Run 16: 2024-05-01 through 2024-05-09 Run 17: 2024-05-28 through 2024-06-03 Run 18: 2024-08-24 through 2024-09-03 Bias: pipetask run --register-dataset-types -j 60 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/calib -o price/pipe2d-1036/bias/run14 -p $DRP_STELLA_DIR/pipelines/bias.yaml -d "exposure IN (103173..103197)" --fail-fast pipetask run --register-dataset-types -j 60 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/calib -o price/pipe2d-1036/bias/run16 -p $DRP_STELLA_DIR/pipelines/bias.yaml -d "exposure IN (109212..109236)" --fail-fast pipetask run --register-dataset-types -j 60 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/calib -o price/pipe2d-1036/bias/run18 -p $DRP_STELLA_DIR/pipelines/bias.yaml -d "exposure IN (113467..113490)" --fail-fast butler certify-calibrations /work/datastore price/pipe2d-1036/bias/run14 PFS/calib bias --begin-date 2000-01-01T00:00:00 --end-date 2024-04-30T23:59:59 butler certify-calibrations /work/datastore price/pipe2d-1036/bias/run16 PFS/calib bias --begin-date 2024-05-01T00:00:00 --end-date 2024-08-23T23:59:59 butler certify-calibrations /work/datastore price/pipe2d-1036/bias/run18 PFS/calib bias --begin-date 2024-08-24T00:00:00 To remove those certifications (e.g., so we can adjust the date): butlerDecertify.py /work/datastore PFS/calib bias --begin-date 2000-01-01T00:00:00 --end-date 2024-04-30T23:59:59 butlerDecertify.py /work/datastore PFS/calib bias --begin-date 2024-05-01T00:00:00 --end-date 2024-08-23T23:59:59 butlerDecertify.py /work/datastore PFS/calib bias --begin-date 2024-08-24T00:00:00 Added a script to remove datasets: butlerCleanRun.py /work/datastore price/pipe2d-1036/bias/* biasProc Dark: pipetask run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/calib -o price/pipe2d-1036/dark/run14 -p $DRP_STELLA_DIR/pipelines/dark.yaml -d "exposure IN (103198..103222)" --fail-fast pipetask run --register-dataset-types -j 60 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/calib -o price/pipe2d-1036/dark/run16 -p $DRP_STELLA_DIR/pipelines/dark.yaml -d "exposure IN (109237..109261)" --fail-fast pipetask run --register-dataset-types -j 60 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/calib -o price/pipe2d-1036/dark/run18 -p $DRP_STELLA_DIR/pipelines/dark.yaml -d "exposure IN (113491..113515)" --fail-fast butler certify-calibrations /work/datastore price/pipe2d-1036/dark/run14 PFS/calib dark --begin-date 2000-01-01T00:00:00 --end-date 2024-04-30T23:59:59 butler certify-calibrations /work/datastore price/pipe2d-1036/dark/run16 PFS/calib dark --begin-date 2024-05-01T00:00:00 --end-date 2024-08-23T23:59:59 butler certify-calibrations /work/datastore price/pipe2d-1036/dark/run18 PFS/calib dark --begin-date 2024-08-24T00:00:00 butlerCleanRun.py /work/datastore price/pipe2d-1036/dark/* darkProc darkScaled (321 GB --> 6.3 GB) Flats: Let's first start with fake flats. mkdir -p $WORKDIR/flats makeFakeFlat.py $WORKDIR/flats --doNir (lsst-scipipe-7.0.1) price@pfsa-usr02-gb:/work/price/pipe2d-1036/flats $ cat flats.ecsv # %ECSV 1.0 # --- # datatype: # - {name: filename, datatype: string} # - {name: instrument, datatype: string} # - {name: arm, datatype: string} # - {name: spectrograph, datatype: int64} # - {name: detector, datatype: int64} # schema: astropy-2.0 filename instrument arm spectrograph detector /work/price/pipe2d-1036/flats/pfsFakeFlat-b1.fits PFS b 1 0 /work/price/pipe2d-1036/flats/pfsFakeFlat-b2.fits PFS b 2 3 /work/price/pipe2d-1036/flats/pfsFakeFlat-b3.fits PFS b 3 6 /work/price/pipe2d-1036/flats/pfsFakeFlat-b4.fits PFS b 4 9 /work/price/pipe2d-1036/flats/pfsFakeFlat-r1.fits PFS r 1 1 /work/price/pipe2d-1036/flats/pfsFakeFlat-r2.fits PFS r 2 4 /work/price/pipe2d-1036/flats/pfsFakeFlat-r3.fits PFS r 3 7 /work/price/pipe2d-1036/flats/pfsFakeFlat-r4.fits PFS r 4 10 /work/price/pipe2d-1036/flats/pfsFakeFlat-n1.fits PFS n 1 2 /work/price/pipe2d-1036/flats/pfsFakeFlat-n2.fits PFS n 2 5 /work/price/pipe2d-1036/flats/pfsFakeFlat-n3.fits PFS n 3 8 /work/price/pipe2d-1036/flats/pfsFakeFlat-n4.fits PFS n 4 11 /work/price/pipe2d-1036/flats/pfsFakeFlat-m1.fits PFS m 1 -1 /work/price/pipe2d-1036/flats/pfsFakeFlat-m2.fits PFS m 2 -4 /work/price/pipe2d-1036/flats/pfsFakeFlat-m3.fits PFS m 3 -7 /work/price/pipe2d-1036/flats/pfsFakeFlat-m4.fits PFS m 4 -10 We haven't built any "fiberFlat" datasets yet, so we need to declare them. butler register-dataset-type /work/datastore fiberFlat ExposureF instrument arm spectrograph --is-calibration butler ingest-files /work/datastore fiberFlat PFS/fakeFlat flats.ecsv --transfer copy butler certify-calibrations $DATASTORE PFS/fakeFlat PFS/calib fiberFlat --begin-date 2000-01-01T00:00:00 --end-date 2050-12-31T23:59:59 Now, let's try to build some fiberFlats for the n arms. n1 flats were built from 91223..91233:2 91294..91334:2 92029 92031 92035..92061:2 92065 92067 92071 92073 (The n2,n3,n4 flats are fake, so we don't need to build them.) Those are from Run 11 (2023 April-May). We'll need to ingest the data. for dd in /data/raw/2023-{0[4-9],10,11}-?? ; do dateobs=$(basename $dd) butler ingest-raws $DATASTORE $dd/{sps,ramps}/PFS[AB]*.fits --ingest-task lsst.obs.pfs.gen3.PfsRawIngestTask --transfer link 2>&1 | tee ingest/raws-$dateobs.log ingestPfsConfig.py $DATASTORE PFS PFS/pfsConfig $dd/pfsConfig/pfsConfig*.fits --transfer link 2>&1 | tee ingest/pfsConfig-$dateobs.log done We'll assume the biases and darks we already have are suitable. pipetask run --register-dataset-types -j 40 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/calib -o price/pipe2d-1036/flat/run11 -p '$DRP_STELLA_DIR/pipelines/flat.yaml' -d "exposure IN (91223..91233:2, 91294..91334:2, 92029, 92031, 92035..92061:2, 92065, 92067, 92071, 92073)" --fail-fast Doesn't look as good as the Gen2 one, which I believe is mainly due to Gen2 rounding the dither values to 4 decimals instead of the standard 5. To fix that with Gen3, I'd need to hack the database (using a "WITH" statement to update multiple tables at once to avoid foreign key problems); and even then, it seems a waste since we're soon going to be using drift flats. So let's just press on with the fake flats. butler remove-runs /work/datastore price/pipe2d-1036/flat/* rm -r /work/datastore/price/pipe2d-1036/flat Next is bootstrap. Arnaud points to the following engineering fiber observations suitable for bootstrap: 113460^113461 neon 113462^113463 krypton 113464^113465 traces pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/b1 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'b' AND spectrograph = 1" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=-10 pfs.bootstrap INFO: Matched 283 lines pfs.bootstrap INFO: Median difference from detectorMap: 19.236114,1.219995 pixels pfs.bootstrap INFO: Fit 84/140 points, rms: x=0.038262 y=0.041298 total=0.020419 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: 5.057339,-2.319753 pixels pfs.bootstrap INFO: Fit 102/143 points, rms: x=0.051761 y=0.035663 total=0.030386 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/b2 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'b' AND spectrograph = 2" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=10 pfs.bootstrap INFO: Matched 285 lines pfs.bootstrap INFO: Median difference from detectorMap: -2.901038,-2.594161 pixels pfs.bootstrap INFO: Fit 78/142 points, rms: x=0.051111 y=0.123377 total=0.046579 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: -4.919652,3.295592 pixels pfs.bootstrap INFO: Fit 112/143 points, rms: x=0.072149 y=0.224434 total=0.116184 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/b3 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'b' AND spectrograph = 3" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=-15 pfs.bootstrap INFO: Matched 238 lines pfs.bootstrap INFO: Median difference from detectorMap: 1.844482,0.342184 pixels pfs.bootstrap INFO: Fit 58/120 points, rms: x=0.026297 y=0.029006 total=0.016305 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: -0.469351,-0.462867 pixels pfs.bootstrap INFO: Fit 87/118 points, rms: x=0.103167 y=0.247826 total=0.127349 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/b4 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'b' AND spectrograph = 4" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=-15 pfs.bootstrap INFO: Matched 269 lines pfs.bootstrap INFO: Median difference from detectorMap: -4.601576,-1.920829 pixels pfs.bootstrap INFO: Fit 98/136 points, rms: x=0.079212 y=0.104790 total=0.060637 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: -7.689013,-0.750501 pixels pfs.bootstrap INFO: Fit 83/133 points, rms: x=0.049719 y=0.044307 total=0.026997 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/r1 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'r' AND spectrograph = 1" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=-5 pfs.bootstrap INFO: Matched 640 lines pfs.bootstrap INFO: Median difference from detectorMap: -14.790795,-1.794825 pixels pfs.bootstrap INFO: Fit 256/318 points, rms: x=0.064467 y=0.063010 total=0.035745 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: -16.526927,-1.424033 pixels pfs.bootstrap INFO: Fit 259/322 points, rms: x=0.121113 y=0.066640 total=0.060894 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/r2 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'r' AND spectrograph = 2" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=0 Urgh, that's hitting a problem in continuum subtraction, due to a bad amplifier in one of the bias or dark. I'll come back to it. pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/r3 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'r' AND spectrograph = 3" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=20 pfs.bootstrap INFO: Matched 481 lines pfs.bootstrap INFO: Median difference from detectorMap: -1.153194,-2.327864 pixels pfs.bootstrap INFO: Fit 144/236 points, rms: x=0.126161 y=0.133572 total=0.064966 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: -3.806621,-2.865443 pixels pfs.bootstrap INFO: Fit 197/245 points, rms: x=0.191780 y=0.391344 total=0.199135 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/r4 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'r' AND spectrograph = 4" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=-15 pfs.bootstrap INFO: Matched 607 lines pfs.bootstrap INFO: Median difference from detectorMap: 11.195547,2.656991 pixels pfs.bootstrap INFO: Fit 213/299 points, rms: x=0.105882 y=0.249671 total=0.131555 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: 7.276445,0.195784 pixels pfs.bootstrap INFO: Fit 185/308 points, rms: x=0.120797 y=0.201150 total=0.099440 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/n1 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'n' AND spectrograph = 1" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=30 pfs.bootstrap INFO: Matched 248 lines pfs.bootstrap INFO: Median difference from detectorMap: -2.566777,-2.124863 pixels pfs.bootstrap INFO: Fit 209/248 points, rms: x=0.100846 y=0.233036 total=0.119256 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/n2 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'n' AND spectrograph = 2" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=15 pfs.bootstrap INFO: Matched 226 lines pfs.bootstrap INFO: Median difference from detectorMap: 0.076737,-3.065470 pixels pfs.bootstrap INFO: Fit 187/226 points, rms: x=0.071591 y=0.217355 total=0.125093 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/n3 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'n' AND spectrograph = 3" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=0 pfs.bootstrap INFO: Matched 147 lines pfs.bootstrap INFO: Median difference from detectorMap: 6.732298,0.621384 pixels pfs.bootstrap INFO: Fit 110/147 points, rms: x=0.054127 y=0.216744 total=0.120836 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/n4 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'n' AND spectrograph = 4" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=0 pfs.bootstrap INFO: Matched 256 lines pfs.bootstrap INFO: Median difference from detectorMap: -12.529434,-3.272514 pixels pfs.bootstrap INFO: Fit 198/256 points, rms: x=0.130914 y=0.175326 total=0.098206 pixels Don't have engineering fiber data for the m arms handy immediately, so we'll skip those for now. Cleaning up the r2 dark. butlerDecertify.py /work/datastore PFS/calib dark --begin-date 2024-08-24T00:00:00 --id instrument=PFS arm=r spectrograph=2 Now we need to extend the certification time for just the r2 dark. The LSST interface doesn't let me do that, so I'll need to write an upgrade. First, we'll decertify the run16 r2 dark, and then we'll certify it with the extended certification time. butlerDecertify.py /work/datastore PFS/calib dark --begin-date 2024-05-01T00:00:00 --end-date 2024-08-23T23:59:59 --id instrument=PFS arm=r spectrograph=2 butlerCertify.py /work/datastore price/pipe2d-1036/dark/run16 PFS/calib dark --begin-date 2024-05-01T00:00:00 --id instrument=PFS arm=r spectrograph=2 OK, now we can try the r2 bootstrap again: pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/bootstrap/r2 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "exposure IN (113460, 113464) AND arm = 'r' AND spectrograph = 2" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=0 pfs.bootstrap INFO: Matched 507 lines pfs.bootstrap INFO: Median difference from detectorMap: 11.728878,1.459944 pixels pfs.bootstrap INFO: Fit 196/263 points, rms: x=0.139400 y=0.213578 total=0.109586 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: 5.489392,3.879591 pixels pfs.bootstrap INFO: Fit 193/244 points, rms: x=0.090054 y=0.291388 total=0.155010 pixels That worked! So now we have all the brn bootstraps. Next we certify them. butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/b1 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/b2 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/b3 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/b4 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/r1 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/r2 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/r3 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/r4 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/n1 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/n2 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/n3 PFS/bootstrap detectorMap_bootstrap butler certify-calibrations /work/datastore price/pipe2d-1036/bootstrap/n4 PFS/bootstrap detectorMap_bootstrap butlerCleanRun.py /work/datastore price/pipe2d-1036/bootstrap/* postISRCCD And now we go from the bootstraps to the detectorMaps proper. Let's only bother with the most recent run. pipetask --log-level .=INFO run --register-dataset-types -j 60 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/bootstrap,PFS/calib -o price/pipe2d-1036/detectorMap/run18 -p '$DRP_STELLA_DIR/pipelines/detectorMap.yaml' -d "exposure IN (113590..113609, 113613)" -c measureCentroids:connections.calibDetectorMap=detectorMap_bootstrap -c fitDetectorMap:fitDetectorMap.doSlitOffsets=True --fail-fast Needed to back off to 6 cores to avoid memory issues: each process can grab 30+ GB of memory temporarily. Now, let's verify that the results are reasonable. price@pfsa-usr02-gb:/work/price/pipe2d-1036 $ grep "Final result" /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/*.json /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b1_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:06:45.275228+00:00", "message": "Final result: chi2=47371532.083258 dof=12216238 xRMS=0.024206 yRMS=0.326200 xSoften=0.022182 ySoften=0.376870 from 12188914 lines (CdI: 592, HgI: 2868, KrI: 8414, NeI: 14734, Trace: 12160342, XeI: 1964)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 25292, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 1}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 1}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b2_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:39:59.060836+00:00", "message": "Final result: chi2=35627005.478551 dof=12158239 xRMS=0.023039 yRMS=0.065054 xSoften=0.021521 ySoften=0.093016 from 12135190 lines (CdI: 593, HgI: 2094, KrI: 7888, NeI: 12069, Trace: 12110891, XeI: 1655)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 29223, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 2}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 2}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b3_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:08:02.540349+00:00", "message": "Final result: chi2=29317936.896433 dof=12604134 xRMS=0.022102 yRMS=0.053337 xSoften=0.018294 ySoften=0.079694 from 12573188 lines (CdI: 398, HgI: 2996, KrI: 9267, NeI: 17261, Trace: 12540972, XeI: 2294)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 25294, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 3}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 3}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b4_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:07:45.187819+00:00", "message": "Final result: chi2=31853291.543231 dof=12576111 xRMS=0.022673 yRMS=0.046479 xSoften=0.019199 ySoften=0.086991 from 12547312 lines (CdI: 484, HgI: 2858, KrI: 8286, NeI: 16205, Trace: 12517261, XeI: 2218)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 25295, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 4}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 4}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n1_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:35:47.043618+00:00", "message": "Final result: chi2=41240171.181280 dof=8742589 xRMS=0.014789 yRMS=0.062374 xSoften=0.025125 ySoften=0.098489 from 8699479 lines (ArI: 10026, KrI: 7676, NeI: 11225, Trace: 8655127, XeI: 15425)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 28926, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 1}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 1}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n2_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:45:36.652794+00:00", "message": "Final result: chi2=61096829.931262 dof=11762209 xRMS=0.020432 yRMS=0.056988 xSoften=0.067733 ySoften=0.088216 from 11701790 lines (ArI: 13805, KrI: 11214, NeI: 16071, Trace: 11640127, XeI: 20573)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 28696, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 2}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 2}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n3_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:34:23.137685+00:00", "message": "Final result: chi2=39056248.204410 dof=8507427 xRMS=0.015338 yRMS=0.059346 xSoften=0.017901 ySoften=0.092319 from 8448998 lines (ArI: 13065, KrI: 10539, NeI: 16138, Trace: 8389305, XeI: 19951)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 28869, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 3}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 3}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n4_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:38:03.829826+00:00", "message": "Final result: chi2=44949185.327945 dof=9712111 xRMS=0.014208 yRMS=0.069085 xSoften=0.023755 ySoften=0.101292 from 9645483 lines (ArI: 15167, KrI: 12315, NeI: 17971, Trace: 9577609, XeI: 22421)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 28834, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 4}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 4}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r1_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:09:45.155243+00:00", "message": "Final result: chi2=41607758.337424 dof=13618881 xRMS=0.017154 yRMS=0.029259 xSoften=0.015670 ySoften=0.034553 from 13473998 lines (ArI: 17307, CdI: 1826, HgI: 1035, HgII: 1, KrI: 12829, NeI: 60250, Trace: 13327867, XeI: 52883)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 25296, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 1}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 1}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r2_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:07:47.272793+00:00", "message": "Final result: chi2=35662996.166436 dof=11750491 xRMS=0.017997 yRMS=0.067572 xSoften=0.016119 ySoften=0.085964 from 11635988 lines (ArI: 14087, CdI: 1233, HgI: 883, KrI: 7911, NeI: 47905, Trace: 11520235, XeI: 43734)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 25293, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 2}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 2}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r3_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:10:11.988813+00:00", "message": "Final result: chi2=35312646.717181 dof=13958586 xRMS=0.017819 yRMS=0.022849 xSoften=0.015296 ySoften=0.020779 from 13816151 lines (ArI: 22276, CdI: 1739, HgI: 988, HgII: 2, KrI: 17203, NeI: 57367, Trace: 13672446, XeI: 44130)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 25290, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 3}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 3}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} /work/datastore/price/pipe2d-1036/detectorMap/run18/20240918T153701Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r4_price_pipe2d-1036_detectorMap_run18_20240918T153701Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-09-18T16:41:39.790055+00:00", "message": "Final result: chi2=37006655.617168 dof=13632377 xRMS=0.017202 yRMS=0.045380 xSoften=0.014975 ySoften=0.049798 from 13496442 lines (ArI: 18777, CdI: 1955, HgI: 1044, KrI: 13493, NeI: 57934, Trace: 13359255, XeI: 43984)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1337, "funcName": "measureQuality", "process": 29150, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 4}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 4}", "RUN": "price/pipe2d-1036/detectorMap/run18/20240918T153701Z"}} Camera xSoften ySoften b1 0.022182 0.376870 b2 0.021521 0.093016 b3 0.018294 0.079694 b4 0.019199 0.086991 r1 0.015670 0.034553 r2 0.016119 0.085964 r3 0.015296 0.020779 r4 0.014975 0.049798 n1 0.025125 0.098489 n2 0.067733 0.088216 n3 0.017901 0.092319 n4 0.023755 0.101292 These look reasonable, except for b1. Somehow, doSlitOffsets=True is making things *worse* for b1. Reducing the rejection threshold to 3 helps a bit: ySoften=0.255754. Also setting order=4 and soften=0.03 helps a lot more: ySoften=0.084966 Setting order=7 again: ySoften=0.122786 Setting rejection=4 again: ySoften=0.131445 With rejection=4 order=4 soften=0.03: ySoften=0.089550 OK, so let's do that. pipetask --log-level .=INFO run --register-dataset-types -j 20 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/bootstrap,PFS/calib,price/pipe2d-1036/detectorMap/run18 -o price/pipe2d-1036/detectorMap/run18/b1 -p '$DRP_STELLA_DIR/pipelines/detectorMap.yaml#fitDetectorMap' -d "exposure IN (113590..113609) AND arm = 'b' and spectrograph = 1" -c fitDetectorMap:fitDetectorMap.doSlitOffsets=True -c fitDetectorMap:fitDetectorMap.order=4 -c fitDetectorMap:fitDetectorMap.soften=0.03 -c fitDetectorMap:connections.slitOffsets=detectorMap_bootstrap.slitOffsets --fail-fast pfs.fitDetectorMap.fitDetectorMap INFO: Final result: chi2=9999556.072221 dof=10813545 xRMS=0.033106 yRMS=0.082613 xSoften=0.027971 ySoften=0.089439 from 10766614 lines (CdI: 2981, HgI: 4981, KrI: 8609, NeI: 29626, Trace: 10718477, XeI: 1940) That'll do. certifyDetectorMaps.py /work/datastore price/pipe2d-1036/detectorMap/run18/b1 PFS/calib --instrument PFS --begin-date 2000-01-01T00:00:00 --end-date 2050-12-31T23:59:59 Note that I'm using price/pipe2d-1036/detectorMap/run18/b1, and that will include all the other cameras as well because it chains back to price/pipe2d-1036/detectorMap/run18. Now, fiberProfiles! defineFiberProfilesInputs.py /work/datastore PFS run18_brn --bright 113855..113863 --dark 113845..113853 --bright 113903..113911 --dark 113893..113901 --bright 114190..114198 --dark 114180..114188 --bright 114238..114246 --dark 114228..114236 pipetask run --register-dataset-types -j 60 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/fiberProfilesInputs,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/fiberProfiles/run18 -p '$DRP_STELLA_DIR/pipelines/fitFiberProfiles.yaml' -d "profiles_run = 'run18_brn'" --fail-fast -c fitProfiles:profiles.profileRadius=10 -c fitProfiles:profiles.profileOversample=3 -c fitProfiles:profiles.profileSwath=500 butler certify-calibrations /work/datastore price/pipe2d-1036/fiberProfiles/run18 PFS/calib fiberProfiles butlerCleanRun.py /work/datastore price/pipe2d-1036/fiberProfiles/run18/* postISRCCD Finally, fiberNorms... pipetask run --register-dataset-types -j 60 -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib -o price/pipe2d-1036/fiberNorms/run18/brn -p '$DRP_STELLA_DIR/pipelines/fiberNorms.yaml' -d "exposure IN (113590..113592)" --fail-fast butler certify-calibrations /work/datastore price/pipe2d-1036/fiberNorms/run18/brn PFS/calib fiberNorms_calib butlerCleanRun.py /work/datastore price/pipe2d-1036/fiberNorms/run18/brn/* postISRCCD Now we can try some science data! defineCombination.py /work/datastore PFS price/pipe2d-1036/ge-faint 110995 110996 111169 111170 111172 111172 111321 111322 111347 111348 111486 111488 111489 111490 111493 111628 111629 111630 111631 111803 111804 111805 pipetask run --register-dataset-types -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/all,PFS/pfsConfig,PFS/calib,PFS/calib -o price/pipe2d-1036/ge-faint -p '$DRP_STELLA_DIR/pipelines/science.yaml' -d "combination = 'price/pipe2d-1036/ge-faint'" --fail-fast -j 60 Quanta Tasks ------ ------------------- 210 isr 210 reduceExposure 21 mergeArms 63 fiberNorms 21 fitPfsFluxReference 21 fitFluxCal 5 coaddSpectra Create a CHAINED collection to shorten the command-line: butler collection-chain /work/datastore PFS/default PFS/raw/all,PFS/pfsConfig,PFS/calib pipetask run --register-dataset-types -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/default -o price/pipe2d-1036/ge-faint -p '$DRP_STELLA_DIR/pipelines/science.yaml' -d "combination = 'price/pipe2d-1036/ge-faint'" --fail-fast -j 60