PIPE2D-1557: Please add the ability to process n > 1 visits together for cosmic ray rejection pipetask --long-log --log-level pfs=INFO run --fail-fast --instrument lsst.obs.pfs.PrimeFocusSpectrograph -b /work/datastore -i PFS/default -o u/price/pipe2d-1557 -p "$DRP_STELLA_DIR/pipelines/reduceExposure.yaml#isr,cosmicray" -d "visit IN (115028, 115029) AND arm = 'r' and spectrograph = 1" -j 2 Need to do another ingest, so that we can (1) change the dimensions to support cosmoc-ray rejection and (2) fix the filename templates (PIPE2D-1572). /work/datastore-20241003 has registry database: postgresql+psycopg2://pfsa-db:5435/drp The current /work/datastore = /work/datastore-20241023 has: postgresql+psycopg2://pfsa-db:5435/drp_test So we'll clobber the 'drp' database for our re-ingest. First, backup the drp database. (lsst-scipipe-7.0.1) price@pfsa-usr02-gb:/work/datastore-20241003 $ /usr/local/postgresql-16.4/bin/pg_dump -h pfsa-db -p 5435 -U pfs drp > drp-20241106.sql (lsst-scipipe-7.0.1) price@pfsa-usr02-gb:/work/datastore-20241003 $ gzip drp-20241106.sql psql -h pfsa-db -p 5435 -U pfs -d drp DO $$ DECLARE r RECORD; BEGIN FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = current_schema()) LOOP EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(r.tablename) || ' CASCADE'; END LOOP; END $$; Set up the new system: price@pfsa-usr02-gb:/work $ mkdir datastore-20241106 (lsst-scipipe-7.0.1) price@pfsa-usr02-gb:/work/price/pipe2d-1557 $ cp ~/pfs/obs_pfs/gen3/butler.yaml . Edit the registry to: registry: db: postgresql+psycopg2://pfsa-db:5435/drp export DATASTORE=/work/datastore-20241106 butler create $DATASTORE --seed-config /work/price/pipe2d-1557/butler.yaml --dimension-config $OBS_PFS_DIR/gen3/dimensions.yaml --override butler register-instrument $DATASTORE lsst.obs.pfs.PrimeFocusSpectrograph makePfsDefects --mko butler write-curated-calibrations $DATASTORE PFS Next, ingest the data. The last datastore put all the raws in a single directory, so we can just copy those. (lsst-scipipe-7.0.1) price@pfsa-usr02-gb:/work/price/pipe2d-1557 $ mkdir ingest for visit in $(ls /work/datastore-20241023/PFS/raw/all/raw/*.fits | sed 's|^.*/raw_PFS_\(......\)_.*|\1|' | uniq); do echo $visit; butler ingest-raws $DATASTORE /work/datastore-20241023/PFS/raw/all/raw/raw_PFS_${visit}_*.fits --ingest-task lsst.obs.pfs.gen3.PfsRawIngestTask --transfer copy 2>&1 | tee ingest/raw-$visit.log ; done for dd in /work/datastore-20241023/PFS/pfsConfig/pfsConfig/* ; do ingestPfsConfig.py $DATASTORE PFS PFS/raw/pfsConfig $dd/*/pfsConfig_*.fits --transfer copy 2>&1 | tee ingest/pfsConfig-$(basename $dd).log ; done I'm going to generate some calibs so we can actually use the new datastore. First, let's set up PFS/calib so it's a CHAINED collection. We first need to remove the PFS/calib CALIBRATION collection, and then create a new PFS/calib CHAINED collection. butler remove-collections /work/datastore-20241106 PFS/calib (lsst-scipipe-7.0.1) price@pfsa-usr02-gb:/work/price/pipe2d-1557 $ butler query-collections $DATASTORE Name Type ---------------------------------- ---- PFS/calib/curated/19700101T000000Z RUN PFS/calib/curated/20240815T000000Z RUN PFS/calib/unbounded RUN PFS/raw/sps RUN butler collection-chain /work/datastore-20241106 PFS/calib PFS/calib/curated/20240815T000000Z PFS/calib/curated/19700101T000000Z PFS/calib/unbounded (lsst-scipipe-7.0.1) price@pfsa-usr02-gb:/work/price/pipe2d-1557 $ butler query-collections $DATASTORE Name Type ------------------------------------ ------- PFS/calib CHAINED PFS/calib/curated/20240815T000000Z RUN PFS/calib/curated/19700101T000000Z RUN PFS/calib/unbounded RUN PFS/calib/curated/19700101T000000Z RUN PFS/calib/curated/20240815T000000Z RUN PFS/calib/unbounded RUN PFS/raw/sps RUN Bias: pipetask run --register-dataset-types -j 60 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/sps,PFS/calib -o PFS/calib/pipe2d-1557/bias/run18 -p $DRP_STELLA_DIR/pipelines/bias.yaml -d "visit IN (113467..113490)" --fail-fast pipetask run --register-dataset-types -j 60 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/raw/sps,PFS/calib -o PFS/calib/pipe2d-1557/bias/run19 -p $DRP_STELLA_DIR/pipelines/bias.yaml -d "visit IN (115562..115586)" --fail-fast butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bias/run18 PFS/calib/pipe2d-1557 bias --begin-date 2024-08-24T00:00:00 --end-date 2024-10-16T00:00:00 butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bias/run19 PFS/calib/pipe2d-1557 bias --begin-date 2024-10-16T00:00:00 butlerCleanRun.py /work/datastore PFS/calib/pipe2d-1557/bias/* biasProc butler collection-chain $DATASTORE PFS/calib --mode=prepend PFS/calib/pipe2d-1557 butler collection-chain $DATASTORE PFS/defaults PFS/raw/sps,PFS/raw/pfsConfig,PFS/calib Dark: pipetask run --register-dataset-types -j 60 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/dark/run18 -p $DRP_STELLA_DIR/pipelines/dark.yaml -d "visit IN (113491..113515)" --fail-fast pipetask run --register-dataset-types -j 60 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/dark/run19 -p $DRP_STELLA_DIR/pipelines/dark.yaml -d "visit IN (115587..115611)" --fail-fast butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/dark/run18 PFS/calib/pipe2d-1557 dark --begin-date 2024-08-24T00:00:00 --end-date 2024-10-16T00:00:00 butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/dark/run19 PFS/calib/pipe2d-1557 dark --begin-date 2024-10-16T00:00:00 butlerCleanRun.py $DATASTORE PFS/calib/pipe2d-1557/dark/* darkProc darkScaled The run18 r2 dark is bad, so cover it with the run19 r2 dark. butlerDecertify.py $DATASTORE PFS/calib/pipe2d-1557 dark --begin-date 2024-08-24T00:00:00 --end-date 2024-10-16T00:00:00 --id instrument=PFS arm=r spectrograph=2 butlerDecertify.py $DATASTORE PFS/calib/pipe2d-1557 dark --begin-date 2024-10-16T00:00:00 --id instrument=PFS arm=r spectrograph=2 butlerCertify.py $DATASTORE PFS/calib/pipe2d-1557/dark/run19 PFS/calib/pipe2d-1557 dark --begin-date 2024-08-24T00:00:00 --id instrument=PFS arm=r spectrograph=2 Flats: Let's first start with fake flats. mkdir -p $WORKDIR/flats makeFakeFlat.py flats --doNir (lsst-scipipe-7.0.1) price@pfsa-usr02-gb:/work/price/pipe2d-1557/flats $ cat flats.ecsv # %ECSV 1.0 # --- # datatype: # - {name: filename, datatype: string} # - {name: instrument, datatype: string} # - {name: arm, datatype: string} # - {name: spectrograph, datatype: int64} # - {name: detector, datatype: int64} # schema: astropy-2.0 filename instrument arm spectrograph detector /work/price/pipe2d-1557/flats/pfsFakeFlat-b1.fits PFS b 1 0 /work/price/pipe2d-1557/flats/pfsFakeFlat-b2.fits PFS b 2 3 /work/price/pipe2d-1557/flats/pfsFakeFlat-b3.fits PFS b 3 6 /work/price/pipe2d-1557/flats/pfsFakeFlat-b4.fits PFS b 4 9 /work/price/pipe2d-1557/flats/pfsFakeFlat-r1.fits PFS r 1 1 /work/price/pipe2d-1557/flats/pfsFakeFlat-r2.fits PFS r 2 4 /work/price/pipe2d-1557/flats/pfsFakeFlat-r3.fits PFS r 3 7 /work/price/pipe2d-1557/flats/pfsFakeFlat-r4.fits PFS r 4 10 /work/price/pipe2d-1557/flats/pfsFakeFlat-n1.fits PFS n 1 2 /work/price/pipe2d-1557/flats/pfsFakeFlat-n2.fits PFS n 2 5 /work/price/pipe2d-1557/flats/pfsFakeFlat-n3.fits PFS n 3 8 /work/price/pipe2d-1557/flats/pfsFakeFlat-n4.fits PFS n 4 11 /work/price/pipe2d-1557/flats/pfsFakeFlat-m1.fits PFS m 1 -1 /work/price/pipe2d-1557/flats/pfsFakeFlat-m2.fits PFS m 2 -4 /work/price/pipe2d-1557/flats/pfsFakeFlat-m3.fits PFS m 3 -7 /work/price/pipe2d-1557/flats/pfsFakeFlat-m4.fits PFS m 4 -10 butler register-dataset-type $DATASTORE fiberFlat ExposureF instrument arm spectrograph --is-calibration butler ingest-files $DATASTORE fiberFlat PFS/fakeFlat flats.ecsv --transfer copy butler certify-calibrations $DATASTORE PFS/fakeFlat PFS/calib/pipe2d-1557 fiberFlat --begin-date 2000-01-01T00:00:00 --end-date 2050-12-31T23:59:59 butler collection-chain $DATASTORE PFS/calib --mode=extend PFS/fakeFlat Bootstrap: pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/b1 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'b' AND spectrograph = 1" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=-10 pfs.bootstrap INFO: Matched 286 lines pfs.bootstrap INFO: Median difference from detectorMap: 19.231057,1.220237 pixels pfs.bootstrap INFO: Fit 112/143 points, rms: x=0.051793 y=0.056004 total=0.032339 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: 5.057333,-2.319753 pixels pfs.bootstrap INFO: Fit 102/143 points, rms: x=0.051764 y=0.035663 total=0.030389 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/b2 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'b' AND spectrograph = 2" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=10 pfs.bootstrap INFO: Matched 286 lines pfs.bootstrap INFO: Median difference from detectorMap: -2.901808,-2.558421 pixels pfs.bootstrap INFO: Fit 79/143 points, rms: x=0.052034 y=0.122948 total=0.046159 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: -4.919651,3.295596 pixels pfs.bootstrap INFO: Fit 112/143 points, rms: x=0.072140 y=0.224435 total=0.116224 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/b3 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'b' AND spectrograph = 3" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=-15 pfs.bootstrap INFO: Matched 242 lines pfs.bootstrap INFO: Median difference from detectorMap: 1.844565,0.342188 pixels pfs.bootstrap INFO: Fit 58/120 points, rms: x=0.026306 y=0.029005 total=0.016306 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: -0.469351,-0.462867 pixels pfs.bootstrap INFO: Fit 105/122 points, rms: x=0.106191 y=0.409511 total=0.219015 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/b4 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'b' AND spectrograph = 4" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=-15 pfs.bootstrap INFO: Matched 272 lines pfs.bootstrap INFO: Median difference from detectorMap: -4.601575,-1.920829 pixels pfs.bootstrap INFO: Fit 98/136 points, rms: x=0.079203 y=0.104790 total=0.060639 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: -7.675594,-0.751151 pixels pfs.bootstrap INFO: Fit 87/136 points, rms: x=0.049697 y=0.044538 total=0.027403 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/r1 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'r' AND spectrograph = 1" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=-5 pfs.bootstrap INFO: Matched 647 lines pfs.bootstrap INFO: Median difference from detectorMap: -14.773185,-1.798937 pixels pfs.bootstrap INFO: Fit 256/320 points, rms: x=0.064464 y=0.063010 total=0.035745 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: -16.441840,-1.454350 pixels pfs.bootstrap INFO: Fit 275/327 points, rms: x=0.131986 y=0.070623 total=0.067473 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/r2 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'r' AND spectrograph = 2" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=0 pfs.bootstrap INFO: Matched 514 lines pfs.bootstrap INFO: Median difference from detectorMap: 11.707699,1.369469 pixels pfs.bootstrap INFO: Fit 193/265 points, rms: x=0.139960 y=0.208104 total=0.106649 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: 5.435626,3.018340 pixels pfs.bootstrap INFO: Fit 219/249 points, rms: x=0.094558 y=0.351264 total=0.189897 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/r3 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'r' AND spectrograph = 3" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=20 pfs.bootstrap INFO: Matched 480 lines pfs.bootstrap INFO: Median difference from detectorMap: -1.153193,-2.396805 pixels pfs.bootstrap INFO: Fit 151/236 points, rms: x=0.131113 y=0.139698 total=0.069743 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: -3.745062,-3.036626 pixels pfs.bootstrap INFO: Fit 193/244 points, rms: x=0.191958 y=0.388329 total=0.198178 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/r4 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'r' AND spectrograph = 4" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=-15 pfs.bootstrap INFO: Matched 609 lines pfs.bootstrap INFO: Median difference from detectorMap: 11.193389,2.600554 pixels pfs.bootstrap INFO: Fit 212/303 points, rms: x=0.106143 y=0.250201 total=0.131397 pixels pfs.bootstrap INFO: Updating detectorMap... pfs.bootstrap INFO: Median difference from detectorMap: 7.279279,0.132381 pixels pfs.bootstrap INFO: Fit 183/306 points, rms: x=0.122556 y=0.202748 total=0.098740 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/n1 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'n' AND spectrograph = 1" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=30 pfs.bootstrap INFO: Matched 274 lines pfs.bootstrap INFO: Median difference from detectorMap: -2.543895,-2.139709 pixels pfs.bootstrap INFO: Fit 231/274 points, rms: x=0.097091 y=0.241702 total=0.121547 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/n2 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'n' AND spectrograph = 2" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=15 pfs.bootstrap INFO: Matched 242 lines pfs.bootstrap INFO: Median difference from detectorMap: 0.085457,-3.131910 pixels pfs.bootstrap INFO: Fit 214/242 points, rms: x=0.077765 y=0.295795 total=0.170332 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/n3 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'n' AND spectrograph = 3" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=0 pfs.bootstrap INFO: Matched 188 lines pfs.bootstrap INFO: Median difference from detectorMap: 6.738138,0.643128 pixels pfs.bootstrap INFO: Fit 165/188 points, rms: x=0.086916 y=0.535395 total=0.336744 pixels pipetask --log-level .=INFO run --register-dataset-types -j 20 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/bootstrap/n4 -p '$DRP_STELLA_DIR/pipelines/bootstrap.yaml' -d "visit IN (113460, 113464) AND arm = 'n' AND spectrograph = 4" --fail-fast -c isr:doCrosstalk=False -c bootstrap:profiles.profileRadius=2 -c bootstrap:profiles.profileSwath=2500 -c bootstrap:profiles.profileOversample=3 -c 'bootstrap:targetType=[ENGINEERING]' -c bootstrap:spectralOffset=0 pfs.bootstrap INFO: Matched 277 lines pfs.bootstrap INFO: Median difference from detectorMap: -12.456475,-3.192794 pixels pfs.bootstrap INFO: Fit 216/277 points, rms: x=0.134419 y=0.176319 total=0.097435 pixels butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/b1 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/b2 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/b3 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/b4 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/r1 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/r2 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/r3 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/r4 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/n1 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/n2 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/n3 PFS/bootstrap/run18 detectorMap_bootstrap butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/bootstrap/n4 PFS/bootstrap/run18 detectorMap_bootstrap butlerCleanRun.py $DATASTORE PFS/calib/pipe2d-1557/bootstrap/* postISRCCD detectorMaps: Need to back off on the number of cores in use in order to prevent memory exhaustion. pipetask --log-level .=INFO run --register-dataset-types -j 12 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults,PFS/bootstrap/run18 -o PFS/calib/pipe2d-1557/detectorMap/run18 -p '$DRP_STELLA_DIR/pipelines/detectorMap.yaml' -d "visit IN (113590..113609, 113613)" -c measureCentroids:connections.calibDetectorMap=detectorMap_bootstrap -c fitDetectorMap:fitDetectorMap.doSlitOffsets=True -c fitDetectorMap:fitDetectorMap.order=4 -c fitDetectorMap:fitDetectorMap.soften=0.03 --fail-fast grep "Final result" /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/*.json | sort /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b1_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:41:13.060659+00:00", "message": "Final result: chi2=12199889.844891 dof=12906818 xRMS=0.033630 yRMS=0.064297 xSoften=0.028666 ySoften=0.071193 from 12857912 lines (CdI: 2525, HgI: 7166, KrI: 8802, NeI: 29684, Trace: 12807800, XeI: 1935)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13742, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 1}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 1}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b2_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:48:17.123848+00:00", "message": "Final result: chi2=13323316.989288 dof=12826024 xRMS=0.035027 yRMS=0.094932 xSoften=0.030439 ySoften=0.101377 from 12778771 lines (CdI: 2919, HgI: 8141, KrI: 8187, NeI: 27529, Trace: 12730310, XeI: 1685)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 14000, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 2}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 2}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b3_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:41:24.159970+00:00", "message": "Final result: chi2=9924669.059913 dof=12869595 xRMS=0.030668 yRMS=0.063488 xSoften=0.024384 ySoften=0.065275 from 12817249 lines (CdI: 2694, HgI: 7900, KrI: 9426, NeI: 31198, Trace: 12763675, XeI: 2356)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13789, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 3}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 3}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b4_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:41:39.284919+00:00", "message": "Final result: chi2=10195953.827115 dof=12933757 xRMS=0.030852 yRMS=0.050845 xSoften=0.025027 ySoften=0.049537 from 12882162 lines (CdI: 3043, HgI: 8291, KrI: 9168, NeI: 30076, Trace: 12829357, XeI: 2227)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13959, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 4}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 4}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n1_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:46:01.279549+00:00", "message": "Final result: chi2=19330873.134109 dof=13330405 xRMS=0.039228 yRMS=0.080703 xSoften=0.040400 ySoften=0.090892 from 13244872 lines (ArI: 19117, HgI: 6090, KrI: 14403, NeI: 19026, Trace: 13158139, XeI: 28097)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13573, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 1}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 1}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n2_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:55:50.757849+00:00", "message": "Final result: chi2=42414147.220432 dof=16395841 xRMS=0.059217 yRMS=0.081980 xSoften=0.071202 ySoften=0.093871 from 16337744 lines (ArI: 13025, HgI: 3970, KrI: 9569, NeI: 13199, Trace: 16278445, XeI: 19536)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13646, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 2}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 2}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n3_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:44:40.786119+00:00", "message": "Final result: chi2=19695097.740476 dof=12674597 xRMS=0.039272 yRMS=0.094656 xSoften=0.039006 ySoften=0.107252 from 12599380 lines (ArI: 16448, HgI: 5636, KrI: 12830, NeI: 17773, Trace: 12522941, XeI: 23752)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13880, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 3}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 3}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n4_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:41:42.120360+00:00", "message": "Final result: chi2=19595012.857482 dof=13572283 xRMS=0.039003 yRMS=0.081056 xSoften=0.040034 ySoften=0.093069 from 13482452 lines (ArI: 19802, HgI: 6467, KrI: 15154, NeI: 20275, Trace: 13391417, XeI: 29337)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13614, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 4}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 4}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r1_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:42:57.298024+00:00", "message": "Final result: chi2=10091381.063074 dof=14506539 xRMS=0.027302 yRMS=0.042906 xSoften=0.023270 ySoften=0.029400 from 14340153 lines (ArI: 23689, CdI: 1041, HgI: 3159, KrI: 20748, NeI: 66393, Trace: 14172561, XeI: 52562)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13518, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 1}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 1}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r2_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:50:02.557497+00:00", "message": "Final result: chi2=8634970.858692 dof=14061774 xRMS=0.025742 yRMS=0.041436 xSoften=0.021303 ySoften=0.027223 from 13909845 lines (ArI: 24145, CdI: 922, HgI: 2673, KrI: 16725, NeI: 60272, Trace: 13756708, XeI: 48400)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13925, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 2}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 2}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r3_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:50:46.609671+00:00", "message": "Final result: chi2=9433727.681864 dof=14469732 xRMS=0.027057 yRMS=0.043238 xSoften=0.021729 ySoften=0.031020 from 14310012 lines (ArI: 25739, CdI: 1047, HgI: 2842, KrI: 20963, NeI: 63787, Trace: 14149064, XeI: 46570)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13686, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 3}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 3}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r4_PFS_calib_pipe2d-1557_detectorMap_run18_20241107T225211Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-07T23:43:25.164740+00:00", "message": "Final result: chi2=9097082.084627 dof=14261742 xRMS=0.026382 yRMS=0.040026 xSoften=0.021584 ySoften=0.025640 from 14108104 lines (ArI: 23736, CdI: 1052, HgI: 3053, KrI: 20805, NeI: 61217, Trace: 13953256, XeI: 44985)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 13843, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 4}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 4}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run18/20241107T225211Z"}} Camera xSoften ySoften b1 0.028666 0.071193 b2 0.030439 0.101377 b3 0.024384 0.065275 b4 0.025027 0.049537 n1 0.040400 0.090892 n2 0.071202 0.093871 n3 0.039006 0.107252 n4 0.040034 0.093069 r1 0.023270 0.029400 r2 0.021303 0.027223 r3 0.021729 0.031020 r4 0.021584 0.025640 certifyDetectorMaps.py $DATASTORE PFS/calib/pipe2d-1557/detectorMap/run18 PFS/calib/pipe2d-1557 --instrument PFS --begin-date 2000-01-01T00:00:00 --end-date 2024-10-16T00:00:00 pipetask --log-level .=INFO run --register-dataset-types -j 12 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/detectorMap/run19 -p '$DRP_STELLA_DIR/pipelines/detectorMap.yaml' -d "visit IN (115824..115838, 115836..115838, 115842..115844)" -c fitDetectorMap:fitDetectorMap.order=4 -c fitDetectorMap:fitDetectorMap.soften=0.03 --fail-fast Missing some pfsConfig files. ingestPfsConfig.py $DATASTORE PFS PFS/raw/pfsConfig /data/raw/2024-10-24/pfsConfig/pfsConfig-0x*.fits --transfer=copy 2>&1 | tee ingest/pfsConfig-2024-10-24.log lsst.daf.butler.registry._exceptions.ConflictingDefinitionError: Existing dataset type and dataId does not match new dataset: {'dataset_type_id': 256, 'instrument': 'PFS', 'visit': 115802, 'dataset_id': UUID('4ce4699c-a6f2-46b1-a802-8a5c3005566d'), 'new dataset_id': UUID('d255b580-04c0-4583-bad1-7db6d24f0074'), 'collection_id': 942, 'new collection_id': 942} Looks like there were problems setting the pfsDesignId with some visits. The following have header fixes defined in pfs_utils: ingestPfsConfig.py $DATASTORE PFS PFS/raw/pfsConfig /data/raw/2024-10-24/pfsConfig/pfsConfig-0x*-11582[12].fits --transfer=copy 2>&1 | tee ingest/pfsConfig-115821-115822.log for vv in $(seq 115824 115838); do ingestPfsConfig.py $DATASTORE PFS PFS/raw/pfsConfig /data/raw/2024-10-24/pfsConfig/pfsConfig-0x*-$vv.fits --transfer=copy 2>&1 | tee ingest/pfsConfig-$vv.log ; done Can't find detectorMap_calib. Looks like I need to mess with the validity ranges. butlerDecertify.py $DATASTORE PFS/calib/pipe2d-1557 detectorMap_calib --begin-date 2000-01-01T00:00:00 --end-date 2024-10-16T00:00:00 butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/detectorMap/run18/certify PFS/calib/pipe2d-1557 detectorMap_calib --begin-date 2000-01-01T00:00:00 OK, that's working now... (lsst-scipipe-7.0.1) price@pfsa-usr02-gb:/work/price/pipe2d-1557 $ grep "Final result" /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z/fitDetectorMap_log/*.json | sort /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b2_PFS_calib_pipe2d-1557_detectorMap_run19_20241108T155505Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-08T16:15:20.040441+00:00", "message": "Final result: chi2=12406797.036583 dof=6714743 xRMS=0.047327 yRMS=0.046063 xSoften=0.044668 ySoften=0.043036 from 6651230 lines (CdI: 6335, HgI: 14004, KrI: 9063, NeI: 32456, Trace: 6587681, XeI: 1691)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 128161, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 2}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 2}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b3_PFS_calib_pipe2d-1557_detectorMap_run19_20241108T155505Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-08T16:14:17.842435+00:00", "message": "Final result: chi2=8717138.849434 dof=6672948 xRMS=0.040213 yRMS=0.053077 xSoften=0.036014 ySoften=0.054511 from 6608292 lines (CdI: 6439, HgI: 12978, KrI: 10200, NeI: 32717, Trace: 6543600, XeI: 2358)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 127877, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 3}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 3}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z/fitDetectorMap_log/fitDetectorMap_log_PFS_b4_PFS_calib_pipe2d-1557_detectorMap_run19_20241108T155505Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-08T16:14:22.232532+00:00", "message": "Final result: chi2=7194061.069218 dof=6741115 xRMS=0.036090 yRMS=0.051128 xSoften=0.031309 ySoften=0.049619 from 6677771 lines (CdI: 6340, HgI: 13473, KrI: 9985, NeI: 31420, Trace: 6614391, XeI: 2162)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 127842, "processName": "task-{instrument: 'PFS', arm: 'b', spectrograph: 4}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'b', spectrograph: 4}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n2_PFS_calib_pipe2d-1557_detectorMap_run19_20241108T155505Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-08T16:13:38.105627+00:00", "message": "Final result: chi2=20252690.919754 dof=5382340 xRMS=0.068366 yRMS=0.146028 xSoften=0.082824 ySoften=0.171787 from 5307491 lines (ArI: 14512, HgI: 9219, KrI: 12644, NeI: 15216, Trace: 5232612, XeI: 23288)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 128017, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 2}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 2}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n3_PFS_calib_pipe2d-1557_detectorMap_run19_20241108T155505Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-08T16:13:02.152974+00:00", "message": "Final result: chi2=11521421.370528 dof=4703202 xRMS=0.049345 yRMS=0.147528 xSoften=0.051659 ySoften=0.171474 from 4609262 lines (ArI: 18129, HgI: 10960, KrI: 15653, NeI: 20368, Trace: 4515292, XeI: 28860)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 127916, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 3}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 3}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z/fitDetectorMap_log/fitDetectorMap_log_PFS_n4_PFS_calib_pipe2d-1557_detectorMap_run19_20241108T155505Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-08T16:14:23.681282+00:00", "message": "Final result: chi2=21308566.898522 dof=6403183 xRMS=0.064812 yRMS=0.159181 xSoften=0.084381 ySoften=0.191179 from 6289213 lines (ArI: 21885, HgI: 14451, KrI: 19603, NeI: 23757, Trace: 6175213, XeI: 34304)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 128092, "processName": "task-{instrument: 'PFS', arm: 'n', spectrograph: 4}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'n', spectrograph: 4}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r2_PFS_calib_pipe2d-1557_detectorMap_run19_20241108T155505Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-08T16:14:53.779192+00:00", "message": "Final result: chi2=10087902.751186 dof=7951962 xRMS=0.037812 yRMS=0.046807 xSoften=0.034976 ySoften=0.035944 from 7772291 lines (ArI: 26882, CdI: 1595, HgI: 4600, KrI: 18852, NeI: 68578, Trace: 7592584, XeI: 59200)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 127808, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 2}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 2}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r3_PFS_calib_pipe2d-1557_detectorMap_run19_20241108T155505Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-08T16:15:03.687825+00:00", "message": "Final result: chi2=16377932.237931 dof=7969332 xRMS=0.048830 yRMS=0.053990 xSoften=0.046878 ySoften=0.045534 from 7798523 lines (ArI: 27054, CdI: 1586, HgI: 4111, KrI: 20481, NeI: 64695, Trace: 7627678, XeI: 52918)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 127950, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 3}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 3}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z"}} /work/datastore-20241106/PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z/fitDetectorMap_log/fitDetectorMap_log_PFS_r4_PFS_calib_pipe2d-1557_detectorMap_run19_20241108T155505Z.json:{"name": "pfs.fitDetectorMap.fitDetectorMap", "asctime": "2024-11-08T16:15:05.040665+00:00", "message": "Final result: chi2=7449556.567050 dof=7903114 xRMS=0.032574 yRMS=0.046407 xSoften=0.028651 ySoften=0.035739 from 7725080 lines (ArI: 26847, CdI: 1569, HgI: 4613, KrI: 20650, NeI: 68020, Trace: 7547010, XeI: 56371)", "levelno": 20, "levelname": "INFO", "filename": "fitDistortedDetectorMap.py", "pathname": "/home/price/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py", "lineno": 1425, "funcName": "measureQuality", "process": 128048, "processName": "task-{instrument: 'PFS', arm: 'r', spectrograph: 4}", "MDC": {"LABEL": "fitDetectorMap:{instrument: 'PFS', arm: 'r', spectrograph: 4}", "RUN": "PFS/calib/pipe2d-1557/detectorMap/run19/20241108T155505Z"}} Camera xSoften ySoften b2 0.044668 0.043036 b3 0.036014 0.054511 b4 0.031309 0.049619 n2 0.082824 0.171787 n3 0.051659 0.171474 n4 0.084381 0.191179 r2 0.034976 0.035944 r3 0.046878 0.045534 r4 0.028651 0.035739 Not good numbers for n2,n3,n4; maybe the detectors were warmed up or something? But we don't care much about n arm right now, so going to push forward. certifyDetectorMaps.py $DATASTORE PFS/calib/pipe2d-1557/detectorMap/run19 PFS/calib/pipe2d-1557 --instrument PFS --begin-date 2024-10-16T00:00:00 lsst.daf.butler.registry._exceptions.ConflictingDefinitionError: 9 validity range conflicts certifying datasets of type detectorMap_calib into PFS/calib/pipe2d-1557 for range [2024-10-16 00:00:00.000000, None). Now we need to remove the open-ended validity range that we put in. butlerDecertify.py $DATASTORE PFS/calib/pipe2d-1557 detectorMap_calib --begin-date 2000-01-01T00:00:00 butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/detectorMap/run18/certify PFS/calib/pipe2d-1557 detectorMap_calib --begin-date 2000-01-01T00:00:00 --end-date 2024-10-16T00:00:00 certifyDetectorMaps.py $DATASTORE PFS/calib/pipe2d-1557/detectorMap/run19 PFS/calib/pipe2d-1557 --instrument PFS --begin-date 2024-10-16T00:00:00 butlerCleanRun.py $DATASTORE PFS/calib/pipe2d-1557/detectorMap/* postISRCCD fiberProfiles: defineFiberProfilesInputs.py $DATASTORE PFS run18_brn --bright 113855..113863 --dark 113845..113853 --bright 113903..113911 --dark 113893..113901 --bright 114190..114198 --dark 114180..114188 --bright 114238..114246 --dark 114228..114236 defineFiberProfilesInputs.py $DATASTORE PFS run19_brn --bright 116168..116176 --dark 116140..116148 --bright 116234..116242 --dark 116206..116214 --bright 116386..116394 --dark 116358..116366 --bright 116584..116592 --dark 116556..116564 pipetask run --register-dataset-types -j 60 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults,PFS/fiberProfilesInputs -o PFS/calib/pipe2d-1557/fiberProfiles/run18 -p '$DRP_STELLA_DIR/pipelines/fitFiberProfiles.yaml' -d "profiles_run = 'run18_brn'" --fail-fast -c fitProfiles:profiles.profileRadius=10 -c fitProfiles:profiles.profileOversample=3 -c fitProfiles:profiles.profileSwath=500 pipetask run --register-dataset-types -j 60 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults,PFS/fiberProfilesInputs -o PFS/calib/pipe2d-1557/fiberProfiles/run19 -p '$DRP_STELLA_DIR/pipelines/fitFiberProfiles.yaml' -d "profiles_run = 'run19_brn'" --fail-fast -c fitProfiles:profiles.profileRadius=10 -c fitProfiles:profiles.profileOversample=3 -c fitProfiles:profiles.profileSwath=500 butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/fiberProfiles/run18 PFS/calib/pipe2d-1557 fiberProfiles --begin-date 2000-01-01T00:00:00 --end-date 2024-10-16T00:00:00 butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/fiberProfiles/run19 PFS/calib/pipe2d-1557 fiberProfiles --begin-date 2024-10-16T00:00:00 butlerCleanRun.py $DATASTORE PFS/calib/pipe2d-1557/fiberProfiles/* postISRCCD fiberNorms: pipetask run --register-dataset-types -j 60 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/fiberNorms/run18/brn -p '$DRP_STELLA_DIR/pipelines/fiberNorms.yaml' -d "visit IN (113590..113592)" --fail-fast pipetask run --register-dataset-types -j 60 -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o PFS/calib/pipe2d-1557/fiberNorms/run19/brn -p '$DRP_STELLA_DIR/pipelines/fiberNorms.yaml' -d "visit IN (115842..115844)" --fail-fast butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/fiberNorms/run18/brn PFS/calib/pipe2d-1557 fiberNorms_calib --begin-date 2000-01-01T00:00:00 --end-date 2024-10-16T00:00:00 butler certify-calibrations $DATASTORE PFS/calib/pipe2d-1557/fiberNorms/run19/brn PFS/calib/pipe2d-1557 fiberNorms_calib --begin-date 2024-10-16T00:00:00 butlerCleanRun.py $DATASTORE PFS/calib/pipe2d-1557/fiberNorms/* postISRCCD Science: defineCombination.py $DATASTORE PFS price/pipe2d-1557/deep2-3 113956 113958 113960 113962 defineVisitGroup.py $DATASTORE PFS 113956 113958 113960 113962 pipetask run --register-dataset-types -b $DATASTORE --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o u/price/pipe2d-1557/deep2-3 -p '$DRP_STELLA_DIR/pipelines/science.yaml' -d "combination = 'price/pipe2d-1557/deep2-3'" --fail-fast -j 60 The new datastore (`/work/datastore-20241106`) is ready for use. We'll need to coordinate merging PIPE2D-1557 and making the datastore primary. If you want to use the new datastore before we merge PIPE2D-1557, you'll need the `tickets/PIPE2D-1557` branches of datamodel, obs_pfs and pfs_pipe2d, and the `tickets/PIPE2D-1557-future` branch of drp_stella (the `tickets/PIPE2D-1557` branch of drp_stella supports CR identification with the old datastore). I've ingested the data from runs 18 and 19, and will add more over the weekend. Some important changes: 1. The `reduceExposure.yaml` (and therefore the `calibrateExposure.yaml` and `science.yaml`) pipelines require defining visit groups, I'm hoping that the grouping information will get put in the raw FITS headers so we can make this more automatic, but for now the visits in a group need to be specified manually, e.g., `defineVisitGroup.py $DATASTORE PFS 113956 113958 113960 113962`. _Even if the visit group is composed of a single visit, it needs to be defined._ 2. There is a new pipeline, `observing.yaml`, for reducing exposures that are not part of a visit group (i.e., using the old cosmic-ray identification task); as the name suggests, this is intended to support observing, when we don't know the visit groups or don't even have the full visit group collected yet. I've moved the `fiberNorms` measurement out of the `reduceExposure.yaml` pipeline and put it in the `observing.yaml` pipeline, since the former is intended to support science observations which don't include quartzes requiring `fiberNorms` measurements. 3. I've replaced the usual `PFS/calibs` `CALIBRATION` collection with a `CHAINED` collection, as I believe the LSST calibration team recommends. I've generated some `brn` calibs for runs 18 and 19 under `PFS/calib/pipe2d-1557` and linked this in. I've defined the `PFS/defaults` `CHAINED` collection to include the `PFS/calibs`, `PFS/raw/sps` and `PFS/raw/pfsConfig` collections. I would ask that the `drpActor` be updated to ingest the pfsConfigs to collection `PFS/raw/pfsConfig`. Now, ingest the remaining data. for dd in $(ls -d /work/drp/202?-* | sort -r) ; do dateobs=$(basename $dd) butler ingest-raws $DATASTORE $dd/PFS[AB]*.fits --ingest-task lsst.obs.pfs.gen3.PfsRawIngestTask --transfer copy 2>&1 | tee ingest/raws-$dateobs.log ingestPfsConfig.py $DATASTORE PFS PFS/raw/pfsConfig /work/drp/pfsConfig/$dateobs/pfsConfig*.fits --transfer copy 2>&1 | tee ingest/pfsConfig-$dateobs.log done