[PIPE2D-1343] Modify reduceProfiles to use dot-roach darks Created: 13/Jan/24  Updated: 24/Apr/24  Resolved: 24/Apr/24

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: EngRun
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PDF File pfsFiberProfilesStats-2023-12-29-104399-b1.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104399-b2.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104399-b3.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104399-b4.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104399-n1.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104399-n2.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104399-n3.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104399-r1.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104399-r2.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104399-r3.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104399-r4.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104447-m1.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104447-m2.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104447-m3.pdf     PDF File pfsFiberProfilesStats-2023-12-29-104447-m4.pdf     PDF File pfsFiberProfilesStats-2024-03-09-106956-b1.pdf     PDF File pfsFiberProfilesStats-2024-03-09-106956-b2.pdf     PDF File pfsFiberProfilesStats-2024-03-09-106956-b4.pdf     PDF File pfsFiberProfilesStats-2024-03-09-106956-n2.pdf     PDF File pfsFiberProfilesStats-2024-03-09-106956-n3.pdf     PDF File pfsFiberProfilesStats-2024-03-09-106956-r1.pdf     PDF File pfsFiberProfilesStats-2024-03-09-106956-r2.pdf     PDF File pfsFiberProfilesStats-2024-03-09-106956-r3.pdf     PDF File pfsFiberProfilesStats-2024-03-09-106956-r4.pdf     PDF File pfsFiberProfilesStats-2024-03-10-107218-m1.pdf     PDF File pfsFiberProfilesStats-2024-03-10-107218-m2.pdf     PDF File pfsFiberProfilesStats-2024-03-10-107218-m3.pdf     PDF File pfsFiberProfilesStats-2024-03-10-107218-m4.pdf    
Issue Links:
Relates
relates to PIPE2D-1405 Generate detectorMaps for Mar. 2024 e... Done
Sprint: PreEngRun15Mar, Engrun15Mar, postRun15Mar_or_preRun16May, EngRun16May
Reviewers: Kiyoto Yabe

 Description   

We have observed quartzes with every fourth fiber exposed (mod4, a.k.a. "shaka"), with slit dithers to measure the profile at high resolution. To deal with fibers that cannot move, we have also observed the quartzes with all fibers hidden, using the same slit dithers. These "dot-roach darks" need to be subtracted from the mod4 exposures.

 Comments   
Comment by price [ 02/Apr/24 ]

I modified how profiles are constructed: we measure the profiles for the different groups (different fiber settings) separately and then merge them. Below are the command-lines I used to construct the profiles from Run 14 (December 2023) and Run 15 (March 2024). I also added some QA metrics and plots for evaluating the profiles, which I'll attach here.

# Run 14

reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run14-group1 --id visit=104399..104407 --darkId visit=104391..104397 -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 12 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run14-group2 --id visit=104351..104359 --darkId visit=104343..104349 -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 12 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run14-group3 --id visit=104303..104311 --darkId visit=104295..104301 -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 12 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run14-group4 --id visit=104253..104261 --darkId visit=104245..104251 -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 12 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run14-group1 --id visit=104447..104455 arm=m --darkId visit=104439..104445 arm=m -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 4 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run14-group2 --id visit=104527..104535 arm=m --darkId visit=104519..104525 arm=m -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 4 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run14-group3 --id visit=104578..104586 arm=m --darkId visit=104570..104576 arm=m -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 4 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run14-group4 --id visit=104626..104634 arm=m --darkId visit=104618..104624 arm=m -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 4 &
wait

for spectrograph in 1 2 3 4; do for arm in b r n; do mergeFiberProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run14-merge --id visit=104409..104411 arm=$arm spectrograph=$spectrograph --profiles /work/drp/rerun/price/pipe2d-1343-20240328-run14-group1/FIBERPROFILES/pfsFiberProfiles-2023-12-29-104399-$arm$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run14-group2/FIBERPROFILES/pfsFiberProfiles-2023-12-29-104351-$arm$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run14-group3/FIBERPROFILES/pfsFiberProfiles-2023-12-29-104303-$arm$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run14-group4/FIBERPROFILES/pfsFiberProfiles-2023-12-29-104253-$arm$spectrograph.fits --visits 104399 104351 104303 104253 --darkVisits 104391 104343 104295 104245 --no-versions & mergeFiberProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run14-merge --id visit=104636..104638 arm=m spectrograph=$spectrograph --profiles /work/drp/rerun/price/pipe2d-1343-20240328-run14-group1/FIBERPROFILES/pfsFiberProfiles-2023-12-29-104447-m$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run14-group2/FIBERPROFILES/pfsFiberProfiles-2023-12-30-104527-m$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run14-group3/FIBERPROFILES/pfsFiberProfiles-2023-12-30-104578-m$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run14-group4/FIBERPROFILES/pfsFiberProfiles-2023-12-30-104626-m$spectrograph.fits --visits 104447 104527 104578 104626 --darkVisits 104439 104519 104570 104618 --no-versions & done ; done ; wait

# Run 15:

reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run15-group1 --id visit=106956..106964 --darkId visit=106948..106954 -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 12 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run15-group2 --id visit=107002..107010 --darkId visit=106994..107000 -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 12 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run15-group3 --id visit=107048..107056 --darkId visit=107040..107046 -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 12 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run15-group4 --id visit=107096..107104 --darkId visit=107088..107094 -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 12 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run15-group1 --id visit=107218..107226 arm=m --darkId visit=107210..107216 arm=m -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 4 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run15-group2 --id visit=107264..107272 arm=m --darkId visit=107256..107262 arm=m -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 4 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run15-group3 --id visit=107372..107380 arm=m --darkId visit=107364..107370 arm=m -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 4 &
reduceProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run15-group4 --id visit=107537..107547 arm=m --darkId visit=107527..107535 arm=m -c reduceExposure.isr.doFlat=False profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=500 -j 4 &
wait

for spectrograph in 1 2 3 4; do for arm in b r n; do mergeFiberProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run15-merge --id visit=106866..106870 arm=$arm spectrograph=$spectrograph --profiles /work/drp/rerun/price/pipe2d-1343-20240328-run15-group1/FIBERPROFILES/pfsFiberProfiles-2024-03-09-106956-$arm$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run15-group2/FIBERPROFILES/pfsFiberProfiles-2024-03-09-107002-$arm$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run15-group3/FIBERPROFILES/pfsFiberProfiles-2024-03-09-107048-$arm$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run15-group4/FIBERPROFILES/pfsFiberProfiles-2024-03-09-107096-$arm$spectrograph.fits --visits 106956 107002 107048 107096 --darkVisits 106948 106994 107040 107088  --no-versions & mergeFiberProfiles.py /work/drp --calib /work/drp/CALIB-2024-03-v2 --rerun price/pipe2d-1343-20240328-run15-merge --id visit=106898..106902 arm=m spectrograph=$spectrograph --profiles /work/drp/rerun/price/pipe2d-1343-20240328-run15-group1/FIBERPROFILES/pfsFiberProfiles-2024-03-10-107218-m$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run15-group2/FIBERPROFILES/pfsFiberProfiles-2024-03-10-107264-m$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run15-group3/FIBERPROFILES/pfsFiberProfiles-2024-03-10-107372-m$spectrograph.fits /work/drp/rerun/price/pipe2d-1343-20240328-run15-group4/FIBERPROFILES/pfsFiberProfiles-2024-03-11-107537-m$spectrograph.fits --visits 107218 107264 107372 107537 --darkVisits 107210 107256 107364 107527 --no-versions & done ; done ; wait
Comment by price [ 02/Apr/24 ]

Kiyoto Yabe, could you please confirm that you understand how to create profiles using the new code (from the posted examples), and that the new profiles are of reasonable quality for inclusion in the calibs (from the attached QA plots)? I don't think it's necessary for you to review the code changes.

Comment by Kiyoto Yabe [ 11/Apr/24 ]

Sorry for being slow. At least, I confirmed that I could make the same profiles with those commands and the branch. I understand what those try to do but probably need to look into the codes to deeply understand what those actually do. I don't have any expertise to do the code review. By the way, is there any reason why the first 2 visits in allHidden for each group (e.g. visit=106946..106947 for group1) are not used?

Comment by price [ 11/Apr/24 ]

I believe the first two visits have the same slit offset, so they don't add value to the process. Perhaps we could try coadding them in the future (so long as the visits are consecutive and the slit hasn't moved).

Comment by arnaud.lefur [ 11/Apr/24 ]

yeah, the dither sequence has always 7 visits with those positions :  [0, -0.4, -0.2, 0, 0.2, 0.4, 0]

and we always add couple of visit in 0, so that we get 5 duplicate of the 0 position.

Those were meant to be coadded, unless we show they are different enough, I don't think so but worth checking.

Comment by Kiyoto Yabe [ 12/Apr/24 ]

Thank you. I made profiles by adding additional visits but the results don't change at all as far as I see the some statistics plots.

Comment by rhl [ 12/Apr/24 ]

Reducing nominally-identical data would provide useful QA; if they are really consistent we'd gain S/N and robustness to CRs by using all the data.

Comment by price [ 24/Apr/24 ]

Merged to master.

Generated at Sat Apr 19 13:28:48 JST 2025 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.