[PIPE2D-1310] Diagnose sky subtraction problems after updating calibs Created: 06/Oct/23  Updated: 18/Nov/23  Resolved: 18/Nov/23

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: EDR
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File 97252-newResid_histograms.png     PNG File 97252r1_v4-v5.png     PNG File 97252_resid.png    
Sprint: EDR+AIT

 Description   

Jared Siegel reports a significant degradation in sky subtraction performance following updating from CALIB-2023-07-v4 to CALIB-2023-07-v5. The update only involved fixing the validity ranges for the fiberProfiles, so all arms should now be using the (correct) July fiberProfiles to reduce his exposure except n3 (which didn't have fiberProfiles for April 2023).

Jared's reduction script:

. /scratch/gpfs/HSC/PFS/stack/current/loadLSST.bash

setup pfs_pipe2d w.2023.37

reduceExposure.py /projects/HSC/PFS/Subaru --calib=/projects/HSC/PFS/Subaru/CALIB --rerun=siegeljc/10 -j 8 --id visit=97252 -c isr.doFlat=False

<Fiber reassignment, to set N% to sky fibers>

reduceExposure.py /projects/HSC/PFS/Subaru --calib=/projects/HSC/PFS/Subaru/CALIB --rerun=siegeljc/10 -j 8 --id visit=97252 -c isr.doFlat=False

mergeArms.py /projects/HSC/PFS/Subaru --calib=/projects/HSC/PFS/Subaru/CALIB --rerun=siegeljc/10 -j 8 --id visit=97252


 Comments   
Comment by price [ 06/Oct/23 ]

Confirmed the changes seen by Jared Siegel. I have done the sky subtraction on 97252r1 separately for v4 calibs and v5 calibs, and subtracted. There are clear differences, as seen in the plot below.

arnaud.lefur has pointed out the existence of "shoulders" in the fiber profiles at +/- 9 pixels with an amplitude of about 1% (both in v4 and v5, but with slight differences). There appears to be a fundamental problem in the construction of the fiber profiles. Given that both v4 and v5 are bad in this regard, I'm going to aim to fix the problem and not worry about the change.

Comment by arnaud.lefur [ 06/Oct/23 ]

I wonder if it comes from flats used for the normalization in the fiberProfiles construction :
96938..96944 and those were taken without the flat screen.
I would try with 97037..97045 which were taken the day after, with the flat screen in.
Might or might not fix the issue but worth trying.

Comment by arnaud.lefur [ 07/Oct/23 ]

Follow PIPE2D-1311 for the shoulder feature investigation.
In the meantime, I've regenerated fiberProfiles for r1, changing the normalization visits.
Jared Siegel is currently re-reducing the data using those and hopefully find improved residuals.

Comment by arnaud.lefur [ 10/Oct/23 ]

Jared Siegel confirm that the new r1 profiles did improve greatly the residuals, the results are now closely in line with April data


I'm going forward and now re-building fiberProfiles for the other arm/spectrograph.
I will add those to CALIB-2023-07-v6

Comment by arnaud.lefur [ 10/Oct/23 ]

for m arm, it's slightly trickier because I don't think we've taken any traces with the flat screen in...

Comment by arnaud.lefur [ 18/Nov/23 ]

Turned out the main issue came from the visits used in the fiberProfiles normalization

Generated at Sat Feb 10 16:05:53 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.