[PIPE2D-1143] sky subtraction is bad for a certain fraction of fibers in the faint galaxy frames taken during Nov run Created: 13/Jan/23  Updated: 17/Feb/23  Resolved: 17/Feb/23

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major
Reporter: Kiyoto Yabe Assignee: price
Resolution: Done Votes: 0
Labels: collab-datarelease
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File bad_subtraction_example1.png     PNG File fibers_with_large_residual_on_pfi.png     PNG File good_subtraction_example1.png     PNG File image (7).png     File reduction_run08_galaxies_example.sh     JPEG File Xnip2023-01-20_17-15-59.jpg    
Issue Links:
Blocks
blocks PIPE2D-1140 Reduce data obtained in Nov run for t... Done
is blocked by DAMD-145 Add facility to hold history/metrics ... Done
Relates
relates to PIPE2D-1165 Engineering Data Release 1 Done
relates to PIPE2D-1140 Reduce data obtained in Nov run for t... Done
Sprint: 2DDRP-2023 A
Reviewers: hassan

 Description   

Reducing the data for faint galaxies (visit=83244..83249) taken during the Nov run, I just noticed that the sky subtraction for some fibers is (still) significantly bad. Commands I ran are in this reduction_run08_galaxies_example.sh. Some examples of good and bad subtraction are shown below. Those are coadded pfsObjects but I see the same things in pfsMerged and in a single visit.



 Comments   
Comment by price [ 14/Jan/23 ]

Could you please tell me how the sky fiber positions were selected?

Comment by Kiyoto Yabe [ 14/Jan/23 ]

Sky fibers used for the subtraction? I didn't specify them so config.selectSky.fiberFilter='ALL'.

Comment by rhl [ 14/Jan/23 ]

I thought that this was a "proper" extra-Galactic field from HSC, so shouldn't we have SKY objects? Will we have SKY objects in the next engineering runs?

I don't think that this can explain the problem, though, as we are over subtracting the OH.

Comment by Kiyoto Yabe [ 14/Jan/23 ]

Murata-san's sky object DB was used and we have 500 SKY fibers (in entire focal plane not SM1&3) in this field.

Comment by rhl [ 15/Jan/23 ]

Ah, OK. So they were specified in the pfsDesign, which is all that matters.

You shouldn't ever need to config.selectSky.fiberFilter except for interactive fiddling (a better solution is to write a pfsConfig to the rerun).

Comment by Kiyoto Yabe [ 17/Jan/23 ]

FWIW, attached is the location of fibers with large residual (using an isolated sky line) and sky fibers on PFI plane. It looks that there is no strong (anti-) correlation?

Comment by price [ 21/Jan/23 ]

This is due to fibers being in the penumbra of nearby black spots. Based on a bunch of sky lines from two exposures (83149 and 83244), the throughput appears to be linear with distance from the black spot (slope of 1.232/mm) up to a distance of 1.108 mm.
I've added a correction for this effect in reduceExposure. The correction modifies the normalisation, so that the flux is still what is extracted from the image.
In order to keep track of the correction (and other reduction operations and quality measurements), I've added a new element to the spectra, currently called "notes". This is an addition to the datamodel, so needs approval via a DAMD ticket.
Using the fixed code, I get the following sky-subtracted merged spectrum for visit=83244 fiberId=478 (which observes objId=43158588759244917, which had the bad sky subtraction):

Comment by price [ 04/Feb/23 ]

This includes the changes of DAMD-145 (adding reduction notes to spectral products) in order to record the values that were used.

Comment by hassan [ 11/Feb/23 ]

Changes approved. Could not see any obvious issues with the proposed changes, including the new notes mechanism. Perhaps a slope to model the penumbra may be too simplistic for the future, but it works now and is a good start.

Comment by price [ 12/Feb/23 ]

Need to check: it looks like I might be making the correction twice (once in extractSpectra, and once in reduceExposure).

Comment by price [ 14/Feb/23 ]

I double-checked, and I am not making the correction twice. The second correction is in pfs.drp.stella.pipelines.extractSpectra (part of the Gen3 pipeline that replaces reduceExposure), not pfs.drp.stella.extractSpectra (the building block).

Comment by price [ 17/Feb/23 ]

Merged to master.

Generated at Sat Feb 10 16:03:20 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.