[PIPE2D-972] Unproperly masked/identified sky line residuals Created: 28/Jan/22  Updated: 20/Jun/23  Resolved: 17/Feb/22

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Normal
Reporter: vlebrun Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File calexp-1000n-crFix.png     JPEG File calexp-1000n.jpg     PNG File fixed_11022.png     PNG File fixed_12286.png     PNG File fixed_12350.png     PNG File Forgotten_skyline_11022.png     PNG File Sky_line_12286.png     PNG File Skyline_12350.png    
Sprint: 2DDRP-2022 A
Reviewers: hassan

 Description   

In the last weekly runs I noticed that some subtraction residuals do not have a properly estimated variance, which makes the 1D pipeline identifying them as real lines. I have already identified four of them at 11020, 11444, 12286, 12350 Angstroms (see attached plots)



 Comments   
Comment by vlebrun [ 28/Jan/22 ]

a comment after some 10s of  spectra, some of these features appear at the same location in several spectra, but other appear at various location.

Comment by price [ 02/Feb/22 ]

The n arm images have many many sky lines flagged as CRs. In the following image, I've put a green circle (right middle) at the approximate position of a strange feature from the pfsArm file. It's a sky line flagged as a CR, as are many other sky lines.

Comment by price [ 03/Feb/22 ]

After a configuration tweak, the lines aren't being flagged as CRs so much. Note that this is specific to the simulated data, where the lines are much sharper than in the images from the real instrument.
I've attached some plots of the troublesome lines identified in the original description, showing the problem is much better. The line at 12350 isn't well subtracted, but maybe that's a separate issue.


Comment by rhl [ 03/Feb/22 ]

Can you quantify, "lines aren't being flagged as CRs so much"?

Also, the original complaint was about sky subtraction residuals. vlebrun: does that mean that you weren't looking at the CR flags, or are they not being properly propagated?

Comment by vlebrun [ 03/Feb/22 ]

Actually I did not check wether it was CR or sky line residuals (did not even realize there was a flag, we should definitely use it to avoid this kind of trouble), but the fact that I found some at various location seems to indicate that CR might be the explanation

Comment by price [ 03/Feb/22 ]

You can see in the original spectra posted that there are missing pixels (most conspicuous in the lower green line).

As for quantifying, I offer the following from my notes:

Try modifying the FWHM value used by the cosmic ray finder.

repair.interp.modelPsf.defaultFwhm=3.0 (default): reduceExposure.repair INFO: Identified 58346 cosmic rays.
repair.interp.modelPsf.defaultFwhm=2.0: reduceExposure.repair INFO: Identified 20666 cosmic rays.
repair.interp.modelPsf.defaultFwhm=1.5: reduceExposure.repair INFO: Identified 12285 cosmic rays.
repair.interp.modelPsf.defaultFwhm=1.0: reduceExposure.repair INFO: Identified 8376 cosmic rays.

Even with defaultFwhm=1.5, getting LOTS of sky lines flagged (though now they tend to be fainter).
defaultFwhm=1.0 still gets some sky lines flagged, but looks much better.
Can we perhaps do defaultFwhm=1.5 and modify cond3_fac? Default cond3_fac is 2.5.

repair.interp.modelPsf.defaultFwhm=1.5 repair.cosmicray.cond3_fac=10: reduceExposure.repair INFO: Identified 1007 cosmic rays.

The result actually looks pretty good. Real CRs are getting flagged, but no (or at least, not many) sky lines.
Comment by price [ 03/Feb/22 ]

Also dropping this figure posted to Slack:

Right: current defaults. Left: repair.interp.modelPsf.defaultFwhm=1.5 repair.cosmicray.cond3_fac=10. The sky lines are left alone, but we’re still finding those real cosmics.

Comment by rhl [ 03/Feb/22 ]

Looks good, thanks. We do flag too many pixels around a CR, but that's a different (and not new?) problem.

Comment by price [ 03/Feb/22 ]

vlebrun, when reporting problems could you also please ensure you record all the necessary information to be able to reproduce the plots? In this case, the nVisit and pfsVisitHash aren't available, and I had to guess since I have two pfsObjects with different pfsVisitHash (I'm guessing because of separate processing in the "weekly" and "science" datasets). Just showing the full filename would be sufficient.

Comment by rhl [ 03/Feb/22 ]

And if you can't specify enough information for Paul then we need to fix our book-keeping.

Comment by vlebrun [ 03/Feb/22 ]

Sure I will take care of this (actually we truncate the file name when converting them into local format so it's easy to correct)

Comment by rhl [ 03/Feb/22 ]

The info you need should be in the header too

Comment by price [ 15/Feb/22 ]

The tickets/PIPE2D-972 contains fixes for PIPE2D-956 and PIPE2D-976 in addition to PIPE2D-972 (all involved improving the quality of science outputs, so I wanted to make sure fixing one didn't break the others).

Comment by hassan [ 17/Feb/22 ]

All changes look good.

Comment by price [ 17/Feb/22 ]

Merged.

Comment by price [ 20/Jun/23 ]

Just discovered that the changes to pfs_pipe2d (CR parameter tuning for sims data) hadn't been merged, and merged them.

Generated at Sat Feb 10 16:00:39 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.