[PIPE2D-972] Unproperly masked/identified sky line residuals Created: 28/Jan/22 Updated: 20/Jun/23 Resolved: 17/Feb/22 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Normal |
| Reporter: | vlebrun | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Sprint: | 2DDRP-2022 A |
| Reviewers: | hassan |
| Description |
|
In the last weekly runs I noticed that some subtraction residuals do not have a properly estimated variance, which makes the 1D pipeline identifying them as real lines. I have already identified four of them at 11020, 11444, 12286, 12350 Angstroms (see attached plots) |
| Comments |
| Comment by vlebrun [ 28/Jan/22 ] |
|
a comment after some 10s of spectra, some of these features appear at the same location in several spectra, but other appear at various location. |
| Comment by price [ 02/Feb/22 ] |
|
The n arm images have many many sky lines flagged as CRs. In the following image, I've put a green circle (right middle) at the approximate position of a strange feature from the pfsArm file. It's a sky line flagged as a CR, as are many other sky lines. |
| Comment by price [ 03/Feb/22 ] |
|
After a configuration tweak, the lines aren't being flagged as CRs so much. Note that this is specific to the simulated data, where the lines are much sharper than in the images from the real instrument. |
| Comment by rhl [ 03/Feb/22 ] |
|
Can you quantify, "lines aren't being flagged as CRs so much"? Also, the original complaint was about sky subtraction residuals. vlebrun: does that mean that you weren't looking at the CR flags, or are they not being properly propagated? |
| Comment by vlebrun [ 03/Feb/22 ] |
|
Actually I did not check wether it was CR or sky line residuals (did not even realize there was a flag, we should definitely use it to avoid this kind of trouble), but the fact that I found some at various location seems to indicate that CR might be the explanation |
| Comment by price [ 03/Feb/22 ] |
|
You can see in the original spectra posted that there are missing pixels (most conspicuous in the lower green line). As for quantifying, I offer the following from my notes: Try modifying the FWHM value used by the cosmic ray finder. repair.interp.modelPsf.defaultFwhm=3.0 (default): reduceExposure.repair INFO: Identified 58346 cosmic rays. repair.interp.modelPsf.defaultFwhm=2.0: reduceExposure.repair INFO: Identified 20666 cosmic rays. repair.interp.modelPsf.defaultFwhm=1.5: reduceExposure.repair INFO: Identified 12285 cosmic rays. repair.interp.modelPsf.defaultFwhm=1.0: reduceExposure.repair INFO: Identified 8376 cosmic rays. Even with defaultFwhm=1.5, getting LOTS of sky lines flagged (though now they tend to be fainter). defaultFwhm=1.0 still gets some sky lines flagged, but looks much better. Can we perhaps do defaultFwhm=1.5 and modify cond3_fac? Default cond3_fac is 2.5. repair.interp.modelPsf.defaultFwhm=1.5 repair.cosmicray.cond3_fac=10: reduceExposure.repair INFO: Identified 1007 cosmic rays. The result actually looks pretty good. Real CRs are getting flagged, but no (or at least, not many) sky lines. |
| Comment by price [ 03/Feb/22 ] |
|
Also dropping this figure posted to Slack: |
| Comment by rhl [ 03/Feb/22 ] |
|
Looks good, thanks. We do flag too many pixels around a CR, but that's a different (and not new?) problem. |
| Comment by price [ 03/Feb/22 ] |
|
vlebrun, when reporting problems could you also please ensure you record all the necessary information to be able to reproduce the plots? In this case, the nVisit and pfsVisitHash aren't available, and I had to guess since I have two pfsObjects with different pfsVisitHash (I'm guessing because of separate processing in the "weekly" and "science" datasets). Just showing the full filename would be sufficient. |
| Comment by rhl [ 03/Feb/22 ] |
|
And if you can't specify enough information for Paul then we need to fix our book-keeping. |
| Comment by vlebrun [ 03/Feb/22 ] |
|
Sure I will take care of this (actually we truncate the file name when converting them into local format so it's easy to correct) |
| Comment by rhl [ 03/Feb/22 ] |
|
The info you need should be in the header too |
| Comment by price [ 15/Feb/22 ] |
|
The tickets/ |
| Comment by hassan [ 17/Feb/22 ] |
|
All changes look good. |
| Comment by price [ 17/Feb/22 ] |
|
Merged. |
| Comment by price [ 20/Jun/23 ] |
|
Just discovered that the changes to pfs_pipe2d (CR parameter tuning for sims data) hadn't been merged, and merged them. |