[PIPE2D-633] Debug false lines in extracted spectra Created: 29/Sep/20  Updated: 29/Sep/20  Resolved: 29/Sep/20

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to PIPE2D-635 Improve FitFocalPlaneTask Done
relates to PIPE1D-14 Run 1D-DRP for the resulting products... Done

 Description   

PIPE1D-14 identifies some false lines in coadded spectra from the integration test. They appear to be coming from negative features in the sky spectra. In particular, visit=25 arm=r has a single pixel in the fiberTrace at row=116 for the eighth spectrum, which is producing a 3-sigma negative value in the extracted spectrum, but inspection of the calExp shows that pixel flux value is well within the expected noise.



 Comments   
Comment by rhl [ 29/Sep/20 ]

So you're looking at why the 2-D data isn't significant, but the 1-D result is?

Is this related to the problem that we use the background as an estimate for its variance? (population v. sample)?

Comment by price [ 29/Sep/20 ]

That's the first thing I'm investigating. There may be other causes that I will chase down as well.

I don't think this is related to how the variance is measured. At the moment, the question is how a single-pixel value of -0.14 with a variance of 21 gets extracted to a spectrum value of -1.0 with a variance of 0.11.

Comment by price [ 29/Sep/20 ]

Ah, I had mismatched the image and spectrum. The actual image value is -10.8 and variance of 12. That's a S/N of about 3, which matches the S/N in the spectrum. So that question is answered.

Now, how is that single 3-sigma low value leaking into everything else?

Comment by price [ 29/Sep/20 ]

The low value is in a sky fiber, and so the low value is an ingredient for the 1D sky subtraction model, and the subtraction of a low value makes a high value.

The 1D sky subtraction is using FitFocalPlaneTask.fit, the docstring for which says:

        This implementation is a placeholder, as it simply averages the input
        vectors instead of doing any real fitting or rejection of outliers, and
        no attention is paid to the position of the fibers on the focal plane.

I should add that there's no treatment of errors either.

So I think I now understand what's causing the problem, and the solution is to replace FitFocalPlaneTask with a proper implementation. We'll always be limited by the mere two sky fibers in the integration test, but we should be able to make that single bad pixel go away.

Comment by rhl [ 29/Sep/20 ]

If we need to add more sky fibres, let's do it.  I'd be OK with a 50:50 ratio for a test suite (as opposed to a guessed 20:80 sky:science on the sky)

Comment by price [ 29/Sep/20 ]

Debugging is done. Will be fixed on PIPE2D-635.

Generated at Sat Feb 10 15:55:37 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.