[PIPE2D-1197] subtractSky1d applied to spectrograph=3 produces excessive noise Created: 31/Mar/23  Updated: 28/Apr/23  Resolved: 28/Apr/23

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Figure 1-5.png     PNG File Figure 1 (60).png     PNG File Figure 1 (61).png     PNG File Figure 1 (62).png     PNG File Figure 1-6.png     File pipe2d-1197.ipynb     JPEG File pipe2d-1197.jpg    

 Description   

rhl reports:

With this how-to-repeat, these plots are pretty-much a smoking gun for a problem with spectrograph != 1:

dataId.update(spectrograph=1)
spec = butler.get('pfsArm', dataId)
pfsConfig = butler.get("pfsConfig", dataId)
​
from pfs.drp.stella.subtractSky1d import subtractSky1d
sky1d = butler.get("sky1d", dataId)
subtractSky1d(spec, pfsConfig, sky1d)
​
spec.flux[(spec.mask & ~spec.flags["REFLINE"]) != 0x0] = np.NaN
​
fig = 1; plt.close(fig); fig = plt.figure(fig)
spec.plot(spec.fiberId[100:101], figure=fig, show=False)
plt.ylim(-1000, 1000)
​
plt.title(dataId);


 Comments   
Comment by price [ 31/Mar/23 ]

I have a fix that I'm going to put on PIPE2D-907 with the other changes for the engineering data release.

(It's a completely boring, accounting bug.)

This fix will invalidate existing sky1d products (they cannot be accessed from the butler any more).

Comment by rhl [ 02/Apr/23 ]

The normalisation is still wrong, I think. Here are the calexp, the spectra-subtracted, and the sky-subtracted images for 83149.

You can also see artefacts from the background subtraction model, but that's a different issue.  The code for the last two plots only differs in doing the sky subtraction:

pfsArm = butler.get("pfsArm", dataId)
if subtractSky:
    from pfs.drp.stella.subtractSky1d import subtractSky1d 
    sky1d = butler.get("sky1d", dataId)
    subtractSky1d(pfsArm, pfsConfig, sky1d)
Comment by price [ 04/Apr/23 ]

Your code doesn't match the figures, so it's not clear to me what you're doing.

One thing that I noticed is that when we make an image from spectra, the image includes the values in the sky arrays. That's probably not what we want (we only want the flux), so I've changed that (on PIPE2D-907).

Perhaps that, combined with using the old branch might explain the figures you posted. If not, please provide me detailed instructions on how to reproduce the problem. I've attached my notebook in case it's helpful.

Comment by rhl [ 05/Apr/23 ]

The "image includes the values in the sky arrays" comment was the clue.  Yes, the code works with the tip of PIPE2D-907; thanks.

One thing that would be helpful would be an option to spectra.makeImage(exp.getDimensions(), traces) that subtracted the .sky array rather than the .flux.  Yes;  I can stuff sky into flux, but it's ugly.

 

Comment by price [ 28/Apr/23 ]

This merged as part of PIPE2D-907.

I added a toggle to SpectrumSet.makeImage to have it construct an image from the sky instead of from the flux.

Generated at Sat Feb 10 16:04:10 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.