[PIPE2D-782] Spectra extracted from data used to construct fiberTraces are not flat Created: 17/Mar/21 Updated: 20/Mar/21 Resolved: 19/Mar/21 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | rhl | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
extract_three_slits.png
pipe2d-782-fixed.png
pipe2d-782.png
quartz_spectra.png
three_slits.png
|
| Sprint: | 2DDRP-2021 A3 |
| Reviewers: | hassan |
| Description |
|
If I run the code in constructFiberProfilesTask.py to make new fiberProfiles from a single exposure and then use them to extract the spectra from the same data they should be identically 1.0, but they are not (see import os import numpy as np import matplotlib.pyplot as plt %matplotlib notebook import lsst.daf.persistence as dafPersist # --------------------------------- base, rerun = "/projects/HSC/PFS/Subaru", os.path.join("rhl", "sunss") dataDir = os.path.join(base, "rerun", rerun) calibRoot = os.path.join(base, "CALIB-SuNSS") butler = dafPersist.Butler(dataDir, calibRoot=calibRoot) # --------------------------------- import pfs.drp.stella.buildFiberProfiles BuildFiberProfilesTask = pfs.drp.stella.buildFiberProfiles.BuildFiberProfilesTask config = BuildFiberProfilesTask.ConfigClass() config.doBlindFind = False buildFiberProfilesTask = BuildFiberProfilesTask(config) # --------------------------------- dataId = dict(visit=46243, spectrograph=1, arm='r') # domeflat exp = butler.get("calexp", dataId) detMap = butler.get("detectorMap", dataId) pfsConfig = butler.get("pfsConfig", dataId) results = buildFiberProfilesTask.run(exp, detMap, pfsConfig) profiles, centers = results.profiles, results.centers traces = profiles.makeFiberTracesFromDetectorMap(detMap) spectra0 = traces.extractSpectra(exp.maskedImage) for i, ss in enumerate(spectra0): profiles[ss.fiberId].norm = ss.flux traces = profiles.makeFiberTracesFromDetectorMap(detMap) spectra = traces.extractSpectra(exp.maskedImage) # --------------------------------- fig = 1; plt.close(fig); fig = plt.figure(fig) for ss in spectra: if 4 < ss.fiberId < 140: plt.plot(ss.flux/np.median(ss.flux), alpha=0.5) plt.ylim(1 + 0.01*np.array([-1, 1])) plt.xlabel("pixel") plt.ylabel("Extracted flux") plt.title(f"{'%(visit)d %(arm)s%(spectrograph)d' % dataId}"); |
| Comments |
| Comment by price [ 17/Mar/21 ] |
|
Note that profiles.makeFiberTracesFromDetectorMap(detMap) doesn't use the exact same centers used in building the fiberProfiles. For that, you want to use profiles.makeFiberTraces(detMap.bbox.getDimensions(), centers). But I don't think that causes what you're seeing, because you're being consistent later on. Your config has profileRadius=5, which is going to cause overlapping profiles in the dense fiber region (e.g., try profiles[167].plot()). Therefore, changing the normalisation has a ripple effect through all the overlaps (the off-diagonal values in the tri-di matrix are about 1/10 of the diagonal). The fibers outside the dense fiber region have values close to unity. If you iterate once (for ss in spectra: profiles[ss.fiberId].norm *= ss.flux), the resultant spectra are flat: |
| Comment by rhl [ 17/Mar/21 ] |
|
I know about the centres difference (I think my use of the detMap is better). I thought about the overlaps, but only showed results from the non-overlapping fibres below fiberId 140 — why do problems in the dense pack part of the slit propagate out to the ends? I'd expect the matrix to be blocked. |
| Comment by rhl [ 17/Mar/21 ] |
|
Here are two more figures, showing just three fibres from the non-dense part of the slit. |
| Comment by price [ 19/Mar/21 ] |
|
I reproduced this with a simple setup (using the synthetic package in drp_stella, which uses straight Gaussian traces). When the traces are perfectly vertical, the extractions are perfect (setting the profile normalisation to the extracted spectrum results in unity when re-extracting). When the traces are slanted, the extractions are not perfect (I see some 1% deviations when the slope is 0.02). I believe this is due to the profile not accurately modeling the data. |
| Comment by rhl [ 19/Mar/21 ] |
|
I don't understand this, as we define the normalisation of the traces from spectra extracted using the same traces, so errors in the modelling should cancel out. |
| Comment by price [ 19/Mar/21 ] |
|
Of course you're right, and that was the important clue that I kept forgetting: it turns out that when there was no normalisation specified, the traces did not sum to unity, but when there was a normalisation specified, then I multiplied by the normalisation divided by the sum across the trace: there was an extra normalisation. After fixing this, I'm getting 5e-8 precision, even with a slope of 0.05. This will be ready for review soon. |
| Comment by price [ 19/Mar/21 ] |
|
Demonstration that it's fixed: |
| Comment by hassan [ 19/Mar/21 ] |
|
Fix looks fine. No additional comments. |
| Comment by price [ 19/Mar/21 ] |
|
Merged. |
| Comment by price [ 20/Mar/21 ] |
|
Added another commit to move from profiles.makeFiberTraces to profiles.makeFiberTracesFromDetectorMap, and merged after RHL signed off on Slack. |
