[PIPE2D-807] pfsObject variance is much too large Created: 08/Apr/21 Updated: 20/Apr/21 Resolved: 20/Apr/21 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | price | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Story Points: | 1 |
| Sprint: | 2DDRP-2021 A 4 |
| Reviewers: | hassan |
| Description |
|
Kazumi Murata reports that the variance for a pfsObject from the integration test is much too large compared to the flux. >>> import numpy as np >>> import matplotlib.pyplot as plt >>> from lsst.daf.persistence import Butler >>> from pfs.datamodel.utils import calculatePfsVisitHash >>> butler = Butler("INTEGRATION/rerun/integration/pipeline") >>> visits = [24, 25] >>> spectrum = butler.get("pfsObject", catId=1, tract=0, patch="0,0", objId=55, nVisit=len(visits), pfsVisitHash=calculatePfsVisitHash(visits)) >>> np.median(spectrum.flux/np.sqrt(spectrum.variance)) 1.2713986241730218e-07 >>> plt.plot(spectrum.wavelength, spectrum.flux, "k-") >>> plt.plot(spectrum.wavelength, np.sqrt(spectrum.variance), "r-") >>> plt.plot(spectrum.wavelength, spectrum.sky, "b-") >>> plt.show() |
| Comments |
| Comment by rhl [ 09/Apr/21 ] |
|
I've just taken a look at the noise in the pfsArm files from SuNSS, and it looks pretty good, maybe 10% low. So is this coming from the combination? Can Murata-san check if he's happy with the noise in the single-arm extractions for the data going into the pfsObject file? |
| Comment by Kazumi Murata [ 19/Apr/21 ] |
|
Sorry for the late response. Yes, this comes from the combination process. Single spectra have much lower variances thouth not high (order of 1). I found the main cause is dupulication of _itruediv_ }}in {{pfs.datamodel.PfsFiberArray and pfs.drp.stella.datamodel.PfsFiberArray. This leads to an extra multiplication of CALIB factor in the covariance. This is why the resultant spectra have an extremely low S/N. Avoiding the dupulication, the S/N became ~0.35, similar to the pfsSingle spectra. |
| Comment by price [ 20/Apr/21 ] |
|
Excellent detective work, Kazumi Murata, thanks! It seems I've been bitten by python's multiple inheritance (again). I'm working on a fix. |
| Comment by price [ 20/Apr/21 ] |
|
After the fix: >>> import numpy as np >>> import matplotlib.pyplot as plt >>> from lsst.daf.persistence import Butler >>> from pfs.datamodel.utils import calculatePfsVisitHash >>> butler = Butler("INTEGRATION/rerun/integration/pipeline") CameraMapper INFO: Loading exposure registry from /scratch/pprice/jenkins/integrationTest/138/INTEGRATION/registry.sqlite3 CameraMapper INFO: Loading calib registry from /scratch/pprice/jenkins/integrationTest/138/INTEGRATION/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /scratch/pprice/jenkins/integrationTest/138/INTEGRATION/CALIB/calibRegistry.sqlite3 >>> visits = [24, 25] >>> spectrum = butler.get("pfsObject", catId=1, tract=0, patch="0,0", objId=55, nVisit=len(visits), pfsVisitHash=calculatePfsVisitHash(visits)) >>> np.median(spectrum.flux/np.sqrt(spectrum.variance)) 0.37628223117773874 |
| Comment by hassan [ 20/Apr/21 ] |
|
Changes look good, thanks. |
| Comment by price [ 20/Apr/21 ] |
|
Merged. |