[DAMD-113] Write pfsArm in electrons Created: 12/Jun/21 Updated: 31/Mar/22 Resolved: 21/Jul/21 |
|
| Status: | Done |
| Project: | Data Model |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | price | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Story Points: | 4 | ||||||||
| Sprint: | 2DDRP-2021 A 6, 2DDRP-2021 A 7 | ||||||||
| Reviewers: | hassan | ||||||||
| Description |
|
It would be useful for engineering and debugging pipelines if the extracted spectra in the pfsArm files were in units of electrons; currently they are relative to the fiberProfile normalisation, which comes from extracting a quartz). Similarly, it would similarly be useful if the spectra in the pfsMerged files were close to electrons (or, at least, normalised like they are close to electrons). This will require small documentation changes in the datamodel, and some modest changes to the pipeline to support. |
| Comments |
| Comment by price [ 12/Jun/21 ] |
|
This issue replaces |
| Comment by price [ 08/Jul/21 ] |
|
I gave up having the pfsMerged files be in units of (something close to) counts, because that requires choosing a normalisation using multiple arms, and then choosing the exact same normalisation later for coaddition when there's the potential to have a different input set. But pfsArm is now in counts. |
| Comment by rhl [ 08/Jul/21 ] |
|
I think you should probably have discussed this decision on slack. I don't quite understand why the book-keeping can't be done in a safe way (probably by including some extra information in the files). Can you explain? |
| Comment by price [ 08/Jul/21 ] |
|
I don't like the idea of writing the normalization to a file that's uninvolved in the coaddition process, but I guess we could write a new dataset type that contains the normalisation. |
| Comment by rhl [ 08/Jul/21 ] |
|
I must not understand. What information would you need to write to an extra file, and when would it be read? I'd have thought it'd go to the pfsMerged file and everything would be transparent |
| Comment by price [ 09/Jul/21 ] |
|
The new plan is to add an extra HDU to pfsArm containing the normalisation. |
| Comment by price [ 15/Jul/21 ] |
|
I think I've got this working. Below I show me poking around the results of the integration test. Note that pfsArm and pfsMerged fluxes of the flux standard star (about i=18 mag) are in the thousands, and the normalised fluxes (ratio of the flux to the quartz) are about 0.08. For the single-object products, the flux in the pfsSingle and pfsObject are about 2.25e5 nJy = 18 mag. I'm going to clean up the commit history and then submit for review. >>> import numpy as np >>> from lsst.daf.persistence import Butler >>> butler = Butler("INTEGRATION/rerun/integration/pipeline") CameraMapper INFO: Loading exposure registry from /scratch/pprice/damd-113/INTEGRATION/registry.sqlite3 CameraMapper INFO: Loading calib registry from /scratch/pprice/damd-113/INTEGRATION/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /scratch/pprice/damd-113/INTEGRATION/CALIB/calibRegistry.sqlite3 >>> index = 0 >>> pfsArm = butler.get("pfsArm", visit=24, arm="b") >>> select = (pfsArm.wavelength[index] > 500) & (pfsArm.wavelength[index] < 600) >>> print(np.median(pfsArm.flux[index][select])) 3193.8977383679285 >>> print(np.median(pfsArm.flux[index][select]/pfsArm.norm[index][select])) 0.08098331564220362 >>> pfsMerged = butler.get("pfsMerged", visit=24) >>> select = (pfsMerged.wavelength[index] > 500) & (pfsMerged.wavelength[index] < 600) >>> print(np.median(pfsMerged.flux[index][select])) 4250.031433514167 >>> print(np.median(pfsMerged.flux[index][select]/pfsMerged.norm[index][select])) 0.07860174756195284 >>> pfsSingle = butler.get("pfsSingle", catId=0, tract=0, patch="0,0", objId=1, visit=24) >>> select = (pfsSingle.wavelength > 600) & (pfsSingle.wavelength < 700) >>> print(np.median(pfsSingle.flux[select])) 226348.42616116628 >>> select = (pfsSingle.fluxTable.wavelength > 600) & (pfsSingle.fluxTable.wavelength < 700) >>> print(np.median(pfsSingle.fluxTable.flux[select])) 225535.49838282933 >>> pfsObject = butler.get("pfsObject", catId=0, tract=0, patch="0,0", objId=1, nVisit=2, pfsVisitHash=0x60241444a9af118c) >>> select = (pfsObject.wavelength > 600) & (pfsObject.wavelength < 700) >>> print(np.median(pfsObject.flux[select])) 226177.83216109022 >>> select = (pfsObject.fluxTable.wavelength > 600) & (pfsObject.fluxTable.wavelength < 700) & (pfsObject.fluxTable.mask == 0) >>> print(np.median(pfsObject.fluxTable.flux[select])) 224241.48945935623 |
| Comment by price [ 15/Jul/21 ] |
|
Everything appears to be working. |
| Comment by hassan [ 20/Jul/21 ] |
|
No additional comments on proposed changes. |
| Comment by price [ 21/Jul/21 ] |
|
Merged. |