[DAMD-63] Redesign and implement pfsObject FluxTbl Created: 26/Jun/19  Updated: 25/Oct/19  Resolved: 25/Oct/19

Status: Done
Project: Data Model
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File fluxTable.png    
Issue Links:
Blocks
blocks PIPE2D-438 Implement fluxtbl Done
Relates
relates to DAMD-65 Retrospective approval of 2D pipeline... Done
Story Points: 3
Sprint: 2DDRP-2019 F, 2DDRP-2019 G, 2DDRP-2019 H
Reviewers: hassan

 Description   

I screwed up the implementation of the pfsObject FITS HDUs when I implemented the end-to-end pipeline, since I didn't understand what FLUXTBL was meant for. We need to rename what is now FLUXTBL to FLUX, redesign FLUXTBL (the original design did not support multiple visits) and provide an implementation for populating it, and update datamodel.txt.



 Comments   
Comment by price [ 26/Jun/19 ]

hassan points out that the CONFIG and MASK extensions have been removed to. CONFIG has been removed in favour of pfsConfig, and I believe MASK has been subsumed into what will be FLUX. Let's fix these up in datamodel.txt at the same time.

Comment by price [ 30/Jul/19 ]

Here's a plot of a coadd (black) and the fluxTable contents (purple, yellow, red, cyan).

>>> from lsst.daf.persistence import Butler
>>> butler = Butler("INTEGRATION/rerun/integration/pipeline")
>>> obj = butler.get("pfsObject", catId=1, tract=0, patch="0,0", objId=0x5d, nVisit=4, pfsVisitHash=0x42cfc4af8cc94582)
>>> fig, axes = obj.fluxTable.plot(show=False)
>>> axes.plot(obj.wavelength, obj.flux, 'k-')
>>> fig.show()
Comment by rhl [ 30/Jul/19 ]

I am confused by the comment about CONFIG.  It is not replaced by the pfsConfig file, it was explicitly denormalised for the benefit of end users.

I also don't understand the remark about multiple epochs.  The pfsObject file is the combined spectrum; if you want the input visits you need to go back to the pfsArm files.

Comment by price [ 30/Jul/19 ]

Now I'm confused too. You say that pfsObject is a combined spectrum, but a spectrum that has been coadded from multiple observations doesn't (necessarily) have a single top-end configuration. So what are you expecting to find in the CONFIG extension?

What is the point of having the FLUXTBL as opposed to just the flux vector of the coadd? My understanding is:

    A coadded spectrum is a convenient representation of the spectrum of an
    object with multiple observations, but the act of coaddition necessarily
    involves a reduction in the amount of data through resampling in wavelength
    and replacing a distribution of fluxes with a statistical representation.
    The intention of a ``FluxTable`` is to provide the full set of original data
    so that the user can have the option to inspect and fit the observations
    themselves rather than a reduced, resampled and censored version of the
    observations. This may be useful for identifying artifacts that have leaked
    into the coadds (e.g., cosmic rays) or transient spectral features, or for
    doing Hogg-ish Bayesian fitting.
Comment by price [ 06/Aug/19 ]

I believe this is now ready for people to look at. rhl and Kiyoto Yabe have expressed an interest, but I welcome comments from anyone.

Comment by price [ 02/Oct/19 ]

I believe I have addressed concerns about changes to the datamodel:

  • I had consolidated arrays from independent image HDUs into a binary table HDU.
  • I was providing a wavelength array instead of specifying it via WCS.
    I have undone these changes in datamodel.txt, and changed the implementation so it matches.

Please have another look.

Comment by price [ 25/Oct/19 ]

Merged to master.

Generated at Sat Feb 10 15:33:51 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.