https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/DAMD-63 Data Model <p>I screwed up the implementation of the pfsObject FITS HDUs when I implemented the end-to-end pipeline, since I didn't understand what <tt>FLUXTBL</tt> was meant for. We need to rename what is now <tt>FLUXTBL</tt> to <tt>FLUX</tt>, redesign <tt>FLUXTBL</tt> (the original design did not support multiple visits) and provide an implementation for populating it, and update <tt>datamodel.txt</tt>.</p> DAMD-63

Redesign and implement pfsObject FluxTbl

Story Normal Done Done price price Tue, 25 Jun 2019 16:24:07 +0000 Thu, 24 Oct 2019 20:32:24 +0000 Thu, 24 Oct 2019 16:38:10 +0000 0 3 <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a> points out that the <tt>CONFIG</tt> and <tt>MASK</tt> extensions have been removed to. <tt>CONFIG</tt> has been removed in favour of pfsConfig, and I believe <tt>MASK</tt> has been subsumed into what will be <tt>FLUX</tt>. Let's fix these up in <tt>datamodel.txt</tt> at the same time.</p> <p>Here's a plot of a coadd (black) and the fluxTable contents (purple, yellow, red, cyan).</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/12100/12100_fluxTable.png" style="border: 0px solid black" /></span></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> &gt;&gt;&gt; from lsst.daf.persistence <span class="code-keyword">import</span> Butler &gt;&gt;&gt; butler = Butler(<span class="code-quote">"INTEGRATION/rerun/integration/pipeline"</span>) &gt;&gt;&gt; obj = butler.get(<span class="code-quote">"pfsObject"</span>, catId=1, tract=0, patch=<span class="code-quote">"0,0"</span>, objId=0x5d, nVisit=4, pfsVisitHash=0x42cfc4af8cc94582) &gt;&gt;&gt; fig, axes = obj.fluxTable.plot(show=False) &gt;&gt;&gt; axes.plot(obj.wavelength, obj.flux, <span class="code-quote">'k-'</span>) &gt;&gt;&gt; fig.show() </pre> </div></div> <p>I am confused by the comment about CONFIG.  It is not replaced by the pfsConfig file, it was explicitly denormalised for the benefit of end users.</p> <p>I also don't understand the remark about multiple epochs.  The pfsObject file is the combined spectrum; if you want the input visits you need to go back to the pfsArm files.</p> <p>Now <em>I'm</em> confused too. You say that <tt>pfsObject</tt> is a combined spectrum, but a spectrum that has been coadded from multiple observations doesn't (necessarily) have a single top-end configuration. So what are you expecting to find in the <tt>CONFIG</tt> extension?</p> <p>What is the point of having the <tt>FLUXTBL</tt> as opposed to just the flux vector of the coadd? My understanding is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> A coadded spectrum is a convenient representation of the spectrum of an object with multiple observations, but the act of coaddition necessarily involves a reduction in the amount of data through resampling in wavelength and replacing a distribution of fluxes with a statistical representation. The intention of a ``FluxTable`` is to provide the full set of original data so that the user can have the option to inspect and fit the observations themselves rather than a reduced, resampled and censored version of the observations. This may be useful <span class="code-keyword">for</span> identifying artifacts that have leaked into the coadds (e.g., cosmic rays) or <span class="code-keyword">transient</span> spectral features, or <span class="code-keyword">for</span> doing Hogg-ish Bayesian fitting. </pre> </div></div> <p>I believe this is now ready for people to look at. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> and <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> have expressed an interest, but I welcome comments from anyone.</p> <p>I believe I have addressed concerns about changes to the datamodel:</p> <ul> <li>I had consolidated arrays from independent image HDUs into a binary table HDU.</li> <li>I was providing a wavelength array instead of specifying it via WCS.<br/> I have undone these changes in datamodel.txt, and changed the implementation so it matches.</li> </ul> <p>Please have another look.</p> <p>Merged to master.</p> Blocks PIPE2D-438 Relates DAMD-65 Development Rank 0|02qpt6:508yl03z0c Reviewers hassan Sprint 2DDRP-2019 F 2DDRP-2019 G 2DDRP-2019 H Story Points 3.0