https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/DAMD-113 Data Model <p>It would be useful for engineering and debugging pipelines if the extracted spectra in the pfsArm files were in units of electrons; currently they are relative to the fiberProfile normalisation, which comes from extracting a quartz). Similarly, it would similarly be useful if the spectra in the pfsMerged files were close to electrons (or, at least, normalised like they are close to electrons).</p> <p>This will require small documentation changes in the datamodel, and some modest changes to the pipeline to support.</p> DAMD-113

Write pfsArm in electrons

Story Normal Done Done price price Fri, 11 Jun 2021 15:02:35 +0000 Wed, 30 Mar 2022 23:43:09 +0000 Tue, 20 Jul 2021 19:07:53 +0000 0 4 <p>This issue replaces <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-811" title="Provide an option to make the unit of `pfsArm` in real counts" class="issue-link" data-issue-key="PIPE2D-811"><del>PIPE2D-811</del></a>.</p> <p>I gave up having the pfsMerged files be in units of (something close to) counts, because that requires choosing a normalisation using multiple arms, and then choosing the exact same normalisation later for coaddition when there's the potential to have a different input set. But pfsArm is now in counts.</p> <p>I think you should probably have discussed this decision on slack. I don't quite understand why the book-keeping can't be done in a safe way (probably by including some extra information in the files). Can you explain?</p> <p>I don't like the idea of writing the normalization to a file that's uninvolved in the coaddition process, but I guess we could write a new dataset type that contains the normalisation.</p> <p>I must not understand. What information would you need to write to an extra file, and when would it be read? I'd have thought it'd go to the pfsMerged file and everything would be transparent</p> <p>The new plan is to add an extra HDU to pfsArm containing the normalisation.</p> <p>I think I've got this working. Below I show me poking around the results of the integration test. Note that pfsArm and pfsMerged fluxes of the flux standard star (about i=18 mag) are in the thousands, and the normalised fluxes (ratio of the flux to the quartz) are about 0.08. For the single-object products, the flux in the pfsSingle and pfsObject are about 2.25e5 nJy = 18 mag.</p> <p>I'm going to clean up the commit history and then submit for review.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> &gt;&gt;&gt; <span class="code-keyword">import</span> numpy as np &gt;&gt;&gt; from lsst.daf.persistence <span class="code-keyword">import</span> Butler &gt;&gt;&gt; butler = Butler(<span class="code-quote">"INTEGRATION/rerun/integration/pipeline"</span>) CameraMapper INFO: Loading exposure registry from /scratch/pprice/damd-113/INTEGRATION/registry.sqlite3 CameraMapper INFO: Loading calib registry from /scratch/pprice/damd-113/INTEGRATION/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /scratch/pprice/damd-113/INTEGRATION/CALIB/calibRegistry.sqlite3 &gt;&gt;&gt; index = 0 &gt;&gt;&gt; pfsArm = butler.get(<span class="code-quote">"pfsArm"</span>, visit=24, arm=<span class="code-quote">"b"</span>) &gt;&gt;&gt; select = (pfsArm.wavelength[index] &gt; 500) &amp; (pfsArm.wavelength[index] &lt; 600) &gt;&gt;&gt; print(np.median(pfsArm.flux[index][select])) 3193.8977383679285 &gt;&gt;&gt; print(np.median(pfsArm.flux[index][select]/pfsArm.norm[index][select])) 0.08098331564220362 &gt;&gt;&gt; pfsMerged = butler.get(<span class="code-quote">"pfsMerged"</span>, visit=24) &gt;&gt;&gt; select = (pfsMerged.wavelength[index] &gt; 500) &amp; (pfsMerged.wavelength[index] &lt; 600) &gt;&gt;&gt; print(np.median(pfsMerged.flux[index][select])) 4250.031433514167 &gt;&gt;&gt; print(np.median(pfsMerged.flux[index][select]/pfsMerged.norm[index][select])) 0.07860174756195284 &gt;&gt;&gt; pfsSingle = butler.get(<span class="code-quote">"pfsSingle"</span>, catId=0, tract=0, patch=<span class="code-quote">"0,0"</span>, objId=1, visit=24) &gt;&gt;&gt; select = (pfsSingle.wavelength &gt; 600) &amp; (pfsSingle.wavelength &lt; 700) &gt;&gt;&gt; print(np.median(pfsSingle.flux[select])) 226348.42616116628 &gt;&gt;&gt; select = (pfsSingle.fluxTable.wavelength &gt; 600) &amp; (pfsSingle.fluxTable.wavelength &lt; 700) &gt;&gt;&gt; print(np.median(pfsSingle.fluxTable.flux[select])) 225535.49838282933 &gt;&gt;&gt; pfsObject = butler.get(<span class="code-quote">"pfsObject"</span>, catId=0, tract=0, patch=<span class="code-quote">"0,0"</span>, objId=1, nVisit=2, pfsVisitHash=0x60241444a9af118c) &gt;&gt;&gt; select = (pfsObject.wavelength &gt; 600) &amp; (pfsObject.wavelength &lt; 700) &gt;&gt;&gt; print(np.median(pfsObject.flux[select])) 226177.83216109022 &gt;&gt;&gt; select = (pfsObject.fluxTable.wavelength &gt; 600) &amp; (pfsObject.fluxTable.wavelength &lt; 700) &amp; (pfsObject.fluxTable.mask == 0) &gt;&gt;&gt; print(np.median(pfsObject.fluxTable.flux[select])) 224241.48945935623 </pre> </div></div> <p>Everything appears to be working.</p> <p>No additional comments on proposed changes.</p> <p>Merged.</p> Relates PIPE2D-1008 Development Rank 0|zzs4wj:zz9 Reviewers hassan Sprint 2DDRP-2021 A 6 2DDRP-2021 A 7 Story Points 4.0