https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-404 DRP 2-D Pipeline <p>This issue is raised following a discussion on slack channel #drp-2d on 2019-04-09 and today.</p> <p>The FITS keyword <tt>W_PFDSGN</tt> has been set in raw LAM exposures since approx 2010-02-18. The current 2D DRP pipeline currently supports data with this keyword, but not well in scenarios without this keyword. There are a number of developers needing to process LAM exposures before and after this change, so the 2D DRP pipeline needs to be updated to support both scenarios.</p> <p>So the scenario where <tt>W_PFSDSGN</tt> is <em>not</em> set needs to be supported. It needs to be agreed under this scenario what fibers are assumed to be illuminated (all? none?). If it is assumed that no fibers are illuminated, then <tt>PfiDesign</tt> needs to be updated, as it currently will not persist an instance if itself if the <tt>fiberId</tt> is an empty list.</p> PIPE2D-404

Support for raw exposures with W_PFDSGN not set

Story Normal Done Done price hassan Wed, 10 Apr 2019 19:03:43 +0000 Wed, 1 May 2019 13:28:19 +0000 Sat, 27 Apr 2019 13:44:50 +0000 0 5 <p>The following is the initial setup of the pipeline on <tt>gfarm.ipmu.jp</tt>.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ source /opt/local/pfs/loadLSST.bash $ setup psf_pipe2d $ mkdir /gpfs02/work/yasuda/PFS/repo $ echo <span class="code-quote">"lsst.obs.pfs.PfsMapper"</span> &gt; /gpfs02/work/yasuda/PFS/repo/_mapper </pre> </div></div> <p>When I try to ingest the frame without <tt>W_PFDSGN</tt>, <tt>ingestPfsImages.py</tt> complains non availability of <tt>pfiDesign</tt> file.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ ingestPfsImages.py /gpfs02/work/yasuda/PFS/repo /gpfs01/pfs/rawdata/2019-02-15/PFLA01194412.fits CameraMapper INFO: Loading Posix exposure registry from /gpfs02/work/yasuda/PFS/repo ingestPfs INFO: /gpfs01/pfs/rawdata/2019-02-15/PFLA01194412.fits --&lt;link&gt;--&gt; /gpfs02/work/yasuda/PFS/repo/2019-02-15/PFLA011944r1.fits Traceback (most recent call last): File <span class="code-quote">"/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_pfs/master/python/lsst/obs/pfs/ingest.py"</span>, line 275, in runFile self.ingestPfsConfig(os.path.dirname(infile), hduInfoList[0], args) File <span class="code-quote">"/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_pfs/master/python/lsst/obs/pfs/ingest.py"</span>, line 242, in ingestP fsConfig (pfiDesignId,)) RuntimeError: Unable to find PfsConfig or PfiDesign <span class="code-keyword">for</span> pfiDesignId=0x0000000000000000 ingestPfs WARN: Failed to ingest file /gpfs01/pfs/rawdata/2019-02-15/PFLA01194412.fits: Unable to find PfsConfig or PfiDesign <span class="code-keyword">for</span> pfiDesignId=0x0000000000000000 </pre> </div></div> <p>So we will try to create <tt>pfiDesign</tt> file without specifying any fibers but <tt>makeLamDesign.py</tt> cannot create null <tt>pfiDesign</tt> file.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ makeLamDesign.py {} usage: makeLamDesign.py [-h] [--directory DIRECTORY] {red1,red2,red3,red4,red5,red6,red7,red8,orange,blue,green,yellow} [{red1,red2,red3,red4,red5,red6,red7,red8,orange,blue,green,yellow} ...] makeLamDesign.py: error: argument colors: invalid choice: <span class="code-quote">'{}'</span> (choose from <span class="code-quote">'red1'</span>, <span class="code-quote">'red2'</span>, <span class="code-quote">'red3'</span>, <span class="code-quote">'red4'</span>, <span class="code-quote">'red5'</span>, <span class="code-quote">'red6'</span>, <span class="code-quote">'red7'</span>, <span class="code-quote">'red8'</span>, <span class="code-quote">'orange'</span>, <span class="code-quote">'blue'</span>, <span class="code-quote">'green'</span>, <span class="code-quote">'yellow'</span>) </pre> </div></div> <p>If we use <tt>ingestImages.py</tt> instead of <tt>ingestPfsImages.py</tt>, the file will be registered normally. But I'm not sure how we can extract 1d arc spectra using <tt>pipe2d</tt> in this case.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ ingestImages.py /gpfs02/work/yasuda/PFS/repo /gpfs01/pfs/rawdata/2019-02-15/PFLA01194412.fits root INFO: Loading config overrride file <span class="code-quote">'/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_pfs/master/config/ingest.py'</span> CameraMapper INFO: Loading Posix exposure registry from /gpfs02/work/yasuda/PFS/repo ingest INFO: /gpfs01/pfs/rawdata/2019-02-15/PFLA01194412.fits --&lt;link&gt;--&gt; /gpfs02/work/yasuda/PFS/repo/2019-02-15/PFLA011944r1.fits </pre> </div></div> <p>Should we ask LAM poeple to provide the information of illuminated fibers for every dates?</p> <p>Another problem is <tt>NO CURRENT VALUE</tt> in fits header. In this case, <tt>W_PFDSGN</tt> value is <tt>NO CURRENT VALUE</tt> and we cannot process this data properly. Should we also ask LAM people to fix the files with correct values or just abandon these files?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ ingestPfsImages.py /gpfs02/work/yasuda/PFS/repo /gpfs01/pfs/rawdata/2019-03-29/PFLA01471912.fits CameraMapper INFO: Loading exposure registry from /gpfs02/work/yasuda/PFS/repo/registry.sqlite3 Traceback (most recent call last): File <span class="code-quote">"/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_pfs/master/python/lsst/obs/pfs/ingest.py"</span>, line 274, in runFile hduInfoList = <span class="code-keyword">super</span>().runFile(infile, registry, args) File <span class="code-quote">"/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0+1/python/lsst/pipe/tasks/ingest.py"</span>, line 508, in runFile <span class="code-keyword">if</span> registry is not None and self.register.check(registry, fileInfo): File <span class="code-quote">"/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0+1/python/lsst/pipe/tasks/ingest.py"</span>, line 318, in check values = [self.typemap[self.config.columns[col]](info[col]) <span class="code-keyword">for</span> col in self.config.unique] File <span class="code-quote">"/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0+1/python/lsst/pipe/tasks/ingest.py"</span>, line 318, in &lt;listcomp&gt; values = [self.typemap[self.config.columns[col]](info[col]) <span class="code-keyword">for</span> col in self.config.unique] ValueError: invalid literal <span class="code-keyword">for</span> <span class="code-object">int</span>() with base 10: <span class="code-quote">'NO CURRENT VALUE'</span> </pre> </div></div> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a>, please try the fix on <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-404" title="Support for raw exposures with W_PFDSGN not set" class="issue-link" data-issue-key="PIPE2D-404"><del>PIPE2D-404</del></a></tt> of obs_pfs. You'll want to set the default value of <tt>parse.pfsDesignId</tt>. Here's an example:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pprice@tiger2-sumire:~/pfs/obs_pfs[tickets/PIPE2D-404] $ ingestPfsImages.py DATA PFLA01194412.fits PFLA01471912.fits -c parse.pfsDesignId=0x0000000100100011 CameraMapper INFO: Loading Posix exposure registry from /home/pprice/pfs/obs_pfs/DATA ingestPfs.parse WARN: No value set <span class="code-keyword">for</span> pfsDesignId; using <span class="code-keyword">default</span> (0x0000000100100011) ingestPfs INFO: PFLA01194412.fits --&lt;link&gt;--&gt; /home/pprice/pfs/obs_pfs/DATA/2019-02-15/PFLA011944r1.fits ingestPfs INFO: pfsConfig-0x0000000100100011-011944.fits --&lt;link&gt;--&gt; /home/pprice/pfs/obs_pfs/DATA/pfsConfig/2019-02-15/pfsConfig-0x0000000100100011-011944.fits ingestPfs.parse WARN: No value set <span class="code-keyword">for</span> pfsDesignId; using <span class="code-keyword">default</span> (0x0000000100100011) ingestPfs INFO: PFLA01471912.fits --&lt;link&gt;--&gt; /home/pprice/pfs/obs_pfs/DATA/2019-03-29/PFLA014719r1.fits ingestPfs INFO: pfsConfig-0x0000000100100011-014719.fits --&lt;link&gt;--&gt; /home/pprice/pfs/obs_pfs/DATA/pfsConfig/2019-03-29/pfsConfig-0x0000000100100011-014719.fits </pre> </div></div> <p>Thanks. This works fine if we know the fiber configuration used. For old data, it is not easy to guess the fiber configuration but this is not the problem of software.</p> <p>Slightly related: <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-354" title="On missing, unconverted, or invalid numeric values, write a string FITS complaint" class="issue-link" data-issue-key="INSTRM-354"><del>INSTRM-354</del></a> covers a discussion of what to do with missing or invalid card values. The FITS standard says to use an empty value, but that causes just as much trouble for consumers which expect typed values as semi-descriptive strings ('NO CURRENT VALUE', etc), so it looks like we will agree to use magic values. I propose using 0 for <tt>W_PFDSGN</tt>, and not to distinguish between "missing" and "unknown/invalid". Please object now if you will.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a>, would you mind reviewing the changes, please? Then I'll get this into master.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> price@MacBook:~/pfs/obs_pfs (tickets/PIPE2D-404=) $ git sub commit 1ec884e458ec635af20e317db202df1831044cf6 (HEAD -&gt; tickets/PIPE2D-404, origin/tickets/PIPE2D-404) Author: Paul Price &lt;price@astro.princeton.edu&gt; Date: Mon Apr 15 15:24:22 2019 -0400 ingest: make <span class="code-keyword">default</span> pfsDesignId value configurable If the pfsDesignId is not set, we need to be able to fall back to a reasonable value (perhaps determined via manual inspection of the image), rather than a hard-coded value. python/lsst/obs/pfs/ingest.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) </pre> </div></div> <p>I have reviewed the change and it looks fine. Please merge to master.</p> <p>Thanks <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a>!</p> <p>Merged to master.</p> Blocks PIPE2D-339 Relates INSTRM-354 INSTRM-648 Development Rank 0|02qpt6:508yjvi Reviewers naoki.yasuda Sprint 2DDRP-2019 D Story Points 4.0