https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-35 DRP 2-D Pipeline <p>The data model product provides code to write (and read) files the conform to the data model. Please use it to write files.</p> <p>There are objects in data model that contain the information that will be written to pfsArm files. The simplest thing to do would be to pack your Spectrum object into one of these objects, and call the write method.</p> PIPE2D-35

Read & write 2-D outputs according to the data model

Bug Major Done Done rhl rhl Fri, 8 Jul 2016 17:53:56 +0000 Thu, 13 Oct 2016 21:27:45 +0000 Thu, 13 Oct 2016 21:27:45 +0000 0 3 <p>According to Jim Bosh for the butler to write/read these pfsArm files the write method should be renamed to writeFits and the read method needs to be a static method called readFits returning a new pfsArm object. Apparently this is by far the easiest way to make the butler call the appropriate methods. Shall I make these changes?</p> <p>See tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-35" title="Read &amp; write 2-D outputs according to the data model " class="issue-link" data-issue-key="PIPE2D-35"><del>PIPE2D-35</del></a> for drp_stella and datamodel.<br/> The mystery of the empty file was caused by the butler wanting to write a temporary file and then rename that to the output file name. The PfsArm.write(...) method determined the file name independently of the butler what resulted in an empty file overwriting the output file.</p> <p>Should be rebased on top of master after <a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-36" title="Write quick-start guide for Stella" class="issue-link" data-issue-key="INFRA-36"><del>INFRA-36</del></a> has been merged, then put it back in review.</p> <p>That change to the data model implementation is not acceptable &#8211; you're hardcoding details about your particular case. I understand why you need a method called readFits on some object, but the pfsArm object in the data model is not the right one. Please come up with some other solution.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> @staticmethod def readFits(fileName): dirName = os.path.dirname(fileName) visit = 4 spectrograph = 2 arm = <span class="code-quote">'r'</span> pfsArm = PfsArm( visit, spectrograph, arm ) pfsArm.read(dirName=dirName) <span class="code-keyword">return</span> pfsArm </pre> </div></div> <p>We do need a readFits(fileName) method for PfsArm but I added a getInfo(fileName) function that extracts the spectrograph, arm, and visit number from the fileName. drp_stella's reduceArc.py writes the correct PfsArm object now but when trying to read it back in with "myPfsArm = pfsArm.PfsArm.readFits(fileName)" I get the error</p> <p>IOError: <span class="error">&#91;Errno 2&#93;</span> No such file or directory: '/Users/azuri/spectra/pfs/PFS/rerun/azuri/tmp/pfsArm/2015-12-22/v0000004/pfsConfig-0x0bef64dc.fits'</p> <p>Should the write method write the PfsConfig object as well?</p> <p>You may not add a readFits method. Please use one of the other mechanisms available in the butler/mapper (using a bypass will work); I think you need</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> def bypass_spArm(self, datasetType, pythonType, location, dataId): from pfs.datamodel.pfsArm <span class="code-keyword">import</span> PfsArm <span class="code-keyword">for</span> fn in location.locationList: dirName = os.path.dirname(fn) pfsa = PfsArm(spectrograph=dataId[<span class="code-quote">"spectrograph"</span>], arm=dataId[<span class="code-quote">"arm"</span>], visit=dataId[<span class="code-quote">"visit"</span>]) <span class="code-keyword">try</span>: pfsa.read(dirName=dirName) except Exception as e: pass <span class="code-keyword">else</span>: <span class="code-keyword">return</span> pfsa raise RuntimeError(<span class="code-quote">"Unable to read pfsArm <span class="code-keyword">for</span> %s: %s"</span> % (dataId.items(), e)) </pre> </div></div> <p>(where the import needs to be at file scope). Note that that returns a pfsArm object, and you may need to unpack that into one of your internal objects (SpectrumSet) in your code after retrieval. Note that you'll have to change the name of this function when you fix the name from "spArm" to "pfsArm".</p> <p>Did you read the data model? Yes, you do need that file, but it should be written by the simulator. There is code in datamodel to write it for you.</p> <p>We discussed this at the meeting of 2016-09-19. We believe the code <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> added below should contain all we need here. This should be closed (or set to review) in time for the next meeting.</p> <p>Actually, we believe <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a>'s code will fix <b>reading</b>, but we need to do something else for writing. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> himself has volunteered to take a look at this.</p> <p>This appears to be a weakness in the butler. While my fix for reading works beautifully, there's no equivalent for writing.</p> <p>So we need to write a dummy class (pfsArmIO) that has readFits/writeFits methods and calls the pfsArm code. </p> <p>I'll do it</p> <p>Merged to master.</p> <p>N.b. The work was done on branch tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-35" title="Read &amp; write 2-D outputs according to the data model " class="issue-link" data-issue-key="PIPE2D-35"><del>PIPE2D-35</del></a>-rhl. There is also a branch tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-35" title="Read &amp; write 2-D outputs according to the data model " class="issue-link" data-issue-key="PIPE2D-35"><del>PIPE2D-35</del></a> that has a variety of changes, many unrelated to this ticket. I've filed <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-101" title="Salvage needed patches from tickets/PIPE2D-35" class="issue-link" data-issue-key="PIPE2D-101"><del>PIPE2D-101</del></a> to salvage anything we need.</p> Development Rank 0|02qpuc:zv Reviewers rhl Sprint 2014-15 2014-16