https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/DAMD-48 Data Model <p>in pfsObject, covar shape should be (length, 3).</p> <p>also : expId is not available in TargetObservations when writing spectrum, preventing it to work.</p> DAMD-48

in pfsObject, covar shape should be (length, 3), fix expId in TargetObservations too

Task Normal Won't Fix Won't Fix hassan Guillaume PERNOT Fri, 22 Mar 2019 09:25:33 +0000 Mon, 13 May 2019 16:52:35 +0000 Wed, 1 May 2019 15:52:09 +0000 0 5 <p>Hi all,</p> <p>I attached the <tt>test_dmd48.py</tt> file to explain, with an example, the issue we encounter with current DM.</p> <p>Feel free to comment.</p> <p>PYves</p> <p>In <tt>datamodel.txt</tt> the dimensions of the <tt>covar</tt> HDU are NROW*3:<br/> <a href="https://github.com/Subaru-PFS/datamodel/blob/6930c77dc7b26daa6904bce59c5dce5320e32b8f/datamodel.txt#L369" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/datamodel/blob/6930c77dc7b26daa6904bce59c5dce5320e32b8f/datamodel.txt#L369</a><br/> but the assertion below checks for 3*NCOL:<br/> <a href="https://github.com/Subaru-PFS/datamodel/blob/6930c77dc7b26daa6904bce59c5dce5320e32b8f/python/pfs/datamodel/pfsSpectrum.py#L275" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/datamodel/blob/6930c77dc7b26daa6904bce59c5dce5320e32b8f/python/pfs/datamodel/pfsSpectrum.py#L275</a></p> <p>this should be changed to <br/> <tt>assert self.covar.shape == (self.length, 3)</tt></p> <p>Using a <tt>covar</tt> array of dimensions (10, 3) <span class="error">&#91;which is legitimate according to datamodel.txt for constructing a PsfObject &#93;</span>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">covar=np.array([np.ones(10), np.zeros(10), np.zeros(10)]) ct = covar.transpose() obj = PfsObject(target=target, observations=observations, wavelength=np.array(wavels), flux=np.array(fluxls), mask=np.zeros(10), sky=np.zeros(10), covar=ct, covar2=np.random.rand(2, 2), flags=MaskHelper()) </pre> </div></div> <p>the following error is raised:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) &lt;ipython-input-19-3ab70bf49470&gt; in &lt;module&gt;() 7 covar=ct, 8 covar2=np.random.rand(2, 2), ----&gt; 9 flags=MaskHelper()) ~/LSST/stack/v16/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/datamodel/5.0-13-g6930c77/python/pfs/datamodel/pfsSpectrum.py in __init__(self, target, observations, wavelength, flux, mask, sky, covar, covar2, flags) 249 self.covar2 = covar2 250 self.numExp = len(self.observations) --&gt; 251 <span class="code-keyword">super</span>().__init__(target, wavelength, flux, mask, flags) 252 253 @property ~/LSST/stack/v16/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/datamodel/5.0-13-g6930c77/python/pfs/datamodel/pfsSpectrum.py in __init__(self, target, wavelength, flux, mask, flags) 39 40 self.length = len(wavelength) ---&gt; 41 self.validate() 42 43 def validate(self): ~/LSST/stack/v16/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/datamodel/5.0-13-g6930c77/python/pfs/datamodel/pfsSpectrum.py in validate(self) 273 <span class="code-keyword">assert</span> len(self.observations) == self.numExp 274 <span class="code-keyword">assert</span> self.sky.shape == (self.length,) --&gt; 275 <span class="code-keyword">assert</span> self.covar.shape == (3, self.length) 276 <span class="code-keyword">assert</span> self.covar2.ndim == 2 277 AssertionError: </pre> </div></div> <p>Could you check this?  The datamodel discusses FITS specs which are transposed with respect to C/python and I thought that I'd tested the datamodel code</p> <p> </p> <p>I checked again, and indeed I was wrong: the code as it stands is correct, and the <tt>covar</tt> numpy should have a shape of <tt>(3, length)</tt>.</p> <p>The <tt>datamodel.txt</tt> specifies the dimensions of the <tt>covar</tt> HDU array as <tt>NROW*3</tt>. These are the dimensions for the FITS image, which means that <tt>NAXIS1=NROW</tt>, and <tt>NAXIS2=3</tt>. The corresponding numpy array would be transposed, so would have a shape <tt>(3, length)</tt>. Which is what the code as it stands asserts.</p> <p>This is the first of two points in this issue. The second point, about the non-presence of <tt>expId</tt> is the subject of <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-49" title="Clarify use of expId" class="issue-link" data-issue-key="DAMD-49"><del>DAMD-49</del></a>. </p> <p>So I will close this ticket with no further action in a day or so, to allow the creator to comment.</p> <p>I think the point is not that it isn't correct, it's that "COVAR would be easier to use if given as columns instead of rows" (per <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=gpernot" class="user-hover" rel="gpernot">Guillaume PERNOT</a>).</p> <p>Ok, I used BinTable HDU for covar, instead of Image. I was not aware of the difference between BinTable and Image. I'll modify my code in the light of this.</p> <p>Some extra information in relation to the intended <tt>COVAR</tt> shape in python of <tt>(3, length)</tt>. This may be useful if we revisit this issue in the future.</p> <p><a href="https://github.com/Subaru-PFS/datamodel/blob/6930c77dc7b26daa6904bce59c5dce5320e32b8f/datamodel.txt#L298" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/datamodel/blob/6930c77dc7b26daa6904bce59c5dce5320e32b8f/datamodel.txt#L298</a><br/> <a href="https://github.com/Subaru-PFS/datamodel/blob/6930c77dc7b26daa6904bce59c5dce5320e32b8f/python/pfs/datamodel/utils.py#L54" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/datamodel/blob/6930c77dc7b26daa6904bce59c5dce5320e32b8f/python/pfs/datamodel/utils.py#L54</a></p> <p>Will close the ticket as proposed last week. I will add some additional information in <tt>datamodel.txt</tt> to emphasize further the shape convention for image data further, but that will be a separate ticket.</p> Relates DAMD-49 Development Rank 0|zx002w: