https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-1367 Instrument control development <p>Following discussions in the #obslog slack channel during Engineering Run 3, it would be very useful to be able perform a global search on all raw FITS files for a keyword value matching a certain condition. For example:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> W_CLNEOT&gt;14 </pre> </div></div> <p>would return all visits/files matching the case where the keyword <tt>W_CLNEOT</tt> (Neon lamp request time) has a value of 14 or greater.</p> INSTRM-1367

obslog: global search on header keyword

Task Normal Done Done michitaro hassan EngRun Sat, 18 Sep 2021 01:04:28 +0000 Fri, 14 Oct 2022 05:40:47 +0000 Fri, 14 Oct 2022 05:40:47 +0000 spt_operational_database 0 5 <p>Another example is </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> W_AITNEO=<span class="code-quote">'T'</span> </pre> </div></div> <p>(or corresponding correct boolean expression)</p> <p>to find all exposures which have been taken under the Neon DCB lamp.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a>-san,<br/> This task requires some changes on opdb and the component inserting exposure records into opdb.<br/> I can provide codes for extra schema definition for opdb and a routine to insert FITS header information from a FITS file into the database.</p> <p>How do we go ahead?</p> <p>I still don't know how larger the fix is, but anyway if you make a fix easily please make a pull request.</p> <p>I'm thinking about how to store FITS headers in the opdb.<br/> There are the following 3 feasible ways to do so as far as I think.</p> <h2><a name="Method1%28naive%29"></a>Method 1 (naive)</h2> <p>To make a wide table with columns for all keywords the FITS headers have.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-python"> <span class="code-keyword">class </span>FitsHeader(Base): __tablename__ = <span class="code-quote">'fitsheader'</span> filename = Column(String, primary_key=<span class="code-keyword"><span class="code-object">True</span></span>) hdu_index = Column(Integer, primary_key=<span class="code-keyword"><span class="code-object">True</span></span>) SIMPLE = Column(String) BITPIX = Column(Integer) NAXIS = Column(Integer) EXTEND = Column(Boolean) DATA_TYP = Column(String) FRAMEID = Column(String) EXP_ID = Column(String) DETECTOR = Column(String) GAIN = Column(Float) DET_TMP = Column(Float) DET_ID = Column(String) DISPAXIS = Column(Integer) DISPRSR = Column(Float) WAV_MIN = Column(Float) WAV_MAX = Column(FLoat) ... </pre> </div></div> <h3><a name="Pros"></a>Pros</h3> <p>Simple and no violation of SQL standards.</p> <h3><a name="Cons"></a>Cons</h3> <p>Schema change always will be required when the set of keywords changes.<br/> We cannot store cards with a keyword that does not exist in the table.</p> <h2><a name="Method2%28EAV%29"></a>Method 2 (EAV)</h2> <p>To make a table whose each record represents a FITS card.<br/> This method is known as the EAV pattern (considered a bad design in general.)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-python"> <span class="code-keyword">class </span>FitsHdu(Base): __tablename__ = <span class="code-quote">'fits_hdu'</span> <span class="code-object">id</span> = Column(Integer, primary_key=<span class="code-keyword"><span class="code-object">True</span></span>) <span class="code-keyword">class </span>FitsCardString(Base): __tablename__ = <span class="code-quote">'fits_card_string'</span> hdu_id = Column(Integer, ForeignKey(<span class="code-quote">'fits_hdu.<span class="code-object">id</span>'</span>)) keyword = Column(String, nullable=<span class="code-keyword"><span class="code-object">False</span></span>) value = Column(String, nullable=<span class="code-keyword"><span class="code-object">False</span></span>) <span class="code-keyword">class </span>FitsCardFloat(Base): __tablename__ = <span class="code-quote">'fits_card_float'</span> hdu_id = Column(Integer, ForeignKey(<span class="code-quote">'fits_hdu.<span class="code-object">id</span>'</span>)) keyword = Column(String, nullable=<span class="code-keyword"><span class="code-object">False</span></span>) value = Column(Float, nullable=<span class="code-keyword"><span class="code-object">False</span></span>) <span class="code-keyword">class </span>FitsCardBoolean(Base): __tablename__ = <span class="code-quote">'fits_card_boolean'</span> hdu_id = Column(Integer, ForeignKey(<span class="code-quote">'fits_hdu.<span class="code-object">id</span>'</span>)) keyword = Column(String, nullable=<span class="code-keyword"><span class="code-object">False</span></span>) value = Column(Boolean, nullable=<span class="code-keyword"><span class="code-object">False</span></span>) </pre> </div></div> <h3><a name="Pros"></a>Pros</h3> <p>No violation of SQL standards.<br/> No schema change will be required when the set of keywords changes.</p> <h3><a name="Cons"></a>Cons</h3> <p>This approach requires a complicated implementation.</p> <h2><a name="Method3%28JSON%29"></a>Method 3 (JSON)</h2> <p>To make a table with a column of JSON(B) type to store the header cards.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-python"> <span class="code-keyword">class </span>FitsHeader(Base): __tablename__ = <span class="code-quote">'fitsheader'</span> <span class="code-object">id</span> = Column(Integer, primary_key=<span class="code-keyword"><span class="code-object">True</span></span>) cards = Column(JSONB) <span class="code-keyword">with</span> Session(future=<span class="code-keyword"><span class="code-object">True</span></span>) <span class="code-keyword">as</span> session: records = [ FitsHeader(cards={ <span class="code-quote">'FRAMEID'</span>: <span class="code-quote">'PFS001'</span>, <span class="code-quote">'favorite_food'</span>: <span class="code-quote">'rice'</span>, }), FitsHeader(cards={ <span class="code-quote">'FRAMEID'</span>: <span class="code-quote">'PFS002'</span>, <span class="code-quote">'favorite_food'</span>: <span class="code-quote">'noodle'</span>, }), ] session.add_all(records) session.commit() q = session.query(FitsHeader).<span class="code-object">filter</span>( FitsHeader.cards[<span class="code-quote">'FRAMEID'</span>].astext == <span class="code-quote">'PFS002'</span>) r = <span class="code-object">list</span>(q)[0] <span class="code-keyword">assert</span> r.cards[<span class="code-quote">'favorite_food'</span>] == <span class="code-quote">'noodle'</span> <span class="code-comment"># <span class="code-object">type</span>: ignore</span> </pre> </div></div> <h3><a name="Pros"></a>Pros</h3> <p>Simple implementation.<br/> Expression index for a particular keyword helps quick search.</p> <h3><a name="Cons"></a>Cons</h3> <p>Less potable. A JSON column is out of SQL standards (, but Postgres, MySQL, SQLite and MSSQLServer support it.)<br/> (<a href="https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON" class="external-link" rel="nofollow">https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON</a>)<br/> We may have to write some codes for specific DB backend.</p> <p>If we have no plan to run the code on DBMS other than Postgres, I like the 3rd method.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>-san, what do you think about introducing JSON to opdb?</p> <p>I'm now positively thinking about using JSON columns. We do not probably use any backends other than Postgres.</p> <p>Thank you for the comment.</p> <p>I will make a prototype using JSON column approach.</p> <p>I have made a PR for this.</p> <p><a href="https://github.com/Subaru-PFS/spt_operational_database/pull/54" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/spt_operational_database/pull/54</a></p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> san, to be able to search by FITS headers, it's necessary to add an `obslog_fits_header` record when a new `pfs_visit` record is issued.</p> <p>As the PR shows, codes to add a `obslog_fits_header ` record from a `pyfits.HDUList` will be something like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-python"> <span class="code-keyword">import</span> opdb.obslog <span class="code-keyword">import</span> astropy.io.fits <span class="code-keyword">as</span> pyfits <span class="code-keyword">from</span> opdb.models <span class="code-keyword">import</span> pfs_visit db = SessionClass() pfs_visit_id = 66453 filestem = <span class="code-quote">'PFSA06645311'</span> hdul = pyfits.HDUList([ pyfits.PrimaryHDU(header=pyfits.Header([ (<span class="code-quote">'PI'</span>, 3.14, <span class="code-quote">"ratio of a circle's circumference to its diameter"</span>), ])), ]) db.add_all(opdb.obslog.fits_headers_from_hdulist(hdul, pfs_visit_id, filestem): db.commit() </pre> </div></div> <p>Is it feasible to add some codes like the above to your code?</p> <p>Sorry for the delayed progress.</p> <p>[ Sorry, I missed your much much earlier request. Yes, I'm fine with the clever JSONB encoding. ]</p> <p>I have opened <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-1463" title="Insert FITS headers into opdb" class="issue-link" data-issue-key="INSTRM-1463">INSTRM-1463</a> to insert the headers into opdb just before gen2 archiving &#8211; that is not the earliest possible time but close.</p> <p>Thanks! <img class="emoticon" src="https://pfspipe.ipmu.jp/jira/images/icons/emoticons/biggrin.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I have added the header cards in existing sps and mcs FITS files to opdb.</p> <p>Now we can query visits by their FITS header value on obslog like this:</p> <p><span class="image-wrap" style=""><a id="14530_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/14530/14530_Screen+Shot+2021-12-09+at+4.15.03.png" title="Screen Shot 2021-12-09 at 4.15.03.png" file-preview-type="image" file-preview-id="14530" file-preview-title="Screen Shot 2021-12-09 at 4.15.03.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/14530/_thumb_14530.png" style="border: 0px solid black" /></a></span></p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> where fits_header[<span class="code-quote">'CHECKSUM'</span>] = <span class="code-quote">'V4faV3fYV3faV3fW'</span> where fits_header[<span class="code-quote">'OBSERVER'</span>] like <span class="code-quote">'%tamura%'</span> where fits_header[<span class="code-quote">'DET-TMP'</span>]::<span class="code-object">float</span> between 163 and 164 where fits_header[<span class="code-quote">'W_ENIISH'</span>] = <span class="code-quote">'<span class="code-keyword">true</span>'</span> -- not <span class="code-quote">'T'</span> or <span class="code-quote">'F'</span></pre> </div></div> <p>Query on a boolean value, we should use 'true' and 'false' instead of 'T' or 'F.'</p> <p> </p> <p> </p> <p>Thank you, Koike-san! That would be very useful.</p> <p>Can we combine the previous search method (for instance just typing `defocused` to select defocused data) and this new query method (for instance, `where fits_header<span class="error">&#91;&#39;EXPTIME&#39;&#93;</span>::float &lt; 10`)?</p> <p>In other words, how can we query to obslog columns such as description and notes? Sorry, but I probably don't understand the search mechanism.</p> <p>I have to make some documents for this.</p> <p>If the text in the search box starts with `where ` (like the example above), the search query will be sent to the server as is.</p> <p>But if the search query does not start with `where ` (for instance, just `defocused`) the search query will be compiled into a particular format before the query will be sent to the server.</p> <p>For the case of `defocused`, it will be compiled into `where any_column like '%defocused%'.</p> <p>`any_column` is a special virtual column that will be expanded to some real columns in the opdb.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # any_column will be expanded the columns below M.pfs_visit.pfs_visit_id, <span class="code-object">String</span>, M.pfs_visit.pfs_visit_description, M.obslog_visit_note.body, visit_note_user.account_name, M.obslog_visit_set_note.body, visit_set_note_user.account_name, M.iic_sequence.name, M.iic_sequence.sequence_type, M.sps_annotation.notes, M.obslog_mcs_exposure_note.body, mcs_exposure_note_user.account_name, </pre> </div></div> <p>In addition to `fits_header`, we can use more virtual columns below.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (ast.<span class="code-object">String</span>(<span class="code-quote">'any_column'</span>),): AnyColumn, (ast.<span class="code-object">String</span>(<span class="code-quote">'visit_id'</span>),): M.pfs_visit.pfs_visit_id, (ast.<span class="code-object">String</span>(<span class="code-quote">'id'</span>),): M.pfs_visit.pfs_visit_id, (ast.<span class="code-object">String</span>(<span class="code-quote">'sequence_type'</span>),): M.iic_sequence.sequence_type, (ast.<span class="code-object">String</span>(<span class="code-quote">'issued_at'</span>),): M.pfs_visit.issued_at, (ast.<span class="code-object">String</span>(<span class="code-quote">'is_sps_visit'</span>),): M.sps_visit.pfs_visit_id != None, (ast.<span class="code-object">String</span>(<span class="code-quote">'is_mcs_visit'</span>),): M.mcs_exposure.mcs_frame_id != None, (ast.<span class="code-object">String</span>(<span class="code-quote">'is_agc_visit'</span>),): M.agc_exposure.agc_exposure_id != None, (ast.<span class="code-object">String</span>(<span class="code-quote">'fits_header'</span>),): M.obslog_fits_header.cards_dict,</pre> </div></div> <p> <br/> &gt; Can we combine the previous search method<br/> Yes, `where any_column like '%defocus%' and fits_header<span class="error">&#91;&#39;EXPTIME&#39;&#93;</span>::float &lt; 10` will work for you.<br/>  <br/> Anyway, I will make documents for this soon.<br/>  </p> <p>Thank you! That information is very helpful. And thank you for documenting.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=michitaro" class="user-hover" rel="michitaro">michitaro</a> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> Is this ticket complete? Can we close this?</p> <p>Yes, we can close this ticket, but we need to register FITS files on the DB manually periodically.<br/> The automatic registration will be handled in this ticket↓.<br/> <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-1463" class="external-link" rel="nofollow">https://pfspipe.ipmu.jp/jira/browse/INSTRM-1463</a></p> <p>Following the above comments, I close this ticket.</p> Development Rank 0|02qpjd:00r3000zw Story Points 6.0