https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-978 Instrument control development <p>Description: As we discussed in previous ICS/MCS-PFI telecon, I will collect the functional requirements and possible use cases in each site of software development to figure out what kind of function we need in the standard library to access opDB and how we should generalized it. Please provide a list of your required function for this library and example of major use cases from the view point of DRP (and more general aspects if possible).<br/>  <br/> Below are possible examples for MCS/PFI-related tables: * <br/>  insert (pandas/numpy/…) array of measured data into `mcs_data`</p> <ul> <li> get information from `cobra_geometry`</li> </ul> <ul> <li> get target information joining `cobra_status`, `pfs_config_fiber`, and `target`</li> </ul> <ul> <li>...</li> </ul> <p> <br/> If you can provide any performance requirements (speed/stability/…), that would be great.<br/>  <br/> I will compile the requirements and update the proposal of the standard library:<br/> <a href="https://sumire.pbworks.com/w/file/fetch/139861650/PFS-DAT-IPM003007-01_opdb_proposal_library.pdf" class="external-link" rel="nofollow">https://sumire.pbworks.com/w/file/fetch/139861650/PFS-DAT-IPM003007-01_opdb_proposal_library.pdf</a><br/>  </p> INSTRM-978

Provide functional requirement and use cases of the opDB standard library from the viewpoint of DRP (and more general aspects)

Task Normal Done Done hassan Kiyoto Yabe opDB Mon, 11 May 2020 08:33:43 +0000 Tue, 10 Nov 2020 05:02:34 +0000 Tue, 10 Nov 2020 05:02:34 +0000 spt_operational_database 0 3 <h2><a name="ExperimentalSPSLog"></a>Experimental SPS Log</h2> <p>We need to continue to be able to extract SPS log information from the opDB and present that from a web server to enable users to view what exposures have been made for a given visit. Current query performed is (provided by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> select seq.visit_set_id as set_id, sequence_type, name, min(visits.pfs_visit_id) as min_visit, max(visits.pfs_visit_id) as max_visit, cmd_str
from sps_sequence seq
join visit_set visits on visits.visit_set_id = seq.visit_set_id 
where seq.visit_set_id &gt;= 361
group by seq.visit_set_id 
order by seq.visit_set_id; </pre> </div></div> <p>Or this query (provided by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a> in #drp-2d channel 2020-06-10):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> select sequence_type, name, comments, sps_sequence.visit_set_id, visit_set.pfs_visit_id, exp_type, sps_module_id, arm, notes, data_flag from sps_sequence \ <span class="code-keyword">inner</span> join visit_set on visit_set.visit_set_id=sps_sequence.visit_set_id \ <span class="code-keyword">inner</span> join sps_visit on sps_visit.pfs_visit_id=visit_set.pfs_visit_id \ <span class="code-keyword">inner</span> join sps_exposure on sps_exposure.pfs_visit_id=visit_set.pfs_visit_id \ <span class="code-keyword">inner</span> join sps_camera on sps_camera.sps_camera_id=sps_exposure.sps_camera_id \ left <span class="code-keyword">outer</span> join sps_annotation on sps_exposure.pfs_visit_id=sps_annotation.pfs_visit_id order by visit_set.pfs_visit_id desc </pre> </div></div> <h2><a name="ExtractYAMLconfigurationfile"></a>Extract YAML configuration file</h2> <p>For 2D DRP processing, it is important to be able to determine which biases, darks, flats and arcs need to be processed at a given point. This work is described in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-423" title="Regular DRP processing of LAM and Subaru data" class="issue-link" data-issue-key="PIPE2D-423">PIPE2D-423</a> and related tickets. There will be specific child tickets for opDB interactions raised shortly. No example queries are yet available on this specific work, but the query provided by A Lefur in the earlier comment may be a basis. </p> <p>Here is the code I have for now to create log pages from opdb. I will clean that code and update pfs_utils lib to add the second page.</p> <p>but it will give you an idea:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> with psycopg2.connect(<span class="code-quote">"host=<span class="code-quote">'{}'</span> port={} dbname=<span class="code-quote">'{}'</span> user={}"</span>.format(host, port, dbname, user)) as conn: sql = "select seq.visit_set_id as visit_set_id, sequence_type, name,\ min(visits.pfs_visit_id) as min_visit,\ max(visits.pfs_visit_id) as max_visit, \ cmd_str, comments, status \ from sps_sequence seq \ join visit_set visits on visits.visit_set_id = seq.visit_set_id \ where seq.visit_set_id &gt;= 361 \ group by seq.visit_set_id \ order by seq.visit_set_id DESC; \ " dat = pd.read_sql_query(sql, conn, index_col=<span class="code-quote">"visit_set_id"</span>)dat.to_html(<span class="code-quote">"logbook_opdb_sm1_subaru.html"</span>) </pre> </div></div> <p>and the all visits page:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> sql_all = "select cmd_str,sps_exposure.pfs_visit_id,\ sps_sequence.visit_set_id, \ exp_type, time_exp_start,\ notes, data_flag, \ comments, name, status, sequence_type,\ sps_module_id,arm,sps_exposure.sps_camera_id \ from sps_exposure \ <span class="code-keyword">inner</span> join visit_set on sps_exposure.pfs_visit_id=visit_set.pfs_visit_id \ <span class="code-keyword">inner</span> join sps_sequence on visit_set.visit_set_id=sps_sequence.visit_set_id \ <span class="code-keyword">inner</span> join sps_visit on sps_exposure.pfs_visit_id=sps_visit.pfs_visit_id \ <span class="code-keyword">inner</span> join sps_camera on sps_exposure.sps_camera_id = sps_camera.sps_camera_id \ left <span class="code-keyword">outer</span> join sps_annotation on sps_exposure.pfs_visit_id=sps_annotation.pfs_visit_id \ where sps_sequence.visit_set_id &gt;= 361 \ order by sps_exposure.pfs_visit_id DESC; \ with psycopg2.connect(<span class="code-quote">"host=<span class="code-quote">'{}'</span> port={} dbname=<span class="code-quote">'{}'</span> user={}"</span>.format(host, port, dbname, user)) as conn: df_all = pd.read_sql_query(sql_all, conn) df_all.sort_values(<span class="code-quote">"pfs_visit_id"</span>, inplace=True) df_all.set_index([<span class="code-quote">'visit_set_id'</span>, <span class="code-quote">'pfs_visit_id'</span>], inplace=True) df_all.to_html(<span class="code-quote">"logbook_opdb_visit_sm1_subaru.html"</span>) </pre> </div></div> Blocks INSTRM-735 Development Rank 0|zzs1kg: