https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/OBSPROC-29 PFS observation processing/procedure <p>Since the number of records of `sky` table in targetDB is so large, it takes so long to get sky targets in the process of pfsDesign generation. Consider efficient way for the query. One option is to use `q3c_radial_query`, which is used for gaiaDB.</p> OBSPROC-29

Consider efficient query for sky targets

Task Normal Done Done monodera Kiyoto Yabe EngRun Mon, 19 Sep 2022 23:59:38 +0000 Fri, 3 Mar 2023 05:04:32 +0000 Fri, 3 Mar 2023 05:04:32 +0000 0 2 <p>Now we fixed this using `q3c`. Could you briefly summarize what you did with the SQL commands, please? <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=monodera" class="user-hover" rel="monodera">monodera</a></p> <p>Also, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=msyktnk" class="user-hover" rel="msyktnk">Masayuki Tanaka</a> mentioned that we might need to think about distributed querying using such as Citus when targetDB expands in future.</p> <p><tt>q3c</tt> was installed by Furusawa-san as an extension of PostgreSQL. After the installation, I logged into the current targetdb (currently <tt>targetdb_comm2022may</tt> database) and executed the following SQL commands.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-sql"> <span class="code-keyword">CREATE</span> EXTENSION q3c; <span class="code-keyword">CREATE</span> <span class="code-keyword">INDEX</span> <span class="code-keyword">ON</span> sky (q3c_ang2ipix(ra,<span class="code-keyword">dec</span>)); <span class="code-keyword">CLUSTER</span> sky_q3c_ang2ipix_idx <span class="code-keyword">ON</span> sky; <span class="code-keyword">ANALYZE</span> sky; </pre> </div></div> <p> </p> <p>There are various functions in the q3c extension, but at this point only <tt>q3c_radial_query()</tt> which simply returns true if (ra, dec) is in a search radius. An example query is the following.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-sql"> <span class="code-keyword">SELECT</span> * <span class="code-keyword">FROM</span> {tablename} <span class="code-keyword">WHERE</span> q3c_radial_query(ra, <span class="code-keyword">dec</span>, {ra}, {<span class="code-keyword">dec</span>}, {search_radius}) </pre> </div></div> <p>Documentation can be found at the following GitHub repository.</p> <p><a href="https://github.com/segasai/q3c" class="external-link" rel="nofollow">https://github.com/segasai/q3c</a><br/>  </p> <p> </p> <p>BTW, the q3c indexing is only done in the sky table.</p> <p>FYI, Murata-san commented that the HSC SSP database uses the earthdistance extension in postgresql for indexing. Coord column is used as index.</p> <p>According to <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=monodera" class="user-hover" rel="monodera">monodera</a> efficient query was achieved by indexing with q4c. If other method needs to be used (see the above comment), it will be handled in another ticket.</p> Development Rank 0|zzsupb: