https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-66 Instrument control development <p>The archiver takes a long time to start, evidently because of checking/crawling all the existing rows. At LAM, the raw tables have ~170M rows:</p> <blockquote> <p>2017-01-05 17:31:36+0000 Log opened.<br/> 2017-01-05 17:31:36+0000 importing postgres dbapi module psycopg2<br/> 2017-01-05 17:31:36+0000 database: table enu__slit contains 1 rows<br/> 2017-01-05 17:33:50+0000 database: table reply_raw contains 170736146 rows<br/> 2017-01-05 17:35:06+0000 database: table reply_hdr contains 170686655 rows<br/> 2017-01-05 17:35:06+0000 database: table hub__commanders contains 104527 rows<br/> 2017-01-05 17:35:06+0000 database: table hub__users contains 104526 rows<br/> 2017-01-05 17:35:06+0000 database: table actors contains 44 rows</p></blockquote> <p>It is not clear whether that scan is necessary. If not perhaps we can remove it. If it is, perhaps we need to use a union database or something.</p> INSTRM-66

archiver startup is slow

Task Major Done Done arnaud.lefur cloomis Thu, 5 Jan 2017 17:48:21 +0000 Mon, 2 Jul 2018 19:24:14 +0000 Mon, 2 Jul 2018 19:24:12 +0000 0 3 <p> <b>"select count<img class="emoticon" src="https://pfspipe.ipmu.jp/jira/images/icons/emoticons/star_yellow.png" height="16" width="16" align="absmiddle" alt="" border="0"/> from %s" % tableName"</b> is really slow with a billion row.<br/> But it appears that's not really necessary to do it every single tables.</p> <p>However it's required for <em>reply_raw</em>, and <em>reply_hdr</em>, in that case, i'm just taking the last row_id, and it seems ok.</p> <p><a href="https://github.com/Subaru-PFS/ics_archiver/commit/ea01a7421cba16da3054e7ac3d0d3316191ca266" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/ics_archiver/commit/ea01a7421cba16da3054e7ac3d0d3316191ca266</a></p> <p>I'll merge later, if it's fine.</p> <p>Or if it is int/bigint type, you could use MAX() instead of??</p> <ol> <li>In pgsql, count<img class="emoticon" src="https://pfspipe.ipmu.jp/jira/images/icons/emoticons/star_yellow.png" height="16" width="16" align="absmiddle" alt="" border="0"/> could be quite slow if index only scan is not enabled, due to MVCC... (I don't have clear idea that MAX() can solve this,,, sorry.)</li> </ol> <p>I agree with using MAX():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-sql"> archiver=# <span class="code-keyword">explain</span> <span class="code-keyword">select</span> <span class="code-keyword">count</span>(id) <span class="code-keyword">from</span> reply_raw; QUERY <span class="code-keyword">PLAN</span> <span class="code-comment">---------------------------------------------------------------------------- </span> <span class="code-keyword">Aggregate</span> (cost=1695406.31..1695406.32 <span class="code-keyword">rows</span>=1 width=8) -&gt; Seq Scan <span class="code-keyword">on</span> reply_raw (cost=0.00..1533464.05 <span class="code-keyword">rows</span>=64776905 width=8) (2 <span class="code-keyword">rows</span>) </pre> </div></div> <p>versus:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-sql"> archiver=# <span class="code-keyword">explain</span> <span class="code-keyword">select</span> <span class="code-keyword">max</span>(id) <span class="code-keyword">from</span> reply_raw; QUERY <span class="code-keyword">PLAN</span> <span class="code-comment">------------------------------------------------------------------------------------------------------------------------- </span> <span class="code-keyword">Result</span> (cost=0.61..0.62 <span class="code-keyword">rows</span>=1 width=0) InitPlan 1 (<span class="code-keyword">returns</span> $0) -&gt; <span class="code-keyword">Limit</span> (cost=0.56..0.61 <span class="code-keyword">rows</span>=1 width=8) -&gt; <span class="code-keyword">Index</span> <span class="code-keyword">Only</span> Scan <span class="code-keyword">Backward</span> <span class="code-keyword">using</span> reply_raw_pkey <span class="code-keyword">on</span> reply_raw (cost=0.56..2732366.40 <span class="code-keyword">rows</span>=64776905 width=8) <span class="code-keyword">Index</span> Cond: (id <span class="code-keyword">IS</span> <span class="code-keyword">NOT</span> <span class="code-keyword">NULL</span>) (5 <span class="code-keyword">rows</span>) </pre> </div></div> <p>Note that we should perhaps be using postgresql's `BIGSERIAL` type for the ids. Probably does not matter for single-writer tables. But it does make the monotonically increasing integer constraint explicit.</p> <p>Due to MVCC used by pgsql, you need to use some trick or configuration to get rid from seq scan into index only scan for count(xxx). For just getting max number from auto_increment-ish column, the easiest way in pgsql is max()...</p> <p>Yes, I should have used max(), what I did is equivalent but not very clean.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a> close this?</p> <p>This has been fixed for a year.</p> Development Rank 0|ii01h3: