PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 [INSTRM-1180] ingest using shell call to ingestPfsImages https://pfspipe.ipmu.jp/jira/browse/INSTRM-1180 Instrument control development <p>We currently ingesting using a Task. Switch to a call to ingestPfsImages. </p> <p>Hoping to make new file processing faster and less prone to registry failures.</p> INSTRM-1180 ingest using shell call to ingestPfsImages Task Normal Done Done cloomis cloomis Fri, 12 Feb 2021 17:16:08 +0000 Mon, 15 Feb 2021 17:30:22 +0000 Mon, 15 Feb 2021 17:30:22 +0000 ics_drpActor 0 2 <p>Doesn't really help. Delay Is dependent on the size of the registry, so we do have to figure this out. Not obvious that the copy-update-rename hack fixed by the DM ticket is the problem, as much as I hate and want to blame that. Does seem to be at least partially tied up with a sequential scan of the db right before closing. More to come.</p> <p>Good news: ingest really does take long time due to a fixable obs_pfs problem updating the <tt>raw_visit</tt> table. <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-715" title="Remove updates to raw_visit table from ingestPfsImages.py" class="issue-link" data-issue-key="PIPE2D-715"><del>PIPE2D-715</del></a></p> <p>Bad news: the drpActor does have associated sequencing problems which still need to be addressed. But I think addressing those will either take care of <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-1075" title="drpActor still hangs" class="issue-link" data-issue-key="INSTRM-1075"><del>INSTRM-1075</del></a> or at least make things less confusing.</p> <p>Great, it used to be ~atomic, so that makes sense.<br/> Yeah, basically the command thread is the bottleneck, as a result ingest + detrend are done sequentially.<br/> I though it was safer to avoid dealing with threads, especially with DB behind the scene.<br/> We still want to make sure that detrend is fired once after all files are ingested right ?</p> <p>This is still a mess. I am running <tt>obs_pfs cpl</tt> on top of <tt>pfs_pipe2d w.2021.06b</tt>, which disables the <tt>raw_visit</tt> updates which cause ingest to slow down to 10s of seconds (everywhere, not just on shell2-ics). There may still be gains to be made on ingest.</p> <p>I re-ordered the internal processing of ingest&amp;detrend in order to guarantee that only one operation is running at a time.</p> <p>Added <tt>isr.doDefect=False</tt> to get around a temporary detrend bug. See <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-715" title="Remove updates to raw_visit table from ingestPfsImages.py" class="issue-link" data-issue-key="PIPE2D-715"><del>PIPE2D-715</del></a>.</p> <p>Even though shell-2 is under provisioned (4-core, 4GB)I do <b>not</b> see bad system contention (per <tt>dstat 1</tt>), but detrend is still slow (12-30s).</p> <p>There are two oddities which need to be figured out before I will trust or suspect <em>anything</em> else. One of the loggers (not the main actorcore/python logger, nor the LSST logger) multiplies itself, adding one line for each file processed. And when you quit the drpActor, there are a correspondingly increasing number of complaints from sqlite and from the multiplying logger. But according to the main thread, we only ever get four threads (not many many), and the thread ids printed by the sqlite complaint at the end are all the same:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2021-02-14 18:40:49.316Z actor 20 Actor.py:550 reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... actor INFO: reactor dead, cleaning up... Exception ignored in: &lt;function SqlRegistry.__del__ at 0x7fd4fea88400&gt; Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/registries.py", line 323, in __del__ self.conn.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140551400564480 and this is thread id 140553341876032. Exception ignored in: &lt;function SqlRegistry.__del__ at 0x7fd4fea88400&gt; Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/registries.py", line 323, in __del__ self.conn.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140551400564480 and this is thread id 140553341876032. Exception ignored in: &lt;function SqlRegistry.__del__ at 0x7fd4fea88400&gt; Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/registries.py", line 323, in __del__ self.conn.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140551400564480 and this is thread id 140553341876032. Exception ignored in: &lt;function SqlRegistry.__del__ at 0x7fd4fea88400&gt; Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/registries.py", line 323, in __del__ self.conn.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140551400564480 and this is thread id 140553341876032. Exception ignored in: &lt;function SqlRegistry.__del__ at 0x7fd4fea88400&gt; Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/registries.py", line 323, in __del__ self.conn.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140551400564480 and this is thread id 140553341876032. Exception ignored in: &lt;function SqlRegistry.__del__ at 0x7fd4fea88400&gt; Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/registries.py", line 323, in __del__ self.conn.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140551400564480 and this is thread id 140553341876032. Exception ignored in: &lt;function SqlRegistry.__del__ at 0x7fd4fea88400&gt; Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/registries.py", line 323, in __del__ self.conn.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140551400564480 and this is thread id 140553341876032. Exception ignored in: &lt;function SqlRegistry.__del__ at 0x7fd4fea88400&gt; Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/registries.py", line 323, in __del__ self.conn.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140551400564480 and this is thread id 140553341876032. Exception ignored in: &lt;function SqlRegistry.__del__ at 0x7fd4fea88400&gt; Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/registries.py", line 323, in __del__ self.conn.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140551400564480 and this is thread id 140553341876032. Exception ignored in: &lt;function SqlRegistry.__del__ at 0x7fd4fea88400&gt; Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/registries.py", line 323, in __del__ self.conn.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140551400564480 and this is thread id 140553341876032. </pre> </div></div> <p>A cute thing to notice in that last log: the "actor" logger is an actorcore thing, along with the "cmds" logger. Both always use the formatting you see in that top line: the date, logger name, etc. But after the detrend task is called the first time, every output line on the actor logger then also generates the multiplying lines as formatted by the LSST logging system: the logger name, the level, the message.</p> <p>Also, detrend was being called on each camera, but without specifying the arm: the second call processed both arms. Fixed.</p> <p>FWIW, with the ingest raw_visit problem fixed, ingest+detrend take ~20-25s, which is fast enough to avoid piling up when taking, say, a sequence of biases. So I propose:</p> <ul class="alternate" type="square"> <li>I'll close and merge this ticket, which includes some hack-y crap to simplify and track the problems.</li> <li>Arnaud starts from that and works on the detrend ticket, which <em>might</em> fix this stealing and multiplying problem.</li> <li>If that does not work by the day SuNSS goes back on the sky, move detrend out into an external process, like ingest now is. Sure, we eat some overhead, but at least we can stay ahead of the incoming data and not eat ourselves.</li> <li>Separately, have the listener gather all files for a given visit and process them all at once. Yes, that requires knowing which cameras are live, but that is doable.</li> </ul> <p>Frankly, grouping all exposures fo a visit <b>and</b> calling detrend.py externally is probably as good as we can do in any case: the detrend.py overhead will be taken up by the 8 exposures.</p> <p>And finally, I will request that shell2-ics be beefed up some more. This configuration is silly.</p> <p><em>Also, detrend was being called on each camera, but without specifying the arm: the second call processed both arms. Fixed.</em><br/> that's right, but I did it on purpose. The idea is that since I'm calling detrend without arm, the first call was processing all matching ingested files.<br/> But in fact, I was checking the file existence before actually calling detrend, so when the second call kicks in, the file is already there, hence it just generates the filepath keyword for gingaActor. </p> <p>Agreed, will try to fix <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-1178" title="replace drpActor parseAndRun" class="issue-link" data-issue-key="INSTRM-1178"><del>INSTRM-1178</del></a> asap and external call will definitely my safety net.</p> <p>Merged at 76ee97a. Intended to be a sloppy mess.</p> Relates PIPE2D-715 Development Rank 0|zzs42g: Story Points 1.0