https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-643 Instrument control development <p>ccd r1 crashed during biases exposure</p> <p> </p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> </pre> </div></div> <p>{{2019-04-04 10:40:19 ccd_r1 f text="command failed: KeyError('fee',) in fee() at python/ccdActor/main.py:47" }}</p> <p> </p> <p> </p> <p>then tried disconnect/connect</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2019-04-04 10:40:48 cmdIn=ccd_r1 connect controller=fee 2019-04-04 10:40:49 ccd_r1 w controllers="ccd" 2019-04-04 10:40:49 ccd_r1 w text="failed to connect controller fee" 2019-04-04 10:40:49 ccd_r1 f text="command failed: RuntimeError('failed to arm for readout)',) in pyFPGA.FPGA.setClockLevels() at fpga/pyFPGA.pyx:127" </pre> </div></div> <p> </p> <p> </p> <p>then tried power off/on and connect but:</p> <p> </p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2019-04-04 10:45:14 ccd_r1 w controllers="ccd" 2019-04-04 10:45:14 ccd_r1 w text="failed to connect controller fee" 2019-04-04 10:45:14 ccd_r1 f text="command failed: RuntimeError('failed to arm for readout)',) in pyFPGA.FPGA.setClockLevels() at fpga/pyFPGA.pyx:127" </pre> </div></div> <p> </p> <p> </p> <p> </p> <p> </p> INSTRM-643

ccd r1 fee crash

Bug Major In Progress Unresolved cloomis fmadec Thu, 4 Apr 2019 10:52:09 +0000 Wed, 23 Dec 2020 20:09:14 +0000 ics_ccdActor 0 3 <p>a restart of ccd actor solved the issue...</p> <p> </p> <p>Slightly confused by the output; will look more carefully at the logs.</p> <p>This is an amusing cluster of problems....</p> <p>Problem 0: the exposure after 15317 "hung". Specifically, the background thread started to run the integration and readout never started the readout (hence no <tt>exposureState=readout</tt>). This command/code path (running exposures in a background thread) is not used in production (or will not be after <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-626" title="give up using ccd expose bias|dark command" class="issue-link" data-issue-key="INSTRM-626"><del>INSTRM-626</del></a>). But the thread holds a reference to an instance of the FEE object, which caused the later problems. No, I do nto know why readout did not start, but it feels thread-y. There is a small chance that <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-640" title="values following nan are not reported in FITS header" class="issue-link" data-issue-key="INSTRM-640"><del>INSTRM-640</del></a> is implicated, but since the transition to integration requires that the <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-640" title="values following nan are not reported in FITS header" class="issue-link" data-issue-key="INSTRM-640"><del>INSTRM-640</del></a> header be complete I don't see how right now.</p> <p>Problem 1: the exposure did not finish, so the exposure was cleared and the fee reconnected. But it couldn't be reconnected, because the exposure thread was not really dead (python cannot kill stuck threads) and still had a reference to an old fee object, which had the hardware connection open.</p> <p>The two problems explain why there are two "concurrent" and very very different complaints about the FEE.</p> <p>What to do? I'm inclined to instrument a few things better and make fee disconnection more violent/effective, but that's it. At first these look like very serious problems, but it is only because of the background thread which is going away. So understanding exactly <em>why</em> that thread hung up is not worth working on. If there is an underlying problem, it will be much easier to see and fix without the thread.</p> <p>It happened today, but I had an issue with spsaitActor, it's probably my fault, I have to check the logs...</p> <p>Bump, maybe. Tx, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a>. The 2020-12-17 problems might well have happened because the file writing took forever, so the per-exposure thread never closed out. </p> <p>At the very least, new exposures should be rejected if an old exposure thread exists. What to do to better kill a stuck thread is still not clear.</p> Blocks INSTRM-626 Development Rank 0|zx004o: