https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-1993 Instrument control development <p>Ramps sporadically fail, where the actor cannot get enough pixels from the DAQ to complete the ramp. There is no per-read framing, so this is only discovered at the end of the ramp, and the drops could have come anytime.</p> <p>On n1 at Subaru, we are seeing quite a few of these: 19/516 from 2023-05 to now, ~4%. It turns out that most/all ramps are simply missing pixels at the <b>beginning</b> of the ramp, visible even in the RESET frame. This is probably good news: even if we cannot fully recover that frame, it is essentially never used.</p> <p>I have not yet found anything stupid on my side, sadly, nor seen warnings of SAM FIFO overflows. There are various counters on the SAM and the ASIC which can be reported more enthusiastically; I'll work on adding those under this ticket. </p> INSTRM-1993

Fix pixel drops at start of ramps

Bug Major Done Done cloomis cloomis near-term Thu, 15 Jun 2023 13:58:04 +0000 Fri, 28 Jul 2023 10:21:50 +0000 Fri, 28 Jul 2023 10:21:50 +0000 hxhal ics_hxActor 0 1 <p>Partly inspired by <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-2013" title="reported readTimes incorrect" class="issue-link" data-issue-key="INSTRM-2013"><del>INSTRM-2013</del></a>, I will completely validate the ASIC configuration which we read back at the start of each ramp. It can also be rewritten/reloaded, but that can take a while (10s of seconds in some cases) so we do not want to reconfigure unnecessarily. These values are not supposed to change, but I have seen the row window and number of outputs registers unexpectedly change. Both of which would completely mess things up.</p> <p>About a third of the failures happened because the number of pixels expected for the ramp is calculated based on values read back from the ASIC, and sometimes those values are read back wrong. Simple fix for these particular ramps: calculate based on the values we wrote to the ASIC when configuring it.</p> <p>This seems moderately likely to be from errors on the SAM (ASIC register reads are done by programming some SAM registers; in one case the value of the register was the <b>address</b> of the register): I think we will find similar problems with some of the other failed ramps.</p> <p>I have made most of the changes required to notice failed ramps quickly, and restructured the inner <tt>takeRamp</tt> method to support resuming a stopped ramp. But need a failure at this point before I can actually implement that. </p> <p>I am going to close this one, although I do expect to have smaller followup tickets. Did four things, basically:</p> <ul class="alternate" type="square"> <li>ignore read back of the geometry registers, and instead use the values we wrote to them. This is the one problem which looks like hardware (<b>our</b> hardware, at least).</li> <li>be gentler about stopping a ramp in the middle, as we often do from sunss. Basically, set the ASIC to idle, but do <b>not</b> reconfigure it. I still don't think I got this entirely right for all cases.</li> <li>be more aggressive about initializing the SAM right before starting a ramp, and reporting errors.</li> <li>start cleaning up after total failures in the inner loop. I will make a new ticket for finishing this work.</li> </ul> <p>Yeah, close it, but also look at 98434 on n1.</p> <p>As commented, there will be more work, but this ticket should be closed.</p> <p>hxhal: tagged 3.1.1<br/> ics_hxActor: tagged 2.7.9</p> <p>which is what we have been running for a couple of nights at the end of the 2023-07 run.</p> Relates INSTRM-2013 Development Rank 0|02qpjd:00r20060i200186403c Sprint Eng12July