[INSTRM-1111] Add file writing process to hxActors Created: 16/Nov/20 Updated: 02/Mar/22 Resolved: 02/Mar/22 |
|
| Status: | Done |
| Project: | Instrument control development |
| Component/s: | ics_hxActor |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Normal |
| Reporter: | cloomis | Assignee: | cloomis |
| Resolution: | Done | Votes: | 0 |
| Labels: | SPS | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Story Points: | 3 |
| Sprint: | SM1PD-2021 A 12, SM1PD-2022 B |
| Description |
|
When reading ramps we need to stay ahead of the DAQ. I currently read from the DAQ and write to the FITS file in the same process, but any delay causes the DAQ to run out of buffer and for the reads to fail. A recent change in the readout sequence to minimize the effect of some electronic noise is making us hit the hardware limits. Add a process which handles the file writing. The tricky part is that we promise to generate keywords when we have flushed individual reads and the entire PFSA and PFSB files, and decoupling readouts from file writes will make that less trivial. |
| Comments |
| Comment by cloomis [ 27/Nov/20 ] |
|
If we want this can also be added to the ccdActor/mcsActor. Can't say I like adding this kind of complexity (a process for the file writing and a thread for tracking progress from the process) in the core of data acquisition, but it does seem to work. Need to walk through all the new failure modes. |
| Comment by cloomis [ 05/Dec/20 ] |
|
This works in the sense of noticeably improving the performance of the reads from the SAM. But the two processes as a whole perform quite badly: either image transfers over the Queue are slow or something is bad about the i/o being done on the external process. Nothing obvious with the system. Will test with multiprocessing.shared_memory |
| Comment by cloomis [ 27/Jan/22 ] |
|
Working cleanly with multiprocessing.shared_memory. That only comes with python 3.8 though. I am using a conda 3.9 environment and the actor is working fine. If that is not acceptable I can implement something similar using mmap. But would prefer to use a standard library routine. |
| Comment by cloomis [ 02/Mar/22 ] |
|
Merged at 20898d1, then transferred to ics_utils under |