https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/DAMD-29 Data Model <p>Besides the to-be-archived PFSD guider files and the strange hex image for the Mitsubishi displays, we need to define the files made specifically for displaying while guiding. It turns out that the mechanism for doing this is new at Subaru, so this is a white-sheet design – we should be able to design something which works well for both Subaru and PFS, and it will be collaboratively designed.</p> <p>A brief discussion led to a few starting suggestions:</p> <ul> <li>N (6) HDUs for the images</li> <li>zero-ed out background. RICE compression. Or maybe in this case HCOMP?</li> <li>display scaling parameters in the headers, as well as the WCS.</li> <li>a binary table with object and offset properties, good enough for useful annotations.</li> <li>either HDU cards or a binary table for the final guiding/focus.</li> <li>filename is not bound to the PFS visit</li> <li>file is not expected to be used for anything but live display.</li> </ul> <p> Even though this is purely intended for observing, since it is a published interface I'm putting this ticket in DAMD.</p> DAMD-29

Define AGCC files for display while guiding

Task Normal Open Unresolved Yoshida, Hiroshige cloomis CanBeClosed FITS PFI Mon, 30 Apr 2018 18:00:59 +0000 Wed, 1 Feb 2023 20:22:50 +0000 0 5 <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>, this is a pretty close start to what we want, but a couple of points:</p> <ul> <li>the data is used both during observation to monitor the guiding, and evaluate the sky level, etc. We absolutely want to get raw data, not a zeroed out background. </li> <li>data is updated periodically in a display gui, but can also be saved as a "VGW" prefixed file by the SA or Operator dynamically. This all happens on the Gen2 side, so apart from sending the data at a periodic interval there are should be no impact to PFS side file operations.  VGW files are archived, and can be used for engineering purposes, troubleshooting, understanding guiding problems during observation at specific points, etc.</li> <li>a table HDU attached for metadata would be really useful, and we should itemize in this ticket what kinds of interesting metadata that could be attached, but we should not hold up the periodic transfer of the images waiting for particular items like final focus to be decided; I'm assuming that will also be available as a status item in gen2 as well, since that will be the preferred method for transmitting it in a telescope command for setting the Z position. VGW files can also collect status items when writing out files, so things can be added later from status items in Gen2 as needed.  </li> </ul> <p>Each image is 1024 * 1024 * 2, correct?</p> <p>Ideally, you would not need to actually write out these FITS images as files to disk on PFS side. We'd have to check the memory constraints, but what I imagine is an actor (or process) has an astropy or fitsio HDUList in memory and is simply taking the image buffers on a periodic basis and overwriting the same numpy arrays and maybe updating a few items in a table HDU, then grabbing the whole thing as a buffer and sending it over the network.  </p> <p>Assuming these are 1x1 K 16 bit images, then with basic headers and a table HDU attached you are looking at something around 15-20MB in size in memory, being updated periodically.</p> <p>If we decide to transfer them via file transfer, then we could also just use a small circular number of FITS files and sending a URL for Gen2 to copy them.</p> <p>Interesting. We had not thought in those terms. I guess I'd say that if there is little real cost (threshold at ~1-2 s, maybe?) to using writing and transferring FITS files we should use them just to keep the number of maintained interfaces down. </p> <p>But since this is use-once-then-discard data, sure. Use zmq or something? Do you have a preference?</p> <p><span class="error">&#91;Aside: Apologies for the late comment: apparently I am not getting notified when a ticket I comment on is updated. I sort of assumed that Jira worked a bit like Github issues, but apparently not. I did check that I am a watcher on this ticket so I don&#39;t know what happened. If there is any suggestion for how to make that the default behavior in my settings, let me know.&#93;</span></p> <p>&gt; I guess I'd say that if there is little real cost (threshold at ~1-2 s, maybe?) to using writing and transferring FITS files we should use them just to keep the number of maintained interfaces down.</p> <p>&gt; But since this is use-once-then-discard data, sure. Use zmq or something? Do you have a preference?</p> <p>I agree.  And I think its best not to over-engineer something until we see a good reason for it.  So I suggest just writing a FITS file and notifying Gen2 of the path.  For notification, how about using the messaging hub? If making a client to that does not involve anything complicated (I'm assuming not) then we can just write a small, Gen2-side receiver that will listen on the channel/topic for "gen2-ag" (or whatever you want to call it) and then FTP/SFTP/SCP the file.</p> <p>A <em>test</em> message to see if Eric sees Jira updates as a watcher.</p> <p>I got a notification. </p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=eric" class="user-hover" rel="eric">eric</a> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hiro" class="user-hover" rel="hiro">Yoshida, Hiroshige</a>: can this ticket be closed now?</p> <p>I don't think we have heard back from @cloomis or @Yoshida. Also, it is not clear yet (to me) who is assigned this work and what actor it happens in.</p> <p> </p> <p>My understanding is that if the AGCC actor writes out MEF files with images and a binary table extension that contains metrics of detected candidate guide stellar objects, they can be sent straightaway to Gen2. If the actual candidate guide stellar objects used and/or the field acquisition/guiding offsets determined are also needed, they can either be sent as MHS then Gen2 status values to be captured by the Gen2-side client for annotation, or the AG client, or likely another new actor, will have to add an another binary table extension with the information before image files are sent out to Gen2.</p> <p>Following discussions during the ICS/PFI telecon 2020-10-16: this ticket can be closed.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=eric" class="user-hover" rel="eric">eric</a> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hiro" class="user-hover" rel="hiro">Yoshida, Hiroshige</a> still want this ticket &#8211; we are at the point where the details can and need to be pinned down. And we think we understand the system well enough to do that. The only significant changes from the original proposal are:</p> <ul> <li>do <b>not</b> cleverly compress the background. We will instead save the RICE-compressed background-subtracted image: the exact image that centroiding operates on.</li> <li>do use a visit- and sequence-based filename. Really, why not?</li> </ul> <p>Since the nice VGW mechanism snapshots these files, I think we should stick with keeping everything in one file. </p> <p>For implementation purposes, I think the agActor should write and announce the file: it knows the guide star catalog in use, the outputs from Kawanomoto-san's routines, and the final telescope and focus offsets. It is possible (likely) that the agccActor will write the image HDUs first, but that is an internal implementation detail. Once the file is finished, some "agDisplayFile" MHS keyword would get generated, which the gen2Actor would pick up on to generate a Gen2 status keyword, or whatever <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=eric" class="user-hover" rel="eric">eric</a> wants to see.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=eric" class="user-hover" rel="eric">eric</a> Can we agree on a naming convention for these? Is there something you already use? I am under the impression that the "VGW" names are only for the snapshotted file &#8211; is that true?</p> <p>If there are no constraints, I'll suggest including the PFS visit and guider sequence number within that. Oh, say, <tt>pfsGuidingDisplay-%06d-%03d.fits</tt>. </p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hiro" class="user-hover" rel="hiro">Yoshida, Hiroshige</a> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>: Discussing this with Moritani-san, I believe that this ticket, including file name convention, might already be covered by <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-138" title="Add AGCC filename format" class="issue-link" data-issue-key="DAMD-138"><del>DAMD-138</del></a>, and already implemented. Can you confirm? </p> Blocks INSTRM-1001 Relates INSTRM-1006 DAMD-82 DAMD-2 DAMD-138 Development Rank 0|02qpjd:00r3000k0014 Sprint PreEngRun4 PreEngRun05 F Story Points 4.0