https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/DAMD-2 Data Model <p>In definition of the current datamodel, AG is assigned name as <br/> &gt; "PF%1sD%06d%1d%1d.fits" % (site, visit, spectrograph, armNum); category=D<br/> but there is no plan to save AG image as archive in PFS* namespace at Subaru.<br/> File transfer interface to Gen2 for AG FITS files are normally unnamed, and operator <b>can</b> save an image in interest as VGW* namespace (AFAIK).</p> <p>So, we may have options:</p> <ul class="alternate" type="square"> <li>delete category=D</li> <li>add interface definition to save a or some AG image as PFSD%6d00.fits (00 = no SM, no arm; or just sequential number in 2 digits)</li> <li>add interface definition to save a stacked AG image over one exposure period as PFSD%6d00.fits</li> </ul> <p><b>Combining constraint originally described in <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-82" title="AG FrameId Allocation" class="issue-link" data-issue-key="DAMD-82"><del>DAMD-82</del></a></b>: Following discussions about the AG frameID during the DRP telecon 2020-05-22, the archiving should cover AG exposures for all 6 cameras and take into the account that there may be more than 100 exposures per visit.</p> DAMD-2

AG image output to archive

Task Major In Progress Unresolved cloomis shimono EngRun FITS PFI Thu, 2 Jun 2016 04:56:23 +0000 Fri, 6 Jan 2023 04:24:12 +0000 0 8 <p>I do not see the corresponding git branch <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-2" title="AG image output to archive" class="issue-link" data-issue-key="DAMD-2">DAMD-2</a> &#8212; did you forget to push it? It is not possible to understand this proposal until the details are added to datamodel.txt (note that no implementation is required until this issue is accepted) (RHL)</p> <p>After some discussion by email and on the software fuze con the conclusion was that we should maintain the PFSD AG images, but make them multi-extension fits files with one extension for each guide exposure. Philip Tait confirms that Subaru is able to handle MEFs (e.g. FMOS and MOIRCS).</p> <p>Because we are using windowed reads we'll write NaNs into the un-read pixels and compress the files; that way the format will be the same for all exposures, although the number of extensions will vary with the exposure time.</p> <p>I think current plan is not to use windowed reads on AG cameras, but just use windowed region for calculation of object statistics. We are possible to mark outside of that windowed region as NaNs for archived data, of course. Also we may use multiple AG cameras, and need FITS files in MEF with multi HDU, right?</p> <p>One another item. While writing document on AG communication and data output rate now, I noticed that we need to have a plan on this FITS file for exposure sequence with different exposure time per arm, like 15min red/IR and 7.5min blue. We could have only one FITS file over entire exposure sequence. In any case, we need to have some lines in this datamodel definition.</p> <p>The data model proposes:</p> <blockquote> <p>Raw Data from the metrology camera and autoguider (AG)</p> <p> "PF%1s%1s%06d%02d.fits" % (site, category, visit, cameraNum)</p> <p>There are many exposures for each visit. These files will accordingly be multi-extension fits files (MEFs)<br/> with one extension per read and one file per camera.</p> <p>The AG data (PF?D files) is a bit different as we only process small windows. The images will be the<br/> full size of the detectors (1024x1024) and unsigned 16-bit, but with all unprocessed-pixels<br/> represented as 0 (we will set the keyword BLANK = 0 &#8211; we'd use NaN, except that these files will<br/> contain integers). We will use FITS (probably fpack) compression on these files, so the unprocessed<br/> pixels will only have a negligible effect upon the total image size.</p> <p>The first (and possible last) AG exposures will be complete, with no masking applied. An advantage<br/> of the proposed format is that the masked and unmasked HDUs will be identical.</p></blockquote> <p>I thought about making one large image, and decided that it wasn't worth while for two reasons:</p> <ol> <li>It would require getting all the images in memory on one machine. I'm not sure if that's easy for the DA</li> <li>There's no natural wcs for the mosaic. Each chip is well defined, but the 6 chips aren't; worse, the chips can move relative to each other.</li> </ol> <p>Assigned issue to @rhl (as he was the last person to commit changes to branch)</p> <p>Adding comment originally provided by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoyuki.tamura" class="user-hover" rel="naoyuki.tamura">naoyuki.tamura</a> to <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-82" title="AG FrameId Allocation" class="issue-link" data-issue-key="DAMD-82"><del>DAMD-82</del></a> 2020-08-06:</p> <p>At the telecon dedicated to AG on June 23/24 2020 ( <a href="https://sumire.pbworks.com/w/page/140496828/WG%204%20(AG)%20telecon%20on%2023-24%20June%202020" class="external-link" rel="nofollow">https://sumire.pbworks.com/w/page/140496828/WG%204%20(AG)%20telecon%20on%2023-24%20June%202020</a> ), eric suggested and we agreed on implementing multiple HDUs to pack as many exposures as we can, rather than limiting them below 100 where somebody might have to decide a selection of 100 exposures.</p> <p>Adding comment originally provided by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> to <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-82" title="AG FrameId Allocation" class="issue-link" data-issue-key="DAMD-82"><del>DAMD-82</del></a> 2020-08-07:</p> <p>Those will get pretty unwieldy, but that may be fine if they are only used for archiving and SMOKA/STARS are happy. 6 HDUs per AG exposure, so at 10s/exposure 540 HDUs for 15 minutes? </p> <p>If we agree that these are only for archiving I think this is fine.</p> <p>We need to get the downstream archive/Subaru FITS people to comment here, but I think MEFs should not cause any great concern so long as the headers are properly addressed, which seems to be the usual bugaboo.  From a file-handling standpoint on the Hilo side we would prefer larger files vs. bunches of smaller files.</p> <p>Presumably these are used for engineering and possibly science/pipeline processing?  And transferred at the end of an exposure?</p> <p>Do you want to have a single FITS file containing all 6 AG detectors as in different HDUs mixing series of AG exposure? Is is worth considering to have 6 separate FITS files for each detector?<br/> I think archiving MEF PFSD*fits would be useful and no problem, but need to check what the expected number of files and size (data rate) would be? We want to look into what keywords should be included in the header, which would be most important for archiving.</p> <p>I need to learn the discussion in <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-29" title="Define AGCC files for display while guiding" class="issue-link" data-issue-key="DAMD-29">DAMD-29</a>, too, to understand correctly how we are going to deal with VGW images as well as this PFSD stuff and their relationship in operation.</p> <p>I don't understand why <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-29" class="external-link" rel="nofollow">DAMD-29</a> was closed.  We need answers on this.</p> <p> </p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=eric" class="user-hover" rel="eric">eric</a> Could you be specific about remaining issues to be addressed on <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-29" title="Define AGCC files for display while guiding" class="issue-link" data-issue-key="DAMD-29">DAMD-29</a>? If you are concerned about implementation details, then yes we agree we should address them but we think we should do so by filing new INSTRM ticket(s) separately. If you think the data model/structure in question itself is still premature even to try going ahead to implementations a little further, could you please write down what specifics we should address, then perhaps new DAMD ticket(s) should be filed accordingly for proposals of solutions and discussions.</p> <p>Sorry, but I think I missed the (verbal?) discussion where this ticket was decided to be closed.  And perhaps I don't understand the scope of the ticket–I thought it should at least contain some responses to my and Furusawa-san's questions above?</p> <p>For this particular ticket, are you still planning to generate PFS "D" frames?  If so, what is the format?</p> <p> </p> <p> </p> <p>And, I don't think we know the exact format and contents of VGW images, yet. As <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=eric" class="user-hover" rel="eric">eric</a> commented, whether we will attach all necessary meta information for assessing observing conditions into them (as an extension table), or keeping as simple with VGW and give complete info to PFSD? I thought <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-29" title="Define AGCC files for display while guiding" class="issue-link" data-issue-key="DAMD-29">DAMD-29</a> was a good place to address this question, but if not, we may want to file a new issue. Also, we want to know the relationship between VGW and PFSD if we generate both - what roles do we expect for each of them?</p> <p><a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-2" title="AG image output to archive" class="issue-link" data-issue-key="DAMD-2">DAMD-2</a> and <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-29" title="Define AGCC files for display while guiding" class="issue-link" data-issue-key="DAMD-29">DAMD-29</a> are indeed interrelated, and <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-29" title="Define AGCC files for display while guiding" class="issue-link" data-issue-key="DAMD-29">DAMD-29</a> has been revived for final discussion and implementation. I would like to restate the goals of the two files, and to propose more of the details for the <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-2" title="AG image output to archive" class="issue-link" data-issue-key="DAMD-2">DAMD-2</a> PFSD files. If no </p> <ul class="alternate" type="square"> <li>The <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-2" title="AG image output to archive" class="issue-link" data-issue-key="DAMD-2">DAMD-2</a> PFSD files are just for long-term archiving of all "raw" guider frames. They are <b>not</b> part of the observing activities.</li> <li>The <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-29" title="Define AGCC files for display while guiding" class="issue-link" data-issue-key="DAMD-29">DAMD-29</a> files are <b>only</b> for observing displays. As a part of that, they can easily be manually selected by the observer as VGW files for further investigation ("guiding was bad here". "that was a horrible guide star", etc). See <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-29" title="Define AGCC files for display while guiding" class="issue-link" data-issue-key="DAMD-29">DAMD-29</a> for more detail.</li> </ul> <p>The agccActor acquires raw images from the six guide cameras. I suggest that it write two files for each exposure:</p> <ul class="alternate" type="square"> <li>this ticket's PFSD file, pretty much as described above. It will hold N sets of seven HDUs: the six raw agcc images (compressed pretty hard), plus a single binary table with the centroids for all six cameras. I should set N=5: a number big enough to allow 500 guide frames per PFS visit, but small enough not to give obnoxious file sizes. Once finished, the agccActor will generate a keyword which will trigger the gen2Actor to request archiving.</li> <li>the <b>starting</b> file for the <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-29" title="Define AGCC files for display while guiding" class="issue-link" data-issue-key="DAMD-29">DAMD-29</a> display file. The <b>uncompressed</b> six agcc frames, plus the same binary table. The agActor will append to this and arrange to pass it on to Gen2 for display, etc. when complete.</li> </ul> <p>As far as headers go, the PFSD PHDU will hold the same gen2 cards as all the other systems, plus a few agcc-specific W_ cards (camera temperatures, serial numbers, etc). The six image HDUs could have pixel WCS headers, or even very very crude sky transforms, but at this point nothing is known about sky coordinates, let alone guide catalog matches or offsets.</p> <p>Additional note: we will always write six image HDUs per "exposure". The images for missing cameras will be 0-filled (<tt>BLANK</tt>-filled) images of the same size as the image from the first working camera.</p> <p>We will start with a pixel-pixel WCS.</p> Blocks DAMD-6 INSTRM-1001 Duplicate DAMD-82 Relates SIM2D-66 DAMD-29 INSTRM-1458 INSTRM-1434 PIPE2D-969 Development Rank 0|02qpjd:00r3000k0013 Sprint PreEngRun4 PreEngRun05 F Story Points 4.0