I do not see the corresponding git branch DAMD-2 — did you forget to push it? It is not possible to understand this proposal until the details are added to datamodel.txt (note that no implementation is required until this issue is accepted) (RHL)

After some discussion by email and on the software fuze con the conclusion was that we should maintain the PFSD AG images, but make them multi-extension fits files with one extension for each guide exposure. Philip Tait confirms that Subaru is able to handle MEFs (e.g. FMOS and MOIRCS).

Because we are using windowed reads we'll write NaNs into the un-read pixels and compress the files; that way the format will be the same for all exposures, although the number of extensions will vary with the exposure time.

I think current plan is not to use windowed reads on AG cameras, but just use windowed region for calculation of object statistics. We are possible to mark outside of that windowed region as NaNs for archived data, of course. Also we may use multiple AG cameras, and need FITS files in MEF with multi HDU, right?

One another item. While writing document on AG communication and data output rate now, I noticed that we need to have a plan on this FITS file for exposure sequence with different exposure time per arm, like 15min red/IR and 7.5min blue. We could have only one FITS file over entire exposure sequence. In any case, we need to have some lines in this datamodel definition.

The data model proposes:

Raw Data from the metrology camera and autoguider (AG)

"PF%1s%1s%06d%02d.fits" % (site, category, visit, cameraNum)

There are many exposures for each visit. These files will accordingly be multi-extension fits files (MEFs)
with one extension per read and one file per camera.

The AG data (PF?D files) is a bit different as we only process small windows. The images will be the
full size of the detectors (1024x1024) and unsigned 16-bit, but with all unprocessed-pixels
represented as 0 (we will set the keyword BLANK = 0 – we'd use NaN, except that these files will
contain integers). We will use FITS (probably fpack) compression on these files, so the unprocessed
pixels will only have a negligible effect upon the total image size.

The first (and possible last) AG exposures will be complete, with no masking applied. An advantage
of the proposed format is that the masked and unmasked HDUs will be identical.

I thought about making one large image, and decided that it wasn't worth while for two reasons:

1. It would require getting all the images in memory on one machine. I'm not sure if that's easy for the DA
2. There's no natural wcs for the mosaic. Each chip is well defined, but the 6 chips aren't; worse, the chips can move relative to each other.

Assigned issue to @rhl (as he was the last person to commit changes to branch)

Adding comment originally provided by naoyuki.tamura to DAMD-82 2020-08-06:

At the telecon dedicated to AG on June 23/24 2020 ( https://sumire.pbworks.com/w/page/140496828/WG%204%20(AG)%20telecon%20on%2023-24%20June%202020 ), eric suggested and we agreed on implementing multiple HDUs to pack as many exposures as we can, rather than limiting them below 100 where somebody might have to decide a selection of 100 exposures.

Adding comment originally provided by cloomis to DAMD-82 2020-08-07:

Those will get pretty unwieldy, but that may be fine if they are only used for archiving and SMOKA/STARS are happy. 6 HDUs per AG exposure, so at 10s/exposure 540 HDUs for 15 minutes?

If we agree that these are only for archiving I think this is fine.

We need to get the downstream archive/Subaru FITS people to comment here, but I think MEFs should not cause any great concern so long as the headers are properly addressed, which seems to be the usual bugaboo.  From a file-handling standpoint on the Hilo side we would prefer larger files vs. bunches of smaller files.

Presumably these are used for engineering and possibly science/pipeline processing?  And transferred at the end of an exposure?

Do you want to have a single FITS file containing all 6 AG detectors as in different HDUs mixing series of AG exposure? Is is worth considering to have 6 separate FITS files for each detector?
I think archiving MEF PFSD*fits would be useful and no problem, but need to check what the expected number of files and size (data rate) would be? We want to look into what keywords should be included in the header, which would be most important for archiving.

I need to learn the discussion in DAMD-29, too, to understand correctly how we are going to deal with VGW images as well as this PFSD stuff and their relationship in operation.

I don't understand why DAMD-29 was closed.  We need answers on this.

eric Could you be specific about remaining issues to be addressed on DAMD-29? If you are concerned about implementation details, then yes we agree we should address them but we think we should do so by filing new INSTRM ticket(s) separately. If you think the data model/structure in question itself is still premature even to try going ahead to implementations a little further, could you please write down what specifics we should address, then perhaps new DAMD ticket(s) should be filed accordingly for proposals of solutions and discussions.

Sorry, but I think I missed the (verbal?) discussion where this ticket was decided to be closed.  And perhaps I don't understand the scope of the ticket–I thought it should at least contain some responses to my and Furusawa-san's questions above?

For this particular ticket, are you still planning to generate PFS "D" frames?  If so, what is the format?

And, I don't think we know the exact format and contents of VGW images, yet. As eric commented, whether we will attach all necessary meta information for assessing observing conditions into them (as an extension table), or keeping as simple with VGW and give complete info to PFSD? I thought DAMD-29 was a good place to address this question, but if not, we may want to file a new issue. Also, we want to know the relationship between VGW and PFSD if we generate both - what roles do we expect for each of them?

DAMD-2 and DAMD-29 are indeed interrelated, and DAMD-29 has been revived for final discussion and implementation. I would like to restate the goals of the two files, and to propose more of the details for the DAMD-2 PFSD files. If no

• The DAMD-2 PFSD files are just for long-term archiving of all "raw" guider frames. They are not part of the observing activities.
• The DAMD-29 files are only for observing displays. As a part of that, they can easily be manually selected by the observer as VGW files for further investigation ("guiding was bad here". "that was a horrible guide star", etc). See DAMD-29 for more detail.

The agccActor acquires raw images from the six guide cameras. I suggest that it write two files for each exposure:

• this ticket's PFSD file, pretty much as described above. It will hold N sets of seven HDUs: the six raw agcc images (compressed pretty hard), plus a single binary table with the centroids for all six cameras. I should set N=5: a number big enough to allow 500 guide frames per PFS visit, but small enough not to give obnoxious file sizes. Once finished, the agccActor will generate a keyword which will trigger the gen2Actor to request archiving.
• the starting file for the DAMD-29 display file. The uncompressed six agcc frames, plus the same binary table. The agActor will append to this and arrange to pass it on to Gen2 for display, etc. when complete.

As far as headers go, the PFSD PHDU will hold the same gen2 cards as all the other systems, plus a few agcc-specific W_ cards (camera temperatures, serial numbers, etc). The six image HDUs could have pixel WCS headers, or even very very crude sky transforms, but at this point nothing is known about sky coordinates, let alone guide catalog matches or offsets.

Additional note: we will always write six image HDUs per "exposure". The images for missing cameras will be 0-filled (BLANK-filled) images of the same size as the image from the first working camera.

We will start with a pixel-pixel WCS.