You've already thought about this.

The experience I've had so far has been with the SDSS spPlate file format, and I'm leaning toward a modified version of it. The short explanation is that it's multi-extension FITS with an extension for flux, inverse variance (we could use variance instead), sky spectrum subtracted (maybe/maybe-not suitable in our case), bit-flags, and metadata.

Here it is described in its modern form for SDSS3:

http://data.sdss3.org/datamodel/files/SPECTRO_REDUX/RUN2D/PLATE4/spPlate.html

And here's the earlier variant from SDSS2:

http://classic.sdss.org/dr7/dm/flatFiles/spPlate.html

For the work I did on spectroscopic variability with Carlos Badenes, we modified the spPlate format to split the spectra into separate files, each containing the co-added spectrum plus the individual spectra which were stacked to produce it. The MEF structure was the same. The result was the so-called spFiber file. I've pinged a few people who've worked with both spFiber files and spPlate files to get some feedback, and by and large the preference is for 1 source per file. The caveat that's been mentioned is that it's often necessary to check neighboring fibers as sources of contamination, and that's easier when all data are in the same file.

In bullets, here's why I like the spFiber:

• It's based on SDSS, which is:
  • well tested, and
  • already familiar to the community.
• It's a per-source format, which is:
  • more easily run in parallel,
  • easier to handle for users who want specific objects.

One concern I have with it is that there's some metadata which is common to all fibers in the focal plane ... exposure time, observing conditions, etc. In our spFiber files, we just repeated that info in each file, which is clearly inefficient (though not particularly badly so ... it's just not that much info, after all).

In any case, I have an example of an spFiber, and a short Python script which loads it to make a trivial plot of the coadd and its sub-spectra. Where is a suitable place to check these in, or to otherwise share them?

Also, although I'm leaning towards the spFiber model, I don't consider it a done deal. If others have suggestions/ideas/concerns, please speak.

Let me channel a pair of questions which I have been asked about using SDSS reductions. The proposed per-fiber model (which I like) opens some choices up.

• Do we provide pre-coadd "spFiber"s, where pixel wavelengths have not been interpolated onto a common vector? I would expect 3 rows or 3 HDUs for the three cameras. Basically: can people get "well-calibrated" detector pixels with unambiguous bitmasks?

• For the coadd spFibers, do we lock the wavelength CRVAL1 to be int*CD1_1, where CD1_1 is a constant for all reductions? Even for coadds of one exposure? Basically: can spFibers be directly compared/grouped without interpolation, which process no one gets right in the face of complex bitmasks?

This work has been moved into the data model project (DAMD), and is tracked in git at https://github.com/Subaru-PFS/datamodel/blob/master/datamodel.txt