Changing the pfsDesignId would break the connection between the exposure (W_PFDSGN header) and the pfsDesign file.

We've agreed that it's not clear exactly how we should handle all this. We think it might be important to be able to write pfsConfig subsets, e.g., for distributing censored raw data when users share fibers within the same exposure. We need input from NAOJ (e.g., Masayuki Tanaka) on how they envision data distribution of shared exposures will work.

One possibility is that when exporting data for shared exposures, we write a new "raw" exposure (manipulating pixel values to remove proprietary spectra, either by masking or subtracting the extracted spectra) with a new pfsDesignId (W_PFDSGN header) that maps to the modified pfsConfig. One thing to consider is flagging censored fibers (e.g., targetType=CENSORED and set target information to meaningless values) rather than removing them completely; I'm not sure what that gains, but it seems more honest and helpful about what's actually in the image.

No matter the exact method we use, it seems to me that data reduction based on censored raw data is always going to be inferior to the original raw data because of optical crosstalk. I hope that the pipeline-extracted spectra will be the primary output data product rather than the raw images.

NAOJ has not decided yet how we distribute raw data.  I think we do not want to modify the raw images.  My current thinking is that we replace the information of censored targets with NULL in pfsDesign and generate multiple pfsDesign files for all PIs in an exposure.  I know this will require modifications to the datamodel.  I have not discussed this yet with the STARS people and I think that is the first thing I should do.  And, what did you mean by optical crosstalk?  Overlapping spatial traces between neigboring fibers?  If so, can we extract all spectra and just not write censored fibers in the output?

Yes, optical crosstalk is the overlap between neighbouring fibers in the spatial dimension. That means that the spectrum in fiber i is required to extract the spectrum from fiber i+1, and vice versa. That means that if users are going to extract spectra themselves, they need access to all spectra on the image in order to extract optimal quality spectra.

Created DAMD-122 to track the issue of raw data distribution policy, and made it a blocker.