[DAMD-122] Determine raw data distribution policy for shared exposures Created: 19/Oct/21  Updated: 14/Jul/23

Status: In Progress
Project: Data Model
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: Masayuki Tanaka
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Blocks
blocks PIPE2D-910 Update pfsDesignIds when subsetting p... Open

 Description   

Optical crosstalk means that all spectra on an image are required in order to optimally extract any one spectrum. If fibers on an image are shared among multiple programs, how are the raw data to be distributed while preserving proprietary information? Note that the raw data includes both the pixels from the spectrograph cameras as well as the pfsConfig file.
Some potential options are:
1. Only distribute pipeline-extracted spectra. Only distribute the raw pixels once the proprietary periods for all fibers on the image have passed. The quality of the distributed spectra is limited by the quality of the pipeline (it might be necessary to rerun the extractions on all data when we find important bugs).
2. Attempt to censor the data. This might be done by masking or by subtracting the extracted spectra from the raw pixels, and manipulating the pfsConfig. Any extraction from the censored raw data would not have access to all the data, and therefore would not be able to reach the quality level of the pipeline operating on the original raw data, while also being dependent upon the quality of the pipeline (depending on how we do the censoring, it might be necessary to rerun the censoring on all data when we find important bugs).
3. Do nothing and trust users not to look at fibers other than their own.



 Comments   
Comment by Masayuki Tanaka [ 14/Jul/23 ]

Our current thinking is that (1) we do not sensor the raw data, (2) we instead make a custom pfsConfig for each PI, in which information of targets of other PIs is reasonably anonymous: (a) catId and objId are replaced with NULL or -1 or some other nonsense number, (b) ra and dec point the fiducial center of each cobra (i.e., ra + dec is different what we actually point to), and (c) pfiCenter and pfiNominal also point at the fiducial center. This does not make targets by other PIs completely anonymous but better than nothing. We also plan to (3) distribute the pipeline processed data for each PI through the science database, and (4) make an observatory policy that if you look at spectra other than your own, you will not be able use PFS in future runs.  I am slightly worried that "(b)" might have some effect on the sky subtraction if we interpolate the sky line intensities acorss the field of view, but they probably do not vary significantly over a scale of 1 arcmin.

Generated at Sat Feb 10 15:34:29 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.