[PIPE2D-404] Support for raw exposures with W_PFDSGN not set Created: 11/Apr/19  Updated: 01/May/19  Resolved: 27/Apr/19

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: hassan Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Blocks
blocks PIPE2D-339 Model arc line amplitudes Done
Relates
relates to INSTRM-354 On missing, unconverted, or invalid n... Done
relates to INSTRM-648 agreeing on dcbActor instance name Done
Story Points: 4
Sprint: 2DDRP-2019 D
Reviewers: naoki.yasuda

 Description   

This issue is raised following a discussion on slack channel #drp-2d on 2019-04-09 and today.

The FITS keyword W_PFDSGN has been set in raw LAM exposures since approx 2010-02-18. The current 2D DRP pipeline currently supports data with this keyword, but not well in scenarios without this keyword. There are a number of developers needing to process LAM exposures before and after this change, so the 2D DRP pipeline needs to be updated to support both scenarios.

So the scenario where W_PFSDSGN is not set needs to be supported. It needs to be agreed under this scenario what fibers are assumed to be illuminated (all? none?). If it is assumed that no fibers are illuminated, then PfiDesign needs to be updated, as it currently will not persist an instance if itself if the fiberId is an empty list.

 Comments   
Comment by naoki.yasuda [ 12/Apr/19 ]

The following is the initial setup of the pipeline on gfarm.ipmu.jp.

$ source /opt/local/pfs/loadLSST.bash
$ setup psf_pipe2d
$ mkdir /gpfs02/work/yasuda/PFS/repo
$ echo "lsst.obs.pfs.PfsMapper" > /gpfs02/work/yasuda/PFS/repo/_mapper

When I try to ingest the frame without W_PFDSGN, ingestPfsImages.py complains non availability of pfiDesign file.

$ ingestPfsImages.py /gpfs02/work/yasuda/PFS/repo /gpfs01/pfs/rawdata/2019-02-15/PFLA01194412.fits
CameraMapper INFO: Loading Posix exposure registry from /gpfs02/work/yasuda/PFS/repo
ingestPfs INFO: /gpfs01/pfs/rawdata/2019-02-15/PFLA01194412.fits --<link>--> /gpfs02/work/yasuda/PFS/repo/2019-02-15/PFLA011944r1.fits
Traceback (most recent call last):
  File "/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_pfs/master/python/lsst/obs/pfs/ingest.py", line 275, in runFile
    self.ingestPfsConfig(os.path.dirname(infile), hduInfoList[0], args)
  File "/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_pfs/master/python/lsst/obs/pfs/ingest.py", line 242, in ingestP                            fsConfig
    (pfiDesignId,))
RuntimeError: Unable to find PfsConfig or PfiDesign for pfiDesignId=0x0000000000000000
ingestPfs WARN: Failed to ingest file /gpfs01/pfs/rawdata/2019-02-15/PFLA01194412.fits: Unable to find PfsConfig or PfiDesign for pfiDesignId=0x0000000000000000

So we will try to create pfiDesign file without specifying any fibers but makeLamDesign.py cannot create null pfiDesign file.

$ makeLamDesign.py {}
usage: makeLamDesign.py [-h] [--directory DIRECTORY]
                        {red1,red2,red3,red4,red5,red6,red7,red8,orange,blue,green,yellow}
                        [{red1,red2,red3,red4,red5,red6,red7,red8,orange,blue,green,yellow} ...]
makeLamDesign.py: error: argument colors: invalid choice: '{}' (choose from 'red1', 'red2', 'red3', 'red4', 'red5', 'red6', 'red7', 'red8', 'orange', 'blue', 'green', 'yellow')

If we use ingestImages.py instead of ingestPfsImages.py, the file will be registered normally. But I'm not sure how we can extract 1d arc spectra using pipe2d in this case.

$ ingestImages.py /gpfs02/work/yasuda/PFS/repo /gpfs01/pfs/rawdata/2019-02-15/PFLA01194412.fits
root INFO: Loading config overrride file '/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_pfs/master/config/ingest.py'
CameraMapper INFO: Loading Posix exposure registry from /gpfs02/work/yasuda/PFS/repo
ingest INFO: /gpfs01/pfs/rawdata/2019-02-15/PFLA01194412.fits --<link>--> /gpfs02/work/yasuda/PFS/repo/2019-02-15/PFLA011944r1.fits

Should we ask LAM poeple to provide the information of illuminated fibers for every dates?

Another problem is NO CURRENT VALUE in fits header. In this case, W_PFDSGN value is NO CURRENT VALUE and we cannot process this data properly. Should we also ask LAM people to fix the files with correct values or just abandon these files?

$ ingestPfsImages.py /gpfs02/work/yasuda/PFS/repo /gpfs01/pfs/rawdata/2019-03-29/PFLA01471912.fits CameraMapper INFO: Loading exposure registry from /gpfs02/work/yasuda/PFS/repo/registry.sqlite3 Traceback (most recent call last): File "/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_pfs/master/python/lsst/obs/pfs/ingest.py", line 274, in runFile hduInfoList = super().runFile(infile, registry, args) File "/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0+1/python/lsst/pipe/tasks/ingest.py", line 508, in runFile if registry is not None and self.register.check(registry, fileInfo): File "/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0+1/python/lsst/pipe/tasks/ingest.py", line 318, in check values = [self.typemap[self.config.columns[col]](info[col]) for col in self.config.unique] File "/opt/local/pfs/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/16.0+1/python/lsst/pipe/tasks/ingest.py", line 318, in <listcomp> values = [self.typemap[self.config.columns[col]](info[col]) for col in self.config.unique] ValueError: invalid literal for int() with base 10: 'NO CURRENT VALUE'
Comment by price [ 16/Apr/19 ]

naoki.yasuda, please try the fix on tickets/PIPE2D-404 of obs_pfs. You'll want to set the default value of parse.pfsDesignId. Here's an example:

pprice@tiger2-sumire:~/pfs/obs_pfs[tickets/PIPE2D-404] $ ingestPfsImages.py DATA PFLA01194412.fits PFLA01471912.fits -c parse.pfsDesignId=0x0000000100100011
CameraMapper INFO: Loading Posix exposure registry from /home/pprice/pfs/obs_pfs/DATA
ingestPfs.parse WARN: No value set for pfsDesignId; using default (0x0000000100100011)
ingestPfs INFO: PFLA01194412.fits --<link>--> /home/pprice/pfs/obs_pfs/DATA/2019-02-15/PFLA011944r1.fits
ingestPfs INFO: pfsConfig-0x0000000100100011-011944.fits --<link>--> /home/pprice/pfs/obs_pfs/DATA/pfsConfig/2019-02-15/pfsConfig-0x0000000100100011-011944.fits
ingestPfs.parse WARN: No value set for pfsDesignId; using default (0x0000000100100011)
ingestPfs INFO: PFLA01471912.fits --<link>--> /home/pprice/pfs/obs_pfs/DATA/2019-03-29/PFLA014719r1.fits
ingestPfs INFO: pfsConfig-0x0000000100100011-014719.fits --<link>--> /home/pprice/pfs/obs_pfs/DATA/pfsConfig/2019-03-29/pfsConfig-0x0000000100100011-014719.fits
Comment by naoki.yasuda [ 24/Apr/19 ]

Thanks. This works fine if we know the fiber configuration used. For old data, it is not easy to guess the fiber configuration but this is not the problem of software.

Comment by cloomis [ 24/Apr/19 ]

Slightly related: INSTRM-354 covers a discussion of what to do with missing or invalid card values. The FITS standard says to use an empty value, but that causes just as much trouble for consumers which expect typed values as semi-descriptive strings ('NO CURRENT VALUE', etc), so it looks like we will agree to use magic values. I propose using 0 for W_PFDSGN, and not to distinguish between "missing" and "unknown/invalid". Please object now if you will.

Comment by price [ 27/Apr/19 ]

naoki.yasuda, would you mind reviewing the changes, please? Then I'll get this into master.

price@MacBook:~/pfs/obs_pfs (tickets/PIPE2D-404=) $ git sub
commit 1ec884e458ec635af20e317db202df1831044cf6 (HEAD -> tickets/PIPE2D-404, origin/tickets/PIPE2D-404)
Author: Paul Price <price@astro.princeton.edu>
Date:   Mon Apr 15 15:24:22 2019 -0400

    ingest: make default pfsDesignId value configurable
    
    If the pfsDesignId is not set, we need to be able to fall back
    to a reasonable value (perhaps determined via manual inspection
    of the image), rather than a hard-coded value.

 python/lsst/obs/pfs/ingest.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
Comment by naoki.yasuda [ 27/Apr/19 ]

I have reviewed the change and it looks fine. Please merge to master.

Comment by price [ 27/Apr/19 ]

Thanks naoki.yasuda!

Merged to master.

Generated at Sat Feb 10 15:52:47 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.