[PIPE2D-418] ConstructFiberTrace.py only identifies 5 of 10 fibers Created: 09/May/19  Updated: 22/May/19  Resolved: 17/May/19

Status: Won't Fix
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: hassan Assignee: price
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2019-05-15-16-40-02-818.png     PNG File image-2019-05-21-22-46-14-548.png     PNG File pfsFiberFlat-2019-03-22-013882-r1.png    
Issue Links:
Blocks
blocks PIPE2D-339 Model arc line amplitudes Done
Duplicate
duplicates PIPE2D-415 Sometimes pfsArm has less elements th... Done
Relates
relates to PIPE2D-415 Sometimes pfsArm has less elements th... Done
Story Points: 2
Sprint: 2DDRP-2019 E

 Description   

naoki.yasuda reports in PIPE2D-339 that for visit=14000 only 5 of 10 fibers are identified. See comment https://pfspipe.ipmu.jp/jira/browse/PIPE2D-339?focusedCommentId=15406&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15406 .

Yasuda-san also mentioned in PIPE2D-415 that this problem occurs in constructFiberTrace.py. See https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415?focusedCommentId=15414&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15414

naoki.yasuda: Can you add the command-line steps you made to this ticket so that the problem can be reproduced please?

This ticket blocks PIPE2D-339.

 Comments   
Comment by ncaplar [ 09/May/19 ]

I am almost positive that the problem that naoki.yasuda reports is the problem described in PIPE2D-414.

Comment by hassan [ 10/May/19 ]

Ok. I'll keep these tickets separate until they are investigated to be sure. If they turn out to be the same PIPE2D-418 can be closed as a duplicate.

Comment by ncaplar [ 10/May/19 ]

Yes, yes, of course. That is only my assumption that needs to be investigated.

Comment by naoki.yasuda [ 10/May/19 ]

I'm sorry but I'm a bit confused what I have done. I will check my procedure and report them soon.

Comment by naoki.yasuda [ 15/May/19 ]

While trying to reproduce the problem I have noticed something is wrong also with constructFiberFlat.py.
I have used branch tickets/PIPE2D-411 of drp_stella and obs_pfs.
The following is the script I have used.

export REPO_DIR=/gpfs02/work/yasuda/PFS/repo_PIPE2D411

mkdir $REPO_DIR
mkdir $REPO_DIR/CALIB

echo "lsst.obs.pfs.PfsMapper" > $REPO_DIR/_mapper
ingestPfsImages.py $REPO_DIR /gpfs01/pfs/rawdata/2019-03-22/PFLA*.fits -c parse.pfsDesignId=0x0000010001001010
ingestPfsImages.py $REPO_DIR /gpfs01/pfs/rawdata/2019-04-04/PFLA*.fits -c parse.pfsDesignId=0x0000010001001010
ingestPfsImages.py $REPO_DIR /gpfs01/pfs/rawdata/2019-04-05/PFLA*.fits -c parse.pfsDesignId=0x0000010001001010

ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $OBS_PFS_DIR/pfs/camera/detectorMap-sim-r1.fits --mode=copy --validity 1000

constructBias.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/bias --id visit=15390..15396:2^15397^15399^15402^15403^15406..15410:2^15411^15414^15415^15417 arm=r --batch-type=smp --cores=1
ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/bias/BIAS/*.fits --validity 1000

constructDark.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/dark --id visit=15419^15421^15424..15428:2^15429..15435:2^15438^15440^15441^15443^15446^15447 arm=r --batch-type=smp --cores=1
ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/dark/DARK/*.fits --validity 1000

constructFiberFlat.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/flat --id visit=13882..14010:3 arm=r --batch-type=smp --cores=1
ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/dark/FLAT/*.fits --validity 1000

The messages of constructFiberFlat.py says

fiberFlat INFO: Scatter combination                                                             
fiberFlat INFO: Combining 43 inputs {'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'filter': 'r', 'calibDate': '2019-03-22', 'visit0': 13882} on fe:10519                                            
fiberFlat INFO: 6 FiberTraces found for DataId(initialdata={'visit': 13882, 'arm': 'r', 'dateObs': '2019-03-22', 'site': 'L', 'category': 'A', 'expId': 13882, 'spectrograph': 1, 'field': 'FLAT', 'ccd': 1, 'filter': 'r', 'expTime': 14.999, 'dataType': 'flat', 'taiObs': '2019-03-22', 'pfiDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set())                                         
fiberFlat INFO: 6 FiberTraces found for DataId(initialdata={'visit': 13885, 'arm': 'r', 'dateObs': '2019-03-22', 'site': 'L', 'category': 'A', 'expId': 13885, 'spectrograph': 1, 'field': 'FLAT', 'ccd': 1, 'filter': 'r', 'expTime': 15.004, 'dataType': 'flat', 'taiObs': '2019-03-22', 'pfiDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set())
...

Only 6 FiberTraces found instead of 10 which can be seen in postIsrCcd and also created fiber flat does not look good.


Left is created fiber flat and right is one of postIsrCcd image.
What's wrong with me?

Comment by price [ 17/May/19 ]

Thank you for the detailed bug report, which made it very easy to reproduce and investigate. The solution I'm about to post for PIPE2D-415 fixes this.

Comment by price [ 17/May/19 ]

Closing as duplicate of PIPE2D-415.

Comment by naoki.yasuda [ 21/May/19 ]

The default setting of ticket/PIPE2D-415 does not solve my problem. I have to set trace.finding.minLength=1000 to find 10 traces in constructFiberFlat.py.
However, the flat created still looks bad. For some fiber, full length is not covered.

 

Comment by price [ 22/May/19 ]

Thanks for checking, naoki.yasuda, but I don't get the same result. I ran the above command to generate flats (no override of trace.finding.minLength=1000), using the tickets/PIPE2D-415 branch of drp_stella, and obtained the following flat-field image:

In particular, I noticed that 10 traces were found on all of the inputs, e.g.:

fiberFlat INFO: 10 FiberTraces found for DataId(initialdata={'visit': 13882, 'arm': 'r', 'dateObs': '2019-03-22', 'site': 'L', 'category': 'A', 'expId': 13882, 'spectrograph': 1, 'field': 'FLAT', 'ccd': 1, 'filter': 'r', 'expTime': 14.999, 'dataType': 'flat', 'taiObs': '2019-03-22', 'pfiDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set())

Previously (without the PIPE2D-415 fix), only 6 traces were found for that image:

fiberFlat INFO: 6 FiberTraces found for DataId(initialdata={'visit': 13882, 'arm': 'r', 'dateObs': '2019-03-22', 'site': 'L', 'category': 'A', 'expId': 13882, 'spectrograph': 1, 'field': 'FLAT', 'ccd': 1, 'filter': 'r', 'expTime': 14.999, 'dataType': 'flat', 'taiObs': '2019-03-22', 'pfiDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set())

Could your please double-check that you have compiled the tickets/PIPE2D-415 branch of drp_stella?

pprice@tiger2-sumire:/tigress/pprice/pipe2d-418 $ eups list -s
afw                   16.0+1            v16_0 current setup
apr                   1.5.2             v16_0 current setup
apr_util              1.5.4             v16_0 current setup
astrometry_net        0.67.123ff3e.lsst1+8      v16_0 current setup
astrometry_net_data   10.0+119          v16_0 current setup
astropy               2.0.1+2           v16_0 current setup
astshim               16.0+1            v16_0 current setup
base                  16.0              v16_0 current setup
boost                 1.66.0+2          v16_0 current setup
cfitsio               3360.lsst5        v16_0 current setup
coadd_chisquared      16.0+1            v16_0 current setup
coadd_utils           16.0+1            v16_0 current setup
ctrl_pool             16.0+1            v16_0 current setup
daf_base              16.0              v16_0 current setup
daf_persistence       16.0              v16_0 current setup
datamodel             5.0-13-g6930c77   setup
doxygen               1.8.13.lsst2      v16_0 current setup
drp_stella            LOCAL:/home/pprice/pfs/drp_stella         setup
eigen                 3.2.5.lsst3       v16_0 current setup
esutil                0.6.2.5.lsst1+2   v16_0 current setup
fftw                  3.3.4.lsst2       v16_0 current setup
flake8                3.5.0+8           v16_0 current setup
geom                  16.0+1            v16_0 current setup
gsl                   2.4               v16_0 current setup
healpy                1.10.3.lsst1+6    v16_0 current setup
ip_diffim             16.0+1            v16_0 current setup
ip_isr                16.0+1            v16_0 current setup
libyaml               0.1.7             v16_0 current setup
lmfit                 0.9.3+8           v16_0 current setup
log                   16.0              v16_0 current setup
log4cxx               0.10.0.lsst7      v16_0 current setup
matplotlib            2.0.2+2           v16_0 current setup
meas_algorithms       16.0+1            v16_0 current setup
meas_astrom           16.0+1            v16_0 current setup
meas_base             16.0+1            v16_0 current setup
meas_deblender        16.0+1            v16_0 current setup
meas_extensions_astrometryNet 16.0+1            v16_0 current setup
minuit2               5.34.14           v16_0 current setup
mpi                   0.0.1+3           v16_0 current setup
mpi4py                2.0.0+6           v16_0 current setup
mpich                 3.2.1             v16_0 current setup
ndarray               1.4.2.lsst2+2     v16_0 current setup
numpy                 1.13.1+2          v16_0 current setup
obs_base              16.0+1            v16_0 current setup
obs_pfs               5.0-6-gaa29ba2    setup
obs_test              16.0+1            v16_0 current setup
pep8_naming           0.4.1+3           v16_0 current setup
pex_config            16.0              v16_0 current setup
pex_exceptions        16.0              v16_0 current setup
pex_policy            16.0              v16_0 current setup
pfs_pipe2d            5.0-10-gff9a74e   setup
pipe_base             16.0+1            v16_0 current setup
pipe_drivers          16.0+1            v16_0 current setup
pipe_tasks            16.0+1            v16_0 current setup
pybind11              2.1.1.lsst2       v16_0 current setup
pycodestyle           2.3.1+3           v16_0 current setup
pyfits                3.4.0+11          v16_0 current setup
pyflakes              1.6.0+2           v16_0 current setup
pykg_config           1.3.0+4           v16_0 current setup
pytest                3.2.0.lsst4+1     v16_0 current setup
pytest_flake8         0.9.1+8           v16_0 current setup
pytest_forked         0.2.lsst4+1       v16_0 current setup
pytest_session2file   0.1.9+9           v16_0 current setup
pytest_xdist          1.20.1.lsst4+1    v16_0 current setup
python                0.0.8             v16_0 current setup
python_d2to1          0.2.12.lsst2+4    v16_0 current setup
python_execnet        1.4.1.lsst4+1     v16_0 current setup
python_future         0.16.0+3          v16_0 current setup
python_mccabe         0.6.1+9           v16_0 current setup
python_psutil         5.4.3+2           v16_0 current setup
pyyaml                3.11.lsst2+2      v16_0 current setup
scipy                 0.0.1.lsst1+7     v16_0 current setup
scons                 3.0.0.lsst1+3     v16_0 current setup
sconsUtils            16.0              v16_0 current setup
skymap                16.0+1            v16_0 current setup
sphgeom               16.0              v16_0 current setup
starlink_ast          lsst-dev-g7ecb00ef7f      v16_0 current setup
stsci_distutils       0.3.7.lsst1+5     v16_0 current setup
utils                 16.0              v16_0 current setup
wcslib                5.13.lsst1+2      v16_0 current setup
pprice@tiger2-sumire:/tigress/pprice/pipe2d-418 $ constructFiberFlat.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/flat --id visit=13882..14010:3 arm=r --batch-type=smp --cores=10
Comment by naoki.yasuda [ 22/May/19 ]

I'm sorry. I have forgotten to recompile. I thought the changes are only in python. It works well with tickets/PIPE2D-415.

Generated at Sat Feb 10 15:52:57 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.