https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-418 DRP 2-D Pipeline <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a> reports in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-339" title="Model arc line amplitudes" class="issue-link" data-issue-key="PIPE2D-339"><del>PIPE2D-339</del></a> that for visit=14000 only 5 of 10 fibers are identified. See comment <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-339?focusedCommentId=15406&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15406" class="external-link" rel="nofollow">https://pfspipe.ipmu.jp/jira/browse/PIPE2D-339?focusedCommentId=15406&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15406</a> .</p> <p>Yasuda-san also mentioned in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415" title="Sometimes pfsArm has less elements than number of fibers in the image" class="issue-link" data-issue-key="PIPE2D-415"><del>PIPE2D-415</del></a> that this problem occurs in <tt>constructFiberTrace.py</tt>. See <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415?focusedCommentId=15414&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15414" class="external-link" rel="nofollow">https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415?focusedCommentId=15414&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15414</a></p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a>: Can you add the command-line steps you made to this ticket so that the problem can be reproduced please?</p> <p>This ticket blocks <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-339" title="Model arc line amplitudes" class="issue-link" data-issue-key="PIPE2D-339"><del>PIPE2D-339</del></a>.</p> PIPE2D-418

ConstructFiberTrace.py only identifies 5 of 10 fibers

Story Normal Won't Fix Won't Fix price hassan Wed, 8 May 2019 22:43:40 +0000 Wed, 22 May 2019 00:22:46 +0000 Thu, 16 May 2019 20:35:14 +0000 0 4 <p>I am almost positive that the problem that <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a> reports is the problem described in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-414" title="reduceExposure.doSubtractContinuum=True does not remove continuum from all fibers" class="issue-link" data-issue-key="PIPE2D-414"><del>PIPE2D-414</del></a>.</p> <p>Ok. I'll keep these tickets separate until they are investigated to be sure. If they turn out to be the same <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-418" title="ConstructFiberTrace.py only identifies 5 of 10 fibers" class="issue-link" data-issue-key="PIPE2D-418"><del>PIPE2D-418</del></a> can be closed as a duplicate.</p> <p>Yes, yes, of course. That is only my assumption that needs to be investigated. </p> <p>I'm sorry but I'm a bit confused what I have done. I will check my procedure and report them soon.</p> <p>While trying to reproduce the problem I have noticed something is wrong also with constructFiberFlat.py.<br/> I have used branch tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-411" title="reduceArc.py does not work with --config reduceExposure.doSubtractContinuum=True reduceExposure.doWriteCalexp=True" class="issue-link" data-issue-key="PIPE2D-411"><del>PIPE2D-411</del></a> of drp_stella and obs_pfs.<br/> The following is the script I have used.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> export REPO_DIR=/gpfs02/work/yasuda/PFS/repo_PIPE2D411 mkdir $REPO_DIR mkdir $REPO_DIR/CALIB echo <span class="code-quote">"lsst.obs.pfs.PfsMapper"</span> &gt; $REPO_DIR/_mapper ingestPfsImages.py $REPO_DIR /gpfs01/pfs/rawdata/2019-03-22/PFLA*.fits -c parse.pfsDesignId=0x0000010001001010 ingestPfsImages.py $REPO_DIR /gpfs01/pfs/rawdata/2019-04-04/PFLA*.fits -c parse.pfsDesignId=0x0000010001001010 ingestPfsImages.py $REPO_DIR /gpfs01/pfs/rawdata/2019-04-05/PFLA*.fits -c parse.pfsDesignId=0x0000010001001010 ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $OBS_PFS_DIR/pfs/camera/detectorMap-sim-r1.fits --mode=copy --validity 1000 constructBias.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/bias --id visit=15390..15396:2^15397^15399^15402^15403^15406..15410:2^15411^15414^15415^15417 arm=r --batch-type=smp --cores=1 ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/bias/BIAS/*.fits --validity 1000 constructDark.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/dark --id visit=15419^15421^15424..15428:2^15429..15435:2^15438^15440^15441^15443^15446^15447 arm=r --batch-type=smp --cores=1 ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/dark/DARK/*.fits --validity 1000 constructFiberFlat.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/flat --id visit=13882..14010:3 arm=r --batch-type=smp --cores=1 ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/dark/FLAT/*.fits --validity 1000 </pre> </div></div> <p>The messages of constructFiberFlat.py says</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> fiberFlat INFO: Scatter combination fiberFlat INFO: Combining 43 inputs {<span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'ccd'</span>: 1, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'calibDate'</span>: <span class="code-quote">'2019-03-22'</span>, <span class="code-quote">'visit0'</span>: 13882} on fe:10519 fiberFlat INFO: 6 FiberTraces found <span class="code-keyword">for</span> DataId(initialdata={<span class="code-quote">'visit'</span>: 13882, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2019-03-22'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'L'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'expId'</span>: 13882, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'field'</span>: <span class="code-quote">'FLAT'</span>, <span class="code-quote">'ccd'</span>: 1, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'expTime'</span>: 14.999, <span class="code-quote">'dataType'</span>: <span class="code-quote">'flat'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2019-03-22'</span>, <span class="code-quote">'pfiDesignId'</span>: 1099528409104, <span class="code-quote">'slitOffset'</span>: 0.0}, tag=set()) fiberFlat INFO: 6 FiberTraces found <span class="code-keyword">for</span> DataId(initialdata={<span class="code-quote">'visit'</span>: 13885, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2019-03-22'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'L'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'expId'</span>: 13885, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'field'</span>: <span class="code-quote">'FLAT'</span>, <span class="code-quote">'ccd'</span>: 1, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'expTime'</span>: 15.004, <span class="code-quote">'dataType'</span>: <span class="code-quote">'flat'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2019-03-22'</span>, <span class="code-quote">'pfiDesignId'</span>: 1099528409104, <span class="code-quote">'slitOffset'</span>: 0.0}, tag=set()) ... </pre> </div></div> <p>Only 6 FiberTraces found instead of 10 which can be seen in postIsrCcd and also created fiber flat does not look good.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/11948/_thumb_11948.png" style="border: 0px solid black" /></span><br/> Left is created fiber flat and right is one of postIsrCcd image.<br/> What's wrong with me?</p> <p>Thank you for the detailed bug report, which made it very easy to reproduce and investigate. The solution I'm about to post for <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415" title="Sometimes pfsArm has less elements than number of fibers in the image" class="issue-link" data-issue-key="PIPE2D-415"><del>PIPE2D-415</del></a> fixes this.</p> <p>Closing as duplicate of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415" title="Sometimes pfsArm has less elements than number of fibers in the image" class="issue-link" data-issue-key="PIPE2D-415"><del>PIPE2D-415</del></a>.</p> <p>The default setting of ticket/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415" title="Sometimes pfsArm has less elements than number of fibers in the image" class="issue-link" data-issue-key="PIPE2D-415"><del>PIPE2D-415</del></a> does not solve my problem. I have to set <tt>trace.finding.minLength=1000</tt> to find 10 traces in <tt>constructFiberFlat.py</tt>.<br/> However, the flat created still looks bad. For some fiber, full length is not covered.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/11951/11951_image-2019-05-21-22-46-14-548.png" height="252" width="514" style="border: 0px solid black" /></span></p> <p> </p> <p>Thanks for checking, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a>, but I don't get the same result. I ran the above command to generate flats (no override of <tt>trace.finding.minLength=1000</tt>), using the <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415" title="Sometimes pfsArm has less elements than number of fibers in the image" class="issue-link" data-issue-key="PIPE2D-415"><del>PIPE2D-415</del></a></tt> branch of drp_stella, and obtained the following flat-field image:<br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/11952/11952_pfsFiberFlat-2019-03-22-013882-r1.png" style="border: 0px solid black" /></span></p> <p>In particular, I noticed that 10 traces were found on all of the inputs, e.g.:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> fiberFlat INFO: 10 FiberTraces found <span class="code-keyword">for</span> DataId(initialdata={<span class="code-quote">'visit'</span>: 13882, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2019-03-22'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'L'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'expId'</span>: 13882, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'field'</span>: <span class="code-quote">'FLAT'</span>, <span class="code-quote">'ccd'</span>: 1, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'expTime'</span>: 14.999, <span class="code-quote">'dataType'</span>: <span class="code-quote">'flat'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2019-03-22'</span>, <span class="code-quote">'pfiDesignId'</span>: 1099528409104, <span class="code-quote">'slitOffset'</span>: 0.0}, tag=set()) </pre> </div></div> <p>Previously (without the <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415" title="Sometimes pfsArm has less elements than number of fibers in the image" class="issue-link" data-issue-key="PIPE2D-415"><del>PIPE2D-415</del></a> fix), only 6 traces were found for that image:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> fiberFlat INFO: 6 FiberTraces found <span class="code-keyword">for</span> DataId(initialdata={<span class="code-quote">'visit'</span>: 13882, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2019-03-22'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'L'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'expId'</span>: 13882, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'field'</span>: <span class="code-quote">'FLAT'</span>, <span class="code-quote">'ccd'</span>: 1, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'expTime'</span>: 14.999, <span class="code-quote">'dataType'</span>: <span class="code-quote">'flat'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2019-03-22'</span>, <span class="code-quote">'pfiDesignId'</span>: 1099528409104, <span class="code-quote">'slitOffset'</span>: 0.0}, tag=set()) </pre> </div></div> <p>Could your please double-check that you have compiled the <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415" title="Sometimes pfsArm has less elements than number of fibers in the image" class="issue-link" data-issue-key="PIPE2D-415"><del>PIPE2D-415</del></a></tt> branch of drp_stella?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pprice@tiger2-sumire:/tigress/pprice/pipe2d-418 $ eups list -s afw 16.0+1 v16_0 current setup apr 1.5.2 v16_0 current setup apr_util 1.5.4 v16_0 current setup astrometry_net 0.67.123ff3e.lsst1+8 v16_0 current setup astrometry_net_data 10.0+119 v16_0 current setup astropy 2.0.1+2 v16_0 current setup astshim 16.0+1 v16_0 current setup base 16.0 v16_0 current setup boost 1.66.0+2 v16_0 current setup cfitsio 3360.lsst5 v16_0 current setup coadd_chisquared 16.0+1 v16_0 current setup coadd_utils 16.0+1 v16_0 current setup ctrl_pool 16.0+1 v16_0 current setup daf_base 16.0 v16_0 current setup daf_persistence 16.0 v16_0 current setup datamodel 5.0-13-g6930c77 setup doxygen 1.8.13.lsst2 v16_0 current setup drp_stella LOCAL:/home/pprice/pfs/drp_stella setup eigen 3.2.5.lsst3 v16_0 current setup esutil 0.6.2.5.lsst1+2 v16_0 current setup fftw 3.3.4.lsst2 v16_0 current setup flake8 3.5.0+8 v16_0 current setup geom 16.0+1 v16_0 current setup gsl 2.4 v16_0 current setup healpy 1.10.3.lsst1+6 v16_0 current setup ip_diffim 16.0+1 v16_0 current setup ip_isr 16.0+1 v16_0 current setup libyaml 0.1.7 v16_0 current setup lmfit 0.9.3+8 v16_0 current setup log 16.0 v16_0 current setup log4cxx 0.10.0.lsst7 v16_0 current setup matplotlib 2.0.2+2 v16_0 current setup meas_algorithms 16.0+1 v16_0 current setup meas_astrom 16.0+1 v16_0 current setup meas_base 16.0+1 v16_0 current setup meas_deblender 16.0+1 v16_0 current setup meas_extensions_astrometryNet 16.0+1 v16_0 current setup minuit2 5.34.14 v16_0 current setup mpi 0.0.1+3 v16_0 current setup mpi4py 2.0.0+6 v16_0 current setup mpich 3.2.1 v16_0 current setup ndarray 1.4.2.lsst2+2 v16_0 current setup numpy 1.13.1+2 v16_0 current setup obs_base 16.0+1 v16_0 current setup obs_pfs 5.0-6-gaa29ba2 setup obs_test 16.0+1 v16_0 current setup pep8_naming 0.4.1+3 v16_0 current setup pex_config 16.0 v16_0 current setup pex_exceptions 16.0 v16_0 current setup pex_policy 16.0 v16_0 current setup pfs_pipe2d 5.0-10-gff9a74e setup pipe_base 16.0+1 v16_0 current setup pipe_drivers 16.0+1 v16_0 current setup pipe_tasks 16.0+1 v16_0 current setup pybind11 2.1.1.lsst2 v16_0 current setup pycodestyle 2.3.1+3 v16_0 current setup pyfits 3.4.0+11 v16_0 current setup pyflakes 1.6.0+2 v16_0 current setup pykg_config 1.3.0+4 v16_0 current setup pytest 3.2.0.lsst4+1 v16_0 current setup pytest_flake8 0.9.1+8 v16_0 current setup pytest_forked 0.2.lsst4+1 v16_0 current setup pytest_session2file 0.1.9+9 v16_0 current setup pytest_xdist 1.20.1.lsst4+1 v16_0 current setup python 0.0.8 v16_0 current setup python_d2to1 0.2.12.lsst2+4 v16_0 current setup python_execnet 1.4.1.lsst4+1 v16_0 current setup python_future 0.16.0+3 v16_0 current setup python_mccabe 0.6.1+9 v16_0 current setup python_psutil 5.4.3+2 v16_0 current setup pyyaml 3.11.lsst2+2 v16_0 current setup scipy 0.0.1.lsst1+7 v16_0 current setup scons 3.0.0.lsst1+3 v16_0 current setup sconsUtils 16.0 v16_0 current setup skymap 16.0+1 v16_0 current setup sphgeom 16.0 v16_0 current setup starlink_ast lsst-dev-g7ecb00ef7f v16_0 current setup stsci_distutils 0.3.7.lsst1+5 v16_0 current setup utils 16.0 v16_0 current setup wcslib 5.13.lsst1+2 v16_0 current setup pprice@tiger2-sumire:/tigress/pprice/pipe2d-418 $ constructFiberFlat.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/flat --id visit=13882..14010:3 arm=r --batch-type=smp --cores=10 </pre> </div></div> <p>I'm sorry. I have forgotten to recompile. I thought the changes are only in python. It works well with tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-415" title="Sometimes pfsArm has less elements than number of fibers in the image" class="issue-link" data-issue-key="PIPE2D-415"><del>PIPE2D-415</del></a>.</p> Blocks PIPE2D-339 Duplicate PIPE2D-415 Relates PIPE2D-415 Development Rank 0|02qpt6:508yjr Sprint 2DDRP-2019 E Story Points 2.0