PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 [SIM2D-88] Compare ETC and SIM2D+DRP2D outputs https://pfspipe.ipmu.jp/jira/browse/SIM2D-88 DRP 2-D Simulator <p>Given some input <tt>pfsSimObject</tt>, we need <tt>drp_instmodel</tt> plus <tt>drp_stella</tt> outputs to match those from <tt>spt_ExposureTimeCalculator</tt>.</p> SIM2D-88 Compare ETC and SIM2D+DRP2D outputs Story Normal Done Done price cloomis Thu, 1 Nov 2018 19:25:40 +0000 Fri, 26 Apr 2019 21:02:54 +0000 Tue, 29 Jan 2019 21:42:17 +0000 5.0 0 2 <p>Running the 1D simulator:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> price@MacBook:~/pfs/spt_ExposureTimeCalculator (tickets/SIM2D-88=) $ python scripts/run_etc.py @scripts/run_etc.defaults --MAG_FILE=22.0 --MR_MODE=no --EXP_TIME=900 --OUTFILE_SNC=out/etc-22mag-900sec.dat --OUTFILE_SNL=- price@MacBook:~/pfs/spt_ExposureTimeCalculator (tickets/SIM2D-88=) $ python scripts/gen_sim_spec.py @scripts/gen_sim_spec.defaults --outDir=out --etcFile=out/etc-22mag-900sec.dat --MAG_FILE=22.0 </pre> </div></div> <p>Running the 2D simulator:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> setup pfs_pipe2d setup -jr ~/pfs/datamodel # tickets/DAMD-32 setup -jr ~/pfs/drp_instmodel # tickets/SIM2D-88 setup -jr ~/pfs/drp_instdata # master cd ~/pfs/SIMS makeSingleSim </pre> </div></div> <p>Processing the 2D simulated data:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lsst@824ef520e03f:~/pfs/SIMS $ echo lsst.obs.pfs.PfsMapper &gt; 20181213/_mapper lsst@824ef520e03f:~/pfs/SIMS $ ingestImages.py /home/lsst/pfs/SIMS/20181213 20181213-raw/PFFA*.fits sqlite&gt; select * from raw; id|site|category|field|visit|ccd|filter|arm|spectrograph|dateObs|expTime|dataType|taiObs|pfsConfigId|slitOffset 1|F|A|BIAS|0|1|r|r|1|2018-12-13|0.0|BIAS|2018-12-13|0|0.0 2|F|A|BIAS|1|1|r|r|1|2018-12-13|0.0|BIAS|2018-12-13|0|0.0 3|F|A|BIAS|2|1|r|r|1|2018-12-13|0.0|BIAS|2018-12-13|0|0.0 4|F|A|BIAS|3|1|r|r|1|2018-12-13|0.0|BIAS|2018-12-13|0|0.0 5|F|A|BIAS|4|1|r|r|1|2018-12-13|0.0|BIAS|2018-12-13|0|0.0 6|F|A|BIAS|5|1|r|r|1|2018-12-13|0.0|BIAS|2018-12-13|0|0.0 7|F|A|BIAS|6|1|r|r|1|2018-12-13|0.0|BIAS|2018-12-13|0|0.0 8|F|A|BIAS|7|1|r|r|1|2018-12-13|0.0|BIAS|2018-12-13|0|0.0 9|F|A|BIAS|8|1|r|r|1|2018-12-13|0.0|BIAS|2018-12-13|0|0.0 10|F|A|BIAS|9|1|r|r|1|2018-12-13|0.0|BIAS|2018-12-13|0|0.0 11|F|A|DARK|10|1|r|r|1|2018-12-13|900.0|DARK|2018-12-13|0|0.0 12|F|A|DARK|11|1|r|r|1|2018-12-13|900.0|DARK|2018-12-13|0|0.0 13|F|A|DARK|12|1|r|r|1|2018-12-13|900.0|DARK|2018-12-13|0|0.0 14|F|A|DARK|13|1|r|r|1|2018-12-13|900.0|DARK|2018-12-13|0|0.0 15|F|A|DARK|14|1|r|r|1|2018-12-13|900.0|DARK|2018-12-13|0|0.0 16|F|A|DARK|15|1|r|r|1|2018-12-13|900.0|DARK|2018-12-13|0|0.0 17|F|A|DARK|16|1|r|r|1|2018-12-13|900.0|DARK|2018-12-13|0|0.0 18|F|A|DARK|17|1|r|r|1|2018-12-13|900.0|DARK|2018-12-13|0|0.0 19|F|A|DARK|18|1|r|r|1|2018-12-13|900.0|DARK|2018-12-13|0|0.0 20|F|A|DARK|19|1|r|r|1|2018-12-13|900.0|DARK|2018-12-13|0|0.0 21|F|A|FLAT|20|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 22|F|A|FLAT|21|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 23|F|A|FLAT|22|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 24|F|A|FLAT|23|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 25|F|A|FLAT|24|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 26|F|A|FLAT|25|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 27|F|A|FLAT|26|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 28|F|A|FLAT|27|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 29|F|A|FLAT|28|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 30|F|A|FLAT|29|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 31|F|A|FLAT|30|1|r|r|1|2018-12-13|30.0|FLAT|2018-12-13|0|0.0 32|F|A|ARC|31|1|r|r|1|2018-12-13|1.0|ARC|2018-12-13|0|0.0 33|F|A|OBJECT|32|1|r|r|1|2018-12-13|900.0|OBJECT|2018-12-13|0|0.0 lsst@824ef520e03f:~/pfs/SIMS $ mkdir 20181213/CALIB/ lsst@824ef520e03f:~/pfs/SIMS $ constructBias.py 20181213 --calib 20181213/CALIB --rerun bias --cores 3 --job bias --id field=BIAS lsst@824ef520e03f:~/pfs/SIMS $ ingestCalibs.py 20181213 --calib 20181213/CALIB --validity 3000 20181213/rerun/bias/BIAS/pfsBias-2018-12-13-0-r1.fits lsst@824ef520e03f:~/pfs/SIMS $ constructDark.py 20181213 --calib 20181213/CALIB --rerun dark --cores 3 --job dark --id field=DARK lsst@824ef520e03f:~/pfs/SIMS $ ingestCalibs.py 20181213 --calib 20181213/CALIB --validity 3000 20181213/rerun/dark/DARK/pfsDark-2018-12-13-0-r1.fits lsst@824ef520e03f:~/pfs/SIMS $ ingestCalibs.py 20181213 --calib 20181213/CALIB --validity 3000 $OBS_PFS_DIR/pfs/camera/pfsDetectorMap-005833-r1.fits --mode=link lsst@824ef520e03f:~/pfs/SIMS $ constructFiberFlat.py 20181213 --calib 20181213/CALIB --rerun flat --cores 3 --job flat --id field=FLAT lsst@824ef520e03f:~/pfs/SIMS $ ingestCalibs.py 20181213 --calib 20181213/CALIB --validity 3000 20181213/rerun/flat/FLAT/pfsFiberFlat-2018-12-13-000020-r1.fits lsst@824ef520e03f:~/pfs/SIMS $ constructFiberTrace.py 20181213 --calib 20181213/CALIB --rerun fiberTrace --cores 1 --job fiberTrace --id visit=21 lsst@824ef520e03f:~/pfs/SIMS $ ingestCalibs.py 20181213 --calib 20181213/CALIB --validity 3000 20181213/rerun/fiberTrace/FIBERTRACE/pfsFiberTrace-2018-12-13-000021-r1.fits lsst@824ef520e03f:~/pfs/SIMS $ reduceArc.py 20181213 --calib=20181213/CALIB/ --rerun arc --id field=ARC lsst@824ef520e03f:~/pfs/SIMS $ reduceExposure.py 20181213 --calib 20181213/CALIB --rerun exp --id field=OBJECT </pre> </div></div> <p>So now I have a pfsArm file from the 1D simulator, and a pfsArm file from the 2D simulator+pipeline, so I can compare them.<br/> The flux values from the 1D simulator are in 1e-17 erg/s/cm2/A (per <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a>), so I've multiplied it by 200 so I can plot it on the same graph as the flux from the 2D simulator+pipeline.<br/> In the attached plot, red is the 1D simulator (fluxes scaled) and blue is the 2D simulator+pipeline. The agreement isn't bad (I was surprised they are as close as they are!), except at the edges, where the flux from the 1D simulator (and the noise) goes up, while the flux from the 2D simulator+pipeline goes down (the latter makes sense: there should be no flux at the ends because of the dichroics).</p> <p> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/11700/11700_signoise.png" style="border: 0px solid black" /></span></p> <p>The remaining thing to do is to review and merge the changes. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>, would you please have a look at the changes in drp_instmodel and drp_stella?</p> <p>Merged with <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-310" title="Modify pipeline flow to match design" class="issue-link" data-issue-key="PIPE2D-310"><del>PIPE2D-310</del></a>.</p> <p>Merged the changes to spt_ExposureTimeCalculator today after discovering they hadn't been merged previously.</p> Development Rank 0|s0023f:j Reviewers cloomis Sprint 2019 A Story Points 8.0