https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-339 DRP 2-D Pipeline <p>Create model spectra by combining LAM arc data for different elements. Then for each combined spectrum, perform a PCA or similar analysis of the line intensities. </p> <p>This analysis would be used in the future to estimate the intensities of lines from actual spectra taken at the summit. Line variability, for example that of the OH lines, can also then be estimated.</p> <p>See also email 'Re: Homework to IPMU for PFS 2D DRP development' from RHL on 2019-01-08</p> PIPE2D-339

Model arc line amplitudes

Task Normal Done Done naoki.yasuda hassan Thu, 7 Feb 2019 13:57:42 +0000 Fri, 28 Jan 2022 17:16:21 +0000 Wed, 17 Mar 2021 22:07:13 +0000 6.0 0 7 <p>Where can I get various arc data taken at LAM?</p> <p>You will in general need access to the Princeton cluster. We will try to arrange that for you. In the meantime, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naka.keigo" class="user-hover" rel="naka.keigo">Keigo Nakamura</a> does have access and should be able to provide you with some of that data.</p> <p>I used to have an access to the Princeton cluster as "naoki.yasuda" but it seems to be deactivated now. Anyway, I will contact Keigo.</p> <p>Recent arc data are taken with the same exposure times. Should I investigate the variability within these exposures? Or should I collect the data from different dates and compare them?</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a> Just in case if you are looking for this information, the comments in this ticket: <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-359" class="external-link" rel="nofollow">https://pfspipe.ipmu.jp/jira/browse/PIPE2D-359</a> has the exposure numbers of the arc observations that we did so far. Soon there will be more. Perhaps we should organize a system to track this better.... </p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a>: in the case of LAM data variability does not need to be estimated. I apologize for the confusion. What you can do is to estimate the line intensity for a) a sample of individual arc exposures and b) combine arc exposures corresponding to different lamps, and then demonstrate that line intensities can be estimated for each element. </p> <p>Once the method of estimated line intensities is demonstrated using arc data, then this method can be applied to 'real' spectra taken at the summit to measure line intensities, and the time-variability of the intensity of certain lines (such as the OH lines).</p> <p>Hassan and I just chatted.  I expect that the lines will vary as the lines warm up and the pressure changes, and probably the Ar to Hg ratio will change.</p> <p>So we can model the arc lines from LAM, looking for variability (e.g. higher PCA components of the line intensities vector).  It's possible that the Ne lines will not change, but that means that we'll measure a variability that's consistent with zero — however, I fear that we'll find evidence that the line measurements aren't quite good enough.</p> <p>When I have run reduceArc.py for visit=15539 arm=r, I will get a lot of warnings like the following.<br/> Is this normal and can I ignore them?<br/> Another thing I noticed is that resulting pfsArm-015539-r1.fits has size of 0 byte.<br/> Do I have to run reduceExposure.py for the same data to see resulting spectra?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=214) - (GaussCoeffs[XC]=211)| = 3 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=233) - (GaussCoeffs[XC]=236)| = 3 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=389) - (GaussCoeffs[XC]=392)| = 3 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=466) - (GaussCoeffs[XC]=469)| = 3 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=971) - (GaussCoeffs[XC]=972.536)| = 1.53619 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=997) - (GaussCoeffs[XC]=995.46)| = 1.5401 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=1007) - (GaussCoeffs[XC]=1004.86)| = 2.13757 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=1489) - (GaussCoeffs[XC]=1490.59)| = 1.59155 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2033) - (GaussCoeffs[XC]=2034.66)| = 1.66064 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2400) - (GaussCoeffs[XC]=2397.87)| = 2.12793 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2423) - (GaussCoeffs[XC]=2420)| = 3 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2461) - (GaussCoeffs[XC]=2458.3)| = 2.7002 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2537) - (GaussCoeffs[XC]=2540)| = 3 &gt;= 1.5 =&gt; Skipping line pfs.drp.stella.Spectrum.identify WARN: fiberId = 3 |(maxPos=2679) - (GaussCoeffs[XC]=2682)| = 3 &gt;= 1.5 =&gt; Skipping line </pre> </div></div> <p> </p> <p>Hopefully this will be helpful for <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a>, especially given he can easily access data on tigeress. This is done with pfs_pipe2d 5.0-10-gff9a74e and checkout of tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-404" title="Support for raw exposures with W_PFDSGN not set" class="issue-link" data-issue-key="PIPE2D-404"><del>PIPE2D-404</del></a>.</p> <p>I noticed the same problem with e.g. <tt>reduceArc.py /tigress/ncaplar/ReducedData/NeonApr_2019 --calib /tigress/ncaplar/ReducedData/NeonApr_2019/CALIB --rerun Apr25_2019/arc --id visit=16292..16297 arm=r -j 20 -c reduceExposure.isr.doFlat=False</tt></p> <p>The first of the pfsArm files will have 0 bytes, e.g., see <tt>/tigress/ncaplar/ReducedData/NeonFeb_2019/rerun/Apr23_2019/arc/pfsArm/2019-02-16/v0012355</tt>.</p> <p> But the others (e.g., <tt>/tigress/ncaplar/ReducedData/NeonFeb_2019/rerun/Apr23_2019/arc/pfsArm/2019-02-16/v0012356</tt>) are 2 mb.</p> <p>Same when running <tt>reduceArc.py /tigress/ncaplar/ReducedData/KrFeb_2019 --calib /tigress/ncaplar/ReducedData/KrFeb_2019/CALIB --rerun Apr25_2019/arc --id visit=13051..13056 -j 20</tt>. </p> <p>This creates empty <tt>/tigress/ncaplar/ReducedData/KrFeb_2019/rerun/Apr25_2019/arc/pfsArm/2019-02-27/v0013051</tt>, while others seem to be full.</p> <p>The following is the complete list of commands to reproduce the problem of 0 byte <tt>pfsArm</tt> file<br/> on <tt>gfarm.ipmu.jp</tt>. I have used master branch for <tt>datamodel</tt>, <tt>drp_stella</tt>, and <tt>pfs_pipe2d</tt> and <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-404" title="Support for raw exposures with W_PFDSGN not set" class="issue-link" data-issue-key="PIPE2D-404"><del>PIPE2D-404</del></a></tt> branch for <tt>obs_pfs</tt>.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> export REPO_DIR=/gpfs02/work/yasuda/PFS/repo echo <span class="code-quote">"lsst.obs.pfs.PfsMapper"</span> &gt; $REPO_DIR/_mapper ingestPfsImages.py $REPO_DIR /gpfs01/pfs/rawdata/2019-04-04/PFLA*.fits -c parse.pfsDesignId=0x0000010001001010 ingestPfsImages.py $REPO_DIR /gpfs01/pfs/rawdata/2019-04-05/PFLA*.fits -c parse.pfsDesignId=0x0000010001001010 ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $OBS_PFS_DIR/pfs/camera/detectorMap-sim-1-r.fits --mode=copy --validity 1000 constructBias.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/bias --id visit=15390..15396:2^15397^15399^15402^15403^15406..15410:2^15411^15414^15415^15417 arm=r --batch-type=smp --cores=1 ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/bias/BIAS/*.fits --validity 1000 constructDark.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/dark --id visit=15419^15421^15424..15428:2^15429..15435:2^15438^15440^15441^15443^15446^15447 arm=r --batch-type=smp --cores=1 ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/dark/DARK/*.fits --validity 1000 constructFiberFlat.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/flat --id visit=15529..15538 arm=r --batch-type=smp --cores=1 ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/flat/FLAT/*.fits --validity 1000 constructFiberTrace.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/fiberTrace --id visit=15529..15538 arm=r slitOffset=0.0 --batch-type=smp --cores=1 ingestCalibs.py $REPO_DIR --calib $REPO_DIR/CALIB $REPO_DIR/rerun/calib/fiberTrace/FIBERTRACE/*.fits --validity 1000 reduceArc.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/arc --id visit=15539 arm=r </pre> </div></div> <p>Thanks. I was able to reproduce the problem, tracked it down and fixed it on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-411" title="reduceArc.py does not work with --config reduceExposure.doSubtractContinuum=True reduceExposure.doWriteCalexp=True" class="issue-link" data-issue-key="PIPE2D-411"><del>PIPE2D-411</del></a>. Please use the <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-411" title="reduceArc.py does not work with --config reduceExposure.doSubtractContinuum=True reduceExposure.doWriteCalexp=True" class="issue-link" data-issue-key="PIPE2D-411"><del>PIPE2D-411</del></a></tt> branch of drp_stella to generate the pfsArm files.</p> <p>I confirmed the psfArm file will be created. But the results does not look like promising.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/11930/11930_image-2019-04-27-21-19-25-398.png" height="366" width="497" style="border: 0px solid black" /></span></p> <p>This figure is made by <tt>pfs.datamodel.drp.PfsArm.plot()</tt>.<br/> I have processed one Hg-Ar arc file (visit=15749) with a command</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceArc.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun calib/arc --id visit=15749 arm=r -c reduceExposure.isr.doWrite=True </pre> </div></div> <p>The output of the command says wavelength calibration is going well.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceArc.calibrateWavelengths INFO: FiberId 650, rms 0.013309 nm (0.156 pix) from 149 (0.102334 nm = 1.200 pix <span class="code-keyword">for</span> 4 reserved points) reduceArc.calibrateWavelengths INFO: Fiber 650: wavelength correction 0.000000 +/- 0.000000 nm reduceArc.calibrateWavelengths INFO: FiberId 588, rms 0.014110 nm (0.165 pix) from 142 (0.040758 nm = 0.477 pix <span class="code-keyword">for</span> 4 reserved points) reduceArc.calibrateWavelengths INFO: Fiber 588: wavelength correction 0.000000 +/- 0.000000 nm reduceArc.calibrateWavelengths INFO: FiberId 526, rms 0.014441 nm (0.168 pix) from 153 (0.099881 nm = 1.165 pix <span class="code-keyword">for</span> 4 reserved points) reduceArc.calibrateWavelengths INFO: Fiber 526: wavelength correction 0.000000 +/- 0.000000 nm reduceArc.calibrateWavelengths INFO: FiberId 465, rms 0.014325 nm (0.167 pix) from 146 (0.069775 nm = 0.815 pix <span class="code-keyword">for</span> 4 reserved points) reduceArc.calibrateWavelengths INFO: Fiber 465: wavelength correction 0.000000 +/- 0.000000 nm reduceArc.calibrateWavelengths INFO: FiberId 402, rms 0.014927 nm (0.174 pix) from 150 (0.113282 nm = 1.324 pix <span class="code-keyword">for</span> 4 reserved points) reduceArc.calibrateWavelengths INFO: Fiber 402: wavelength correction 0.000000 +/- 0.000000 nm reduceArc.calibrateWavelengths INFO: FiberId 340, rms 0.012951 nm (0.151 pix) from 158 (0.038495 nm = 0.450 pix <span class="code-keyword">for</span> 4 reserved points) reduceArc.calibrateWavelengths INFO: Fiber 340: wavelength correction 0.000000 +/- 0.000000 nm reduceArc.calibrateWavelengths INFO: FiberId 256, rms 0.013654 nm (0.160 pix) from 158 (0.043300 nm = 0.506 pix <span class="code-keyword">for</span> 4 reserved points) reduceArc.calibrateWavelengths INFO: Fiber 256: wavelength correction 0.000000 +/- 0.000000 nm reduceArc.calibrateWavelengths INFO: FiberId 193, rms 0.013723 nm (0.160 pix) from 144 (0.090711 nm = 1.059 pix <span class="code-keyword">for</span> 4 reserved points) reduceArc.calibrateWavelengths INFO: Fiber 193: wavelength correction 0.000000 +/- 0.000000 nm reduceArc.calibrateWavelengths INFO: FiberId 64, rms 0.013223 nm (0.155 pix) from 148 (0.103616 nm = 1.211 pix <span class="code-keyword">for</span> 4 reserved points) reduceArc.calibrateWavelengths INFO: Fiber 64: wavelength correction 0.000000 +/- 0.000000 nm reduceArc.calibrateWavelengths INFO: FiberId 3, rms 0.013612 nm (0.160 pix) from 140 (0.113453 nm = 1.331 pix <span class="code-keyword">for</span> 4 reserved points) </pre> </div></div> <p> </p> <p>I have updated flat using dithered flat data (visit=13882..14010). Previously I was using exposed flat data. In addition I have updated fiberTrace using one visit. Previously I was using 10 visits and this results in bad looking trace.</p> <p>I have updated detectormap with the one created by reduceArc.py and rerun reduceArc.py to create pfsArm.</p> <p>This is better than the previous one using the default detectormap ($OBS_PFS_DIR/pfs/camera/detectorMap-sim-1-r.fits) but I'm not sure this is good enough. It seems one fiber is offset by 2nm.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/11931/11931_image-2019-04-28-06-13-26-384.png" height="430" width="599" style="border: 0px solid black" /></span></p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a>: Please use the new detectorMaps in branch <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-411" title="reduceArc.py does not work with --config reduceExposure.doSubtractContinuum=True reduceExposure.doWriteCalexp=True" class="issue-link" data-issue-key="PIPE2D-411"><del>PIPE2D-411</del></a></tt> of obs_pfs as the starting point, and let me know how that goes.</p> <p>I have updated detectorMap but the overall accuracy seems not improved so much.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/11937/11937_image-2019-04-30-13-13-29-311.png" style="border: 0px solid black" /></span></p> <p>The double peaked profile may be come from mismatched fibertrace. So I have tried to make a fibertrace from visit=14000 which have reasonable overlap with visit=15749 which I'm now working on. But for visit=14000 only a half of fibers will be identified (left hand side).</p> <p>How can I access the line lists of each elements from the pipeline?</p> <p>Here's an example: <a href="https://github.com/Subaru-PFS/drp_stella/blob/master/python/pfs/drp/stella/reduceArcTask.py#L159-L163" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/blob/master/python/pfs/drp/stella/reduceArcTask.py#L159-L163</a></p> <p>I have done very preliminary PCA analysis for 18 Xe arc exposures. I'm still not sure this is a required analysis for this ticket. </p> <p><span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/12304/12304_PCA_of_Xe_arc.pdf" title="PCA_of_Xe_arc.pdf attached to PIPE2D-339">PCA_of_Xe_arc.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>I have checked the dependency of each PC components on fiberId and visit number using the same dataset (Xe, 18 visits)  as in the previous comment. The variation of PC components depend more on fiberId.<br/> <span class="image-wrap" style=""><a id="12408_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12408/12408_PCvector_Xe.png" title="PCvector_Xe.png" file-preview-type="image" file-preview-id="12408" file-preview-title="PCvector_Xe.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12406/_thumb_12406.png" style="border: 0px solid black" /></a></span> <span class="image-wrap" style=""><a id="12408_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12408/12408_PCvector_Xe.png" title="PCvector_Xe.png" file-preview-type="image" file-preview-id="12408" file-preview-title="PCvector_Xe.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12407/_thumb_12407.png" style="border: 0px solid black" /></a></span> <span class="image-wrap" style=""><a id="12408_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12408/12408_PCvector_Xe.png" title="PCvector_Xe.png" file-preview-type="image" file-preview-id="12408" file-preview-title="PCvector_Xe.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12408/_thumb_12408.png" style="border: 0px solid black" /></a></span></p> <p>I think this specific ticket has gone as far as it can, and newer tickets will be filed to address the outstanding work.</p> Blocks PIPE2D-345 PIPE2D-404 PIPE2D-418 PIPE2D-333 Relates PIPE2D-332 PIPE2D-580 PIPE2D-329 Development Rank 0|02qpq3:9g Sprint 2DDRP-2019 C 2DDRP-2019 D 2DDRP-2019 E 2DDRP-2019 F 2DDRP-2019 G 2DDRP-2019 H 2DDRP-2019 I 2DDRP-2019 J 2DDRP-2019 K 2DDRP-2020 A 2DDRP-2021 A Story Points 5.0