<!-- 
RSS generated by JIRA (8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b) at Sat Feb 10 15:56:07 JST 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>PFS-JIRA</title>
    <link>https://pfspipe.ipmu.jp/jira</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>8.3.4</version>
        <build-number>803005</build-number>
        <build-date>13-09-2019</build-date>
    </build-info>


<item>
            <title>[PIPE2D-658] Manual processing of stability test data</title>
                <link>https://pfspipe.ipmu.jp/jira/browse/PIPE2D-658</link>
                <project id="10002" key="PIPE2D">DRP 2-D Pipeline</project>
                    <description>&lt;p&gt;In the absence of a working automated processing environment, please perform a manual processing of recent stability test data taken on:&lt;/p&gt;

&lt;p&gt;2020-11-16 (visits 34479..34520)&lt;br/&gt;
2020-11-11  (visits 34153..34205) &lt;/p&gt;

&lt;p&gt;Also provide an estimate of the accuracy of the wavelength solution (which is typically an output of the &lt;tt&gt;reduceArc&lt;/tt&gt; task). &lt;/p&gt;

&lt;p&gt;Later, if additional historical data can be processed, a plot of the wavelength solution accuracy can be made, but that&apos;s a future ticket.&lt;/p&gt;

&lt;p&gt;To help, the output of a recent query of the opDB for stability test data is attached, along with the query itself.&lt;/p&gt;</description>
                <environment></environment>
        <key id="15053">PIPE2D-658</key>
            <summary>Manual processing of stability test data</summary>
                <type id="3" iconUrl="https://pfspipe.ipmu.jp/jira/secure/viewavatar?size=xsmall&amp;avatarId=10518&amp;avatarType=issuetype">Task</type>
                                            <priority id="10000" iconUrl="https://pfspipe.ipmu.jp/jira/images/icons/priorities/medium.svg">Normal</priority>
                        <status id="10002" iconUrl="https://pfspipe.ipmu.jp/jira/images/icons/statuses/generic.png" description="The issue is resolved, reviewed, and merged">Done</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10000">Done</resolution>
                                        <assignee username="price">price</assignee>
                                    <reporter username="hassan">hassan</reporter>
                        <labels>
                    </labels>
                <created>Mon, 30 Nov 2020 22:06:23 +0000</created>
                <updated>Mon, 4 Jan 2021 20:23:23 +0000</updated>
                            <resolved>Mon, 7 Dec 2020 21:42:50 +0000</resolved>
                                                                        <due></due>
                            <votes>0</votes>
                                    <watches>2</watches>
                                                                <comments>
                            <comment id="18102" author="price" created="Tue, 1 Dec 2020 21:12:05 +0000"  >&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
PIPE2D-658: Manual processing of stability test data

This is a &lt;span class=&quot;code-keyword&quot;&gt;new&lt;/span&gt; set of stability test data.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ setup pfs_pipe2d w.2020.48
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ mv /projects/HSC/PFS/Subaru/registry.sqlite3 /projects/HSC/PFS/Subaru/registry.sqlite3.20201130

ingestPfsImages.py /projects/HSC/PFS/Subaru --mode=link --config clobber=True register.ignore=True parse.pfsDesignId=0x0000010001001010 --pfsConfigDir=/projects/HSC/PFS/Subaru/drp/pfsDesign/ -- &lt;span class=&quot;code-quote&quot;&gt;&quot;/projects/HSC/PFS/Subaru/raw&lt;span class=&quot;code-comment&quot;&gt;/*/PFSA*&quot;&lt;/span&gt; &lt;span class=&quot;code-quote&quot;&gt;&quot;/projects/HSC/PFS/Subaru/raw/*/&lt;/span&gt;sps/PFSA*&quot;&lt;/span&gt;

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ mkdir /projects/HSC/PFS/Subaru/CALIB-price/

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ constructPfsBias.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --id field=BIAS dateObs=2020-11-16 --cores 30
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/BIAS/pfsBias-*.fits
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ constructPfsDark.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --id field=DARK dateObs=2020-11-16 --cores 30
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/DARK/pfsDark-*.fits

Doesn&lt;span class=&quot;code-quote&quot;&gt;&apos;t look like there&apos;&lt;/span&gt;s a recent dithered flat. I&lt;span class=&quot;code-quote&quot;&gt;&apos;ll construct one from the previous stability test dataset (it shouldn&apos;&lt;/span&gt;t have changed much, &lt;span class=&quot;code-keyword&quot;&gt;if&lt;/span&gt; any). But to use that, I need biases and darks &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; that.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ constructPfsBias.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --id visit=18218..18227 --cores 30
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/BIAS/pfsBias-*.fits -c clobber=True
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ constructPfsDark.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --id visit=18188..18202 --cores 30
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/DARK/pfsDark-*.fits -c clobber=True

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy ~/pfs/drp_pfs_data/detectorMap/detectorMap-sim-*.fits

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ constructFiberFlat.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --id visit=17839..17982 --cores 30
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/FLAT/pfsFlat-*.fits

Now bootstrap with the first quartz+Neon.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/bootstrap --flatId visit=34481 arm=b --arcId visit=34483 arm=b -c spatialOrder=1

bootstrap INFO: Median difference from detectorMap: 11.249359,-4.296951 pixels
bootstrap INFO: Fit 83/92 points, rms: x=0.707995 y=0.065147 total=0.459968 pixels
bootstrap INFO: Median difference from detectorMap: -0.940176,-6.904669 pixels
bootstrap INFO: Fit 56/62 points, rms: x=0.561787 y=0.041838 total=0.341452 pixels

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/bootstrap --flatId visit=34481 arm=r --arcId visit=34483 arm=r --clobber-config

bootstrap INFO: Median difference from detectorMap: -3.967714,-5.123845 pixels
bootstrap INFO: Fit 207/224 points, rms: x=0.698818 y=0.144028 total=0.451050 pixels
bootstrap INFO: Median difference from detectorMap: -3.629265,-3.667544 pixels
bootstrap INFO: Fit 110/151 points, rms: x=0.289558 y=0.065743 total=0.154679 pixels

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/bootstrap/DETECTORMAP/pfsDetectorMap-034483-*

constructFiberProfiles.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --cores 3 --id visit=34488 &amp;amp;
constructFiberProfiles.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --cores 3 --id visit=34495 &amp;amp;
constructFiberProfiles.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --cores 3 --id visit=34502 &amp;amp;
constructFiberProfiles.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --cores 3 --id visit=34509 &amp;amp;
constructFiberProfiles.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --cores 3 --id visit=34516 &amp;amp;

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/FIBERPROFILES/pfsFiberProfiles-*.fits


Need PIPE2D-653 fixes to drp_stella &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; successful detectorMap fit.

Throughput looks pretty rubbish in b1: no lines in any of the arcs except Neon, and even those lines are faint. The quartzes are faint too. Ignore arm=b &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; now.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ reduceArc.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/arcs --id field=COMPARISON dateObs=2020-11-16 arm=r -j 20

reduceArc.fitDetectorMap INFO: Final fit: chi2=18599.649298 xRMS=0.021147 yRMS=0.023173 (0.001989 nm, 0.742671 km/s) from 8104/10535 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=563520.260577 xRMS=0.043762 yRMS=0.054347 (0.004665 nm, 1.741781 km/s) from 1171 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.014020 pixels (0.001203 nm, 0.449324 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=16197.397955 xRMS=0.024210 yRMS=0.026618 (0.002285 nm, 0.853097 km/s) from 8104/11706 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=391190.307916 xRMS=0.043247 yRMS=0.054002 (0.004635 nm, 1.730709 km/s) from 1171 lines (10.0%)

HgAr has a line at the top-left corner: not the brightest (~2k peak), but might be helpful to see any shifts.
Neon has multiple bright lines in the bottom-left corner.


HgAr: 34485, 34492, 34499, 34506, 34513, 34520
Neon: 34483, 34490, 34497, 34504, 34511, 34518


&lt;span class=&quot;code-keyword&quot;&gt;import&lt;/span&gt; numpy as np
from lsst.daf.persistence &lt;span class=&quot;code-keyword&quot;&gt;import&lt;/span&gt; Butler
butler = Butler(&lt;span class=&quot;code-quote&quot;&gt;&quot;/projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/arcs&quot;&lt;/span&gt;)
visits = np.array([34483, 34490, 34497, 34504, 34511, 34518])
wavelength = 638.4756
fiberId = 650
lines = {vv: butler.get(&lt;span class=&quot;code-quote&quot;&gt;&quot;arcLines&quot;&lt;/span&gt;, visit=vv, arm=&lt;span class=&quot;code-quote&quot;&gt;&quot;r&quot;&lt;/span&gt;) &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; vv in visits}

&lt;span class=&quot;code-keyword&quot;&gt;import&lt;/span&gt; itertools
&lt;span class=&quot;code-keyword&quot;&gt;import&lt;/span&gt; matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2)
&lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; color, space, (fiberId, wavelength) in zip(&lt;span class=&quot;code-quote&quot;&gt;&quot;krbg&quot;&lt;/span&gt;, np.linspace(0, 3.0, len(visits)), itertools.product((2, 650), (638.4756, 885.6298))):
    select = [(lines[vv].wavelength == wavelength) &amp;amp; (lines[vv].fiberId == fiberId) &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; vv in visits]
    xx = np.array([lines[vv].x[ss] &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; vv, ss in zip(visits, select)]).flatten()
    yy = np.array([lines[vv].y[ss] &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; vv, ss in zip(visits, select)]).flatten()
    xErr = np.array([lines[vv].xErr[ss] &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; vv, ss in zip(visits, select)]).flatten()
    yErr = np.array([lines[vv].yErr[ss] &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; vv, ss in zip(visits, select)]).flatten()
    axes[0].errorbar(visits + space, xx - xx.mean(), yerr=xErr, marker=&lt;span class=&quot;code-quote&quot;&gt;&quot;o&quot;&lt;/span&gt;, color=color, ecolor=color)
    axes[0].set_xlabel(&lt;span class=&quot;code-quote&quot;&gt;&quot;Visit&quot;&lt;/span&gt;)
    axes[0].set_ylabel(&lt;span class=&quot;code-quote&quot;&gt;&quot;\\Delta x&quot;&lt;/span&gt;)
    axes[1].errorbar(visits + space, yy - yy.mean(), yerr=yErr, marker=&lt;span class=&quot;code-quote&quot;&gt;&quot;o&quot;&lt;/span&gt;, color=color, ecolor=color, label=f&lt;span class=&quot;code-quote&quot;&gt;&quot;{fiberId}, {wavelength}&quot;&lt;/span&gt;)
    axes[1].set_xlabel(&lt;span class=&quot;code-quote&quot;&gt;&quot;Visit&quot;&lt;/span&gt;)
    axes[1].set_ylabel(&lt;span class=&quot;code-quote&quot;&gt;&quot;\\Delta y&quot;&lt;/span&gt;)
    axes[1].legend()

plt.subplots_adjust(wspace=0.5)
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;</comment>
                            <comment id="18125" author="hassan" created="Mon, 7 Dec 2020 21:42:50 +0000"  >&lt;p&gt;Data processed and corresponding plots were generated.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="13215" name="opdb_query_stabilitytest.txt" size="68568" author="hassan" created="Mon, 30 Nov 2020 22:05:25 +0000"/>
                            <attachment id="13216" name="query_opdb.sh" size="826" author="hassan" created="Mon, 30 Nov 2020 22:05:25 +0000"/>
                            <attachment id="13223" name="stability-20201201.png" size="64120" author="price" created="Tue, 1 Dec 2020 21:11:02 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10500" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10010" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|02qps2:x</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10005" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="93">2DDRP-2021 A</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10002" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        </customfields>
    </item>
</channel>
</rss>