https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INFRA-36 Software Development Infrastructure <p>Provide a document that describes how to install, test and start using Stella. It should be checked in to a git repository.</p> <p>Detailed formatting is not important for this purpose: it should be clear, but whether it's plain text, Doxygen, reStructuredText, LaTeX or something else is immaterial at this stage. Focus on content.</p> INFRA-36

Write quick-start guide for Stella

Story Major Done Done aritter swinbank Fri, 1 Jul 2016 19:16:09 +0000 Fri, 9 Sep 2016 17:20:21 +0000 Fri, 9 Sep 2016 17:20:21 +0000 0 3 <p>What type of installation should be performed for lsst? binary, from source, lsstsw,...?</p> <p>I'd think that <tt>eups distrib install</tt> would be a good starting point.</p> <p>Should I add Biases and Darks to drp_stella_data so people can test all commands in the quick-start guide? Here is what I have now:</p> <p>====================================</p> <ul> <li>Install lsst binary distribution (<a href="https://pipelines.lsst.io/install/conda.html" class="external-link" rel="nofollow">https://pipelines.lsst.io/install/conda.html</a>)</li> <li>Install git-lfs (<a href="https://git-lfs.github.com/" class="external-link" rel="nofollow">https://git-lfs.github.com/</a>)</li> <li>Install matplotlib</li> </ul> <p>conda install matplotlib</p> <ul> <li>Create a directory for the PFS-DRP repositories:</li> </ul> <p>mkdir ~/stella-git<br/> cd ~/stella-git</p> <ul> <li>Clone the relevant git repositories:</li> </ul> <p>git clone <a href="https://github.com/Subaru-PFS/drp_stella.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella.git</a><br/> git clone <a href="https://github.com/Subaru-PFS/drp_stella_data.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella_data.git</a><br/> git clone <a href="https://github.com/Subaru-PFS/obs_pfs.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/obs_pfs.git</a></p> <ul> <li>Install a newer version of eigen:</li> </ul> <p>eups distrib install eigen 3.2.5.lsst2</p> <ul> <li>Declare the git repositories:</li> </ul> <p>eups declare obs_pfs 1.0 -c -f Linux64 -r ~/stella-git/drp_stella<br/> eups declare drp_stella_data 1.0 -c -f Linux64 -r ~/stella-git/drp_stella_data<br/> eups declare drp_stella 1.0 -c -f Linux64 -r ~/stella-git/drp_stella</p> <ul> <li>Setup the pipeline:</li> </ul> <p>setup drp_stella 1.0 -v</p> <ul> <li>Build the pipeline:</li> </ul> <p>cd drp_stella<br/> scons opt=3 -j8</p> <ul> <li>Create the defects database:</li> </ul> <p>cd ../obs_pfs<br/> scons opt=3 -j8</p> <ul> <li>Now for using the pipeline. First we need to create a directory (actually 2) where we want to store pipeline outputs. Let's<br/> assume you have a directory ~/spectra and do everything related to spectra there:</li> </ul> <p>mkdir ~/spectra/PFS<br/> mkdir ~/spectra/PFS/CALIB</p> <ul> <li>Let's further assume your raw fits files are in ~/spectra/raw/. Configuration parameters can be passed to the pipeline task<br/> either on the command line or in a config file (e.g. ~/stella-git/obs_pfs/config/pfs/ingest.py, see following example). You <br/> can list all possible configuration parameters by appending a "--show config" to the parameter list.<br/> To ingest the raw images:</li> </ul> <p>cd ~/spectra<br/> ingestImages.py 'PFS' 'raw/*.fits' --output='PFS' --mode=link -C ~/stella-git/obs_pfs/config/pfs/ingest.py</p> <ul> <li>You can now inspect the created SQL database with e.g. sqlitebrowser (<a href="http://sqlitebrowser.org/" class="external-link" rel="nofollow">http://sqlitebrowser.org/</a>).<br/> Now that we have our database we can start reducing things. We should probably start with the Biases. If the biases we<br/> want to reduce were observed on 2015-12-22 on spectrograph 2, arm red (r) at site S:</li> </ul> <p>reduceBias.py 'PFS' --calib 'PFS/CALIB' --output 'PFS' --calibId calibVersion=bias arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=BIAS dateObs=2015-12-22 --clobber-config --nodes=1<br/> --procs=1</p> <ul> <li>Note the 2 config parameters --nodes and --procs at the end. These parameters are required by tasks which are<br/> parallelized. Sometimes running the code in parallel can lead to problems (in most cases caused by the 3rd-party libraries<br/> used), to setting nodes and procs to 1 is a safe choice. —-clobber-config is needed if you re-run the task with different<br/> config parameters. The config used for each run is written to PFS/config/bias.py and the task won’t run if you change<br/> parameters, unless you say —-clobber-config.</li> </ul> <ul> <li>Now that we have a master bias we need to inject that too into our database:</li> </ul> <p>genCalibRegistry.py --root='PFS/CALIB' --camera PFS --validity 180</p> <ul> <li>Now we can create a bias-subtracted Dark and ingest that into our database:</li> </ul> <p>reduceDark.py 'PFS' --calib 'PFS/CALIB' --output 'PFS' --calibId calibVersion=dark arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=DARK dateObs=2015-12-22 --nodes=1 --procs=1 -C ~/stella-git/obs_pfs/config/pfs/dark.py<br/> genCalibRegistry.py --root='PFS/CALIB' --camera PFS --validity 180</p> <ul> <li>and a bias and dark-subtracted flat (currently only used to trace the apertures of the fiber traces):</li> </ul> <p>reduceFlat.py 'PFS' --calib 'PFS/CALIB' --calibId calibVersion=flat arm=r calibDate=2015-12-22 filter=r spectrograph=2 --do-exec --id field=FLAT arm=r dateObs=2015-12-22 spectrograph=2 --loglevel 'info' --config isr.doBias='True' isr.doDark='True' isr.doLinearize=False --output 'PFS' --nodes=1 --procs=1 --clobber-config</p> <p>genCalibRegistry.py --root='PFS/CALIB' --camera PFS --validity 180</p> <ul> <li>Since we have the Bias and Dark we can also run the Instrumental-Signature Removal (ISR) task for our Arc spectrum:</li> </ul> <p>detrend.py 'PFS' --calib='PFS/CALIB' --id arm=‘r’ spectrograph=2 dateObs='2015-12-22' field='ARC' -C ~/stella-git/obs_pfs/config/pfs/detrend.py --output 'PFS'</p> <ul> <li>We now have the postISRCCD images and can extract and wavelength-calibrate our CdHgKrNeXe Arc with the visit<br/> number 4:</li> </ul> <p>reduceArc.py 'PFS' --id visit=4 --wLenFile $OBS_PFS_DIR'/pfs/RedFiberPixels.fits.gz' --lineList $OBS_PFS_DIR'/pfs/lineLists/CdHgKrNeXe_red.fits' --loglevel 'info' --calib 'PFS/CALIB/' --output 'PFS'</p> <p>Added quick-start guide to drp_stella/doc/quick-start_guide.text in tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-36" title="Write quick-start guide for Stella" class="issue-link" data-issue-key="INFRA-36"><del>INFRA-36</del></a>.<br/> Raw fits files to run the examples are in drp_stella_data/tests/data/raw (tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-36" title="Write quick-start guide for Stella" class="issue-link" data-issue-key="INFRA-36"><del>INFRA-36</del></a>)</p> <p>Looks like something went wrong here: the <a href="https://github.com/Subaru-PFS/drp_stella/commit/db7819a505fa812e6227f86cc91c78fa37afa439" class="external-link" rel="nofollow">most recent commit</a> to <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-36" title="Write quick-start guide for Stella" class="issue-link" data-issue-key="INFRA-36"><del>INFRA-36</del></a></tt> actually <b>removed</b> the existing <tt>doc/install.text</tt> without adding anything.</p> <p>Putting this back in progress until there's something to review!</p> <p>Made sure that install.text is now in drp_stella/doc/install.text (tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-36" title="Write quick-start guide for Stella" class="issue-link" data-issue-key="INFRA-36"><del>INFRA-36</del></a>)...</p> <p>I've left a number of comments on the <a href="https://github.com/Subaru-PFS/drp_stella/pull/3" class="external-link" rel="nofollow">pull request</a>. The brief summary is that the installation instructions seem basically fine, although I've made a few suggestions for places where they could be simplified or clarified. The demo reduction confused me for quite a while until I figured out that the data in drp_stella_data doesn't actually correspond to your example command lines: you keep referring to <tt>arm=r spectrograph=2</tt>, but the data seems to be from <tt>arm=n spectrograph=1</tt>. Even once I'd got that figured out, the final <tt>reduceArc.py</tt> command fails because it has a hard-coded path referring to your home directory in it.</p> <p>More generally, I'm a bit worried about the level of the demo documentation. What is your intended audience for this? It seems to go into a lot of detail in some places (e.g. suggesting that users might want to inspect the registry database with a SQLite browser) but to skim over the top of what you're actually doing and why. A paragraph or two of more introductory material might help, as would cutting out all the complexity related to how tasks are constructed and run. For example, why do you use both config override files and command line options? why use <tt>--clobber-config</tt>? None of this seems relevant to somebody who simply wants to start processing PFS data as quickly as possible.</p> <p>Please make this 7-bit clean; the line</p> <blockquote><p>export DRP_STELLA=&lt;E2&gt;&lt;80&gt;&lt;9C&gt;&lt;your_target_directory&gt;&lt;E2&gt;&lt;80&gt;&lt;9D&gt;</p></blockquote> <p>should read</p> <blockquote><p>export DRP_STELLA="&lt;your_target_directory&gt;"</p></blockquote> <p>Here is an annotated version of install.tst; my comments start with RHL</p> <blockquote> <p>The following notes should allow you to install the LSST stack and the PFS DRP.<br/> After the installation procedure example commands are given to show you how to<br/> use the pipeline. The commands have been tested on an Arch Linux machine as well<br/> as on Mac OS X.</p> <ul> <li>Install the LSST binary distribution (<a href="https://pipelines.lsst.io/install/conda.html" class="external-link" rel="nofollow">https://pipelines.lsst.io/install/conda.html</a>)<br/> Here it should not matter whether you install anaconda or miniconda. Don’t forget<br/> to setup the LSST environment as mentioned on the website:<br/> RHL Please note that there's only a need to install lsst-distrib, not lsst-sims<br/> RHL Which version of LSST code are you assuming? Please at least specify a minimum version</li> </ul> <p>source activate lsst<br/> source eups-setups.sh</p> <p>RHL Explain why we need this, and please provide slightly more detailed instructions<br/> RHL E.g.:<br/> RHL Follow the instructions at <a href="https://git-lfs.github.com" class="external-link" rel="nofollow">https://git-lfs.github.com</a> to installing git-lfs (e.g. using homebrew<br/> RHL on os/x), then type<br/> RHL git lfs install<br/> RHL there's no need to issue any "git lfs track" commands</p> <ul> <li>Install git-lfs (<a href="https://git-lfs.github.com/" class="external-link" rel="nofollow">https://git-lfs.github.com/</a>)</li> <li>Install matplotlib and gcc:</li> </ul> <p>RHL Why? lsst-distrib installs matplotlib. Why are we installing gcc &#8211; we should be using cc<br/> RHL at least on os/x<br/> conda install matplotlib gcc</p> <ul> <li>Create a directory for the PFS-DRP repositories:<br/> RHL Please explain that the use of DRP_STELLA is purely for convenience in this tutorial<br/> export DRP_STELLA=“&lt;your_target_directory&gt;”<br/> RHL It's better to say,<br/> RHL export DRP_STELLA=$HOME/stella-git<br/> RHL as expanding ~ is a bash extension. Also, why the quotes?<br/> RHL That suggested directory name "stella-git" is not all that helpful &#8211; the fact that<br/> RHL it's git is irrelevant, and it's more than stella. Maybe $HOME/PFS?<br/> (e.g. export DRP_STELLA="~/stella-git")<br/> mkdir -p $DRP_STELLA<br/> cd $DRP_STELLA</li> </ul> <ul> <li>Clone the relevant git repositories:</li> </ul> <p>git clone <a href="https://github.com/Subaru-PFS/drp_stella.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella.git</a><br/> git clone <a href="https://github.com/Subaru-PFS/drp_stella_data.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella_data.git</a><br/> git clone <a href="https://github.com/Subaru-PFS/obs_pfs.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/obs_pfs.git</a></p> <ul> <li>Set $EUPS_PKGROOT</li> </ul> <p>export EUPS_PKGROOT="https://sw.lsstcorp.org/eupspkg/"</p> <ul> <li>Install Eigen with the unsupported modules:<br/> RHL I see that lsst-distrib is still installing 3.2.5.lsst1 (I've just prodded square about this).<br/> RHL Please monitor this, and remove these instructions ASAP; in the meantime make it clear<br/> RHL that setting EUPS_PKGROOT and doing this install are temporary workarounds.<br/> eups distrib install eigen 3.2.5.lsst2</li> </ul> <ul> <li>Declare the git repositories:</li> </ul> <p>RHL Why 0.1? You didn't tell them to checkout a 0.1 branch, <br/> RHL Also, you should almost never declare a working version; install and then declare the installed<br/> RHL version<br/> RHL<br/> RHL Before you install, make sure that everythings works &#8211; the versions of the LSST stack that you<br/> RHL were using will be frozen into the installation.</p> <p>eups declare obs_pfs 0.1 -c -r $DRP_STELLA/obs_pfs<br/> eups declare drp_stella_data 0.1 -c -r $DRP_STELLA/drp_stella_data<br/> eups declare drp_stella 0.1 -c -r $DRP_STELLA/drp_stella</p> <ul> <li>Setup the pipeline. As obs_pfs and drp_stella_data are required packages they are set up automatically:<br/> RHL Why specify -v?<br/> RHL Why 0.1 when you made drp_stella current?<br/> setup drp_stella 0.1 -v</li> </ul> <p>During this step the environment variables $DRP_STELLA_DIR, $DRP_STELLA_DATA_DIR, and $OBS_PFS_DIR are <br/> automatically created as well:<br/> RHL set, rather than created.</p> <p>echo $DRP_STELLA_DIR</p> <p>should show you:<br/> RHL No, it'll show you the value of $DRP_STELLA expanded. E.g. /Users/aritter/PFS/drp_stella<br/> RHL But why do they need to know this?<br/> $DRP_STELLA/drp_stella</p> <ul> <li>Build the pipeline:<br/> RHL Why did you have them declare a local version before building it?<br/> RHL See notes about declaring git versions. Please use setup -r . (maybe -j) instead.<br/> RHL<br/> RHL In fact, I think that the order matters.<br/> RHL Don't you need to setup and build obs_pfs first?<br/> RHL then drp_stella_data, then drp_stella?<br/> RHL<br/> RHL obs_pfs generates lots of chatter (e.g.<br/> RHL type(config) = &lt;class 'lsst.afw.cameraGeom.cameraConfig.CameraConfig'&gt;<br/> RHL dir(ccd) = ['_<em>class</em><em>', '</em><em>del</em><em>', '</em><em>delattr</em><em>', '</em><em>dict</em><em>', '</em><em>doc</em><em>', '</em><em>format</em><em>', '</em><em>getattr</em><br/> RHL ...<br/> RHL as well as some silly LSST warnings that we should get fixed and that you should warn people to<br/> RHL ignore:<br/> RHL CameraMapper WARNING: Calibration root directory not found: ./CALIB<br/> RHL ...<br/> RHL<br/> RHL And don't you need a more recent sconsUtils and a --filterWarn flag to deal with warnings about typemaps?<br/> RHL Actually I see that those fixes didn't get into sconsUtils (until tonight!) so there are some unavoidable<br/> RHL warnings &#8212; you should probably mention them.</li> </ul> <p>cd $DRP_STELLA/drp_stella<br/> scons opt=3<br/> cd ../obs_pfs<br/> scons opt=3</p> <p>RHL tests/PSF.py generates warnings<br/> RHL afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE<br/> RHL : Empty WCS extension, using FITS header<br/> RHL what is causing these? Is there a workaround (or a ticket)?</p> <p>RHL tests/FiberTraces.py generates 1217 lines of chatter, and Spectra.py 615. Where is it coming from? Most of it looks<br/> RHL like debugging stuff that should be disabled (is it an incorrect logging level?)</p> <p>RHL tests/testDrp.py fails without returning an error code. It's due to the os/x 10.11 SIP stuff I assume,<br/> RHL but the lack of an error code is a bug. Also, I'm not sure that we should be using python unittest to<br/> RHL test shell scripts. How does ci_hsc do this?</p> <p>RHL Not setting up drp_pipeline_data says, "Nothing to be done for tests"; it should at least print a warning,<br/> RHL and preferably skip them using unittest's decorators &#8211; but I'm not sure LSST does this bit right</p> <ul> <li>Now for using the pipeline. Raw test data are in $DRP_STELLA_DATA_DIR/tests/data/raw/ (3 Biases, 3 Darks, 1 Flat, 1 Arc).<br/> RHL why do we need to know this?<br/> First we need to create a directory (actually 2) where we want to store pipeline outputs. <br/> Let's assume you want to store the pipeline outputs in a directory ~/spectra/PFS:<br/> RHL: Please explain why you are introducing an environment variable.<br/> RHL: If you want them to use this one again, they should either put it in a startup file,<br/> RHL or use eups to manage it (probably a better solution)</li> </ul> <p>export SPECTRA_DIR="~/spectra/PFS"<br/> mkdir -p $SPECTRA_DIR/CALIB</p> <ul> <li>We need to tell the LSST stack which mapper to use:</li> </ul> <p>echo "lsst.obs.pfs.PfsMapper" &gt; $SPECTRA_DIR/_mapper</p> <ul> <li>Configuration parameters can be passed to the pipeline task either on the command line or in a config file<br/> (e.g. $OBS_PFS_DIR/config/pfs/ingest.py). You can list all possible configuration parameters by appending a <br/> "--show config" to the parameter list.<br/> RHL If we're providing this sort of information (and I don't think it should be in an install document)<br/> RHL document --show config=glob<br/> To ingest the raw images:</li> </ul> <p>RHL Why use cd and . rather than specifying the output?<br/> cd $SPECTRA_DIR<br/> ingestImages.py . $DRP_STELLA_DATA_DIR/tests/data/raw/*.fits --output . --mode link</p> <ul> <li>Now that we have our database we can start reducing things. We start with reducing the Biases. The data we<br/> want to reduce were observed on 2015-12-22 on spectrograph 2, arm red (r) at site S.<br/> RHL What does site S mean?<br/> Note the 2 config parameters --nodes and --procs at the end. These parameters are required by tasks which are <br/> parallelized. Sometimes running the code in parallel can lead to problems (in most cases caused by the 3rd-party <br/> libraries used), so setting nodes and procs to 1 is a safe choice.<br/> RHL Under what circumstances and why does it lead to problems? Are there jira issues?<br/> RHL What are the default values of nodes and procs?</li> </ul> <p>reduceBias.py . --calib CALIB --output . --calibId calibVersion=bias arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=BIAS dateObs=2015-12-22 --nodes 1 --procs 1</p> <ul> <li>Now that we have a master bias we need to ingest that too into our database:<br/> RHL What exactly does the genCalibRegistry.py command do? What is the 180?<br/> genCalibRegistry.py --root CALIB --camera PFS --validity 180</li> </ul> <ul> <li>Now we can create a bias-subtracted Dark and ingest that into our database:<br/> RHL presumably trimmed too, and scaled? I.e. a master dark used in further processing<br/> RHL and isn't this a different database? And do you mean database or registry?</li> </ul> <p>reduceDark.py . --calib CALIB --output . --calibId calibVersion=dark arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=DARK dateObs=2015-12-22 --nodes 1 --procs 1<br/> genCalibRegistry.py --root CALIB --camera PFS --validity 180</p> <ul> <li>and a bias and dark-subtracted flat (currently only used to trace the apertures of the fiber traces):<br/> RHL Now I'm confused. What are we making here? A flat field? Why the parenthetical comment? Where<br/> RHL do dithered flats fit into this?<br/> RHL What does the --calib CALIB option mean; is it --CALIB ./CALIB? Does it default?</li> </ul> <p>reduceFlat.py . --calib CALIB --calibId calibVersion=flat arm=r calibDate=2015-12-22 filter=r spectrograph=2 --do-exec --id field=FLAT arm=r dateObs=2015-12-22 spectrograph=2 --output . --nodes 1 --procs 1<br/> genCalibRegistry.py --root=CALIB --camera PFS --validity 180</p> <ul> <li>Since we have the Bias and Dark we can also run the Instrumental-Signature Removal (ISR) task for our Arc spectrum:<br/> RHL Still writing into the same place? Why not using --rerun?<br/> RHL Do I need to specify all those flags?</li> </ul> <p>detrend.py . --calib CALIB --output . --id arm=r spectrograph=2 dateObs=2015-12-22 field=ARC</p> <ul> <li>We now have the postISRCCD images and can extract and wavelength-calibrate our CdHgKrNeXe Arc with the visit<br/> number 4:</li> </ul> <p>RHL Why do I need the --wLenFile? If it's calibration, why isn't it in the calibration registy? Or maybe<br/> RHL in obs_pfs?<br/> RHL Why isn't the line list in the calib registry?<br/> reduceArc.py . --output . --calib CALIB --id visit=4 --wLenFile $OBS_PFS_DIR/pfs/RedFiberPixels.fits.gz --lineList $OBS_PFS_DIR/pfs/lineLists/CdHgKrNeXe_red.fits</p> <p>RHL I didn't run these data commands (yet). How chatty are they at various loglevels (e.g. INFO)?</p></blockquote> <p>Here's an update to that document with new comments noted as RHL2. Andreas has addressed some of these (and some of the RHLs too), but I haven't removed the older comments.</p> <blockquote> <p>The following notes should allow you to install the LSST stack and the PFS DRP.<br/> After the installation procedure example commands are given to show you how to<br/> use the pipeline. The commands have been tested on an Arch Linux machine as well<br/> as on Mac OS X.</p> <ul> <li>Install the LSST binary distribution (<a href="https://pipelines.lsst.io/install/conda.html" class="external-link" rel="nofollow">https://pipelines.lsst.io/install/conda.html</a>)<br/> Here it should not matter whether you install anaconda or miniconda. Don’t forget<br/> to setup the LSST environment as mentioned on the website:<br/> RHL Please note that there's only a need to install lsst-distrib, not lsst-sims<br/> RHL Which version of LSST code are you assuming? Please at least specify a minimum version</li> </ul> <p>source activate lsst<br/> source eups-setups.sh</p> <p>RHL Explain why we need this, and please provide slightly more detailed instructions<br/> RHL E.g.:<br/> RHL Follow the instructions at <a href="https://git-lfs.github.com" class="external-link" rel="nofollow">https://git-lfs.github.com</a> to installing git-lfs (e.g. using homebrew<br/> RHL on os/x), then type<br/> RHL git lfs install<br/> RHL there's no need to issue any "git lfs track" commands</p> <ul> <li>Install git-lfs (<a href="https://git-lfs.github.com/" class="external-link" rel="nofollow">https://git-lfs.github.com/</a>)</li> <li>Install matplotlib and gcc:</li> </ul> <p>RHL Why? lsst-distrib installs matplotlib. Why are we installing gcc &#8211; we should be using cc<br/> RHL at least on os/x<br/> conda install matplotlib gcc</p> <ul> <li>Create a directory for the PFS-DRP repositories:<br/> RHL Please explain that the use of DRP_STELLA is purely for convenience in this tutorial<br/> export DRP_STELLA=“&lt;your_target_directory&gt;”<br/> RHL It's better to say,<br/> RHL export DRP_STELLA=$HOME/stella-git<br/> RHL as expanding ~ is a bash extension. Also, why the quotes?<br/> RHL That suggested directory name "stella-git" is not all that helpful &#8211; the fact that<br/> RHL it's git is irrelevant, and it's more than stella. Maybe $HOME/PFS?<br/> (e.g. export DRP_STELLA="~/stella-git")<br/> mkdir -p $DRP_STELLA<br/> cd $DRP_STELLA</li> </ul> <ul> <li>Clone the relevant git repositories:</li> </ul> <p>git clone <a href="https://github.com/Subaru-PFS/drp_stella.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella.git</a><br/> git clone <a href="https://github.com/Subaru-PFS/drp_stella_data.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella_data.git</a><br/> git clone <a href="https://github.com/Subaru-PFS/obs_pfs.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/obs_pfs.git</a></p> <ul> <li>Set $EUPS_PKGROOT</li> </ul> <p>export EUPS_PKGROOT="https://sw.lsstcorp.org/eupspkg/"</p> <ul> <li>Install Eigen with the unsupported modules:<br/> RHL I see that lsst-distrib is still installing 3.2.5.lsst1 (I've just prodded square about this).<br/> RHL Please monitor this, and remove these instructions ASAP; in the meantime make it clear<br/> RHL that setting EUPS_PKGROOT and doing this install are temporary workarounds.<br/> eups distrib install eigen 3.2.5.lsst2</li> </ul> <ul> <li>Declare the git repositories:</li> </ul> <p>RHL Why 0.1? You didn't tell them to checkout a 0.1 branch, <br/> RHL Also, you should almost never declare a working version; install and then declare the installed<br/> RHL version<br/> RHL<br/> RHL Before you install, make sure that everythings works &#8211; the versions of the LSST stack that you<br/> RHL were using will be frozen into the installation.</p> <p>eups declare obs_pfs 0.1 -c -r $DRP_STELLA/obs_pfs<br/> eups declare drp_stella_data 0.1 -c -r $DRP_STELLA/drp_stella_data<br/> eups declare drp_stella 0.1 -c -r $DRP_STELLA/drp_stella</p> <ul> <li>Setup the pipeline. As obs_pfs and drp_stella_data are required packages they are set up automatically:<br/> RHL Why specify -v?<br/> RHL Why 0.1 when you made drp_stella current?<br/> setup drp_stella 0.1 -v</li> </ul> <p>During this step the environment variables $DRP_STELLA_DIR, $DRP_STELLA_DATA_DIR, and $OBS_PFS_DIR are <br/> automatically created as well:<br/> RHL set, rather than created.</p> <p>echo $DRP_STELLA_DIR</p> <p>should show you:<br/> RHL No, it'll show you the value of $DRP_STELLA expanded. E.g. /Users/aritter/PFS/drp_stella<br/> RHL But why do they need to know this?<br/> $DRP_STELLA/drp_stella</p> <ul> <li>Build the pipeline:<br/> RHL Why did you have them declare a local version before building it?<br/> RHL See notes about declaring git versions. Please use setup -r . (maybe -j) instead.<br/> RHL<br/> RHL In fact, I think that the order matters.<br/> RHL Don't you need to setup and build obs_pfs first?<br/> RHL then drp_stella_data, then drp_stella?<br/> RHL<br/> RHL obs_pfs generates lots of chatter (e.g.<br/> RHL type(config) = &lt;class 'lsst.afw.cameraGeom.cameraConfig.CameraConfig'&gt;<br/> RHL dir(ccd) = ['_<em>class</em><em>', '</em><em>del</em><em>', '</em><em>delattr</em><em>', '</em><em>dict</em><em>', '</em><em>doc</em><em>', '</em><em>format</em><em>', '</em><em>getattr</em><br/> RHL ...<br/> RHL as well as some silly LSST warnings that we should get fixed and that you should warn people to<br/> RHL ignore:<br/> RHL CameraMapper WARNING: Calibration root directory not found: ./CALIB<br/> RHL ...<br/> RHL<br/> RHL And don't you need a more recent sconsUtils and a --filterWarn flag to deal with warnings about typemaps?<br/> RHL Actually I see that those fixes didn't get into sconsUtils (until tonight!) so there are some unavoidable<br/> RHL warnings &#8212; you should probably mention them.</li> </ul> <p>cd $DRP_STELLA/drp_stella<br/> scons opt=3<br/> cd ../obs_pfs<br/> scons opt=3</p> <p>RHL tests/PSF.py generates warnings<br/> RHL afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE<br/> RHL : Empty WCS extension, using FITS header<br/> RHL what is causing these? Is there a workaround (or a ticket)?</p> <p>RHL tests/FiberTraces.py generates 1217 lines of chatter, and Spectra.py 615. Where is it coming from? Most of it looks<br/> RHL like debugging stuff that should be disabled (is it an incorrect logging level?)</p> <p>RHL tests/testDrp.py fails without returning an error code. It's due to the os/x 10.11 SIP stuff I assume,<br/> RHL but the lack of an error code is a bug. Also, I'm not sure that we should be using python unittest to<br/> RHL test shell scripts. How does ci_hsc do this?</p> <p>RHL Not setting up drp_pipeline_data says, "Nothing to be done for tests"; it should at least print a warning,<br/> RHL and preferably skip them using unittest's decorators &#8211; but I'm not sure LSST does this bit right</p> <ul> <li>Now for using the pipeline. Raw test data are in $DRP_STELLA_DATA_DIR/tests/data/raw/ (3 Biases, 3 Darks, 1 Flat, 1 Arc).<br/> RHL why do we need to know this?<br/> First we need to create a directory (actually 2) where we want to store pipeline outputs. <br/> Let's assume you want to store the pipeline outputs in a directory ~/spectra/PFS:<br/> RHL: Please explain why you are introducing an environment variable.<br/> RHL: If you want them to use this one again, they should either put it in a startup file,<br/> RHL or use eups to manage it (probably a better solution)</li> </ul> <p>export SPECTRA_DIR="~/spectra/PFS"<br/> mkdir -p $SPECTRA_DIR/CALIB</p> <ul> <li>We need to tell the LSST stack which mapper to use:<br/> RHL2 Define mapper. Explain what is going on here, including repositories.<br/> RHL2 Have you demonstrated using a raw filesystem with no ingestion? It should work</li> </ul> <p>echo "lsst.obs.pfs.PfsMapper" &gt; $SPECTRA_DIR/_mapper</p> <ul> <li>Configuration parameters can be passed to the pipeline task either on the command line or in a config file<br/> (e.g. $OBS_PFS_DIR/config/pfs/ingest.py). You can list all possible configuration parameters by appending a <br/> "--show config" to the parameter list.<br/> RHL If we're providing this sort of information (and I don't think it should be in an install document)<br/> RHL document --show config=glob<br/> To ingest the raw images:</li> </ul> <p>RHL Why use cd and . rather than specifying the output?<br/> RHL2 Actually, why do you need to specify --output at all?<br/> RHL2 (Paul clarified that: it's for the temp files such as trimmed biases. Please add a note)<br/> RHL2 It'd be better to use --rerun e.g. --rerun USERNAME/tmp (and explain that they should<br/> RHL2 substitute USERNAME)<br/> RHL2<br/> RHL2 I think you want<br/> RHL2 ingestImages.py $SPECTRA_DIR $DRP_STELLA_DATA_DIR/tests/data/raw/*.fits --mode link -L warn<br/> RHL2 And please explain the meaning of "--mode link"<br/> RHL2 Probably it would be better to move some of your info logging up to debug; the default logging<br/> RHL2 is too verbose.<br/> RHL2 There are also print statements. Please remove all of them (some by converting to log messages<br/> RHL2 at appropriate levels of verbosity</p> <p>cd $SPECTRA_DIR<br/> ingestImages.py . $DRP_STELLA_DATA_DIR/tests/data/raw/*.fits --output . --mode link</p> <ul> <li>Now that we have our database we can start reducing things. We start with reducing the Biases. The data we<br/> want to reduce were observed on 2015-12-22 on spectrograph 2, arm red (r) at site S.<br/> RHL What does site S mean?<br/> Note the 2 config parameters --nodes and --procs at the end. These parameters are required by tasks which are <br/> parallelized. Sometimes running the code in parallel can lead to problems (in most cases caused by the 3rd-party <br/> libraries used), so setting nodes and procs to 1 is a safe choice.<br/> RHL Under what circumstances and why does it lead to problems? Are there jira issues?<br/> RHL What are the default values of nodes and procs?<br/> RHL2 Why are there not sensible defaults:<br/> RHL2 RuntimeError: Must specify numNodes+numProcs or numCores<br/> RHL2 This error message doesn't tell you the options names (the "num" is confusing)<br/> RHL2 What does --do-exec do?<br/> RHL2 Why are we running this multi-tasked? Can you file a ticket to get this fixed, please.</li> </ul> <p>reduceBias.py . --calib CALIB --output . --calibId calibVersion=bias arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=BIAS dateObs=2015-12-22 --nodes 1 --procs 1</p> <p>RHL2 Why do I need --calib CALIB? I don't think you need --calib CALIB, so the preferred command would be<br/> RHL2 reduceBias.py $SPECTRA_DIR --output $SPECTRA_DIR --cores 1 --id field=BIAS dateObs=2015-12-22 arm=r spectrograph=2 --calibId calibVersion=bias calibDate=2015-15-22 arm=r spectrograph=2<br/> RHL2 Why do I need to specify the output?</p> <p>RHL2 What is<br/> RHL2 --calibId calibVersion=bias<br/> RHL2 needed? Well, I know why it's because<br/> RHL2 template: "BIAS/NONE/%(calibVersion)s/pfsBias-%(calibDate)s-0-%(spectrograph)1d%(arm)s.fits"<br/> RHL2 but why did you do that?<br/> RHL2 If I omit the --calibId I get the error message<br/> RHL2 RuntimeError: Unable to determine output filename from </p> <div class="error"><span class="error">Unknown macro: {&#39;filter&#39;}</span> </div> <p>: No registry for lookup<br/> RHL2 Why is it not picking up arm and spectrograph from the inputs? The input registry has them.<br/> RHL2 There is no check for a mis-formed date (e.g. 2015-99-22)</p> <p>RHL2 Where is "bias.isr: Set 0 BAD pixels to 0.01" coming from? What does it mean?<br/> RHL2 How about ": Empty WCS extension, using FITS header"<br/> RHL2 And also "bias.isr.assembleCcd WARNING: No WCS found in input exposure" </p> <ul> <li>Now that we have a master bias we need to ingest that too into our database:<br/> RHL What exactly does the genCalibRegistry.py command do? What is the 180?<br/> RHL2 Should be<br/> RHL2 genCalibRegistry.py --root $SPECTRA_DIR/CALIB --camera PFS --validity 180<br/> genCalibRegistry.py --root CALIB --camera PFS --validity 180</li> </ul> <ul> <li>Now we can create a bias-subtracted Dark and ingest that into our database:<br/> RHL presumably trimmed too, and scaled? I.e. a master dark used in further processing<br/> RHL and isn't this a different database? And do you mean database or registry?</li> </ul> <p>RHL2 All comments for bias generation apply here.<br/> RHL2 Why are the arguments to the different reduceXXX.py commands in different orders? It's a bit confusing<br/> reduceDark.py . --calib CALIB --output . --calibId calibVersion=dark arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=DARK dateObs=2015-12-22 --nodes 1 --procs 1<br/> RHL2 Explain why this command needs to be repeated not just run at the end<br/> genCalibRegistry.py --root CALIB --camera PFS --validity 180</p> <ul> <li>and a bias and dark-subtracted flat (currently only used to trace the apertures of the fiber traces):<br/> RHL Now I'm confused. What are we making here? A flat field? Why the parenthetical comment? Where<br/> RHL do dithered flats fit into this?<br/> RHL What does the --calib CALIB option mean; is it --CALIB ./CALIB? Does it default?</li> </ul> <p>RHL2 All comments for bias generation apply here.<br/> RHL2 Why do I need filter=r as part of the calibId?<br/> reduceFlat.py . --calib CALIB --calibId calibVersion=flat arm=r calibDate=2015-12-22 filter=r spectrograph=2 --do-exec --id field=FLAT arm=r dateObs=2015-12-22 spectrograph=2 --output . --nodes 1 --procs 1<br/> genCalibRegistry.py --root=CALIB --camera PFS --validity 180</p> <ul> <li>Since we have the Bias and Dark we can also run the Instrumental-Signature Removal (ISR) task for our Arc spectrum:<br/> RHL Still writing into the same place? Why not using --rerun?<br/> RHL Do I need to specify all those flags?<br/> RHL2 What does detrend.py do?<br/> RHL2 We should note that it doesn't (yet) flat field<br/> RHL2 Why the "type(config)" lines?<br/> RHL2 Why specify --calib CALIB?<br/> RHL2 Why are we doing this in two steps (detrend.py and reduceArc.py)?<br/> RHL2 If we are only processing one arc, why specify it this way, not by visit, e.g.<br/> RHL2 detrend.py $SPECTRA_DIR --rerun USERNAME/tmp --id visit=4<br/> RHL2 If you want to specify reducing all r2 arcs taken on 2015-12-22, please explain<br/> detrend.py . --calib CALIB --output . --id arm=r spectrograph=2 dateObs=2015-12-22 field=ARC<br/> RHL2 What files are produced?</li> </ul> <ul> <li>We now have the postISRCCD images and can extract and wavelength-calibrate our CdHgKrNeXe Arc with the visit<br/> number 4:</li> </ul> <p>RHL Why do I need the --wLenFile? If it's calibration, why isn't it in the calibration registy? Or maybe<br/> RHL in obs_pfs?<br/> RHL Why isn't the line list in the calib registry?<br/> RHL2 Actually I see that I don't. What are the pros and cons?<br/> reduceArc.py . --output . --calib CALIB --id visit=4 --wLenFile $OBS_PFS_DIR/pfs/RedFiberPixels.fits.gz --lineList $OBS_PFS_DIR/pfs/lineLists/CdHgKrNeXe_red.fits</p> <p>RHL2 A generic comment: far too much information is being printed to the screen! At -L warn there should be basically nothing; the LSST code doesn't obey this dictum very well, but drp_stella is worse!</p></blockquote> <p>Lots of the above commentary is useful, and should be mined for tickets both for here and for the LSST stack. However, we should avoid scope creep here: it's not necessary to fix all the above in order for this document to be useful. Obviously, it <b>is</b> necessary to correct any errors in this text before we can proceed.</p> <p>One thing that jumped out:</p> <blockquote> <p>Why are we installing gcc – we should be using cc at least on os/x</p></blockquote> <p>As I recall, the aim here is to avoid ABI incompatibilities since (on Linux, at least) all the Anaconda packages were compiled with gcc 4.small and are incompatible with a modern versions. We should check what the situation is on OSX.</p> <p>I agree with John. Essentially all of my comments need to be addressed, but not all as part of this ticket. Please file tickets as appropriate (e.g. the calibId things are clearly not part of <a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-36" title="Write quick-start guide for Stella" class="issue-link" data-issue-key="INFRA-36"><del>INFRA-36</del></a>)</p> <p>Another example is that </p> <blockquote><p>021dd3f Replace reduceDark with constructDark</p></blockquote> <p>should not have been addressed on this ticket.</p> <p>Here is my response to RHL's comments:</p> <p>The following notes should allow you to install the LSST stack and the PFS DRP.
After the installation procedure example commands are given to show you how to
use the pipeline. The commands have been tested on an Arch Linux machine as well
as on Mac OS X.<br/> • Install the LSST binary distribution (<a href="https://pipelines.lsst.io/install/conda.html" class="external-link" rel="nofollow">https://pipelines.lsst.io/install/conda.html</a>)
Here it should not matter whether you install anaconda or miniconda. Don’t forget
to setup the LSST environment as mentioned on the website:<br/> 
RHL Please note that there's only a need to install lsst-distrib, not lsst-sims<br/> 
RHL Which version of LSST code are you assuming? Please at least specify a minimum version<br/> source activate lsst
source eups-setups.sh<br/> RHL Explain why we need this, and please provide slightly more detailed instructions
<br/> RHL E.g.:<br/> 
RHL Follow the instructions at <a href="https://git-lfs.github.com" class="external-link" rel="nofollow">https://git-lfs.github.com</a> to installing git-lfs (e.g. using homebrew<br/> 
RHL on os/x), then type<br/> 
RHL git lfs install<br/> 
RHL there's no need to issue any "git lfs track" commands</p> <p>AR Done</p> <p> • Install git-lfs (<a href="https://git-lfs.github.com/" class="external-link" rel="nofollow">https://git-lfs.github.com/</a>)<br/> • Install matplotlib and gcc:<br/> RHL Why? lsst-distrib installs matplotlib. Why are we installing gcc – we should be using cc
<br/> RHL at least on os/x</p> <p>AR Removed matplotlib from the list of packages to be installed and explained why for Linux<br/> AR machines we need to install gcc within anaconda</p> <p>
conda install matplotlib gcc<br/> • Create a directory for the PFS-DRP repositories:
<br/> RHL Please explain that the use of DRP_STELLA is purely for convenience in this tutorial
export DRP_STELLA=“&lt;your_target_directory&gt;”<br/> 
RHL It's better to say,<br/> 
RHL export DRP_STELLA=$HOME/stella-git
<br/> RHL as expanding ~ is a bash extension. Also, why the quotes?</p> <p>AR Changed to $HOME, removed the quotes</p> <p>RHL That suggested directory name "stella-git" is not all that helpful – the fact that
<br/> RHL it's git is irrelevant, and it's more than stella. Maybe $HOME/PFS?
</p> <p>AR Changed to $HOME/PFS</p> <p>(e.g. export DRP_STELLA="~/stella-git")
mkdir -p $DRP_STELLA
cd $DRP_STELLA<br/> • Clone the relevant git repositories:<br/> git clone <a href="https://github.com/Subaru-PFS/drp_stella.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella.git</a>
git clone <a href="https://github.com/Subaru-PFS/drp_stella_data.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella_data.git</a>
git clone <a href="https://github.com/Subaru-PFS/obs_pfs.git" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/obs_pfs.git</a><br/> • Set $EUPS_PKGROOT<br/> export EUPS_PKGROOT="https://sw.lsstcorp.org/eupspkg/"<br/> • Install Eigen with the unsupported modules:
<br/> RHL I see that lsst-distrib is still installing 3.2.5.lsst1 (I've just prodded square about this).
<br/> RHL Please monitor this, and remove these instructions ASAP; in the meantime make it clear
<br/> RHL that setting EUPS_PKGROOT and doing this install are temporary workarounds.</p> <p> AR Done</p> <p>
eups distrib install eigen 3.2.5.lsst2<br/> • Declare the git repositories:<br/> RHL Why 0.1? You didn't tell them to checkout a 0.1 branch, <br/> 
RHL Also, you should almost never declare a working version; install and then declare the installed
<br/> RHL version<br/> 
RHL<br/> 
RHL Before you install, make sure that everythings works – the versions of the LSST stack that you
<br/> RHL were using will be frozen into the installation.</p> <p>AR See below</p> <p>eups declare obs_pfs 0.1 -c -r $DRP_STELLA/obs_pfs
eups declare drp_stella_data 0.1 -c -r $DRP_STELLA/drp_stella_data
eups declare drp_stella 0.1 -c -r $DRP_STELLA/drp_stella<br/> • Setup the pipeline. As obs_pfs and drp_stella_data are required packages they are set up automatically:<br/> 
RHL Why specify -v?
<br/> RHL Why 0.1 when you made drp_stella current?
setup drp_stella 0.1 -v<br/> During this step the environment variables $DRP_STELLA_DIR, $DRP_STELLA_DATA_DIR, and $OBS_PFS_DIR are 
automatically created as well:
<br/> RHL set, rather than created.<br/> echo $DRP_STELLA_DIR<br/> should show you:
<br/> RHL No, it'll show you the value of $DRP_STELLA expanded. E.g. /Users/aritter/PFS/drp_stella
<br/> RHL But why do they need to know this?</p> <p>AR Removed<br/> 
$DRP_STELLA/drp_stella<br/> • Build the pipeline:
<br/> RHL Why did you have them declare a local version before building it?
<br/> RHL See notes about declaring git versions. Please use setup -r . (maybe -j) instead.
<br/> RHL</p> <p>AR Removed declarations, now using setup -r .</p> <p>RHL In fact, I think that the order matters.
<br/> RHL Don't you need to setup and build obs_pfs first?<br/> 
RHL then drp_stella_data, then drp_stella?
<br/> RHL</p> <p>AR Didn’t need to setup and build obs_pfs first as setting up drp_stella automatically setup obs_pfs.<br/> AR Now with setup -r . corrected the order.</p> <p>
RHL obs_pfs generates lots of chatter (e.g.<br/> 
RHL type(config) = &lt;class 'lsst.afw.cameraGeom.cameraConfig.CameraConfig'&gt;
<br/> RHL dir(ccd) = ['_class', 'del', 'delattr', 'dict', 'doc', 'format', 'getattr
<br/> RHL ...<br/> 
RHL as well as some silly LSST warnings that we should get fixed and that you should warn people to
<br/> RHL ignore:
<br/> RHL CameraMapper WARNING: Calibration root directory not found: ./CALIB
<br/> RHL ...<br/> 
RHL</p> <p>AR Filed tickets for that.</p> <p>
RHL And don't you need a more recent sconsUtils and a --filterWarn flag to deal with warnings about typemaps?
<br/> RHL Actually I see that those fixes didn't get into sconsUtils (until tonight!) so there are some unavoidable
<br/> RHL warnings — you should probably mention them.</p> <p>AR Included installation of a recent sconsUtils and a —filterWarn flag in the quick-start guide</p> <p>cd $DRP_STELLA/drp_stella
scons opt=3
cd ../obs_pfs
scons opt=3<br/> RHL tests/PSF.py generates warnings
<br/> RHL afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE
<br/> RHL : Empty WCS extension, using FITS header
<br/> RHL what is causing these? Is there a workaround (or a ticket)?</p> <p>AR Ticket filed (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-64" title="get rid of warnings in drp_stella tests PSF.py, FiberTraces.py, Spectra.py" class="issue-link" data-issue-key="PIPE2D-64"><del>PIPE2D-64</del></a>), fixed, and put in review.</p> <p>RHL tests/FiberTraces.py generates 1217 lines of chatter, and Spectra.py 615. Where is it coming from? Most of it looks
<br/> RHL like debugging stuff that should be disabled (is it an incorrect logging level?)</p> <p>AR Ticket filed (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-65" title="Clean up chatter in FiberTraces.py and Spectra.py" class="issue-link" data-issue-key="PIPE2D-65"><del>PIPE2D-65</del></a>)</p> <p>RHL tests/testDrp.py fails without returning an error code. It's due to the os/x 10.11 SIP stuff I assume,
<br/> RHL but the lack of an error code is a bug. Also, I'm not sure that we should be using python unittest to<br/> 
RHL test shell scripts. How does ci_hsc do this?</p> <p>AR Ticket filed (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-69" title="tests/testDrp.py fails without returning an error code on os/x 10.11" class="issue-link" data-issue-key="PIPE2D-69"><del>PIPE2D-69</del></a>)</p> <p>RHL Not setting up drp_pipeline_data says, "Nothing to be done for tests"; it should at least print a warning,<br/> 
RHL and preferably skip them using unittest's decorators – but I'm not sure LSST does this bit right</p> <p>AR I’m pretty sure this is LSST</p> <p> • Now for using the pipeline. Raw test data are in $DRP_STELLA_DATA_DIR/tests/data/raw/ (3 Biases, 3 Darks, 1 Flat, 1 Arc).
<br/> RHL why do we need to know this?</p> <p>AR Well, we don’t really, but I would want to know what kind of raw data we have…</p> <p>
First we need to create a directory (actually 2) where we want to store pipeline outputs. 
Let's assume you want to store the pipeline outputs in a directory ~/spectra/PFS:
<br/> RHL: Please explain why you are introducing an environment variable.
<br/> RHL: If you want them to use this one again, they should either put it in a startup file,
<br/> RHL or use eups to manage it (probably a better solution)</p> <p>AR Explanation added</p> <p>export SPECTRA_DIR="~/spectra/PFS"
mkdir -p $SPECTRA_DIR/CALIB<br/> • We need to tell the LSST stack which mapper to use:
<br/> RHL2 Define mapper. Explain what is going on here, including repositories.</p> <p>AR Explanation added</p> <p>
RHL2 Have you demonstrated using a raw filesystem with no ingestion? It should work</p> <p>AR Craig asked Paul about that, waiting for an answer</p> <p>echo "lsst.obs.pfs.PfsMapper" &gt; $SPECTRA_DIR/_mapper<br/> • Configuration parameters can be passed to the pipeline task either on the command line or in a config file
(e.g. $OBS_PFS_DIR/config/pfs/ingest.py). You can list all possible configuration parameters by appending a 
"--show config" to the parameter list.<br/> 
RHL If we're providing this sort of information (and I don't think it should be in an install document)
<br/> RHL document --show config=glob
</p> <p>AR Well it’s a quick-start guide and I think that this is very helpful information for understanding how to set parameters for tasks.<br/> AR What does --show config=glob do? I only see a few parameters when I do that...</p> <p> • To ingest the raw images:<br/> RHL Why use cd and . rather than specifying the output?</p> <p>AR Removed cd and specified the output</p> <p>
RHL2 Actually, why do you need to specify --output at all?
<br/> RHL2 (Paul clarified that: it's for the temp files such as trimmed biases. Please add a note)
<br/> RHL2 It'd be better to use --rerun e.g. --rerun USERNAME/tmp (and explain that they should
<br/> RHL2 substitute USERNAME)
<br/> RHL2</p> <p>AR Replaced —output with —rerun</p> <p>
RHL2 I think you want
<br/> RHL2 ingestImages.py $SPECTRA_DIR $DRP_STELLA_DATA_DIR/tests/data/raw/*.fits --mode link -L warn</p> <p>AR Changed</p> <p>
RHL2 And please explain the meaning of "--mode link"</p> <p>AR Done</p> <p>
RHL2 Probably it would be better to move some of your info logging up to debug; the default logging
<br/> RHL2 is too verbose.<br/> 
RHL2 There are also print statements. Please remove all of them (some by converting to log messages
<br/> RHL2 at appropriate levels of verbosity</p> <p>AR Ticket filed (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-70" title="In ingest.py replace print statements with debug info" class="issue-link" data-issue-key="PIPE2D-70"><del>PIPE2D-70</del></a>)</p> <p>cd $SPECTRA_DIR
ingestImages.py . $DRP_STELLA_DATA_DIR/tests/data/raw/*.fits --output . --mode link<br/> • Now that we have our database we can start reducing things. We start with reducing the Biases. The data we
want to reduce were observed on 2015-12-22 on spectrograph 2, arm red (r) at site S.<br/> 
RHL What does site S mean?</p> <p>AR Explanation added</p> <p>
Note the 2 config parameters --nodes and --procs at the end. These parameters are required by tasks which are 
parallelized. Sometimes running the code in parallel can lead to problems (in most cases caused by the 3rd-party 
libraries used), so setting nodes and procs to 1 is a safe choice.
<br/> RHL Under what circumstances and why does it lead to problems? Are there jira issues?
</p> <p>AR Asked on HipChat, waiting for Craig to help me getting to the bottom of it</p> <p>RHL What are the default values of nodes and procs?
<br/> RHL2 Why are there not sensible defaults:
<br/> RHL2 RuntimeError: Must specify numNodes+numProcs or numCores
<br/> RHL2 This error message doesn't tell you the options names (the "num" is confusing)<br/> 
RHL2 What does --do-exec do?<br/> 
RHL2 Why are we running this multi-tasked? Can you file a ticket to get this fixed, please.</p> <p>AR Asked Paul about it, waiting for answer</p> <p>reduceBias.py . --calib CALIB --output . --calibId calibVersion=bias arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=BIAS dateObs=2015-12-22 --nodes 1 --procs 1<br/> RHL2 Why do I need --calib CALIB? I don't think you need --calib CALIB, so the preferred command would be<br/> 
RHL2 reduceBias.py $SPECTRA_DIR --output $SPECTRA_DIR --cores 1 --id field=BIAS dateObs=2015-12-22 arm=r spectrograph=2 --calibId calibVersion=bias calibDate=2015-15-22 arm=r spectrograph=2</p> <p>AR Removed —calib CALIB</p> <p>
RHL2 Why do I need to specify the output?<br/> RHL2 What is
<br/> RHL2 --calibId calibVersion=bias
<br/> RHL2 needed? Well, I know why it's because
<br/> RHL2 template: "BIAS/NONE/%(calibVersion)s/pfsBias-%(calibDate)s-0-%(spectrograph)1d%(arm)s.fits"<br/> 
RHL2 but why did you do that?</p> <p>AR Ticket filed (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-66" title="Remove calibVersion from mapper" class="issue-link" data-issue-key="PIPE2D-66"><del>PIPE2D-66</del></a>)</p> <p>
RHL2 If I omit the --calibId I get the error message
<br/> RHL2 RuntimeError: Unable to determine output filename from<br/> Unknown macro: </p> {'filter'} <p>: No registry for lookup
<br/> RHL2 Why is it not picking up arm and spectrograph from the inputs? The input registry has them.</p> <p>AR Gonna investigate (ask Paul). Filed ticket (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-67" title="Why is constructBias.py not picking up arm and spectrograph from the inputs?" class="issue-link" data-issue-key="PIPE2D-67"><del>PIPE2D-67</del></a>)</p> <p>
RHL2 There is no check for a mis-formed date (e.g. 2015-99-22)</p> <p>AR Asked Paul about it, waiting for answer</p> <p>RHL2 Where is "bias.isr: Set 0 BAD pixels to 0.01" coming from? What does it mean?
</p> <p>AR Comes from obs_subaru isr.py setBadRegions(self, exposure)<br/> AR value = afwMath.makeStatistics(exposure.getMaskedImage(), statistic, sctrl).getValue()<br/> AR self.log.info("Set %d BAD pixels to %.2f" % (badPixels.sum(), value))</p> <p>RHL2 How about ": Empty WCS extension, using FITS header"<br/> 
RHL2 And also "bias.isr.assembleCcd WARNING: No WCS found in input exposure"</p> <p>AR These warnings normally come from creating an exposure from a fits file via ExposureF(fileName)<br/> AR Should be makeExposure(makeMaskedImage(ImageF(fileName))). Filed a ticket (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-64" title="get rid of warnings in drp_stella tests PSF.py, FiberTraces.py, Spectra.py" class="issue-link" data-issue-key="PIPE2D-64"><del>PIPE2D-64</del></a> in review)</p> <p> • Now that we have a master bias we need to ingest that too into our database:<br/> 
RHL What exactly does the genCalibRegistry.py command do? What is the 180?</p> <p> AR Explanation added.<br/> 
RHL2 Should be<br/> 
RHL2 genCalibRegistry.py --root $SPECTRA_DIR/CALIB --camera PFS --validity 180
genCalibRegistry.py --root CALIB --camera PFS --validity 180</p> <p>AR Done</p> <p> • Now we can create a bias-subtracted Dark and ingest that into our database:
<br/> RHL presumably trimmed too, and scaled? I.e. a master dark used in further processing</p> <p>AR Added trimmed, scaled, and master</p> <p>
RHL and isn't this a different database? And do you mean database <br/> or registry?</p> <p>AR Changed “database” to “calibration registry”.</p> <p>RHL2 All comments for bias generation apply here.
<br/> RHL2 Why are the arguments to the different reduceXXX.py commands in different orders? It's a bit confusing</p> <p>AR Changed order or arguments</p> <p>
reduceDark.py . --calib CALIB --output . --calibId calibVersion=dark arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=DARK dateObs=2015-12-22 --nodes 1 --procs 1<br/> 
RHL2 Explain why this command needs to be repeated not just run at the end</p> <p>AR Done</p> <p>
genCalibRegistry.py --root CALIB --camera PFS --validity 180<br/> • and a bias and dark-subtracted flat (currently only used to trace the apertures of the fiber traces):<br/> 
RHL Now I'm confused. What are we making here? A flat field? Why the parenthetical comment? Where
<br/> RHL do dithered flats fit into this?
</p> <p>AR Comment added</p> <p>RHL What does the --calib CALIB option mean; is it --CALIB ./CALIB? Does it default?</p> <p>AR Removed —calib CALIB option</p> <p>RHL2 All comments for bias generation apply here.
<br/> RHL2 Why do I need filter=r as part of the calibId?</p> <p>AR Removed filter=r</p> <p>
reduceFlat.py . --calib CALIB --calibId calibVersion=flat arm=r calibDate=2015-12-22 filter=r spectrograph=2 --do-exec --id field=FLAT arm=r dateObs=2015-12-22 spectrograph=2 --output . --nodes 1 --procs 1
genCalibRegistry.py --root=CALIB --camera PFS --validity 180<br/> • Since we have the Bias and Dark we can also run the Instrumental-Signature Removal (ISR) task for our Arc spectrum:
<br/> RHL Still writing into the same place? Why not using --rerun?</p> <p>AR Using —rerun now</p> <p>
RHL Do I need to specify all those flags?</p> <p>AR Changed flags to visit=5</p> <p>
RHL2 What does detrend.py do?</p> <p>AR Comment added</p> <p>
RHL2 We should note that it doesn't (yet) flat field
</p> <p>AR Done</p> <p>RHL2 Why the "type(config)" lines?</p> <p>AR Removed “type(config)” lines from camera.py</p> <p>
RHL2 Why specify --calib CALIB?</p> <p>AR Removed —calib CALIB</p> <p>
RHL2 Why are we doing this in two steps (detrend.py and reduceArc.py)?</p> <p>AR Ticket filed (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-106" title="change reduceArc(RefSpec).py to include detrend" class="issue-link" data-issue-key="PIPE2D-106"><del>INFRA-55</del></a>)</p> <p>
RHL2 If we are only processing one arc, why specify it this way, not by visit, e.g.
<br/> RHL2 detrend.py $SPECTRA_DIR --rerun USERNAME/tmp --id visit=4
<br/> RHL2 If you want to specify reducing all r2 arcs taken on 2015-12-22, please explain
detrend.py . --calib CALIB --output . --id arm=r spectrograph=2 dateObs=2015-12-22 field=ARC</p> <p>AR Replaced “arm=r spectrograph=2 dateObs=2015-12-22 field=ARC” with “visit=4”, added comment on how to reduce all Arcs</p> <p>
RHL2 What files are produced?</p> <p>AR See next line</p> <p> • We now have the postISRCCD images and can extract and wavelength-calibrate our CdHgKrNeXe Arc with the visit
number 4:<br/> RHL Why do I need the --wLenFile? If it's calibration, why isn't it in the calibration registy? Or maybe
<br/> RHL in obs_pfs?
<br/> RHL Why isn't the line list in the calib registry?
<br/> RHL2 Actually I see that I don't. What are the pros and cons?</p> <p>AR Removed wLenFile and lineList as they are defaulted. Ticket filed (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-107" title="Add line list and pixel vs. wavelength map to calibration registry" class="issue-link" data-issue-key="PIPE2D-107"><del>INFRA-56</del></a>)</p> <p>
reduceArc.py . --output . --calib CALIB --id visit=4 --wLenFile $OBS_PFS_DIR/pfs/RedFiberPixels.fits.gz --lineList $OBS_PFS_DIR/pfs/lineLists/CdHgKrNeXe_red.fits<br/> RHL2 A generic comment: far too much information is being printed to the screen! At -L warn there should be basically nothing; the LSST code doesn't obey this dictum very well, but drp_stella is worse!</p> <p>AR Filed individual tickets</p> <p>See tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-36" title="Write quick-start guide for Stella" class="issue-link" data-issue-key="INFRA-36"><del>INFRA-36</del></a> for obs_pfs, drp_stella, drp_stella_data</p> <p>Thanks for taking the time to respond to all those comments in detail.</p> <p>I've added a few more to the <a href="https://github.com/Subaru-PFS/drp_stella/pull/3/files" class="external-link" rel="nofollow">drp_stella PR</a> (and I see <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> has done the same). They're mostly fairly minor, though: I don't think they'll take long, and they'll help make things a bit clearer. After resolving them, my suggestion would be that we go ahead and merge this to <tt>master</tt> as "good enough for now", but continue to evolve and improve this document by filing tickets as we spot things that needs improving.</p> <p>A couple of other points:</p> <ul class="alternate" type="square"> <li>Our <a href="http://pfs-2d-pipeline.readthedocs.io/en/latest/dev.html#workflow" class="external-link" rel="nofollow">workflow guidelines</a> suggest that you should rebase and squash to create a logical history before merging. Certainly, we shouldn't have a series of "respond to XXX comments from RHL" commits! We also shouldn't see merges like <a href="https://github.com/Subaru-PFS/drp_stella/pull/3/commits/5868b8f1f5efdbd4e05816bfa87cc3a31bd260a4" class="external-link" rel="nofollow">5868b8f</a> on the ticket branch (but that will drop out when you rebase).</li> </ul> <ul class="alternate" type="square"> <li>In several places above, you write that you've asked Paul about something. Asking Paul is often a good idea, but can I suggest that for many of these questions it would actually be appropriate to put them on <a href="https://community.lsst.org" class="external-link" rel="nofollow">community.lsst.org</a> so that others can respond (rather than relying on Paul for everything) and everybody can benefit from the answers? You could call Paul out with an "@" if you think he's particularly well qualified to comment.</li> </ul> Blocks PIPE2D-38 Development Epic Link INFRA-35 Rank 0|ii00g7: Reviewers swinbank Sprint 2014-14 2014-15 2014-16 Story Points 2.0