What type of installation should be performed for lsst? binary, from source, lsstsw,...?

I'd think that eups distrib install would be a good starting point.

Should I add Biases and Darks to drp_stella_data so people can test all commands in the quick-start guide? Here is what I have now:

====================================

• Install lsst binary distribution (https://pipelines.lsst.io/install/conda.html)
• Install git-lfs (https://git-lfs.github.com/)
• Install matplotlib

conda install matplotlib

• Create a directory for the PFS-DRP repositories:

mkdir ~/stella-git
cd ~/stella-git

• Clone the relevant git repositories:

git clone https://github.com/Subaru-PFS/drp_stella.git
git clone https://github.com/Subaru-PFS/drp_stella_data.git
git clone https://github.com/Subaru-PFS/obs_pfs.git

• Install a newer version of eigen:

eups distrib install eigen 3.2.5.lsst2

• Declare the git repositories:

eups declare obs_pfs 1.0 -c -f Linux64 -r ~/stella-git/drp_stella
eups declare drp_stella_data 1.0 -c -f Linux64 -r ~/stella-git/drp_stella_data
eups declare drp_stella 1.0 -c -f Linux64 -r ~/stella-git/drp_stella

• Setup the pipeline:

setup drp_stella 1.0 -v

• Build the pipeline:

cd drp_stella
scons opt=3 -j8

• Create the defects database:

cd ../obs_pfs
scons opt=3 -j8

• Now for using the pipeline. First we need to create a directory (actually 2) where we want to store pipeline outputs. Let's
  assume you have a directory ~/spectra and do everything related to spectra there:

mkdir ~/spectra/PFS
mkdir ~/spectra/PFS/CALIB

• Let's further assume your raw fits files are in ~/spectra/raw/. Configuration parameters can be passed to the pipeline task
  either on the command line or in a config file (e.g. ~/stella-git/obs_pfs/config/pfs/ingest.py, see following example). You
  can list all possible configuration parameters by appending a "--show config" to the parameter list.
  To ingest the raw images:

cd ~/spectra
ingestImages.py 'PFS' 'raw/*.fits' --output='PFS' --mode=link -C ~/stella-git/obs_pfs/config/pfs/ingest.py

• You can now inspect the created SQL database with e.g. sqlitebrowser (http://sqlitebrowser.org/).
  Now that we have our database we can start reducing things. We should probably start with the Biases. If the biases we
  want to reduce were observed on 2015-12-22 on spectrograph 2, arm red (r) at site S:

reduceBias.py 'PFS' --calib 'PFS/CALIB' --output 'PFS' --calibId calibVersion=bias arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=BIAS dateObs=2015-12-22 --clobber-config --nodes=1
--procs=1

• Note the 2 config parameters --nodes and --procs at the end. These parameters are required by tasks which are
  parallelized. Sometimes running the code in parallel can lead to problems (in most cases caused by the 3rd-party libraries
  used), to setting nodes and procs to 1 is a safe choice. —-clobber-config is needed if you re-run the task with different
  config parameters. The config used for each run is written to PFS/config/bias.py and the task won’t run if you change
  parameters, unless you say —-clobber-config.

• Now that we have a master bias we need to inject that too into our database:

genCalibRegistry.py --root='PFS/CALIB' --camera PFS --validity 180

• Now we can create a bias-subtracted Dark and ingest that into our database:

reduceDark.py 'PFS' --calib 'PFS/CALIB' --output 'PFS' --calibId calibVersion=dark arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=DARK dateObs=2015-12-22 --nodes=1 --procs=1 -C ~/stella-git/obs_pfs/config/pfs/dark.py
genCalibRegistry.py --root='PFS/CALIB' --camera PFS --validity 180

• and a bias and dark-subtracted flat (currently only used to trace the apertures of the fiber traces):

reduceFlat.py 'PFS' --calib 'PFS/CALIB' --calibId calibVersion=flat arm=r calibDate=2015-12-22 filter=r spectrograph=2 --do-exec --id field=FLAT arm=r dateObs=2015-12-22 spectrograph=2 --loglevel 'info' --config isr.doBias='True' isr.doDark='True' isr.doLinearize=False --output 'PFS' --nodes=1 --procs=1 --clobber-config

genCalibRegistry.py --root='PFS/CALIB' --camera PFS --validity 180

• Since we have the Bias and Dark we can also run the Instrumental-Signature Removal (ISR) task for our Arc spectrum:

detrend.py 'PFS' --calib='PFS/CALIB' --id arm=‘r’ spectrograph=2 dateObs='2015-12-22' field='ARC' -C ~/stella-git/obs_pfs/config/pfs/detrend.py --output 'PFS'

• We now have the postISRCCD images and can extract and wavelength-calibrate our CdHgKrNeXe Arc with the visit
  number 4:

reduceArc.py 'PFS' --id visit=4 --wLenFile $OBS_PFS_DIR'/pfs/RedFiberPixels.fits.gz' --lineList $OBS_PFS_DIR'/pfs/lineLists/CdHgKrNeXe_red.fits' --loglevel 'info' --calib 'PFS/CALIB/' --output 'PFS'

Added quick-start guide to drp_stella/doc/quick-start_guide.text in tickets/INFRA-36.
Raw fits files to run the examples are in drp_stella_data/tests/data/raw (tickets/INFRA-36)

Looks like something went wrong here: the most recent commit to tickets/INFRA-36 actually removed the existing doc/install.text without adding anything.

Putting this back in progress until there's something to review!

Made sure that install.text is now in drp_stella/doc/install.text (tickets/INFRA-36)...

I've left a number of comments on the pull request. The brief summary is that the installation instructions seem basically fine, although I've made a few suggestions for places where they could be simplified or clarified. The demo reduction confused me for quite a while until I figured out that the data in drp_stella_data doesn't actually correspond to your example command lines: you keep referring to arm=r spectrograph=2, but the data seems to be from arm=n spectrograph=1. Even once I'd got that figured out, the final reduceArc.py command fails because it has a hard-coded path referring to your home directory in it.

More generally, I'm a bit worried about the level of the demo documentation. What is your intended audience for this? It seems to go into a lot of detail in some places (e.g. suggesting that users might want to inspect the registry database with a SQLite browser) but to skim over the top of what you're actually doing and why. A paragraph or two of more introductory material might help, as would cutting out all the complexity related to how tasks are constructed and run. For example, why do you use both config override files and command line options? why use --clobber-config? None of this seems relevant to somebody who simply wants to start processing PFS data as quickly as possible.

Please make this 7-bit clean; the line

export DRP_STELLA=<E2><80><9C><your_target_directory><E2><80><9D>

should read

export DRP_STELLA="<your_target_directory>"

Here is an annotated version of install.tst; my comments start with RHL

The following notes should allow you to install the LSST stack and the PFS DRP.
After the installation procedure example commands are given to show you how to
use the pipeline. The commands have been tested on an Arch Linux machine as well
as on Mac OS X.

• Install the LSST binary distribution (https://pipelines.lsst.io/install/conda.html)
  Here it should not matter whether you install anaconda or miniconda. Don’t forget
  to setup the LSST environment as mentioned on the website:
  RHL Please note that there's only a need to install lsst-distrib, not lsst-sims
  RHL Which version of LSST code are you assuming? Please at least specify a minimum version

source activate lsst
source eups-setups.sh

RHL Explain why we need this, and please provide slightly more detailed instructions
RHL E.g.:
RHL Follow the instructions at https://git-lfs.github.com to installing git-lfs (e.g. using homebrew
RHL on os/x), then type
RHL git lfs install
RHL there's no need to issue any "git lfs track" commands

• Install git-lfs (https://git-lfs.github.com/)
• Install matplotlib and gcc:

RHL Why? lsst-distrib installs matplotlib. Why are we installing gcc – we should be using cc
RHL at least on os/x
conda install matplotlib gcc

• Create a directory for the PFS-DRP repositories:
  RHL Please explain that the use of DRP_STELLA is purely for convenience in this tutorial
  export DRP_STELLA=“<your_target_directory>”
  RHL It's better to say,
  RHL export DRP_STELLA=$HOME/stella-git
  RHL as expanding ~ is a bash extension. Also, why the quotes?
  RHL That suggested directory name "stella-git" is not all that helpful – the fact that
  RHL it's git is irrelevant, and it's more than stella. Maybe $HOME/PFS?
  (e.g. export DRP_STELLA="~/stella-git")
  mkdir -p $DRP_STELLA
  cd $DRP_STELLA

• Clone the relevant git repositories:

git clone https://github.com/Subaru-PFS/drp_stella.git
git clone https://github.com/Subaru-PFS/drp_stella_data.git
git clone https://github.com/Subaru-PFS/obs_pfs.git

• Set $EUPS_PKGROOT

export EUPS_PKGROOT="https://sw.lsstcorp.org/eupspkg/"

• Install Eigen with the unsupported modules:
  RHL I see that lsst-distrib is still installing 3.2.5.lsst1 (I've just prodded square about this).
  RHL Please monitor this, and remove these instructions ASAP; in the meantime make it clear
  RHL that setting EUPS_PKGROOT and doing this install are temporary workarounds.
  eups distrib install eigen 3.2.5.lsst2

• Declare the git repositories:

RHL Why 0.1? You didn't tell them to checkout a 0.1 branch,
RHL Also, you should almost never declare a working version; install and then declare the installed
RHL version
RHL
RHL Before you install, make sure that everythings works – the versions of the LSST stack that you
RHL were using will be frozen into the installation.

eups declare obs_pfs 0.1 -c -r $DRP_STELLA/obs_pfs
eups declare drp_stella_data 0.1 -c -r $DRP_STELLA/drp_stella_data
eups declare drp_stella 0.1 -c -r $DRP_STELLA/drp_stella

• Setup the pipeline. As obs_pfs and drp_stella_data are required packages they are set up automatically:
  RHL Why specify -v?
  RHL Why 0.1 when you made drp_stella current?
  setup drp_stella 0.1 -v

During this step the environment variables $DRP_STELLA_DIR, $DRP_STELLA_DATA_DIR, and $OBS_PFS_DIR are
automatically created as well:
RHL set, rather than created.

echo $DRP_STELLA_DIR

should show you:
RHL No, it'll show you the value of $DRP_STELLA expanded. E.g. /Users/aritter/PFS/drp_stella
RHL But why do they need to know this?
$DRP_STELLA/drp_stella

• Build the pipeline:
  RHL Why did you have them declare a local version before building it?
  RHL See notes about declaring git versions. Please use setup -r . (maybe -j) instead.
  RHL
  RHL In fact, I think that the order matters.
  RHL Don't you need to setup and build obs_pfs first?
  RHL then drp_stella_data, then drp_stella?
  RHL
  RHL obs_pfs generates lots of chatter (e.g.
  RHL type(config) = <class 'lsst.afw.cameraGeom.cameraConfig.CameraConfig'>
  RHL dir(ccd) = ['_class', 'del', 'delattr', 'dict', 'doc', 'format', 'getattr
  RHL ...
  RHL as well as some silly LSST warnings that we should get fixed and that you should warn people to
  RHL ignore:
  RHL CameraMapper WARNING: Calibration root directory not found: ./CALIB
  RHL ...
  RHL
  RHL And don't you need a more recent sconsUtils and a --filterWarn flag to deal with warnings about typemaps?
  RHL Actually I see that those fixes didn't get into sconsUtils (until tonight!) so there are some unavoidable
  RHL warnings — you should probably mention them.

cd $DRP_STELLA/drp_stella
scons opt=3
cd ../obs_pfs
scons opt=3

RHL tests/PSF.py generates warnings
RHL afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE
RHL : Empty WCS extension, using FITS header
RHL what is causing these? Is there a workaround (or a ticket)?

RHL tests/FiberTraces.py generates 1217 lines of chatter, and Spectra.py 615. Where is it coming from? Most of it looks
RHL like debugging stuff that should be disabled (is it an incorrect logging level?)

RHL tests/testDrp.py fails without returning an error code. It's due to the os/x 10.11 SIP stuff I assume,
RHL but the lack of an error code is a bug. Also, I'm not sure that we should be using python unittest to
RHL test shell scripts. How does ci_hsc do this?

RHL Not setting up drp_pipeline_data says, "Nothing to be done for tests"; it should at least print a warning,
RHL and preferably skip them using unittest's decorators – but I'm not sure LSST does this bit right

• Now for using the pipeline. Raw test data are in $DRP_STELLA_DATA_DIR/tests/data/raw/ (3 Biases, 3 Darks, 1 Flat, 1 Arc).
  RHL why do we need to know this?
  First we need to create a directory (actually 2) where we want to store pipeline outputs.
  Let's assume you want to store the pipeline outputs in a directory ~/spectra/PFS:
  RHL: Please explain why you are introducing an environment variable.
  RHL: If you want them to use this one again, they should either put it in a startup file,
  RHL or use eups to manage it (probably a better solution)

export SPECTRA_DIR="~/spectra/PFS"
mkdir -p $SPECTRA_DIR/CALIB

• We need to tell the LSST stack which mapper to use:

echo "lsst.obs.pfs.PfsMapper" > $SPECTRA_DIR/_mapper

• Configuration parameters can be passed to the pipeline task either on the command line or in a config file
  (e.g. $OBS_PFS_DIR/config/pfs/ingest.py). You can list all possible configuration parameters by appending a
  "--show config" to the parameter list.
  RHL If we're providing this sort of information (and I don't think it should be in an install document)
  RHL document --show config=glob
  To ingest the raw images:

RHL Why use cd and . rather than specifying the output?
cd $SPECTRA_DIR
ingestImages.py . $DRP_STELLA_DATA_DIR/tests/data/raw/*.fits --output . --mode link

• Now that we have our database we can start reducing things. We start with reducing the Biases. The data we
  want to reduce were observed on 2015-12-22 on spectrograph 2, arm red (r) at site S.
  RHL What does site S mean?
  Note the 2 config parameters --nodes and --procs at the end. These parameters are required by tasks which are
  parallelized. Sometimes running the code in parallel can lead to problems (in most cases caused by the 3rd-party
  libraries used), so setting nodes and procs to 1 is a safe choice.
  RHL Under what circumstances and why does it lead to problems? Are there jira issues?
  RHL What are the default values of nodes and procs?

reduceBias.py . --calib CALIB --output . --calibId calibVersion=bias arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=BIAS dateObs=2015-12-22 --nodes 1 --procs 1

• Now that we have a master bias we need to ingest that too into our database:
  RHL What exactly does the genCalibRegistry.py command do? What is the 180?
  genCalibRegistry.py --root CALIB --camera PFS --validity 180

• Now we can create a bias-subtracted Dark and ingest that into our database:
  RHL presumably trimmed too, and scaled? I.e. a master dark used in further processing
  RHL and isn't this a different database? And do you mean database or registry?

reduceDark.py . --calib CALIB --output . --calibId calibVersion=dark arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=DARK dateObs=2015-12-22 --nodes 1 --procs 1
genCalibRegistry.py --root CALIB --camera PFS --validity 180

• and a bias and dark-subtracted flat (currently only used to trace the apertures of the fiber traces):
  RHL Now I'm confused. What are we making here? A flat field? Why the parenthetical comment? Where
  RHL do dithered flats fit into this?
  RHL What does the --calib CALIB option mean; is it --CALIB ./CALIB? Does it default?

reduceFlat.py . --calib CALIB --calibId calibVersion=flat arm=r calibDate=2015-12-22 filter=r spectrograph=2 --do-exec --id field=FLAT arm=r dateObs=2015-12-22 spectrograph=2 --output . --nodes 1 --procs 1
genCalibRegistry.py --root=CALIB --camera PFS --validity 180

• Since we have the Bias and Dark we can also run the Instrumental-Signature Removal (ISR) task for our Arc spectrum:
  RHL Still writing into the same place? Why not using --rerun?
  RHL Do I need to specify all those flags?

detrend.py . --calib CALIB --output . --id arm=r spectrograph=2 dateObs=2015-12-22 field=ARC

• We now have the postISRCCD images and can extract and wavelength-calibrate our CdHgKrNeXe Arc with the visit
  number 4:

RHL Why do I need the --wLenFile? If it's calibration, why isn't it in the calibration registy? Or maybe
RHL in obs_pfs?
RHL Why isn't the line list in the calib registry?
reduceArc.py . --output . --calib CALIB --id visit=4 --wLenFile $OBS_PFS_DIR/pfs/RedFiberPixels.fits.gz --lineList $OBS_PFS_DIR/pfs/lineLists/CdHgKrNeXe_red.fits

RHL I didn't run these data commands (yet). How chatty are they at various loglevels (e.g. INFO)?

Here's an update to that document with new comments noted as RHL2. Andreas has addressed some of these (and some of the RHLs too), but I haven't removed the older comments.

The following notes should allow you to install the LSST stack and the PFS DRP.
After the installation procedure example commands are given to show you how to
use the pipeline. The commands have been tested on an Arch Linux machine as well
as on Mac OS X.

• Install the LSST binary distribution (https://pipelines.lsst.io/install/conda.html)
  Here it should not matter whether you install anaconda or miniconda. Don’t forget
  to setup the LSST environment as mentioned on the website:
  RHL Please note that there's only a need to install lsst-distrib, not lsst-sims
  RHL Which version of LSST code are you assuming? Please at least specify a minimum version

source activate lsst
source eups-setups.sh

RHL Explain why we need this, and please provide slightly more detailed instructions
RHL E.g.:
RHL Follow the instructions at https://git-lfs.github.com to installing git-lfs (e.g. using homebrew
RHL on os/x), then type
RHL git lfs install
RHL there's no need to issue any "git lfs track" commands

• Install git-lfs (https://git-lfs.github.com/)
• Install matplotlib and gcc:

RHL Why? lsst-distrib installs matplotlib. Why are we installing gcc – we should be using cc
RHL at least on os/x
conda install matplotlib gcc

• Create a directory for the PFS-DRP repositories:
  RHL Please explain that the use of DRP_STELLA is purely for convenience in this tutorial
  export DRP_STELLA=“<your_target_directory>”
  RHL It's better to say,
  RHL export DRP_STELLA=$HOME/stella-git
  RHL as expanding ~ is a bash extension. Also, why the quotes?
  RHL That suggested directory name "stella-git" is not all that helpful – the fact that
  RHL it's git is irrelevant, and it's more than stella. Maybe $HOME/PFS?
  (e.g. export DRP_STELLA="~/stella-git")
  mkdir -p $DRP_STELLA
  cd $DRP_STELLA

• Clone the relevant git repositories:

git clone https://github.com/Subaru-PFS/drp_stella.git
git clone https://github.com/Subaru-PFS/drp_stella_data.git
git clone https://github.com/Subaru-PFS/obs_pfs.git

• Set $EUPS_PKGROOT

export EUPS_PKGROOT="https://sw.lsstcorp.org/eupspkg/"

• Install Eigen with the unsupported modules:
  RHL I see that lsst-distrib is still installing 3.2.5.lsst1 (I've just prodded square about this).
  RHL Please monitor this, and remove these instructions ASAP; in the meantime make it clear
  RHL that setting EUPS_PKGROOT and doing this install are temporary workarounds.
  eups distrib install eigen 3.2.5.lsst2

• Declare the git repositories:

RHL Why 0.1? You didn't tell them to checkout a 0.1 branch,
RHL Also, you should almost never declare a working version; install and then declare the installed
RHL version
RHL
RHL Before you install, make sure that everythings works – the versions of the LSST stack that you
RHL were using will be frozen into the installation.

eups declare obs_pfs 0.1 -c -r $DRP_STELLA/obs_pfs
eups declare drp_stella_data 0.1 -c -r $DRP_STELLA/drp_stella_data
eups declare drp_stella 0.1 -c -r $DRP_STELLA/drp_stella

• Setup the pipeline. As obs_pfs and drp_stella_data are required packages they are set up automatically:
  RHL Why specify -v?
  RHL Why 0.1 when you made drp_stella current?
  setup drp_stella 0.1 -v

During this step the environment variables $DRP_STELLA_DIR, $DRP_STELLA_DATA_DIR, and $OBS_PFS_DIR are
automatically created as well:
RHL set, rather than created.

echo $DRP_STELLA_DIR

should show you:
RHL No, it'll show you the value of $DRP_STELLA expanded. E.g. /Users/aritter/PFS/drp_stella
RHL But why do they need to know this?
$DRP_STELLA/drp_stella

• Build the pipeline:
  RHL Why did you have them declare a local version before building it?
  RHL See notes about declaring git versions. Please use setup -r . (maybe -j) instead.
  RHL
  RHL In fact, I think that the order matters.
  RHL Don't you need to setup and build obs_pfs first?
  RHL then drp_stella_data, then drp_stella?
  RHL
  RHL obs_pfs generates lots of chatter (e.g.
  RHL type(config) = <class 'lsst.afw.cameraGeom.cameraConfig.CameraConfig'>
  RHL dir(ccd) = ['_class', 'del', 'delattr', 'dict', 'doc', 'format', 'getattr
  RHL ...
  RHL as well as some silly LSST warnings that we should get fixed and that you should warn people to
  RHL ignore:
  RHL CameraMapper WARNING: Calibration root directory not found: ./CALIB
  RHL ...
  RHL
  RHL And don't you need a more recent sconsUtils and a --filterWarn flag to deal with warnings about typemaps?
  RHL Actually I see that those fixes didn't get into sconsUtils (until tonight!) so there are some unavoidable
  RHL warnings — you should probably mention them.

cd $DRP_STELLA/drp_stella
scons opt=3
cd ../obs_pfs
scons opt=3

RHL tests/PSF.py generates warnings
RHL afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE
RHL : Empty WCS extension, using FITS header
RHL what is causing these? Is there a workaround (or a ticket)?

RHL tests/FiberTraces.py generates 1217 lines of chatter, and Spectra.py 615. Where is it coming from? Most of it looks
RHL like debugging stuff that should be disabled (is it an incorrect logging level?)

RHL tests/testDrp.py fails without returning an error code. It's due to the os/x 10.11 SIP stuff I assume,
RHL but the lack of an error code is a bug. Also, I'm not sure that we should be using python unittest to
RHL test shell scripts. How does ci_hsc do this?

RHL Not setting up drp_pipeline_data says, "Nothing to be done for tests"; it should at least print a warning,
RHL and preferably skip them using unittest's decorators – but I'm not sure LSST does this bit right

• Now for using the pipeline. Raw test data are in $DRP_STELLA_DATA_DIR/tests/data/raw/ (3 Biases, 3 Darks, 1 Flat, 1 Arc).
  RHL why do we need to know this?
  First we need to create a directory (actually 2) where we want to store pipeline outputs.
  Let's assume you want to store the pipeline outputs in a directory ~/spectra/PFS:
  RHL: Please explain why you are introducing an environment variable.
  RHL: If you want them to use this one again, they should either put it in a startup file,
  RHL or use eups to manage it (probably a better solution)

export SPECTRA_DIR="~/spectra/PFS"
mkdir -p $SPECTRA_DIR/CALIB

• We need to tell the LSST stack which mapper to use:
  RHL2 Define mapper. Explain what is going on here, including repositories.
  RHL2 Have you demonstrated using a raw filesystem with no ingestion? It should work

echo "lsst.obs.pfs.PfsMapper" > $SPECTRA_DIR/_mapper

• Configuration parameters can be passed to the pipeline task either on the command line or in a config file
  (e.g. $OBS_PFS_DIR/config/pfs/ingest.py). You can list all possible configuration parameters by appending a
  "--show config" to the parameter list.
  RHL If we're providing this sort of information (and I don't think it should be in an install document)
  RHL document --show config=glob
  To ingest the raw images:

RHL Why use cd and . rather than specifying the output?
RHL2 Actually, why do you need to specify --output at all?
RHL2 (Paul clarified that: it's for the temp files such as trimmed biases. Please add a note)
RHL2 It'd be better to use --rerun e.g. --rerun USERNAME/tmp (and explain that they should
RHL2 substitute USERNAME)
RHL2
RHL2 I think you want
RHL2 ingestImages.py $SPECTRA_DIR $DRP_STELLA_DATA_DIR/tests/data/raw/*.fits --mode link -L warn
RHL2 And please explain the meaning of "--mode link"
RHL2 Probably it would be better to move some of your info logging up to debug; the default logging
RHL2 is too verbose.
RHL2 There are also print statements. Please remove all of them (some by converting to log messages
RHL2 at appropriate levels of verbosity

cd $SPECTRA_DIR
ingestImages.py . $DRP_STELLA_DATA_DIR/tests/data/raw/*.fits --output . --mode link

• Now that we have our database we can start reducing things. We start with reducing the Biases. The data we
  want to reduce were observed on 2015-12-22 on spectrograph 2, arm red (r) at site S.
  RHL What does site S mean?
  Note the 2 config parameters --nodes and --procs at the end. These parameters are required by tasks which are
  parallelized. Sometimes running the code in parallel can lead to problems (in most cases caused by the 3rd-party
  libraries used), so setting nodes and procs to 1 is a safe choice.
  RHL Under what circumstances and why does it lead to problems? Are there jira issues?
  RHL What are the default values of nodes and procs?
  RHL2 Why are there not sensible defaults:
  RHL2 RuntimeError: Must specify numNodes+numProcs or numCores
  RHL2 This error message doesn't tell you the options names (the "num" is confusing)
  RHL2 What does --do-exec do?
  RHL2 Why are we running this multi-tasked? Can you file a ticket to get this fixed, please.

reduceBias.py . --calib CALIB --output . --calibId calibVersion=bias arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=BIAS dateObs=2015-12-22 --nodes 1 --procs 1

RHL2 Why do I need --calib CALIB? I don't think you need --calib CALIB, so the preferred command would be
RHL2 reduceBias.py $SPECTRA_DIR --output $SPECTRA_DIR --cores 1 --id field=BIAS dateObs=2015-12-22 arm=r spectrograph=2 --calibId calibVersion=bias calibDate=2015-15-22 arm=r spectrograph=2
RHL2 Why do I need to specify the output?

RHL2 What is
RHL2 --calibId calibVersion=bias
RHL2 needed? Well, I know why it's because
RHL2 template: "BIAS/NONE/%(calibVersion)s/pfsBias-%(calibDate)s-0-%(spectrograph)1d%(arm)s.fits"
RHL2 but why did you do that?
RHL2 If I omit the --calibId I get the error message
RHL2 RuntimeError: Unable to determine output filename from

Unknown macro: {'filter'}

: No registry for lookup
RHL2 Why is it not picking up arm and spectrograph from the inputs? The input registry has them.
RHL2 There is no check for a mis-formed date (e.g. 2015-99-22)

RHL2 Where is "bias.isr: Set 0 BAD pixels to 0.01" coming from? What does it mean?
RHL2 How about ": Empty WCS extension, using FITS header"
RHL2 And also "bias.isr.assembleCcd WARNING: No WCS found in input exposure"

• Now that we have a master bias we need to ingest that too into our database:
  RHL What exactly does the genCalibRegistry.py command do? What is the 180?
  RHL2 Should be
  RHL2 genCalibRegistry.py --root $SPECTRA_DIR/CALIB --camera PFS --validity 180
  genCalibRegistry.py --root CALIB --camera PFS --validity 180

• Now we can create a bias-subtracted Dark and ingest that into our database:
  RHL presumably trimmed too, and scaled? I.e. a master dark used in further processing
  RHL and isn't this a different database? And do you mean database or registry?

RHL2 All comments for bias generation apply here.
RHL2 Why are the arguments to the different reduceXXX.py commands in different orders? It's a bit confusing
reduceDark.py . --calib CALIB --output . --calibId calibVersion=dark arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=DARK dateObs=2015-12-22 --nodes 1 --procs 1
RHL2 Explain why this command needs to be repeated not just run at the end
genCalibRegistry.py --root CALIB --camera PFS --validity 180

• and a bias and dark-subtracted flat (currently only used to trace the apertures of the fiber traces):
  RHL Now I'm confused. What are we making here? A flat field? Why the parenthetical comment? Where
  RHL do dithered flats fit into this?
  RHL What does the --calib CALIB option mean; is it --CALIB ./CALIB? Does it default?

RHL2 All comments for bias generation apply here.
RHL2 Why do I need filter=r as part of the calibId?
reduceFlat.py . --calib CALIB --calibId calibVersion=flat arm=r calibDate=2015-12-22 filter=r spectrograph=2 --do-exec --id field=FLAT arm=r dateObs=2015-12-22 spectrograph=2 --output . --nodes 1 --procs 1
genCalibRegistry.py --root=CALIB --camera PFS --validity 180

• Since we have the Bias and Dark we can also run the Instrumental-Signature Removal (ISR) task for our Arc spectrum:
  RHL Still writing into the same place? Why not using --rerun?
  RHL Do I need to specify all those flags?
  RHL2 What does detrend.py do?
  RHL2 We should note that it doesn't (yet) flat field
  RHL2 Why the "type(config)" lines?
  RHL2 Why specify --calib CALIB?
  RHL2 Why are we doing this in two steps (detrend.py and reduceArc.py)?
  RHL2 If we are only processing one arc, why specify it this way, not by visit, e.g.
  RHL2 detrend.py $SPECTRA_DIR --rerun USERNAME/tmp --id visit=4
  RHL2 If you want to specify reducing all r2 arcs taken on 2015-12-22, please explain
  detrend.py . --calib CALIB --output . --id arm=r spectrograph=2 dateObs=2015-12-22 field=ARC
  RHL2 What files are produced?

• We now have the postISRCCD images and can extract and wavelength-calibrate our CdHgKrNeXe Arc with the visit
  number 4:

RHL Why do I need the --wLenFile? If it's calibration, why isn't it in the calibration registy? Or maybe
RHL in obs_pfs?
RHL Why isn't the line list in the calib registry?
RHL2 Actually I see that I don't. What are the pros and cons?
reduceArc.py . --output . --calib CALIB --id visit=4 --wLenFile $OBS_PFS_DIR/pfs/RedFiberPixels.fits.gz --lineList $OBS_PFS_DIR/pfs/lineLists/CdHgKrNeXe_red.fits

RHL2 A generic comment: far too much information is being printed to the screen! At -L warn there should be basically nothing; the LSST code doesn't obey this dictum very well, but drp_stella is worse!

Lots of the above commentary is useful, and should be mined for tickets both for here and for the LSST stack. However, we should avoid scope creep here: it's not necessary to fix all the above in order for this document to be useful. Obviously, it is necessary to correct any errors in this text before we can proceed.

One thing that jumped out:

Why are we installing gcc – we should be using cc at least on os/x

As I recall, the aim here is to avoid ABI incompatibilities since (on Linux, at least) all the Anaconda packages were compiled with gcc 4.small and are incompatible with a modern versions. We should check what the situation is on OSX.

I agree with John. Essentially all of my comments need to be addressed, but not all as part of this ticket. Please file tickets as appropriate (e.g. the calibId things are clearly not part of INFRA-36)

Another example is that

021dd3f Replace reduceDark with constructDark

should not have been addressed on this ticket.

Here is my response to RHL's comments:

The following notes should allow you to install the LSST stack and the PFS DRP.
After the installation procedure example commands are given to show you how to
use the pipeline. The commands have been tested on an Arch Linux machine as well
as on Mac OS X.
• Install the LSST binary distribution (https://pipelines.lsst.io/install/conda.html)
Here it should not matter whether you install anaconda or miniconda. Don’t forget
to setup the LSST environment as mentioned on the website:

RHL Please note that there's only a need to install lsst-distrib, not lsst-sims

RHL Which version of LSST code are you assuming? Please at least specify a minimum version
source activate lsst
source eups-setups.sh
RHL Explain why we need this, and please provide slightly more detailed instructions

RHL E.g.:

RHL Follow the instructions at https://git-lfs.github.com to installing git-lfs (e.g. using homebrew

RHL on os/x), then type

RHL git lfs install

RHL there's no need to issue any "git lfs track" commands

AR Done

• Install git-lfs (https://git-lfs.github.com/)
• Install matplotlib and gcc:
RHL Why? lsst-distrib installs matplotlib. Why are we installing gcc – we should be using cc

RHL at least on os/x

AR Removed matplotlib from the list of packages to be installed and explained why for Linux
AR machines we need to install gcc within anaconda


conda install matplotlib gcc
• Create a directory for the PFS-DRP repositories:

RHL Please explain that the use of DRP_STELLA is purely for convenience in this tutorial
export DRP_STELLA=“<your_target_directory>”

RHL It's better to say,

RHL export DRP_STELLA=$HOME/stella-git

RHL as expanding ~ is a bash extension. Also, why the quotes?

AR Changed to $HOME, removed the quotes

RHL That suggested directory name "stella-git" is not all that helpful – the fact that

RHL it's git is irrelevant, and it's more than stella. Maybe $HOME/PFS?


AR Changed to $HOME/PFS

(e.g. export DRP_STELLA="~/stella-git")
mkdir -p $DRP_STELLA
cd $DRP_STELLA
• Clone the relevant git repositories:
git clone https://github.com/Subaru-PFS/drp_stella.git
git clone https://github.com/Subaru-PFS/drp_stella_data.git
git clone https://github.com/Subaru-PFS/obs_pfs.git
• Set $EUPS_PKGROOT
export EUPS_PKGROOT="https://sw.lsstcorp.org/eupspkg/"
• Install Eigen with the unsupported modules:

RHL I see that lsst-distrib is still installing 3.2.5.lsst1 (I've just prodded square about this).

RHL Please monitor this, and remove these instructions ASAP; in the meantime make it clear

RHL that setting EUPS_PKGROOT and doing this install are temporary workarounds.

AR Done


eups distrib install eigen 3.2.5.lsst2
• Declare the git repositories:
RHL Why 0.1? You didn't tell them to checkout a 0.1 branch,

RHL Also, you should almost never declare a working version; install and then declare the installed

RHL version

RHL

RHL Before you install, make sure that everythings works – the versions of the LSST stack that you

RHL were using will be frozen into the installation.

AR See below

eups declare obs_pfs 0.1 -c -r $DRP_STELLA/obs_pfs
eups declare drp_stella_data 0.1 -c -r $DRP_STELLA/drp_stella_data
eups declare drp_stella 0.1 -c -r $DRP_STELLA/drp_stella
• Setup the pipeline. As obs_pfs and drp_stella_data are required packages they are set up automatically:

RHL Why specify -v?

RHL Why 0.1 when you made drp_stella current?
setup drp_stella 0.1 -v
During this step the environment variables $DRP_STELLA_DIR, $DRP_STELLA_DATA_DIR, and $OBS_PFS_DIR are 
automatically created as well:

RHL set, rather than created.
echo $DRP_STELLA_DIR
should show you:

RHL No, it'll show you the value of $DRP_STELLA expanded. E.g. /Users/aritter/PFS/drp_stella

RHL But why do they need to know this?

AR Removed

$DRP_STELLA/drp_stella
• Build the pipeline:

RHL Why did you have them declare a local version before building it?

RHL See notes about declaring git versions. Please use setup -r . (maybe -j) instead.

RHL

AR Removed declarations, now using setup -r .

RHL In fact, I think that the order matters.

RHL Don't you need to setup and build obs_pfs first?

RHL then drp_stella_data, then drp_stella?

RHL

AR Didn’t need to setup and build obs_pfs first as setting up drp_stella automatically setup obs_pfs.
AR Now with setup -r . corrected the order.


RHL obs_pfs generates lots of chatter (e.g.

RHL type(config) = <class 'lsst.afw.cameraGeom.cameraConfig.CameraConfig'>

RHL dir(ccd) = ['_class', 'del', 'delattr', 'dict', 'doc', 'format', 'getattr

RHL ...

RHL as well as some silly LSST warnings that we should get fixed and that you should warn people to

RHL ignore:

RHL CameraMapper WARNING: Calibration root directory not found: ./CALIB

RHL ...

RHL

AR Filed tickets for that.


RHL And don't you need a more recent sconsUtils and a --filterWarn flag to deal with warnings about typemaps?

RHL Actually I see that those fixes didn't get into sconsUtils (until tonight!) so there are some unavoidable

RHL warnings — you should probably mention them.

AR Included installation of a recent sconsUtils and a —filterWarn flag in the quick-start guide

cd $DRP_STELLA/drp_stella
scons opt=3
cd ../obs_pfs
scons opt=3
RHL tests/PSF.py generates warnings

RHL afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE

RHL : Empty WCS extension, using FITS header

RHL what is causing these? Is there a workaround (or a ticket)?

AR Ticket filed (PIPE2D-64), fixed, and put in review.

RHL tests/FiberTraces.py generates 1217 lines of chatter, and Spectra.py 615. Where is it coming from? Most of it looks

RHL like debugging stuff that should be disabled (is it an incorrect logging level?)

AR Ticket filed (PIPE2D-65)

RHL tests/testDrp.py fails without returning an error code. It's due to the os/x 10.11 SIP stuff I assume,

RHL but the lack of an error code is a bug. Also, I'm not sure that we should be using python unittest to

RHL test shell scripts. How does ci_hsc do this?

AR Ticket filed (PIPE2D-69)

RHL Not setting up drp_pipeline_data says, "Nothing to be done for tests"; it should at least print a warning,

RHL and preferably skip them using unittest's decorators – but I'm not sure LSST does this bit right

AR I’m pretty sure this is LSST

• Now for using the pipeline. Raw test data are in $DRP_STELLA_DATA_DIR/tests/data/raw/ (3 Biases, 3 Darks, 1 Flat, 1 Arc).

RHL why do we need to know this?

AR Well, we don’t really, but I would want to know what kind of raw data we have…


First we need to create a directory (actually 2) where we want to store pipeline outputs. 
Let's assume you want to store the pipeline outputs in a directory ~/spectra/PFS:

RHL: Please explain why you are introducing an environment variable.

RHL: If you want them to use this one again, they should either put it in a startup file,

RHL or use eups to manage it (probably a better solution)

AR Explanation added

export SPECTRA_DIR="~/spectra/PFS"
mkdir -p $SPECTRA_DIR/CALIB
• We need to tell the LSST stack which mapper to use:

RHL2 Define mapper. Explain what is going on here, including repositories.

AR Explanation added


RHL2 Have you demonstrated using a raw filesystem with no ingestion? It should work

AR Craig asked Paul about that, waiting for an answer

echo "lsst.obs.pfs.PfsMapper" > $SPECTRA_DIR/_mapper
• Configuration parameters can be passed to the pipeline task either on the command line or in a config file
(e.g. $OBS_PFS_DIR/config/pfs/ingest.py). You can list all possible configuration parameters by appending a 
"--show config" to the parameter list.

RHL If we're providing this sort of information (and I don't think it should be in an install document)

RHL document --show config=glob


AR Well it’s a quick-start guide and I think that this is very helpful information for understanding how to set parameters for tasks.
AR What does --show config=glob do? I only see a few parameters when I do that...

• To ingest the raw images:
RHL Why use cd and . rather than specifying the output?

AR Removed cd and specified the output


RHL2 Actually, why do you need to specify --output at all?

RHL2 (Paul clarified that: it's for the temp files such as trimmed biases. Please add a note)

RHL2 It'd be better to use --rerun e.g. --rerun USERNAME/tmp (and explain that they should

RHL2 substitute USERNAME)

RHL2

AR Replaced —output with —rerun


RHL2 I think you want

RHL2 ingestImages.py $SPECTRA_DIR $DRP_STELLA_DATA_DIR/tests/data/raw/*.fits --mode link -L warn

AR Changed


RHL2 And please explain the meaning of "--mode link"

AR Done


RHL2 Probably it would be better to move some of your info logging up to debug; the default logging

RHL2 is too verbose.

RHL2 There are also print statements. Please remove all of them (some by converting to log messages

RHL2 at appropriate levels of verbosity

AR Ticket filed (PIPE2D-70)

cd $SPECTRA_DIR
ingestImages.py . $DRP_STELLA_DATA_DIR/tests/data/raw/*.fits --output . --mode link
• Now that we have our database we can start reducing things. We start with reducing the Biases. The data we
want to reduce were observed on 2015-12-22 on spectrograph 2, arm red (r) at site S.

RHL What does site S mean?

AR Explanation added


Note the 2 config parameters --nodes and --procs at the end. These parameters are required by tasks which are 
parallelized. Sometimes running the code in parallel can lead to problems (in most cases caused by the 3rd-party 
libraries used), so setting nodes and procs to 1 is a safe choice.

RHL Under what circumstances and why does it lead to problems? Are there jira issues?


AR Asked on HipChat, waiting for Craig to help me getting to the bottom of it

RHL What are the default values of nodes and procs?

RHL2 Why are there not sensible defaults:

RHL2 RuntimeError: Must specify numNodes+numProcs or numCores

RHL2 This error message doesn't tell you the options names (the "num" is confusing)

RHL2 What does --do-exec do?

RHL2 Why are we running this multi-tasked? Can you file a ticket to get this fixed, please.

AR Asked Paul about it, waiting for answer

reduceBias.py . --calib CALIB --output . --calibId calibVersion=bias arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=BIAS dateObs=2015-12-22 --nodes 1 --procs 1
RHL2 Why do I need --calib CALIB? I don't think you need --calib CALIB, so the preferred command would be

RHL2 reduceBias.py $SPECTRA_DIR --output $SPECTRA_DIR --cores 1 --id field=BIAS dateObs=2015-12-22 arm=r spectrograph=2 --calibId calibVersion=bias calibDate=2015-15-22 arm=r spectrograph=2

AR Removed —calib CALIB


RHL2 Why do I need to specify the output?
RHL2 What is

RHL2 --calibId calibVersion=bias

RHL2 needed? Well, I know why it's because

RHL2 template: "BIAS/NONE/%(calibVersion)s/pfsBias-%(calibDate)s-0-%(spectrograph)1d%(arm)s.fits"

RHL2 but why did you do that?

AR Ticket filed (PIPE2D-66)


RHL2 If I omit the --calibId I get the error message

RHL2 RuntimeError: Unable to determine output filename from
Unknown macro:

: No registry for lookup

RHL2 Why is it not picking up arm and spectrograph from the inputs? The input registry has them.

AR Gonna investigate (ask Paul). Filed ticket (PIPE2D-67)


RHL2 There is no check for a mis-formed date (e.g. 2015-99-22)

AR Asked Paul about it, waiting for answer

RHL2 Where is "bias.isr: Set 0 BAD pixels to 0.01" coming from? What does it mean?


AR Comes from obs_subaru isr.py setBadRegions(self, exposure)
AR value = afwMath.makeStatistics(exposure.getMaskedImage(), statistic, sctrl).getValue()
AR self.log.info("Set %d BAD pixels to %.2f" % (badPixels.sum(), value))

RHL2 How about ": Empty WCS extension, using FITS header"

RHL2 And also "bias.isr.assembleCcd WARNING: No WCS found in input exposure"

AR These warnings normally come from creating an exposure from a fits file via ExposureF(fileName)
AR Should be makeExposure(makeMaskedImage(ImageF(fileName))). Filed a ticket (PIPE2D-64 in review)

• Now that we have a master bias we need to ingest that too into our database:

RHL What exactly does the genCalibRegistry.py command do? What is the 180?

AR Explanation added.

RHL2 Should be

RHL2 genCalibRegistry.py --root $SPECTRA_DIR/CALIB --camera PFS --validity 180
genCalibRegistry.py --root CALIB --camera PFS --validity 180

AR Done

• Now we can create a bias-subtracted Dark and ingest that into our database:

RHL presumably trimmed too, and scaled? I.e. a master dark used in further processing

AR Added trimmed, scaled, and master


RHL and isn't this a different database? And do you mean database
or registry?

AR Changed “database” to “calibration registry”.

RHL2 All comments for bias generation apply here.

RHL2 Why are the arguments to the different reduceXXX.py commands in different orders? It's a bit confusing

AR Changed order or arguments


reduceDark.py . --calib CALIB --output . --calibId calibVersion=dark arm=r calibDate=2015-12-22 spectrograph=2 --do-exec --id field=DARK dateObs=2015-12-22 --nodes 1 --procs 1

RHL2 Explain why this command needs to be repeated not just run at the end

AR Done


genCalibRegistry.py --root CALIB --camera PFS --validity 180
• and a bias and dark-subtracted flat (currently only used to trace the apertures of the fiber traces):

RHL Now I'm confused. What are we making here? A flat field? Why the parenthetical comment? Where

RHL do dithered flats fit into this?


AR Comment added

RHL What does the --calib CALIB option mean; is it --CALIB ./CALIB? Does it default?

AR Removed —calib CALIB option

RHL2 All comments for bias generation apply here.

RHL2 Why do I need filter=r as part of the calibId?

AR Removed filter=r


reduceFlat.py . --calib CALIB --calibId calibVersion=flat arm=r calibDate=2015-12-22 filter=r spectrograph=2 --do-exec --id field=FLAT arm=r dateObs=2015-12-22 spectrograph=2 --output . --nodes 1 --procs 1
genCalibRegistry.py --root=CALIB --camera PFS --validity 180
• Since we have the Bias and Dark we can also run the Instrumental-Signature Removal (ISR) task for our Arc spectrum:

RHL Still writing into the same place? Why not using --rerun?

AR Using —rerun now


RHL Do I need to specify all those flags?

AR Changed flags to visit=5


RHL2 What does detrend.py do?

AR Comment added


RHL2 We should note that it doesn't (yet) flat field


AR Done

RHL2 Why the "type(config)" lines?

AR Removed “type(config)” lines from camera.py


RHL2 Why specify --calib CALIB?

AR Removed —calib CALIB


RHL2 Why are we doing this in two steps (detrend.py and reduceArc.py)?

AR Ticket filed (INFRA-55)


RHL2 If we are only processing one arc, why specify it this way, not by visit, e.g.

RHL2 detrend.py $SPECTRA_DIR --rerun USERNAME/tmp --id visit=4

RHL2 If you want to specify reducing all r2 arcs taken on 2015-12-22, please explain
detrend.py . --calib CALIB --output . --id arm=r spectrograph=2 dateObs=2015-12-22 field=ARC

AR Replaced “arm=r spectrograph=2 dateObs=2015-12-22 field=ARC” with “visit=4”, added comment on how to reduce all Arcs


RHL2 What files are produced?

AR See next line

• We now have the postISRCCD images and can extract and wavelength-calibrate our CdHgKrNeXe Arc with the visit
number 4:
RHL Why do I need the --wLenFile? If it's calibration, why isn't it in the calibration registy? Or maybe

RHL in obs_pfs?

RHL Why isn't the line list in the calib registry?

RHL2 Actually I see that I don't. What are the pros and cons?

AR Removed wLenFile and lineList as they are defaulted. Ticket filed (INFRA-56)


reduceArc.py . --output . --calib CALIB --id visit=4 --wLenFile $OBS_PFS_DIR/pfs/RedFiberPixels.fits.gz --lineList $OBS_PFS_DIR/pfs/lineLists/CdHgKrNeXe_red.fits
RHL2 A generic comment: far too much information is being printed to the screen! At -L warn there should be basically nothing; the LSST code doesn't obey this dictum very well, but drp_stella is worse!

AR Filed individual tickets

See tickets/INFRA-36 for obs_pfs, drp_stella, drp_stella_data

Thanks for taking the time to respond to all those comments in detail.

I've added a few more to the drp_stella PR (and I see rhl has done the same). They're mostly fairly minor, though: I don't think they'll take long, and they'll help make things a bit clearer. After resolving them, my suggestion would be that we go ahead and merge this to master as "good enough for now", but continue to evolve and improve this document by filing tickets as we spot things that needs improving.

A couple of other points:

• Our workflow guidelines suggest that you should rebase and squash to create a logical history before merging. Certainly, we shouldn't have a series of "respond to XXX comments from RHL" commits! We also shouldn't see merges like 5868b8f on the ticket branch (but that will drop out when you rebase).

• In several places above, you write that you've asked Paul about something. Asking Paul is often a good idea, but can I suggest that for many of these questions it would actually be appropriate to put them on community.lsst.org so that others can respond (rather than relying on Paul for everything) and everybody can benefit from the answers? You could call Paul out with an "@" if you think he's particularly well qualified to comment.