https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1038 DRP 2-D Pipeline <p>The Gen3 middleware will allow us to run our pipeline over multiple nodes. We need a middleware plugin that will work with our Slurm (or PBS) clusters. The leading contender is <a href="https://github.com/LSSTDESC/gen3_workflow" class="external-link" rel="nofollow">the parsl plugin developed by DESC</a>.</p> PIPE2D-1038

Investigate parsl plugin for processing on cluster

Story Normal Done Done price price Mon, 2 May 2022 17:36:44 +0000 Fri, 8 Jul 2022 17:12:54 +0000 Fri, 8 Jul 2022 17:12:54 +0000 0 1 <p>Running BPS on Tiger:</p> <ul> <li>Stack requirements:<br/> + Recent LSST stack<br/> + tickets/DM-35494 of ctrl_mpexec and pipe_base<br/> + ctrl_bps_parsl<br/> + Parsl (see ctrl_bps_parsl README)</li> <li>Docs:<br/> + BPS guide: <a href="https://pipelines.lsst.io/modules/lsst.ctrl.bps/index.html" class="external-link" rel="nofollow">https://pipelines.lsst.io/modules/lsst.ctrl.bps/index.html</a><br/> + Parsl plugin guide: <a href="https://github.com/lsst/ctrl_bps_parsl/blob/main/README.md" class="external-link" rel="nofollow">https://github.com/lsst/ctrl_bps_parsl/blob/main/README.md</a></li> <li>The choice of filesystem is very important for efficient cluster use: see <a href="https://researchcomputing.princeton.edu/support/knowledge-base/data-storage" class="external-link" rel="nofollow">https://researchcomputing.princeton.edu/support/knowledge-base/data-storage</a> . <tt>/projects</tt> and <tt>/tigress</tt> are slow to access on cluster nodes, so you should work on <tt>/scratch/gpfs</tt> and include <tt>execution_butler_copy_files.yaml</tt> in your BPS config. This copies the necessary data from the primary butler repo on <tt>/projects</tt> to an “execution butler” in your working directory (on <tt>/scratch/gpfs</tt>), and copies the results back afterwards. This adds some overhead to the beginning and end of the processing, but it allows the processing to be efficient (and not degrade the system for other users).</li> <li>When running flat construction, got about 100 ISR jobs/hour/core.</li> <li>Consider strategies to obtain cluster resources:<br/> + Parsl launches Slurm jobs to obtain cluster resources; these jobs connect to the process running on the head node and execute the various tasks.<br/> + Don’t want to have more resources than can be used at once, but don’t want to be waiting a long time for resources.<br/> + Smaller and shorter jobs are easier to schedule, but are less capable and introduce overhead.<br/> + One strategy might be to use a 6 hour “singleton” job, so only one job is allowed to run at a time, while other jobs hold their place in line.<br/> + Another strategy might be to use multiple 2 hour jobs, and then you use those jobs as they become available.</li> <li>Clusters make the pipeline more efficient: when running DRP, I recommend including <tt>${DRP_PIPE_DIR}/ingredients/clusters.yaml</tt> (requires branch u/price/20220628 of drp_pipe).</li> <li>Example BPS config file for flat construction: <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pipelineYaml: <span class="code-quote">"${CP_PIPE_DIR}/pipelines/DarkEnergyCamera/cpFlat.yaml"</span> wmsServiceClass: lsst.ctrl.bps.parsl.ParslService #computeSite: local computeSite: tiger includeConfigs: - ${CTRL_BPS_PARSL_DIR}/etc/execution_butler_copy_files.yaml payload: output: u/price/20220629/flat butlerConfig: /projects/MERIAN/repo inCollection: DECam/defaults/merian,DECam/calib/curated/19700101T000000Z,DECam/calib/unbounded,u/price/20220629/calib dataQuery: <span class="code-quote">"instrument=<span class="code-quote">'DECam'</span> AND exposure IN (970228, 970229, 970230, 970231, 970232, 970233, 970234, 970235, 970236, 970237, 970238, 970501, 970502, 970503, 970504, 970505, 970506, 970507, 970508, 970509, 970510, 970511, 970836, 970837, 970838, 970839, 970840, 970841, 970842, 970843, 970844, 970845, 970846, 971174, 971175, 971176, 971177, 971178, 971179, 971180, 971181, 971182, 971183, 971184, 971554, 971555, 971556, 971557, 971558, 971559, 1052706, 1052707, 1052708, 1052709, 1052710, 1052711, 1052712, 1052713, 1052714, 1052715, 1052716, 1053093, 1053094, 1053095, 1053096, 1053097, 1053098, 1053099, 1053100, 1053101, 1053102, 1053103, 1053485, 1053486, 1053487, 1053488, 1053489, 1053490, 1053491, 1053492, 1053493, 1053494, 1053495, 1053858, 1053859, 1053860, 1053861, 1053862, 1053863, 1053864, 1053865, 1053866, 1053867, 1053868, 1054287, 1054288, 1054289, 1054290, 1054291, 1054292)"</span> payloadName: flat clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering saveClusteredQgraph: <span class="code-keyword">true</span> cluster: exposure_detector: pipetasks: isr,cpFlatMeasure dimensions: exposure,detector site: local: class: lsst.ctrl.bps.parsl.sites.Local cores: 12 tiger: class: lsst.ctrl.bps.parsl.sites.princeton.Tiger nodes: 1 cores_per_node: 40 walltime: <span class="code-quote">"0:59:59"</span> # Get into tiger-test queue </pre> </div></div></li> <li>Example BPS config file for DRP on ci_hsc: <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> includeConfigs: - ${DRP_PIPE_DIR}/ingredients/clusters.yaml pipelineYaml: <span class="code-quote">"${DRP_PIPE_DIR}/pipelines/HSC/DRP-ci_hsc.yaml"</span> wmsServiceClass: lsst.ctrl.bps.parsl.ParslService computeSite: local #computeSite: tiger site: local: class: lsst.ctrl.bps.parsl.sites.Local cores: 20 tiger: class: lsst.ctrl.bps.parsl.sites.princeton.Tiger nodes: 1 walltime: <span class="code-quote">"0:59:59"</span> </pre> </div></div></li> </ul> <p>The parsl plugin works nicely. I extracted the useful parts into a plugin package: <a href="https://github.com/lsst/ctrl_bps_parsl" class="external-link" rel="nofollow">ctrl&#95;bps&#95;parsl</a>, and was able to run workflows on the Princeton Tiger cluster.</p> Development Rank 0|02qpjd:00xznup2c5 Sprint 2DDRP-2022 E