https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-127 DRP 2-D Pipeline <p>Profile the pipeline to identify hotspots and low-hanging fruit.</p> PIPE2D-127

Profile the pipeline

Story Major Done Done price price Fri, 18 Nov 2016 18:34:03 +0000 Thu, 22 Jun 2017 20:30:35 +0000 Thu, 22 Jun 2017 20:30:35 +0000 0 3 <p>I'll post the details soon, but here are the main results:</p> <p>General gains could be made by looking at ISR:</p> <ul> <li><tt>calcEffectiveGain</tt>, <tt>setGain</tt> aren't useful in my opinion</li> <li>Statistics gets called a lot, for the same kind of need (background level)</li> <li>Need to investigate remaining parts (CCD assembly, interpolation) to see if we can squeeze some performance</li> </ul> <p><tt>pfs::drp::stella::findCenterPositionsOneTrace</tt> is calling <tt>pfs::drp::stella::math::insertSorted</tt>, which is calling <tt>std::vector::insert</tt>. In general, you should never use <tt>std::vector::insert</tt>. cplusplus.com says:</p> <blockquote> <p>Because vectors use an array as their underlying storage, inserting elements in positions other than the vector end causes the container to relocate all the elements that were after position to their new positions. This is generally an inefficient operation compared to the one performed for the same operation by other kinds of sequence containers (such as list or forward_list).</p></blockquote> <p>Instead, push everything onto the end of the <tt>std::vector</tt> and then use <tt>std::sort</tt> at the end. Or use a list instead of an array, but that may have performance implications in other places.</p> <p>Cache the <tt>end()</tt> iterator in loops over <tt>ndarray</tt> (especially 2D arrays, which have more complicated iterators) because that's showing up in the profiles.</p> <p>In <tt>ReduceArcTask</tt>, this snippet is using the lion's share of the runtime:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Line # Hits Time Per Hit % Time Line Contents ============================================================== 165 30412812 67429737 2.2 30.5 <span class="code-keyword">for</span> j in range(traceIds.shape[0]): 166 30412800 72818425 2.4 32.9 <span class="code-keyword">if</span> traceIds[j] != l: 167 7200 16710 2.3 0.0 l = traceIds[j] 168 30412800 74841313 2.5 33.8 <span class="code-keyword">if</span> traceIds[j] == traceIdsUnique[traceId]: 169 50688 125401 2.5 0.1 wLenTemp[k] = wavelengths[j] 170 50688 116528 2.3 0.1 k = k+1 </pre> </div></div> <ul> <li><tt>l</tt> isn't used for anything, so can immediately save 33%.</li> <li>Use an iterator for clarity and to save dereferencing everything, e.g., <tt>for trace in traceIds</tt></li> <li>Iterating over individual vector (or image) values in python is something to avoid like the plague.</li> </ul> <p>The code is clearer and faster (18.774 sec --&gt; 6.077 sec) if written as:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> indices = np.where(traceIds == traceIdsUnique[traceId])[0] <span class="code-keyword">assert</span> len(indices) == len(wLenTemp) wLenTemp[:] = wavelengths[indices] </pre> </div></div> <p>(The <tt>assert</tt> is there because it wasn't obvious to me that the two vectors should have the same length &#8212; I suggest checking the original assignment of <tt>wLenTemp</tt>. In fact, you could drop the <tt>assert</tt>, the original assignment of <tt>wLenTemp</tt> and the <tt>[:]</tt> in the above and it should work just fine, even a tad faster.)</p> <p>Some additional thoughts:</p> <ul> <li>fiberTrace files are being read using pyfits. Can we use <tt>lsst::afw::table</tt> instead? pyfits is notoriously slow.</li> <li>Why is there a single fiberTrace for all time, rather than versioning it like a bias or flat?</li> </ul> <p>Oh, and I forgot to mention, constructDark is taking a long time hunting cosmics (190 of 224 sec), which seems excessive.</p> <p>Details are attached in <tt>profile.txt</tt>.</p> <p>Code changes made as part of this work:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> price@price-laptop:~/PFS/pfs_pipe2d (tickets/PIPE2D-127=) $ git sub commit 116be6508ce3f6f0f201fce6e458b279aadfaa3e Author: Paul Price &lt;price@astro.princeton.edu&gt; Date: Thu Dec 1 15:06:37 2016 -0500 add support <span class="code-keyword">for</span> python profiling in integration test bin/pfs_integration_test.sh | 45 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) commit 79bd9dd1d94526023d5d07f25884ef221dc74bef Author: Paul Price &lt;price@astro.princeton.edu&gt; Date: Fri Dec 2 08:46:07 2016 -0500 travis: build with 2 cores bin/pfs_travis.sh | 2 ++ 1 file changed, 2 insertions(+) </pre> </div></div> <p>And:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> price@price-laptop:~/PFS/obs_pfs (tickets/PIPE2D-127=) $ git sub commit 2377391fc9c0875d8914eee93611b8154e7e2ebf Author: Paul Price &lt;price@astro.princeton.edu&gt; Date: Thu Dec 1 17:19:25 2016 -0500 ingest: clean up <span class="code-keyword">for</span> style python/lsst/obs/pfs/ingest.py | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) </pre> </div></div> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a>, could you please verify this is the level of detail you were after? I haven't yet made tickets to implement these suggestions. Would you like multiple tickets or one ticket to rule them all?</p> <p>That's useful. I don't know how <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=swinbank" class="user-hover" rel="swinbank">swinbank</a> wants to track this, but probably creating the tickets would be useful</p> <p>Please go ahead and create multiple tickets.</p> <p>I think we need to close this &#8211; you did the work. I'll open a new ticket to repeat the work with multiprocessing turned off (<tt>--batch-type None</tt>) and hope/pray that some of the hot spots have been eliminated.</p> <p>Also, we should process the <tt>runPipeline</tt> script.</p> Development Rank 0|ii011b: Reviewers rhl Sprint 2014-16