https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-677 DRP 2-D Pipeline <p><tt>generateReductionSpec.py</tt> currently puts hundreds of visits in a single <tt>detectorMap</tt> block (example below). This does not reflect how we want to process the data, so these blocks should be broken up into individual runs.</p> <p>It's not entirely clear how to identify those runs. For the present, let's make it a continguous set of visits, and restrict the maximum length to 10 visits.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - name: calib_r_59129.655758_0000010001001010 detectorMap: id: - visit=29583..29586^29590..29593^29597..29600^29604..29607^29611..29614^29618..29621^29625..29628^29632..29635^29639..29642^29646..29649^29653..29656^29660..29663^29667..29670^29674..29677^29681..29684^29688..29691^29695..29698^29702..29705^29709..29712^29716..29719^29723..29726^29730..29733^29737..29740^29744..29747^29751..29754^29758..29761^29765..29768^29772..29775^29779..29782^29786..29789^29793..29796^29800..29803^29807..29810^29814..29817^29821..29824^29828..29831^29835..29838^29842..29845^29849..29852^29856..29859^29863..29866^29870..29873^29877..29880^29884..29887^29891..29894^29898..29901^29905..29908^29912..29915^29919..29922^29926..29929^29933..29936^29940..29943^29947..29950^29954..29957^29961..29964^29968..29971^29975..29978^29982..29985^29989..29992^29996..29999^30003..30006^30010..30013^30017..30020^30024..30027^30031..30034^30038..30041^30045..30048^30052..30055^30059..30062^30066..30069^30073..30076^30080..30083^30087..30090^30094..30097^30101..30104^30108..30111^30115..30118^30122..30125^30129..30132^30136..30139^30143..30146^30150..30153^30157..30160^30164..30167^30171..30174^30178..30181^30185..30188^30192..30195^30199..30202^30206..30209^30213..30216^30220..30223^30227..30230^30234..30237^30241..30244^30248..30251^30255..30258^30262..30265^30269..30272^30276..30279^30283..30286^30290..30293^30297..30300^30304..30307^30311..30314^30318..30321^30325..30328^30332..30335^30339..30342^30346..30349^30353..30356^30360..30363^30367..30370^30374..30377^30381..30384^30388..30391^30395..30398^30402..30405^30409..30412^30416..30419^30423..30426^30430..30433^30437..30440^30444..30447^30451..30454^30458..30461^30465..30468^30472..30475^30479..30482^30486..30489^30493..30496^30500..30503^30507..30510^30514..30517^30521..30524^30528..30531^30535..30538^30542..30545^30549..30552^30556..30559^30563..30566^30570..30573^30577..30580^30584..30587^30591..30594^30598..30601^30605..30608^30612..30615^30619..30622^30626..30629^30633..30636^30640..30643^30647..30650^30654..30657^30661..30664^30668..30671^30675..30678^30682..30685^30689..30692^30696..30699^30703..30706^30710..30713^30717..30720^30724..30727^30731..30734^30738..30741^30745..30748^30752..30755^30759..30762^30766..30769^30773..30776^30780..30783^30787..30790^30794..30797^30801..30804^30808..30811^30815..30818^30822..30825^30829..30832^30836..30839^30843..30846^30850..30853^30857..30860^30864..30867^30871..30874^30878..30881^30885..30888^30892..30895^30899..30902^30906..30909^30913..30916^30920..30923^30927..30930^30934..30937^30941..30944^30948..30951^30955..30958^30962..30965^30969..30972^30976..30979^30983..30986^30990..30993^30997..31000^31004..31007^31011..31014^31018..31021^31025..31028^31032..31035^31039..31042^31046..31049^31053..31056^31060..31063^31067..31070^31074..31077^31081..31084^31088..31091^31095..31098^31102..31105^31109..31112^31116..31119^31123..31126^31130..31133^31137..31140^31144..31147^31151..31154^31158..31161^31165..31168^31172..31175^31179..31182^31186..31189^31193..31196^31200..31203^31207..31210^31214..31217^31221..31224^31228..31231^31235..31238^31242..31245^31249..31252^31256..31259^31263..31266^31270..31273^31277..31280^31284..31287^31291..31294^31298..31301^31305..31308^31312..31315^31319..31322^31326..31329^31333..31336^31340..31343^31347..31350^31354..31357^31361..31364^31368..31371^31375..31378^31382..31385^31389..31392^31396..31399^31403..31406^31410..31413^31417..31420^31424..31427^31431..31434^31438..31441^31445..31448^31452..31455^31459..31462^31466..31469^31473..31476^31480..31483^31487..31490^31494..31497^31501..31504^33109..33112^33116..33119^33123..33126^33130..33133^33137..33140^33144..33147^33151..33154^33158..33161^33165..33168^33172..33175^33179..33182^33186..33189^33193..33196^33200..33203^33207..33210^33214..33217^33221..33224^33228..33231^33235..33238^33242..33245^33249..33252^33256..33259^33263..33266^33270..33273^33277..33280^33284..33287^33291..33294^33298..33301^33305..33308^33312..33315^33319..33322^33326..33329^33333..33336^33340..33343^33347..33350^33354..33357^33361..33364^33368..33371^33375..33378^33382..33385^33389..33392^33396..33399^33403..33406^33410..33413^33417..33420^33424..33427^33431..33434^33438..33441 - arm=r - spectrograph=1 </pre> </div></div> PIPE2D-677

generateReductionSpec produces detectorMap blocks with hundreds of visits

Story Normal Done Done sogo.mineo price Wed, 9 Dec 2020 16:30:17 +0000 Fri, 28 Jan 2022 17:16:25 +0000 Tue, 12 Jan 2021 03:00:33 +0000 0 2 <p>We discussed a value in the DB that would change when the configuration was changed, identifying blocks of data to be processed together. See <a href="https://sumire-pfs.slack.com/archives/C3AU3VCMU/p1602162363017900" class="external-link" rel="nofollow">https://sumire-pfs.slack.com/archives/C3AU3VCMU/p1602162363017900</a> and <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-1026" title="Add counter for physical SM configuration changes" class="issue-link" data-issue-key="INSTRM-1026"><del>INSTRM-1026</del></a></p> <p>I think <tt>beam_config_date</tt> is the key, but I'd need to check</p> <p>Yes, this is already respecting <tt>beam&#95;config&#95;date</tt>.</p> <p>Then I don't understand why we are having a problem. Is this old data for which <tt>beam_config_date</tt> wasn't set?</p> <p>I believe the particular example above contains a stability test, where we took lots of arcs over a period with the same beam. But we don't want to process all of those visits into a single <tt>detectorMap</tt>.</p> <p>I added --maxarcs option which defaults to 10. Could you review it?</p> <p>Your implementation takes a long list of arcs and returns a list of arcs with a maximum length of <tt>N</tt>. Instead, I think we need to return multiple lists (and generate separate <tt>reduceArc</tt> runs on each), where each list consists of a contiguous set of arcs with a maximum length of <tt>N</tt>. For example, the visit list <tt>1,2,3,4,5,6,7,8,11,12,13,14</tt> and N=3 should return <tt>(1,2,3),(4,5,6),(7,8),(11,12),(13,14)</tt> instead of just <tt>(1,2,3)</tt>.</p> <p>If the number of contiguous arcs divided by N leaves one, how should I treat the remainder? Should I also construct a detectorMap from the single arc, or should I discard such a remainder if it is less than a certain limit?</p> <p>I think the list should be divided evenly as much as possible (e.g., my list of <tt>11,12,13,14</tt> above divided into two sets of two rather than 3+1). If that leaves a single remainder then I think that's probably due to low <tt>N</tt>, and we should just accept it.</p> <p>Why wouldn't we merge the remainder into one of the blocks?  I don't think we need N to be the same in all blocks.</p> <p>I pushed a new version. If N (=maxarcs) is 5, then 6 is split into 3+3, and 11 is split into 4+4+3.</p> <p>Looks good, thanks!</p> <p>Merged to master. Thank you.</p> Relates PIPE2D-423 Development Rank 0|02qpq3:r1b Reviewers price