[PIPE2D-677] generateReductionSpec produces detectorMap blocks with hundreds of visits Created: 10/Dec/20 Updated: 29/Jan/22 Resolved: 12/Jan/21 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | price | Assignee: | sogo.mineo |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Reviewers: | price | ||||||||
| Description |
|
generateReductionSpec.py currently puts hundreds of visits in a single detectorMap block (example below). This does not reflect how we want to process the data, so these blocks should be broken up into individual runs. It's not entirely clear how to identify those runs. For the present, let's make it a continguous set of visits, and restrict the maximum length to 10 visits.
- name: calib_r_59129.655758_0000010001001010
detectorMap:
id:
- visit=29583..29586^29590..29593^29597..29600^29604..29607^29611..29614^29618..29621^29625..29628^29632..29635^29639..29642^29646..29649^29653..29656^29660..29663^29667..29670^29674..29677^29681..29684^29688..29691^29695..29698^29702..29705^29709..29712^29716..29719^29723..29726^29730..29733^29737..29740^29744..29747^29751..29754^29758..29761^29765..29768^29772..29775^29779..29782^29786..29789^29793..29796^29800..29803^29807..29810^29814..29817^29821..29824^29828..29831^29835..29838^29842..29845^29849..29852^29856..29859^29863..29866^29870..29873^29877..29880^29884..29887^29891..29894^29898..29901^29905..29908^29912..29915^29919..29922^29926..29929^29933..29936^29940..29943^29947..29950^29954..29957^29961..29964^29968..29971^29975..29978^29982..29985^29989..29992^29996..29999^30003..30006^30010..30013^30017..30020^30024..30027^30031..30034^30038..30041^30045..30048^30052..30055^30059..30062^30066..30069^30073..30076^30080..30083^30087..30090^30094..30097^30101..30104^30108..30111^30115..30118^30122..30125^30129..30132^30136..30139^30143..30146^30150..30153^30157..30160^30164..30167^30171..30174^30178..30181^30185..30188^30192..30195^30199..30202^30206..30209^30213..30216^30220..30223^30227..30230^30234..30237^30241..30244^30248..30251^30255..30258^30262..30265^30269..30272^30276..30279^30283..30286^30290..30293^30297..30300^30304..30307^30311..30314^30318..30321^30325..30328^30332..30335^30339..30342^30346..30349^30353..30356^30360..30363^30367..30370^30374..30377^30381..30384^30388..30391^30395..30398^30402..30405^30409..30412^30416..30419^30423..30426^30430..30433^30437..30440^30444..30447^30451..30454^30458..30461^30465..30468^30472..30475^30479..30482^30486..30489^30493..30496^30500..30503^30507..30510^30514..30517^30521..30524^30528..30531^30535..30538^30542..30545^30549..30552^30556..30559^30563..30566^30570..30573^30577..30580^30584..30587^30591..30594^30598..30601^30605..30608^30612..30615^30619..30622^30626..30629^30633..30636^30640..30643^30647..30650^30654..30657^30661..30664^30668..30671^30675..30678^30682..30685^30689..30692^30696..30699^30703..30706^30710..30713^30717..30720^30724..30727^30731..30734^30738..30741^30745..30748^30752..30755^30759..30762^30766..30769^30773..30776^30780..30783^30787..30790^30794..30797^30801..30804^30808..30811^30815..30818^30822..30825^30829..30832^30836..30839^30843..30846^30850..30853^30857..30860^30864..30867^30871..30874^30878..30881^30885..30888^30892..30895^30899..30902^30906..30909^30913..30916^30920..30923^30927..30930^30934..30937^30941..30944^30948..30951^30955..30958^30962..30965^30969..30972^30976..30979^30983..30986^30990..30993^30997..31000^31004..31007^31011..31014^31018..31021^31025..31028^31032..31035^31039..31042^31046..31049^31053..31056^31060..31063^31067..31070^31074..31077^31081..31084^31088..31091^31095..31098^31102..31105^31109..31112^31116..31119^31123..31126^31130..31133^31137..31140^31144..31147^31151..31154^31158..31161^31165..31168^31172..31175^31179..31182^31186..31189^31193..31196^31200..31203^31207..31210^31214..31217^31221..31224^31228..31231^31235..31238^31242..31245^31249..31252^31256..31259^31263..31266^31270..31273^31277..31280^31284..31287^31291..31294^31298..31301^31305..31308^31312..31315^31319..31322^31326..31329^31333..31336^31340..31343^31347..31350^31354..31357^31361..31364^31368..31371^31375..31378^31382..31385^31389..31392^31396..31399^31403..31406^31410..31413^31417..31420^31424..31427^31431..31434^31438..31441^31445..31448^31452..31455^31459..31462^31466..31469^31473..31476^31480..31483^31487..31490^31494..31497^31501..31504^33109..33112^33116..33119^33123..33126^33130..33133^33137..33140^33144..33147^33151..33154^33158..33161^33165..33168^33172..33175^33179..33182^33186..33189^33193..33196^33200..33203^33207..33210^33214..33217^33221..33224^33228..33231^33235..33238^33242..33245^33249..33252^33256..33259^33263..33266^33270..33273^33277..33280^33284..33287^33291..33294^33298..33301^33305..33308^33312..33315^33319..33322^33326..33329^33333..33336^33340..33343^33347..33350^33354..33357^33361..33364^33368..33371^33375..33378^33382..33385^33389..33392^33396..33399^33403..33406^33410..33413^33417..33420^33424..33427^33431..33434^33438..33441
- arm=r
- spectrograph=1
|
| Comments |
| Comment by rhl [ 10/Dec/20 ] |
|
We discussed a value in the DB that would change when the configuration was changed, identifying blocks of data to be processed together. See https://sumire-pfs.slack.com/archives/C3AU3VCMU/p1602162363017900 and I think beam_config_date is the key, but I'd need to check |
| Comment by price [ 10/Dec/20 ] |
|
Yes, this is already respecting beam_config_date. |
| Comment by rhl [ 10/Dec/20 ] |
|
Then I don't understand why we are having a problem. Is this old data for which beam_config_date wasn't set? |
| Comment by price [ 10/Dec/20 ] |
|
I believe the particular example above contains a stability test, where we took lots of arcs over a period with the same beam. But we don't want to process all of those visits into a single detectorMap. |
| Comment by sogo.mineo [ 07/Jan/21 ] |
|
I added --maxarcs option which defaults to 10. Could you review it? |
| Comment by price [ 08/Jan/21 ] |
|
Your implementation takes a long list of arcs and returns a list of arcs with a maximum length of N. Instead, I think we need to return multiple lists (and generate separate reduceArc runs on each), where each list consists of a contiguous set of arcs with a maximum length of N. For example, the visit list 1,2,3,4,5,6,7,8,11,12,13,14 and N=3 should return (1,2,3),(4,5,6),(7,8),(11,12),(13,14) instead of just (1,2,3). |
| Comment by sogo.mineo [ 08/Jan/21 ] |
|
If the number of contiguous arcs divided by N leaves one, how should I treat the remainder? Should I also construct a detectorMap from the single arc, or should I discard such a remainder if it is less than a certain limit? |
| Comment by price [ 08/Jan/21 ] |
|
I think the list should be divided evenly as much as possible (e.g., my list of 11,12,13,14 above divided into two sets of two rather than 3+1). If that leaves a single remainder then I think that's probably due to low N, and we should just accept it. |
| Comment by rhl [ 08/Jan/21 ] |
|
Why wouldn't we merge the remainder into one of the blocks? I don't think we need N to be the same in all blocks. |
| Comment by sogo.mineo [ 08/Jan/21 ] |
|
I pushed a new version. If N (=maxarcs) is 5, then 6 is split into 3+3, and 11 is split into 4+4+3. |
| Comment by price [ 09/Jan/21 ] |
|
Looks good, thanks! |
| Comment by sogo.mineo [ 12/Jan/21 ] |
|
Merged to master. Thank you. |