https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/FIBERALLOC-42 Target to fiber allocation and configuration <p>As we discussed back in April/May, we would like to distribute the standard fibers and sky fibers uniformly in the FoV.  I vaguely remember that netflow did not have that capability, but it would be trivial to implement it (correct me if I am wrong). He-san recently pointed out that I did not file a ticket for that. So, here it is.</p> FIBERALLOC-42

distribute calib object uniformly over the FoV in netflow

Task Normal Done Done Martin Reinecke Masayuki Tanaka Tue, 5 Oct 2021 02:22:18 +0000 Tue, 7 Mar 2023 09:06:57 +0000 Tue, 7 Mar 2023 09:06:56 +0000 0 6 <p>I will try to add this capability, but to start I'll need a quantification of what "uniform" means, such that we can formulate it as additional constraints to the netflow optimizer.</p> <p>As far as I recall, there was also the additional constraint that calibration targets are uniformly distributed along the slits of the spectrographs. For this, we also need a clear specification what this means.</p> <p>Dear Martin, dear all, yes we should be able to model this in the netflow framework.</p> <p>I would propose that the the decouple the implementation side (how do we do this in netflow) from the actual practical side (which parts of the focal plane should be covered how and what are the specific fiber to spectrograph&amp;slit mappings).</p> <p>The simple thing to do is to subdivide the calibrators (=sky) according to whether they fall into a particular subregion of the focal plane. This really would just subdivide the calibrator target call into subclasses, directly equivalent to the different priority classes for galaxies. This is really easy way to solve the problem to good approximation, but it is not precise because calibrators may be reachable by multiple cobras, which belong to different spectrographs even.</p> <p>BUT I think we can actually solve it precisely:</p> <p>For this we would define a generic set of groups of cobras. The list of groups should be dynamic and we should be reading it from an ascii file. My gut feeling is that we will probably have more than 10 and less than 50 such groups. Also we should define a minimum required number of calibrators per group (=N, this might be the same number for all groups but really we can just as easily have one number per group in the same ascii file.</p> <p>netflow should now enforce that there is at least a than minimum number of calibrators allocated per group by adding the term</p> <p>(N - input_flow) x LARGE_COST to the cost function for each group. Input_flow is the sum over all calibrator flows into a group node. Now for this to work we need to distinguish between galaxy-flow and calibrator flow which is a bit of a rewrite of netflow but well possible.  </p> <p> </p> <p>Finally for each cobra group node we need to limit the input flow to N by adding another constraint equation:</p> <p>input_flow &lt;= N</p> <p> </p> <p>This will be generic and allow us to distribute cobras across fibers/spectrographs/slits in whatever-way we like.</p> <p> </p> <p>All the information for the spatial uniformity of the calibs is already present, so the input would be the definition of the regions which define the groups. The mapping of cobras to positions along the slits is present in the <tt>FiberId</tt> object in <tt>fps_utils</tt>, so that's also known to Netflow (at least in theory). I believe that <tt>pfs_utils</tt> brings in few or no extra dependencies, but that should be checked.</p> <p>Perfect, this should make the start easy then. I added a sketch of the proposed solution here</p> <p><span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/14159/14159_skyfiber-dist.pdf" title="skyfiber-dist.pdf attached to FIBERALLOC-42">skyfiber-dist.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>^{and can walk through it tomorrow if useful.}</p> <p> </p> <p>I have been thinking about this some more, and I think we can perhaps formulate this without adding more flow variables:</p> <p>when building the problem, we are adding the "Tv_Cv" flows to the system, and for every one of these flows we know if it comes from a sky calib target and we can also determine to which "location group" the corresponding Cobra belongs. So, let's assume we are adding a flow going from a sky calib target to, say, a Cobra in location group 5: can't we just add this flow to a new constraint for location group 5 and require that the sum of this constraint is &gt;= some user-defined limit?</p> <p>(If we don't want a hard limit but rather penalize not reaching the required number, we can add an "overflow source" with a high cost, right?)</p> <p>Does this make sense? A first sketch is on branch "tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/FIBERALLOC-42" title="distribute calib object uniformly over the FoV in netflow" class="issue-link" data-issue-key="FIBERALLOC-42"><del>FIBERALLOC-42</del></a>".</p> <p> </p> <p>Sorry, my proposal wasn't complete. We will need one additional constraint per location group <em>per exposure</em>. I haven't taken this into account yet, but this should not make the implementation harder.</p> <p>I have now added overflow arcs, and I believe that the code now does (in principle) what it s supposed to do.</p> <p>We still need to discuss how to pass the required additional information to the assigner. As far as I see, we need</p> <ul> <li>the mapping from Cobra index to region index (integer array)</li> <li>the minimum number of sky targets per region</li> <li>the penalty cost for every necessary sky fiber that we didn't manage to allocate</li> </ul> <p>Adding something analogous for spectrograph slit regions should be straightforward once we have nailed down the solution for the focal plane regions.</p> <p>I think that <tt>the mapping from Cobra index to region index (integer array)</tt> can come from the Grand Fibre Map which is available from <tt>pfs_utils</tt>. Is that a dependency for you?</p> <p>The assigner already depends on `pfs.utils`, so this won't cause any additional dependencies. The contents of the Grand Fibre Map look extremely useful for someone defining the regions, but I think someone still needs to decide how many regions to use and which Cobras actually fall into them.</p> <p>That is to say, someone still has to build this integer array and pass it to the assigner ... or is your suggestion to use a single, hard-wired one?</p> <p>Do you think that my general approach makes sense, or do you see any complication I missed? I'm still a bit worried because Max thought we have to (almost) double the number of flow variables, and I seem to manage with just adding {{nvisit*nregion }}extra constraints ...</p> <p>I pushed a few fixes and added a very basic "demo_netflow_regions.py" script.</p> <p> </p> <p>OK, this seems to work just beautifully.  Please see the attached slides. I also added my modifications to the demo script. </p> <p>From my point of view we should discuss merging this to master in the next meeting and also modify commissioning.py to include this.</p> <p> </p> <p><span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/15084/15084_20220124_region_based_fiberallocation.pdf" title="20220124_region_based_fiberallocation.pdf attached to FIBERALLOC-42">20220124_region_based_fiberallocation.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p> </p> <p> </p> <p>Thank you for the extensive testing!</p> <p> </p> <p>As we discussed, there is now analogous functionality for distributing sky targets evenly along the spectrograph slits as well.</p> <p>Tested, again works beautifully, slides and test code attached.</p> <p> </p> <p>Added analysis code to </p> <p>./misc/test_codes/demo_netflow_regions_with_analysis.py </p> <p> </p> <p> <span class="nobr"><a href="https://pfspipe.ipmu.jp/jira/secure/attachment/15085/15085_20220127_region_based_fiberallocation.pdf" title="20220127_region_based_fiberallocation.pdf attached to FIBERALLOC-42">20220127_region_based_fiberallocation.pdf^{<img class="rendericon" src="https://pfspipe.ipmu.jp/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p> </p> <p> </p> <p>Unless there are objections, I propose to merge and close this at the end of the week.</p> <p>I agree entirely!   </p> <p>Merged!</p> <p>I deleted `requirements.txt` from master for the time being, since it pulls in more packages than strictly needed for the assigner.</p> <p>This has the status "review complete", but at the same the resolution is "unresolved", which is a bit confuing.<br/> I think the issue can be closed, but I don't have a button in my Jira interface to do it.</p> <p>I just realised that your account was created Jira internally, not via LDAP... so I need to look into it.</p> <p>I'm sorry, I just realised how it can be done: I just set the workflow to "Done", but don't select any reviewers. That seems to do the trick!</p> Development Epic Link OBSPROC-5 Rank 0|zzsa3s: