https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/FIBERALLOC-28 Target to fiber allocation and configuration <p>Shuffle should select suitable guide stars in each of the six PFS guide cameras. </p> <p> </p> <p>The primary catalog will be GAIA and shuffle should automatically query GAIA (online) for each setup.</p> <p> </p> <p>But also shuffle must allow for other catalogs (sdss, vizier) and manually supplied catalogs (e.g. from HSC pre-imaging).</p> <p> </p> <p>Magnitude cutoffs, and minimum separation criteria should be freely configurable in the config file.</p> FIBERALLOC-28

Update code to find guide star candidates in guide camera footprints

Task Normal Done Done Martin Reinecke mxhf Fri, 10 Jul 2020 13:27:57 +0000 Wed, 3 Nov 2021 08:24:45 +0000 Wed, 3 Nov 2021 08:24:45 +0000 ets_shuffle 0 4 <p>I'm trying to formulate the necessary queries to the star catalogs.<br/> Are there any rough guidelines on how we can constrain the queries, so that we do not stress the servers unnecessarily? I'm thinking of</p> <ul class="alternate" type="square"> <li>what is the minimum magnitude</li> <li>which band should be use</li> <li>any other exclusion criteria?</li> </ul> <p>Also, if we decide to use GAIA when looking for guide star candidates, this would be convenient, since `astroquery.gaia` allows ADQL queries, and I could directly use the polygonal shapes of the guide cam FOVs in these queries. As far as I understood, `astroquery.vizier` (which is currently used by shuffle) only allows simple cone and box searches, so that I would have to do more post-processing on the results.</p> <p>Dear Martin,</p> <p> </p> <p>this is an incomplete answer. We should be able to configure the magnitude cutoffs. I think this functionality exists (somewhat crudely) in shuffle already. This should be per band. So for GAIA G,Gbp,Grb. The syntax of the current configuration file does not lend itself to this right away as it is tied to sdss magnitudes. </p> <p>Ideally we would allow for a freeform filter like "(G &gt; 10 &amp; G &lt; 24) " or similar.</p> <p> </p> <p>As for other limits. We may want to restrict too large peculiar motions (not sure) and close pairs.</p> <p>All TBD.</p> <p>Max</p> <p> </p> <p>Thank you very much! I think the freeform filter is not a problem since the constraints can be passed to the query as a string.</p> <p>I was mainly interested to have some approximate, but realistic numbers to play with, since (as a numerics person) I have absolutely no intuition what kind of magnitude range might be required. With the example you gave me I can be sure that I don't torture the server unduly.</p> <p>By the way, it seems that ADQL even has constraints which can filter out close pairs directly in the query, which could be extremely convenient!</p> <p>Hi Martin,</p> <p> </p> <p>great. The values in the config are actually for a 10m telescope. So they are probably good starting points.</p> <p> </p> <p>Max</p> <p> </p> <p>Small update: I managed to formulate an ADQL query which allows specifying a region of interest (circle or polygon) and magnitude limits, and which allows selecting only stars with proper motion data. Such a more complicated and flexible query is necessary, I think, because without some magnitude preselection the returned tables for the full FOV would just become too large.</p> <p>Using this query syntax we can</p> <ul> <li>get a (presumably short) list of uncomfortably bright stars in the full telescope FOV (expressed as a large circle), which we can test for proximity with any allocated fiber, and</li> <li>for each guide camera FOV (expressed as a polygon) obtain a list of guide star candidates (here the magnitude limits will most likely be different)</li> </ul> <p>I thought I had seen a way that would already eliminate close pairs on the server side when looking for guide stars, but unfortunately cannot find it again at the moment. In any case, doing that locally is not a big effort and does not have a big impact on the size of the query results.</p> <p>One problem with this approach is however that it only works for catalogs that support ADQL queries. So far I have only tested this for GAIA, but judging from pages like <a href="http://tapvizier.u-strasbg.fr/adql/" class="external-link" rel="nofollow">http://tapvizier.u-strasbg.fr/adql/</a> I assume that this also works for other catalogs if we should need them.</p> <p>We only care about proper motions for the stars that fall on the guide chips which are a pretty small area. Are you doing this for the entire focal plane in order to visualise the field? If so, can't we use a magnitude limit instead? </p> <p>Max's idea was to obtain a list of bright stars in the entire field to make sure no fiber was in close vicinity to one of those. It's no problem to use different queries for this purpose and for the small areas seen by the guide chips. In fact, I've just verified that polygon queries work fine for that purpose; proper motion could then be requested for the small-area queries only.</p> <p>I agree with Max about bright stars, although I don't think we know what the bright limit is yet. Hmm, we could put the brightest star on a black spot, but I don't know if that's actually better then letting its light scatter around in the PFI's guts.</p> <p>Do you happen to know if it is possible to put a magnitude limit to a simple Vizier cone search? At least in hetdex_shuffle, this wasn't done, and I couldn't find any examples for this on the net either. So I guessed that I would have to go to full-fledged ADQL... I'd be happy if I'm wrong, of course, because that makes using other catalogs much easier.</p> <p>There is a Vizier service at <a href="http://tapvizier.u-strasbg.fr/adql/" class="external-link" rel="nofollow">http://tapvizier.u-strasbg.fr/adql/</a> which allows full ADQL queries on all catalogs they have available, and that seems to include everything we need. I have proof-of-concept code which performs SDSS and GAIA queries using this service, and I verified that the GAIA query results match those from the official GAIA server.</p> <p>Using ADQL to access this server (and, as a backup, the official Gaia server) is a very elegant and powerful way to resolve this issue, but I see two drawbacks:</p> <ul> <li>if both of these servers are down simultaneously, I'm not sure if there is any other backup service (there might be more ADQL servers around, but I couldn't verify this so far)</li> <li>I don't know how to deal with manually supplied catalogs (e.g. in form of Python dictionaries) in this context</li> </ul> <p>There is now a script called `demo_shuffle.py` in branch `demo` of <a href="https://github.com/Subaru-PFS/ets_shuffle" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/ets_shuffle</a>, which showcases the queries, corrections, and spatial selections I intend to do in order to get the lists of guide star candidates. This is an isolated script with hardwired input and no output, since I don't yet know how the code will be integrated into the higher-level components (e.g. as a library via function call, or as a separate script with command line arguments, etc.).</p> <p>Please have a look if you have the time; any feedback is welcome!</p> <p>P.S.: to make this script easy to run, I copied a few files (which contain the coordinate conversion code) from `pfs_utils`, since I didn't manage to install this locally. This is temporary and will hopefully change soon!</p> <p>Additional question on "close pair" avoidance for guide star candidates:</p> <ul> <li>I assume that we also want to avoid close neighbors that are somewhat fainter than the magnitude limit for our guide stars (i.e. which would not be guide star candidates themselves). Is that correct?</li> <li>We probably should also exclude close pairs of which one star is outside the camera's field of view.</li> </ul> <p>If that is the case, the underlying queries will become a bit more complicated, but should still be fairly straightforward.<br/> Please let me know what you think! </p> <p>I think this is right in principle, but then if we select a guide star right on the edge of the field of view of a guide image we have done something wrong in the first place. An autoguider can not guide off a stare that falls right on the edge of a detector. </p> <p>Guide stars should only be allowed to fall into a footprint that is effectively a <b>bit</b> smaller than the guide camera footprint.</p> <p>In practice we will probably just pretend that the guid chips are a bit smaller than they are?  </p> <p>But so yes also guide stars at the edge of the field of view should not have a close neighbor outside of the field of view.</p> <p>Also yes on the magnitude limit, probably best again to define a parameter.</p> <p>Max</p> <p> </p> <p>Thanks, I'll make the necessary adjustments!</p> <p>Done!</p> <p> </p> <p>Getting the following error</p> <p>Installing collected packages: astroplan</p> <p>Successfully installed astroplan-0.6</p> <p>mxhf@lapex-07:~/work/MPE/pfs/src/ets_shuffle$ python demo_shuffle.py </p> <p>Traceback (most recent call last):</p> <p>  File "demo_shuffle.py", line 2, in &lt;module&gt;</p> <p>    from coordinates.CoordTransp import CoordinateTransform as ctrans</p> <p>  File "/Users/mxhf/ownCloudRZG/work/MPE/pfs/src/ets_shuffle/coordinates/CoordTransp.py", line 12, in &lt;module&gt;</p> <p>    import astroplan</p> <p>  File "/Users/mxhf/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/astroplan/_<em>init</em>_.py", line 32, in &lt;module&gt;</p> <p>    get_IERS_A_or_workaround()</p> <p>  File "/Users/mxhf/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/astroplan/utils.py", line 57, in get_IERS_A_or_workaround</p> <p>    if IERS_A_in_cache():</p> <p>  File "/Users/mxhf/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/astroplan/utils.py", line 67, in IERS_A_in_cache</p> <p>    urls = (iers.conf.iers_auto_url, iers.conf.iers_auto_url_mirror)</p> <p>AttributeError: 'Conf' object has no attribute 'iers_auto_url_mirror'</p> <p>Interesting. I also have astroplan 0.6.0 installed and don't get this error. However I do get the following warning, which could perhaps indicate a solution:</p> <p> </p> <p><tt>WARNING: OldEarthOrientationDataWarning: Your version of the IERS Bulletin A is 36.4 days old. For best precision (on the order of arcseconds), you must download an up-to-date IERS Bulletin A table. To do so, run:</tt></p> <p><tt>&gt;&gt;&gt; from astroplan import download_IERS_A</tt><br/> <tt>&gt;&gt;&gt; download_IERS_A()</tt></p> <p>Perhaps you just need to execute the above lines to populate the data structures.</p> <p> </p> <p>Solution for a similar problem at <a href="https://github.com/astropy/astroplan/issues/443" class="external-link" rel="nofollow">https://github.com/astropy/astroplan/issues/443</a></p> <p> </p> <p>No, unfortunately even the import of astroplan fails....hmmm</p> <p> </p> <p> </p> <p>In <span class="error">&#91;*2*&#93;</span>: <b>import</b> <b>astroplan</b></p> <p>---------------------------------------------------------------------------</p> <p>AttributeError                            Traceback (most recent call last)</p> <p>&lt;ipython-input-2-4b2ee34d882b&gt; in &lt;module&gt;()</p> <p>----&gt; 1 import astroplan</p> <p> </p> <p>~/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/astroplan/_<em>init</em>_.py in &lt;module&gt;()</p> <p>     <b>30</b>     from .periodic import *</p> <p>     <b>31</b></p> <p>---&gt; 32     get_IERS_A_or_workaround()</p> <p> </p> <p>~/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/astroplan/utils.py in get_IERS_A_or_workaround()</p> <p>     <b>55</b>     the end of the IERS table.</p> <p>     <b>56</b>     """</p> <p>---&gt; 57     if IERS_A_in_cache():</p> <p>     <b>58</b>         iers.IERS.iers_table = _get_IERS_A_table()</p> <p>     <b>59</b>     else:</p> <p> </p> <p>~/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/astroplan/utils.py in IERS_A_in_cache()</p> <p>     <b>65</b>     Check if the IERS Bulletin A table is locally cached.</p> <p>     <b>66</b>     """</p> <p>---&gt; 67     urls = (iers.conf.iers_auto_url, iers.conf.iers_auto_url_mirror)</p> <p>     <b>68</b>     for url_key in urls:</p> <p>     <b>69</b>         # The below code which accesses ``urlmapfn`` is stolen from</p> <p> </p> <p>AttributeError: 'Conf' object has no attribute 'iers_auto_url_mirror'</p> <p> </p> <p>In <span class="error">&#91;*3*&#93;</span>:</p> <p>Ah, thanks. Trying ... though ....hmm</p> <p> </p> <p>  File "ets_shuffle/demo_shuffle.py", line 3, in &lt;module&gt;</p> <p>    from ets_shuffle import query_utils</p> <p>ImportError: cannot import name 'query_utils'</p> <p> </p> <p>is query utils not in git yet, maybe?</p> <p> </p> <p>Max</p> <p> </p> <p>Aargh, sorry ... now it is!</p> <p> </p> <p>OK, got it to run. So I am getting a plot of the possible guide stars.</p> <p>I also executed the skyimage.py which (surprise) produces an image of the sky.</p> <p>So, I guess we need to combine them yet (or make the production of an image into a pdf (or similar) optional?</p> <p> </p> <p>Max</p> <p> </p> <p>Yes, at the moment the output is a simple plot (in PFI coordinates).</p> <p>Creating lists of PFI coordinates and GAIA IDs for all the candidates is trivial and is my next step.</p> <p>Skyimage is really just a proof of concept. It cannot yet be combined with the output of the guide star selection, because I don't have the transforms from PFI to sky coordinates yet. Of course I could overplot red dots where the guide star candidates are, but at the moment there is no way to plot, e.g., AG camera FOVs or patrol regions.</p> <p>Max, if you would prefer a more flexible demo that allows, say, providing parameters on the command line or interactively, please let me know which method of providing parameters would be most suitable for you.  As I said, depending on how this will be integrated with the rest of the software, this will be wildly different, so I didn't even start on this part of the code. Possible scenarios are:</p> <ul> <li>code is used as a Python module =&gt; interface is a documented public function which takes the parameters as arguments and returns the results</li> <li>code is used as a standalone script =&gt; parameters come from the command line or via a parameter file, or via entries in a data base, output is written to disk or to a data base in a TBD format</li> </ul> <p>Overall, I don't expect we'll have to use configuration files specific for this task (as is done in HETDEX shuffle), as the number of parameters is fairly small, and the only really complicated ones (AG camera geometry and fiber locations) will be available via library or DB calls.</p> <p> </p> <p>The code now outputs the pixel coordinates of the guide star candidates.</p> <p>Can you please have a look whether this works as intended, Max?</p> <p>OK, this seems to work. I propose to merge to master.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=Martin.Reinecke" class="user-hover" rel="Martin.Reinecke">Martin Reinecke</a> can we close this ticket?</p> <p>Branch has been merged; I think this is OK to close.</p> Blocks FIBERALLOC-26 FIBERALLOC-33 Development Rank 0|zzsgpr: Reviewers mxhf Sprint EngRun03