https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INFRA-93 Software Development Infrastructure <p>Aim of this repository are:</p> <ul> <li>to provide base code modules (class/API) on conversion among various coordinates, both their functions (e.g. polynomial) and parameters</li> <li>to store analysis results from simulation or modeling as reference</li> </ul> <p>Also it is better to provide:</p> <ul> <li>test data set and unit test (using test data set) of code modules for CI</li> <li>jupyter notebook sample</li> </ul> <p>This could be used from AG coordinate converter or fiber allocation software.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> I don't think we should merge this kind of items into datamodel, but be better to have separated. Is it ok for you on management point of view?<br/> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=yuki.moritani" class="user-hover" rel="yuki.moritani">yuki.moritani</a> Could you help once we will have this repo?<br/> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoyuki.tamura" class="user-hover" rel="naoyuki.tamura">naoyuki.tamura</a> Let us know if you have any comment/suggestion on project management aspect.<br/> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> Just let you know on this.</p> INFRA-93

Add new repository for coordinate transformation and its data

Story Major Done Done shimono shimono GitHub MCS Tue, 6 Dec 2016 20:13:38 +0000 Mon, 1 Oct 2018 20:10:13 +0000 Mon, 1 Oct 2018 20:03:18 +0000 0 7 <p>I'm more than happy to help it.<br/> I think we should discuss what exactly I should push to them (format, etc.).</p> <p>Could you make me sure.<br/> You mean by fiber allocation software both ETS and FPS?<br/> I mean, the repository includes not only sky-F3C transformation, but also MCS-F3C transformation?</p> <p>From recent discussions and updates, we'd make this happen.</p> <ul> <li>repository owner: <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=yuki.moritani" class="user-hover" rel="yuki.moritani">yuki.moritani</a></li> <li>repository name: pfs_coordinates</li> </ul> <p>This shall include</p> <ul> <li>documents on functions and parameters in standard docs/ directory</li> <li>conversion code (APIs) which can be used from software modules in eups way</li> </ul> <p>Given that coordinate transformation will be used mainly by ETS and FPS, I would like to hear what <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=Martin.Reinecke" class="user-hover" rel="Martin.Reinecke">Martin Reinecke</a> and <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=chyan" class="user-hover" rel="chyan">chyan</a> think of it.<br/> Is it OK to you two?</p> <p>Considering that mathematical formats of coordinate transformation will be updated at the commissioning (and might be even in scientific use), it would be better to have one repository, which accommodates transportation functions.</p> <p>I think we should discuss argument values and return values, once we agree to have this repository.</p> <p>I would be extremely happy to have this functionality available in a central place!</p> <p>What ETS needs is basically the transformation from target RA/DEC to x/y PFS coordinates (for a given telescope pointing, position angle and observation time).</p> <p>If I can help in any way, please let me know!</p> <p>Let us not reinvent the wheel. There is a standard description of the camera distortions in the HSC camera description, and we could probably reuse the format. It is about to be upgraded to support layers of distortions (e.g. camera optics, detectors, atmosphere), and this layering may well be useful for the MCS.</p> <p>&gt; <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=Martin.Reinecke" class="user-hover" rel="Martin.Reinecke">Martin Reinecke</a><br/> Well, I think we should determine which should deal with telescope position, position angle, atmosphere etc.<br/> Does current ver. of ETS assume these things? If I remember correctly, ETS changes position angle, but I'm not sure about other points.</p> <p>&gt; <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a><br/> If we could reuse or refer the HSC format, it would be nice.<br/> PFS optics is not the same as HSC optics (e.g. PFS has thicker glass than HSC's dewar window and + filters.<br/> Also, PFS should determine it for entire region, whereas HSC possibly determines distortion detector by detector,<br/> Considering these, PFS could use some of layers.</p> <p>Currently, ETS takes RA/DEC of targets as input, which means that it (at least at the moment) has to deal with the full transformation machinery. Also, it is expected that ETS will try not only different position angles, but also slightly different telescope pointings in order to find good fiber assignments.</p> <p>The current transformation from RA/DEC to x/y in ETS is not perfect; especially the distortions due to telescope optics are only very rough approximations.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> it is quite nice if possible and was actually in some plan, to rely on the code of the HSC camera description. we are just at the starting point of studies on our distortion models among coordinates, and also we just started from fomulas of HSC with parameters from our calculation/simulation. As <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=yuki.moritani" class="user-hover" rel="yuki.moritani">yuki.moritani</a> pointed, we plan to include "ALL" into this model, so we may want to have more parameters, and also could need some special handling of calculation, such as to deal with differential changes by so called 'mod 60 degree' which is handled by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> and <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=yuki.moritani" class="user-hover" rel="yuki.moritani">yuki.moritani</a>, or AG calculation.</p> <p>If F3C-MCSC transformation is included in this repository, then you need to take care of the distortion from the small telescope in front of metrology camera. IAA could provide such information.</p> <p>Well, when the spot on MCS from PFI was calculated, both the optics on MCS (e.g. primary mirror, flattner etc.) and PFI (e.g. WFC) are included. <br/> The Subaru engineer used ZEMAX file provided from ASIAA.</p> <p>Sorry that I don't know you have the ZEMAX file. Jennifer has done some study on fiber identification. If there are multiple fibers locate in the overlapping regions, then there is no confidential way to identify fibers. That's why we need to take images while COBRA is moving. Also this enables us to check if a missing fiber is hidden behind the dots.</p> <p>Just asking for my clearer understanding:</p> <p>Can I expect that at some point in the future there will be a Python function I can call, passing target RA/DEC, time, telescope pointing, position angle (and maybe a few more technical parameters), which returns target x/y positions on the focal plane?</p> <p>If this is the case, I would redesign the ETS package with this in mind.</p> <p>Ah .. a few latest conversations are getting out of the scope this ticket.<br/> I'm really sorry...</p> <p>Yes. I'm expecting this is the case.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=chihyi" class="user-hover" rel="chihyi">chihyi</a><br/> Putting aside which way you need and we can provide for F3C-MCSC conversion, do you agree that we prepare a repository?<br/> And let's discuss transformation itself in another ticket (or ML).</p> <p>ok, let me finish this specific ticket.</p> <p>following <a href="https://github.com/Subaru-PFS/doc/blob/master/development-management/request.rst#add-new-github-repository" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/doc/blob/master/development-management/request.rst#add-new-github-repository</a></p> <ul> <li>Name of new repository: pfs_coordinates</li> <li>Short description of new repository: PFS Coordinate definitions and conversion codes</li> <li>Responsible institution: IPMU</li> <li>Corresponding JIRA project and component: INSTRM (or new project?), create new component with repository name</li> <li>License: GPLv2</li> </ul> <p>I don't think that this is a good idea; you almost always regret mixing code and data (as von Neumann pointed out); we need to split the code from the data (which will need to be versioned; we already have a way of doing this for e.g. flat fields and I think we should use the same mechanism).</p> <p>Who owns the schema for these files &#8211; they need to be in the data model. As discussed above there are ways of composing transforms in the PFS codebase (based on starlink's AST library). Are these sufficient?</p> <p>What are the proposed APIs for all these transformations?</p> <p>Why not a BSD license? In general GPL turns out to be a mistake, so why continue making it?</p> <p>I agree that mixing code and data is usually not a good idea; but I don't see the problem with code and data sitting alongside each other in the same repository.</p> <blockquote><p>Why not a BSD license? In general GPL turns out to be a mistake, so why continue making it?</p></blockquote> <p>I don't agree with your assertion about GPL.</p> <p>But independent of that: if GPL indeed turns out to be a mistake, it is trivial to re-license the code in question to BSD.</p> <p>Going from BSD to GPL (for released code) is impossible, however. You can't revoke rights that you have granted before.</p> <p>The problem is that you need to version the data independently of the code.</p> <p>I see your point.</p> <p>Assuming that both code and data quality strictly improve over time, a joint version number could work, but that's not the world we live in <img class="emoticon" src="https://pfspipe.ipmu.jp/jira/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I'm not sure how separate two entities have to be in git in oder to tag them independently, but I agree that this should be possible.</p> <p>&gt; I don't think that this is a good idea; you almost always regret mixing code and data (as von Neumann pointed out); we need to split the code from the data (which will need to be versioned; we already have a way of doing this for e.g. flat fields and I think we should use the same mechanism).<br/> Really mourning to be said as so - always regret mixing.<br/> I REALLY trying to getting things to be tracked and separate data/code/config etc. to be well tracked, within this project. I REALLY FRUSTRATED MANY PEOPLE EVEN IN THE PROJECT OFFICE DOES NOT TRY TO USE TICKETS NOR REPOSITORIES, OR EVEN THEY HESITATE TO USE THINGS! <br/> In this ticket, at least, I want to gain things to be tracked, that's already a difficult issue.<br/> If I want to go lovely world, and I can propose anything, I would want to have </p> <ul> <li>formulas and data formats with well described documents into datamodel</li> <li>transformation code to one repository</li> <li>parameters of transformation in an another repository<br/> I think datamodel itself is something to be considered as definition (or coded reference), and configuration/parameter shall be split from it.</li> </ul> <p>&gt; Who owns the schema for these files – they need to be in the data model. As discussed above there are ways of composing transforms in the PFS codebase (based on starlink's AST library). Are these sufficient?<br/> I agree this. I just thought about parameters and codes for transformation.</p> <p>&gt; What are the proposed APIs for all these transformations?<br/> Shall be by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=yuki.moritani" class="user-hover" rel="yuki.moritani">yuki.moritani</a></p> <p>&gt; Why not a BSD license? In general GPL turns out to be a mistake, so why continue making it?<br/> I actually love MIT or BSD (which are used for some pfs_* repos). GPL is just follow princeton's GPLv3, but I believe v2 is better for our life than v3.<br/> If we all agree on moving from GPL to another, it is possible at anytime. of course older version need to remain, although.</p> <p>The need for at least one independent repository for PFS data for development and test purposes was highlighted during the recent PFI meeting in Taipei 14-15 May. I plan to look into options over the coming months and will discuss options already considered with <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a>l and <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=atsushi.shimono" class="user-hover" rel="atsushi.shimono">shimono</a> and will raise a separate ticket if we find the need for that.</p> <p>As far as the code repo goes, <tt>pfs_utils</tt> was created for modules which 1) we expect will be used by disparate parts of the project (ICS, DRP, ETS) and 2) are unlikely to depend on many other packages. Currently the only module is <tt>fiberids</tt>, which contains the (static) fiber mapping (center PFI coordinates, slit positions, module number, etc). </p> <p>I suggest that this would be a reasonable place to put distortion code. And at least for now perhaps the data files.</p> <p>And if this is deemed reasonable, I'd be glad to move the existing code over from <tt>ics_fpsActor</tt> and neaten it up.</p> <p>Actually, since I sent the code to Chi-Hung, I've modified the code to merge with sky-PFI transformation (not finished yet, though). So if possible, I'd like to upload the new one.</p> <p>I think I can replace after Craig's work, although I'm not sure which is easier to us.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=yuki.moritani" class="user-hover" rel="yuki.moritani">yuki.moritani</a> Is this just a new version of <tt>ics_fpsActor</tt>'s <tt>CoordTransp.py</tt> plus similar files? If so, I'll suggest that you update that now and move it later. </p> <p>(<a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=Martin.Reinecke" class="user-hover" rel="Martin.Reinecke">Martin Reinecke</a>, this is basically <tt>MCSxy_to_PFIxy(zenithAngle)</tt>, plus whatever new <tt>sky_to_PFI()</tt> routines have been added.</p> <p>I use the same routines for all kind of coordinate transformations, just changing coefficients via dictionary.</p> <p>So I personally don't think it suitable to update one to ics_fps Actor, actually..</p> <p>For the same reason, I think . only MCS label isnot enough &gt; <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a></p> <p>OK. Please add to <tt>pfs_utils</tt>, maybe in <tt>pfs/utils/transforms/</tt> or <tt>pfs/utils/coordinates/</tt>?</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=yuki.moritani" class="user-hover" rel="yuki.moritani">yuki.moritani</a> agree that MCS label is not enough. I added that as this issue is relevant for MCS activities, but doesn't mean it is not relevant for others also.</p> <p>In addition as I was discussing with Craig earlier today, this particular issue should probably be resolved based on the GitHub repo creation alone, and separate tickets should be raised on the activities that have been discussed in the comments here. </p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> , All right. maybe the latter?? Anyway I'll push my current code next week.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a> , thank you for explanation... for (my) future reference, could you kindly tell where I can see all available labels? <br/> And I agree, the original goal of this ticket is indeed to make a place for coordinate transformation I'm afraid...</p> <p>Sorry <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=yuki.moritani" class="user-hover" rel="yuki.moritani">yuki.moritani</a> for not responding to your question earlier. There is no obvious means of listing all the available labels. One option is to install the labels gadget (<a href="https://confluence.atlassian.com/jiracoreserver073/gadgets-for-jira-applications-861257078.html?_ga=2.22050875.1837736392.1531921220-800630600.1525207411" class="external-link" rel="nofollow">https://confluence.atlassian.com/jiracoreserver073/gadgets-for-jira-applications-861257078.html?_ga=2.22050875.1837736392.1531921220-800630600.1525207411</a>), but all I do is to search for issues by label and a pop-up appears of all the labels. </p> <p>At this point the labels that currently exist are FITS, GitHub, MCS and SM1.</p> <p>This thread is a little out of context for this issue, so I will also copy my response to a slack channel so that this side-discussion is preserved and is easy to access.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> I have uploaded the current code <br/> <a href="https://github.com/Subaru-PFS/pfs_utils/tree/distortion_201808ym" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/pfs_utils/tree/distortion_201808ym</a></p> <p>At ICS teeconf. today, you told you think separating coefficients and code itself. <br/> Could you please tell how to proceed with the codes?<br/> By the way, the code has local path ... it should be modified at least.</p> <p>From discussions with @cloomis, this ticket is done with the actions of <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=yuki.moritani" class="user-hover" rel="yuki.moritani">yuki.moritani</a>.</p> <p>Well I'm sorry ... I have not merged the brunch yet... so I've done now.</p> <p>(Just for recording.)</p> Relates INFRA-111 Development Rank 0|ii015x: