https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-890 Instrument control development <p>Postgres does not provide unsigned integer types (nor does the SQL standard). At least one of our column types (for <tt>pfsDesignIds</tt>, etc.) requires 64-bit unsigned ints, which do not have a larger type to upgrade to. Besides that, sqlAlchemy has no existing mechanism for synthesizing a new type. </p> <p>There is a <tt>pguint</tt> extension which people seem to use. Or we could convert signed&lt;-&gt;unsigned above the SQL/sqlAlchemy layer. Others? MySQL has unsigned types, FWIW.</p> <p>This is not a problem for FITS. For tables, TSCAL and TZERO do the conversion, and for header cards the content of the integer field is (surprisingly) not constrained.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a>? <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a>? <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a>?</p> INSTRM-890

Decide on mechanism for saving unsigned integers to SQL.

Task Normal Open Unresolved Unassigned cloomis opDB Mon, 27 Jan 2020 22:27:01 +0000 Fri, 30 Oct 2020 15:05:13 +0000 0 3 <p>I was proposing to use `NUMERIC` instead but this may not be a good idea because of performance etc.</p> <p>I can't believe performance would really matter: we do not <em>operate</em> on the values besides = and maybe !=. </p> <p>But sqlalchemy wants the python side of Numerics to be a float or a python Decimal, neither of which is directly useable. If we need to subclass or otherwise add sqlalchemy conversion/cast logic, I think we would be better off doing that for their Integer or BigInt types, where we could easily convert between unsigned and signed values. </p> <p>That said, I do not <em>actually</em> know the right way do do this. Maybe something in <a href="https://docs.sqlalchemy.org/en/13/core/custom_types.html#creating-new-types" class="external-link" rel="nofollow">https://docs.sqlalchemy.org/en/13/core/custom_types.html#creating-new-types</a> maybe, with <tt>signed = unsigned ^ (1&lt;&lt;63)</tt> as the cast?</p> <p>I tested the Postgres extension <a href="https://github.com/petere/pguint" class="external-link" rel="nofollow">pguint</a> in my local Linux machine with PostgreSQL 10.14. Although the installation was OK, I got unknown segfault when I put any value into a table with `uint8` datatype. The repository seems to be maintained by only one person and I'm just worried about using that unstable extension in opDB. </p> <p>I also tested a signed - unsigned conversion in sqlalchemy side while the datatype of the column is `bigint` implementing `model.py` based on the link Craig suggested like this:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">class </span>UnsignedBigInteger(TypeDecorator): impl = BigInteger def __init__(self, *args, **kwargs): TypeDecorator.__init__(self, *args, **kwargs) def process_bind_param(self, value, dialect): <span class="code-keyword">if</span> value is not None: <span class="code-keyword">return</span> value - (1 &lt;&lt; 63) def process_result_value(self, value, dialect): <span class="code-keyword">if</span> value is not None: <span class="code-keyword">return</span> value + (1 &lt;&lt; 63) <span class="code-keyword">class </span>pfs_visit(Base): '''Tracks the Gen2 visit identifier. This is the fundamental identifier <span class="code-keyword">for</span> all instrument exposures (MCS, AGC, SPS) ''' __tablename__ = <span class="code-quote">'pfs_visit'</span> pfs_visit_id = Column(<span class="code-object">Integer</span>, primary_key=True, unique=True, autoincrement=False) pfs_visit_description = Column(<span class="code-object">String</span>) pfs_design_id = Column(UnsignedBigInteger) def __init__(self, pfs_visit_id, pfs_visit_description, pfs_design_id): self.pfs_visit_id = pfs_visit_id self.pfs_visit_description = pfs_visit_description self.pfs_design_id = pfs_design_id </pre> </div></div> <p>I think this works well. I can insert pfsDesignId=1 like </p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> utils.insert(url, <span class="code-quote">'pfs_visit'</span>, pd.DataFrame({<span class="code-quote">'pfs_visit_id'</span>: [120981, 120982, 120983], <span class="code-quote">'pfs_visit_description'</span>: [<span class="code-quote">'test'</span>, <span class="code-quote">'test'</span>, <span class="code-quote">'test'</span>], <span class="code-quote">'pfs_design_id'</span>: [pfsDesignId, pfsDesignId, pfsDesignId]})) </pre> </div></div> <p> </p> <p>and see</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> develop=# select * from pfs_visit; pfs_visit_id | pfs_visit_description |    pfs_design_id      --------------+-----------------------+----------------------        120981 | test                  | -9223372036854775807        120982 | test                  | -9223372036854775807        120983 | test                  | -9223372036854775807 (3 rows) </pre> </div></div> <p> </p> <p>I fetch the table like </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> res = utils.fetch_all(url, <span class="code-quote">'pfs_visit'</span>) print(res) </pre> </div></div> <p>and see </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">    pfs_visit_id pfs_visit_description  pfs_design_id 0        120981                  test              1 1        120982                  test              1 2        120983                  test              1 </pre> </div></div> <p>So I think, as long as we use opDB package, we can treat unsigned bigint without caring about the conversion. We need to worry about that if we want access opDB directly. What do you think, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> ?</p> <p> </p> <p> </p> <p>I was hoping that (using signed integers inside the db, and converting simply on select/insert) would work. The <em>bits</em> are the same and should not change if we do that right. </p> <p>So yes, I like that.</p> Blocks INSTRM-1096 Development Rank 0|zzs0kw: