[INSTRM-1078] Install Gaia DR2 Catalogue at Subaru Created: 30/Sep/20  Updated: 13/Aug/21  Resolved: 13/Aug/21

Status: Done
Project: Instrument control development
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: hassan Assignee: eric
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File gaia_schema.sql    
Issue Links:
Relates
relates to FIBERALLOC-33 How to deal with manually provided/ca... Open
relates to INSTRM-1081 Predicting stellar flux on AG camera ... Won't Fix
Story Points: 4

 Description   

Following discussions in the last ICS/PFI telecon (https://sumire.pbworks.com/w/page/141309765/ICS-PFI-MCS%20telecon%20on%20Sep%2018%202020) we should have a local installation of the Gaia DR2 catalogue at Subaru. This would be a useful for guide stars, for fiber allocation software (FIBERALLOC-33) etc.

Having a local copy would preclude problems accessing guide stars due to external network issues.

The catalogue should be queryable, so ingested into a postgres or equivalent database instance.

The size of the Gaia DR2 in CSV format is 550 GB. If the size is an issue, it may be possible to store a only subset of fields. But this needs to be discussed further.

 Comments   
Comment by hassan [ 16/Oct/20 ]

The Gaia Archives team were contacted to confirm that the contents of the Gaia DR2 CSV files are consistent with that in the Gaia Archive. They have confirmed that.

Comment by hassan [ 30/Oct/20 ]

Re-assigned ticket to Eric Jeschke, as he will be the responsible for ingesting the downloaded CSV files to the postgres database.

Comment by eric [ 02/Dec/20 ]

The Gaia CSV files were ingested into the Subaru Gen2 "star_catalog" Postgres database as table "gaia".  I have attached the schema.  The table has been clustered and an index made using the Q3C Postgres extension to allow very fast searches on RA/DEC/radius coordinates (all in degrees).  The table contains the entirety of the CSV files.

Here is an example SQL query:
psql (11.2)
Type "help" for help.

star_catalog=# SELECT source_id, ra, dec, phot_rp_mean_mag, parallax, pmra, pmdec  FROM gaia WHERE q3c_radial_query(ra, dec, 10.3, 0.4, 0.076);
      source_id      |        ra        |        dec        | phot_rp_mean_mag |
      parallax       |        pmra        |        pmdec        
----------------------------------------------------------------------
----------------------------------------------------------
 2543114289888915712 | 10.3237413085593 | 0.330040149611898 |          14.7414 |
    1.57129893515104 |   2.50609492529233 |   -10.8380693317631
 2543302920556836096 | 10.2431758026912 | 0.354154114747787 |          17.2081 |
   0.327389891246227 |  -2.21040371907556 |   -1.80819707672227
 2543115251960840960 | 10.2655912179137 | 0.347096406226759 |          10.0711 |
     3.4463085813933 |  -37.6061949980034 |   -27.3090372266146
 2543302924852598016 | 10.2546094351212 | 0.358317143282865 |          15.9077 |
    2.76051410620393 |  -4.91943190679037 |   -43.7051987345144
...
...
 
The q3c_radial_query requires you to specify the words "ra, "dec" then ra, dec, radius (in degrees).
 gaia_schema.sql

Comment by hassan [ 13/Aug/21 ]

Installed in Dec 2020.

Generated at Sat Feb 10 16:31:40 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.