[OBSPROC-29] Consider efficient query for sky targets Created: 20/Sep/22  Updated: 03/Mar/23  Resolved: 03/Mar/23

Status: Done
Project: PFS observation processing/procedure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: Kiyoto Yabe Assignee: monodera
Resolution: Done Votes: 0
Labels: EngRun
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Since the number of records of `sky` table in targetDB is so large, it takes so long to get sky targets in the process of pfsDesign generation. Consider efficient way for the query. One option is to use `q3c_radial_query`, which is used for gaiaDB.



 Comments   
Comment by Kiyoto Yabe [ 22/Sep/22 ]

Now we fixed this using `q3c`. Could you briefly summarize what you did with the SQL commands, please? monodera

Also, Masayuki Tanaka mentioned that we might need to think about distributed querying using such as Citus when targetDB expands in future.

Comment by monodera [ 23/Sep/22 ]

q3c was installed by Furusawa-san as an extension of PostgreSQL. After the installation, I logged into the current targetdb (currently targetdb_comm2022may database) and executed the following SQL commands.

 

CREATE EXTENSION q3c;
CREATE INDEX ON sky (q3c_ang2ipix(ra,dec));
CLUSTER sky_q3c_ang2ipix_idx ON sky;
ANALYZE sky;

 

There are various functions in the q3c extension, but at this point only q3c_radial_query() which simply returns true if (ra, dec) is in a search radius. An example query is the following.

 

SELECT * FROM {tablename} WHERE q3c_radial_query(ra, dec, {ra}, {dec}, {search_radius})

Documentation can be found at the following GitHub repository.

https://github.com/segasai/q3c
 

 

Comment by monodera [ 23/Sep/22 ]

BTW, the q3c indexing is only done in the sky table.

Comment by monodera [ 06/Oct/22 ]

FYI, Murata-san commented that the HSC SSP database uses the earthdistance extension in postgresql for indexing. Coord column is used as index.

Comment by yuki.moritani [ 03/Mar/23 ]

According to monodera efficient query was achieved by indexing with q4c. If other method needs to be used (see the above comment), it will be handled in another ticket.

Generated at Sat Feb 10 16:44:32 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.