Uploaded image for project: 'DRP 2-D Pipeline'
  1. DRP 2-D Pipeline
  2. PIPE2D-1749

Move all common database interactions into pfs_utils and optimize

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Won't Fix (View Workflow)
    • Priority: Normal
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: pfs_utils
    • Labels:
    • Epic Name:
      Consolidation of DB access and usage

      Description

      Currently we are accessing our databases (opdb, qaDB, and sort of gaia) via a number of different methods and functions that are spread throughout the codebase. These various connection methods need to be used in a number of places in DRP and ICS (and potentially others, e.g. obsproc). The most logical place for this is pfs_utils.

      This scope of this epic will be to consolidate those accessor methods and utility functions within pfs_utils and then change their usage within the various repositories accordingly. Some of these areas still need to be identified. Most of the identified issues are within ICS however there are various connection methods in DRP.

      Currently the various accessor methods use a variety of connection methods, namely sqlalchemy and psycopg2. As part of this epic we can determine which one is preferred but the first step would be putting them all into one place, which makes the conversion from one method to another fairly easy as our accessor methods will be agnostic for the user.

      Note that the spt_operational_database was (I think) originally designed for this purpose but that repository seems to be dealing more with the creation and maintenance of the database itself and so the connection and utility functions for that repo should come from within pfs_utils while the management, scripts, diagrams, etc. should stay within that repo.

      There already exist a number of outstanding tickets for various parts of this process so those will be placed under this epic however this list does not cover all current usages, which still need to be identified.

      Eventually these database access methods should also make it easy to use "test" databases, although that may be beyond the scope of this epic. For consolidation purposes some of those tickets will be listed here as well.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. INSTRM-2509 obslog remains many `idle in transaction` in opDB

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-387 Added database for MCS.

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-490 Build test db instance or container for MHS tests

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-890 Decide on mechanism for saving unsigned integers to SQL.

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-1108 switch gen2Actor to proper opdb library

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-1244 Put all data interfaces used for targeting into pfs_utils

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-1334 Add efficient DataFrame <--> opdb methods

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-2207 Neaten up mcsActor db access

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-2219 Add AGCC reprocessing to test database

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-2251 Allow redirecting agccActor to a test database

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-2341 Correctly wrap opdb transactions

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-2355 Update opdb.insert_kw to be SQLAlchemy 1.4/2.0-compatible.

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-2413 add opdb access to drpActor

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-2598 improve agc_match insertion

          • Normal - Normal priorities
          • Open

          Task - A task that needs to be done. INSTRM-2614 Implement DB connection pooling in ics_utils

          • Normal - Normal priorities
          • In Progress

            Activity

              People

              • Assignee:
                wtgee Wilfred Gee
                Reporter:
                wtgee Wilfred Gee
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: