[INSTRM-1323] Use context managers ("with") as much as possible in db code. Created: 14/Jul/21  Updated: 18/May/23

Status: In Review
Project: Instrument control development
Component/s: spt_operational_database
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: cloomis Assignee: arnaud.lefur
Resolution: Unresolved Votes: 0
Labels: EngRun
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Story Points: 2
Sprint: EngRun03

 Description   

[Pulled from INSTRM-1298]

Please use context managers as much as possible in any database code. Both sqlalchemy and pandas support them. For sqlalchemy I believe the link is https://docs.sqlalchemy.org/en/14/orm/session_basics.html#using-a-sessionmaker 

 



 Comments   
Comment by cloomis [ 18/Jul/21 ]

It appears that significant improvements were made to context manager support between sqlalchemy 1.3.something and 1.4.something – that link does not apply to 1.3. We are running 1.3.20 at Subaru.

The update would be clean (per mamba update sqalchemy, only sqlalchemy and greenlet would be touched).

If people are nervous we could clone the conda-ics environment and update that for testing. I'm running one now and will report back.

[ Note: mamba update --dry-run sqlalchemy took 15s and offered the current 1.4.21; conda update --dry-run sqlalchemy took 5 minutes and only offered 1.4.4. Both using the same repos. Someone's solver is wrong. ]

Comment by cloomis [ 03/Aug/21 ]

I have run sqlalchemy 1.4.21 in a test conda environment for MCS testing, and it did not obviously break anything in mcs or fps. Can I update the version in conda-ics?

Comment by Kiyoto Yabe [ 03/Aug/21 ]

I just updated the version and tested in my side and did not see any problems so far, so I think you can updete on the mountain.

Comment by cloomis [ 03/Aug/21 ]

Thanks. Done (actually, 1.4.22). I believe the following idiom (from the linked document) for the Opdb class should now work:

# we can now construct a Session() and include begin()/commit()/rollback()
# at once
with self.session.begin() as session:
    session.add(some_object)
    session.add(some_other_object)
# commits the transaction, closes the session
Comment by Kiyoto Yabe [ 18/Aug/21 ]

I pushed a test implementation like this on the branch, but not sure if it's a right way so need a review.

def insert(self, tablename, dataframe):  
    model = getattr(models, tablename)
    with self.session.begin() as s:
        s.session.bulk_insert_mappings(model, dataframe.to_dict(orient="records"))
Comment by Kiyoto Yabe [ 24/Dec/21 ]

Can we close this ticket? If the current opDB codes (at the summit) deal with this properly, I will check them and try to make them merged.

Comment by hassan [ 24/Dec/21 ]

Kiyoto Yabe I think all opDB code (as well as other software) at the summit needs to be merged to master. So we should best close this ticket after the merge.

Comment by Kiyoto Yabe [ 06/Jan/22 ]

I think now the current master branch is (almost) compatible with the summit change. Could somebody check at summit whether it works as we expected?

Comment by arnaud.lefur [ 14/Oct/22 ]

I did the most I could.
I haven't tested yet, but I've mostly followed sqlalchemy prescription

Comment by hassan [ 06/Jan/23 ]

arnaud.lefur is there any quick test you could do to close this ticket?

Generated at Sat Feb 10 16:34:10 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.