[INSTRM-1323] Use context managers ("with") as much as possible in db code. Created: 14/Jul/21 Updated: 18/May/23 |
|
| Status: | In Review |
| Project: | Instrument control development |
| Component/s: | spt_operational_database |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Normal |
| Reporter: | cloomis | Assignee: | arnaud.lefur |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | EngRun | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Story Points: | 2 |
| Sprint: | EngRun03 |
| Description |
|
[Pulled from INSTRM-1298] Please use context managers as much as possible in any database code. Both sqlalchemy and pandas support them. For sqlalchemy I believe the link is https://docs.sqlalchemy.org/en/14/orm/session_basics.html#using-a-sessionmaker
|
| Comments |
| Comment by cloomis [ 18/Jul/21 ] |
|
It appears that significant improvements were made to context manager support between sqlalchemy 1.3.something and 1.4.something – that link does not apply to 1.3. We are running 1.3.20 at Subaru. The update would be clean (per mamba update sqalchemy, only sqlalchemy and greenlet would be touched). If people are nervous we could clone the conda-ics environment and update that for testing. I'm running one now and will report back. [ Note: mamba update --dry-run sqlalchemy took 15s and offered the current 1.4.21; conda update --dry-run sqlalchemy took 5 minutes and only offered 1.4.4. Both using the same repos. Someone's solver is wrong. ] |
| Comment by cloomis [ 03/Aug/21 ] |
|
I have run sqlalchemy 1.4.21 in a test conda environment for MCS testing, and it did not obviously break anything in mcs or fps. Can I update the version in conda-ics? |
| Comment by Kiyoto Yabe [ 03/Aug/21 ] |
|
I just updated the version and tested in my side and did not see any problems so far, so I think you can updete on the mountain. |
| Comment by cloomis [ 03/Aug/21 ] |
|
Thanks. Done (actually, 1.4.22). I believe the following idiom (from the linked document) for the Opdb class should now work: # we can now construct a Session() and include begin()/commit()/rollback()
# at once
with self.session.begin() as session:
session.add(some_object)
session.add(some_other_object)
# commits the transaction, closes the session
|
| Comment by Kiyoto Yabe [ 18/Aug/21 ] |
|
I pushed a test implementation like this on the branch, but not sure if it's a right way so need a review.
def insert(self, tablename, dataframe):
model = getattr(models, tablename)
with self.session.begin() as s:
s.session.bulk_insert_mappings(model, dataframe.to_dict(orient="records"))
|
| Comment by Kiyoto Yabe [ 24/Dec/21 ] |
|
Can we close this ticket? If the current opDB codes (at the summit) deal with this properly, I will check them and try to make them merged. |
| Comment by hassan [ 24/Dec/21 ] |
|
Kiyoto Yabe I think all opDB code (as well as other software) at the summit needs to be merged to master. So we should best close this ticket after the merge. |
| Comment by Kiyoto Yabe [ 06/Jan/22 ] |
|
I think now the current master branch is (almost) compatible with the summit change. Could somebody check at summit whether it works as we expected? |
| Comment by arnaud.lefur [ 14/Oct/22 ] |
|
I did the most I could. |
| Comment by hassan [ 06/Jan/23 ] |
|
arnaud.lefur is there any quick test you could do to close this ticket? |