postgres-bump.txt
[INSTRM-121] Archiver needs to handle database restarts Created: 10/Jun/17 Updated: 10/Feb/21 |
|
| Status: | Open |
| Project: | Instrument control development |
| Component/s: | ics_archiver |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Normal |
| Reporter: | cloomis | Assignee: | cloomis |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
postgres-bump.txt
|
| Description |
|
The archiver lost its connection to the database, due to a (security?) update made to the database server. I don't think it ever tried to reconnect, but kept caching new rows after the failure in the attached log segment. I don't think there is a trivial way to restart, but the hint at https://github.com/psycopg/psycopg2/issues/419 is probably good enough to implement from. |
| Comments |
| Comment by cloomis [ 17/Jun/17 ] |
|
The archiver keeps files of unapplied updates in a /tmp/archiver-$PID/directory. With postgreql, those are suitable for COPY $table FROM '$file' CSV QUOTE '''' commands, which are exactly what the archiver uses. Since the archiver was complaining about the lost connection, I assumed I would have to do that semi-manually. But when I killed the archiver it applied all the updates itself. So I think we just to make the reconnect logic a bit smarter, and/or add an interrupt handler to force reconnections and updates. |
| Comment by cloomis [ 10/Feb/21 ] |
|
Bump. Recent database work caused reconnection and other issues. The archiver does respond well to SIGINTs: it unwedges itself and and flushes all pending transactions from the transaction log files. So it already knows how to to the right things. It just needs to do that automatically. |