Uploaded image for project: 'Instrument control development'
  1. Instrument control development
  2. INSTRM-2422

Survive i/o failures to archiver spool files

    XMLWordPrintable

    Details

      Description

      After some confused bumping of NFS servers and other services, the archiver stopped updating its spool files and therefore the database itself. With this:

      2024-11-13 06:20:13-1000 [-] Starting factory <__main__.startServer.<locals>.HubFactory object at 0x7f6f3ecd4550>
      2024-11-13 06:20:14-1000 [ReplyReceiver,client] Unhandled Error
              Traceback (most recent call last):
                File "/software/condaRoot/envs/rubin8-ics/lib/python3.11/site-packages/twisted/python/log.py", line 96, in callWithLogger
                  return callWithContext({"system": lp}, func, *args, **kw)
                File "/software/condaRoot/envs/rubin8-ics/lib/python3.11/site-packages/twisted/python/log.py", line 80, in callWithContext
                  return context.call({ILogContext: newCtx}, func, *args, **kw)
                File "/software/condaRoot/envs/rubin8-ics/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
                  return self.currentContext().callWithContext(ctx, func, *args, **kw)
                File "/software/condaRoot/envs/rubin8-ics/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
                  return func(*args, **kw)
              --- <exception caught here> ---
                File "/software/condaRoot/envs/rubin8-ics/lib/python3.11/site-packages/twisted/internet/posixbase.py", line 482, in _doReadOrWrite
                  why = selectable.doRead()
                File "/software/condaRoot/envs/rubin8-ics/lib/python3.11/site-packages/twisted/internet/tcp.py", line 250, in doRead
                  return self._dataReceived(data)
                File "/software/condaRoot/envs/rubin8-ics/lib/python3.11/site-packages/twisted/internet/tcp.py", line 255, in _dataReceived
                  rval = self.protocol.dataReceived(data)
                File "/software/condaRoot/envs/rubin8-ics/lib/python3.11/site-packages/twisted/protocols/basic.py", line 440, in dataReceived
                  self.lineReceived(line)
                File "/software/mhs/products/Linux64/ics_archiver/1.2.3h/python/archiver/protocol.py", line 54, in lineReceived
                  self.messageReceived(message)
                File "/software/mhs/products/Linux64/ics_archiver/1.2.3h/python/archiver/protocol.py", line 196, in messageReceived
                  self.replyRaw.record(rawID,tai,message)
                File "/software/mhs/products/Linux64/ics_archiver/1.2.3h/python/archiver/database.py", line 520, in record
                  print(rowString, file=self.bufferFile)
              builtins.ValueError: I/O operation on closed file.
      
      2024-11-13 06:20:14-1000 [ReplyReceiver,client] <twisted.internet.tcp.Connector instance at 0x7f6f3ec76010 disconnected IPv4Address(type='TCP', host='mhs-ics', port=6095)> will retry in 19 seconds
      2024-11-13 06:20:14-1000 [-] Stopping factory <__main__.startServer.<locals>.HubFactory object at 0x7f6f3ecd4550>
      

        Attachments

          Activity

            People

            • Assignee:
              cloomis cloomis
              Reporter:
              cloomis cloomis
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: