Handle temporary RPC failures and daemon restarts without corrupting state (#35)

* Handle temporary RPC failures and daemon restarts without corrupting state

watch_status()'s per-iteration RPC read had no error handling: an
unexpected failure (the daemon crashing or restarting mid-call, or any
other transient RPC hiccup) propagated straight out of the loop and
crashed the whole container, indistinguishable from a real fatal
error.

The read is now wrapped: an unexpected exception logs a warning and
retries on the next tick, leaving prev_status untouched so the next
successful read still reports every real change. A stop signal or
another DscError still ends the loop exactly as before - the wrapper
explicitly re-raises DscError first, since GracefulShutdown is one and
must not be swallowed by the broader except below it.

Verified against the real image: a container survives seaf-daemon
being killed outright, logging the failure instead of exiting.

* Log a traceback on RPC failure and strengthen the state test

Copilot review:
- The RPC failure warning logged only the message; add exc_info so a
  repeated failure in production is actually debuggable.
- The 'prior state is not corrupted' test asserted the warning fired
  but never proved a real change after the failure was still detected
  and logged, which is the actual property being claimed.

---------

Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
This commit is contained in:
Maksim ShulgaandMaxim Shulga authored and GitHub committed 2026-08-03 18:55:34 +00:00
1 parent 28db575c92
commit 123705aa53
4 files changed
+139 -5

No files matched your search

+8 -2
View File
@@ -214,8 +214,14 @@ Acceptance criteria:
Each check is a fresh root-spawned process (unlike the long-lived
`start.py`), so it drops to the `seafile` user with `runuser` before
touching the RPC socket - the one place that mechanism still belongs.
- [ ] Handle temporary RPC failures and daemon restarts without corrupting
state.
- [x] Handle temporary RPC failures and daemon restarts without corrupting
state. `watch_status()`'s per-iteration RPC read is now wrapped so an
unexpected failure (the daemon crashing or restarting mid-call) logs a
warning and retries on the next tick instead of crashing the container;
a stop signal or another `DscError` still ends the loop as before. Prior
state is left untouched across a failed read, so the next successful one
still reports every real change. Verified against the real image: the
container survives `seaf-daemon` being killed outright.
- [ ] Document supported CPU architectures instead of implying multi-platform
support.