Handle temporary RPC failures and daemon restarts without corrupting state (#35)

* Handle temporary RPC failures and daemon restarts without corrupting state

watch_status()'s per-iteration RPC read had no error handling: an
unexpected failure (the daemon crashing or restarting mid-call, or any
other transient RPC hiccup) propagated straight out of the loop and
crashed the whole container, indistinguishable from a real fatal
error.

The read is now wrapped: an unexpected exception logs a warning and
retries on the next tick, leaving prev_status untouched so the next
successful read still reports every real change. A stop signal or
another DscError still ends the loop exactly as before - the wrapper
explicitly re-raises DscError first, since GracefulShutdown is one and
must not be swallowed by the broader except below it.

Verified against the real image: a container survives seaf-daemon
being killed outright, logging the failure instead of exiting.

* Log a traceback on RPC failure and strengthen the state test

Copilot review:
- The RPC failure warning logged only the message; add exc_info so a
  repeated failure in production is actually debuggable.
- The 'prior state is not corrupted' test asserted the warning fired
  but never proved a real change after the failure was still detected
  and logged, which is the actual property being claimed.

---------

Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>

dsc/client.py

@@ -13,7 +13,7 @@ from urllib3.util.retry import Retry
 import seafile
 
 from dsc import const
-from dsc.errors import ConfigError, DaemonError, DaemonTimeout, NetworkError
+from dsc.errors import ConfigError, DaemonError, DaemonTimeout, DscError, NetworkError
 from dsc.misc import config_value, create_dir, hide_secrets, user_cmd
 
 _lg = logging.getLogger(__name__)
@@ -302,7 +302,21 @@ class SeafileClient:
         fmt = "Library {:%ds} {}" % max_name_len
         while True:
             time.sleep(const.STATUS_POLL_PERIOD)
-            cur_status = self.get_status()
+            try:
+                cur_status = self.get_status()
+            except DscError:
+                # A stop signal or another DscError must still end the loop;
+                # only an unexpected RPC failure below is worth retrying.
+                raise
+            except Exception as err:
+                # The daemon can restart under us; a status read caught mid
+                # restart must not take the whole process down with it.
+                # prev_status is left untouched, so the next successful read
+                # still reports every real change since it, not just since
+                # this failure.
+                _lg.warning("Could not read sync status, retrying: %s", err,
+                            exc_info=err)
+                continue
             for library, state in cur_status.items():
                 if state != prev_status.get(library):
                     if 30 > len(library) > max_name_len: