Handle temporary RPC failures and daemon restarts without corrupting state (#35)

* Handle temporary RPC failures and daemon restarts without corrupting state

watch_status()'s per-iteration RPC read had no error handling: an
unexpected failure (the daemon crashing or restarting mid-call, or any
other transient RPC hiccup) propagated straight out of the loop and
crashed the whole container, indistinguishable from a real fatal
error.

The read is now wrapped: an unexpected exception logs a warning and
retries on the next tick, leaving prev_status untouched so the next
successful read still reports every real change. A stop signal or
another DscError still ends the loop exactly as before - the wrapper
explicitly re-raises DscError first, since GracefulShutdown is one and
must not be swallowed by the broader except below it.

Verified against the real image: a container survives seaf-daemon
being killed outright, logging the failure instead of exiting.

* Log a traceback on RPC failure and strengthen the state test

Copilot review:
- The RPC failure warning logged only the message; add exc_info so a
  repeated failure in production is actually debuggable.
- The 'prior state is not corrupted' test asserted the warning fired
  but never proved a real change after the failure was still detected
  and logged, which is the actual property being claimed.

---------

Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>

tests/test_status.py

@@ -0,0 +1,70 @@
+"""A transient RPC failure must not take the whole process down with it."""
+
+import time
+
+import pytest
+
+from dsc.client import SeafileClient
+from dsc.errors import GracefulShutdown
+
+
+@pytest.fixture
+def client(monkeypatch):
+    monkeypatch.setattr(time, "sleep", lambda seconds: None)
+    return SeafileClient("seafile.example", "user", "pw", "/dsc", token="tok-en")
+
+
+def test_watch_status_survives_a_transient_rpc_failure(client, monkeypatch):
+    """The loop keeps going instead of crashing the container."""
+    calls = []
+
+    def flaky_get_status():
+        calls.append(1)
+        if len(calls) == 1:
+            raise ConnectionError("daemon restarting")
+        raise GracefulShutdown("SIGTERM")  # ends the otherwise-infinite loop
+
+    monkeypatch.setattr(client, "get_status", flaky_get_status)
+
+    with pytest.raises(GracefulShutdown):
+        client.watch_status()
+
+    assert len(calls) == 2, "the loop must retry after the first failure"
+
+
+def test_watch_status_still_propagates_a_stop_signal(client, monkeypatch):
+    """A DscError, such as a stop signal, must still end the loop."""
+    monkeypatch.setattr(client, "get_status",
+                         lambda: (_ for _ in ()).throw(GracefulShutdown("SIGINT")))
+
+    with pytest.raises(GracefulShutdown):
+        client.watch_status()
+
+
+def test_watch_status_keeps_prior_state_across_a_failed_read(client, monkeypatch, caplog):
+    """
+    A failed read is logged and skipped; it must not corrupt prior state,
+    so a real change right after the failure is still detected and logged.
+    """
+    calls = []
+
+    def flaky_get_status():
+        calls.append(1)
+        if len(calls) == 1:
+            return {"Docs": "syncing"}
+        if len(calls) == 2:
+            raise ConnectionError("daemon restarting")
+        if len(calls) == 3:
+            return {"Docs": "done"}
+        raise GracefulShutdown("SIGTERM")
+
+    monkeypatch.setattr(client, "get_status", flaky_get_status)
+
+    with caplog.at_level("INFO"):
+        with pytest.raises(GracefulShutdown):
+            client.watch_status()
+
+    assert "Could not read sync status" in caplog.text
+    assert "syncing" in caplog.text
+    assert "done" in caplog.text
+    assert len(calls) == 4