Maksim ShulgaandMaxim Shulga
123705aa53
* Handle temporary RPC failures and daemon restarts without corrupting state watch_status()'s per-iteration RPC read had no error handling: an unexpected failure (the daemon crashing or restarting mid-call, or any other transient RPC hiccup) propagated straight out of the loop and crashed the whole container, indistinguishable from a real fatal error. The read is now wrapped: an unexpected exception logs a warning and retries on the next tick, leaving prev_status untouched so the next successful read still reports every real change. A stop signal or another DscError still ends the loop exactly as before - the wrapper explicitly re-raises DscError first, since GracefulShutdown is one and must not be swallowed by the broader except below it. Verified against the real image: a container survives seaf-daemon being killed outright, logging the failure instead of exiting. * Log a traceback on RPC failure and strengthen the state test Copilot review: - The RPC failure warning logged only the message; add exc_info so a repeated failure in production is actually debuggable. - The 'prior state is not corrupted' test asserted the warning fired but never proved a real change after the failure was still detected and logged, which is the actual property being claimed. --------- Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
71 lines
2.2 KiB
Python
71 lines
2.2 KiB
Python
"""A transient RPC failure must not take the whole process down with it."""
|
|
|
|
import time
|
|
|
|
import pytest
|
|
|
|
from dsc.client import SeafileClient
|
|
from dsc.errors import GracefulShutdown
|
|
|
|
|
|
@pytest.fixture
|
|
def client(monkeypatch):
|
|
monkeypatch.setattr(time, "sleep", lambda seconds: None)
|
|
return SeafileClient("seafile.example", "user", "pw", "/dsc", token="tok-en")
|
|
|
|
|
|
def test_watch_status_survives_a_transient_rpc_failure(client, monkeypatch):
|
|
"""The loop keeps going instead of crashing the container."""
|
|
calls = []
|
|
|
|
def flaky_get_status():
|
|
calls.append(1)
|
|
if len(calls) == 1:
|
|
raise ConnectionError("daemon restarting")
|
|
raise GracefulShutdown("SIGTERM") # ends the otherwise-infinite loop
|
|
|
|
monkeypatch.setattr(client, "get_status", flaky_get_status)
|
|
|
|
with pytest.raises(GracefulShutdown):
|
|
client.watch_status()
|
|
|
|
assert len(calls) == 2, "the loop must retry after the first failure"
|
|
|
|
|
|
def test_watch_status_still_propagates_a_stop_signal(client, monkeypatch):
|
|
"""A DscError, such as a stop signal, must still end the loop."""
|
|
monkeypatch.setattr(client, "get_status",
|
|
lambda: (_ for _ in ()).throw(GracefulShutdown("SIGINT")))
|
|
|
|
with pytest.raises(GracefulShutdown):
|
|
client.watch_status()
|
|
|
|
|
|
def test_watch_status_keeps_prior_state_across_a_failed_read(client, monkeypatch, caplog):
|
|
"""
|
|
A failed read is logged and skipped; it must not corrupt prior state,
|
|
so a real change right after the failure is still detected and logged.
|
|
"""
|
|
calls = []
|
|
|
|
def flaky_get_status():
|
|
calls.append(1)
|
|
if len(calls) == 1:
|
|
return {"Docs": "syncing"}
|
|
if len(calls) == 2:
|
|
raise ConnectionError("daemon restarting")
|
|
if len(calls) == 3:
|
|
return {"Docs": "done"}
|
|
raise GracefulShutdown("SIGTERM")
|
|
|
|
monkeypatch.setattr(client, "get_status", flaky_get_status)
|
|
|
|
with caplog.at_level("INFO"):
|
|
with pytest.raises(GracefulShutdown):
|
|
client.watch_status()
|
|
|
|
assert "Could not read sync status" in caplog.text
|
|
assert "syncing" in caplog.text
|
|
assert "done" in caplog.text
|
|
assert len(calls) == 4
|