* Make dry-run genuinely non-mutating
fs.rsync() now takes a real dry_run code path: every mutating call
(copy, delete, chmod, chown, utime) is skipped, and a missing dst_dir
is never created - the same comparison logic still runs so the yielded
actions match what a real run would do.
backup.initiate_backup() diffs each source directly against the latest
completed snapshot on dry-run, instead of building and discarding a
throwaway staging directory with real hardlinks. cleanup_old_backups()
no longer creates the repo-id file as a side effect of a dry-run.
cli.py skips lock acquisition entirely on dry-run (set_backups_lock()
writes PID/timestamp metadata, itself a mutation) and exposes the
retention policy via --keep-all/--keep-daily/--keep-weekly/
--keep-monthly/--keep-yearly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Address Copilot review: guard restore lock release, reject negative retention values
- Guard restore --dry-run's finally block the same way backup's is
guarded, for symmetry (release_backups_lock() was already a safe
no-op when unheld, but this makes the dry-run contract explicit
rather than relying on that implementation detail).
- Reject negative --keep-* values via a dedicated argparse type: a
negative threshold shifts retention comparisons into the future and
can make cleanup treat every existing backup as deletable.
- Add a restore --dry-run lock-skip regression test, mirroring the
existing backup one.
- Fix _snapshot_all_stats() in the new dry-run non-mutation test to
also record the backups_dir root's own stat, not just its children.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Test nanosecond mtime precision on file and symlink copies
Existing preserves-times tests only set whole-second timestamps via
os.utime(path, (sec, sec)), so a backend that silently truncated to
second resolution would still pass them. The Python backend's
nanosecond-precision guarantee (documented in README's Limitations
section) was otherwise untested at sub-second granularity.
Combined with CI already running the full suite on both ubuntu-latest
and macos-latest, this closes the "Test supported behavior on Linux
and macOS" item in IMPROVEMENTS.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Rename nanosecond timestamp tests to cover atime too
Copilot review feedback on PR #22: both tests assert st_atime_ns and
st_mtime_ns, but the names/docstrings said "mtime" only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Assert symlink entry is found before use in nanosecond test
Copilot review feedback on PR #22: if the scandir() loop didn't find
the symlink, entry stayed None and copy_direntry() would fail with a
less-informative error than an explicit assertion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Document Python/external-rsync backend differences
- Note that mixing --external-rsync (which can recreate device/socket
entries via --archive's -D) with the default Python hardlink-forward
step is unsafe: _recursive_hardlink() only understands files,
symlinks, and directories and errors on anything else.
- Note that --dry-run is I/O-free with --external-rsync but still
fully copies into a throwaway staging directory with the Python
backend, though neither leaves a lasting change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Clarify external-rsync dry-run description
Copilot review feedback on PR #20: rsync's --dry-run still reads and
stats source files to compute what would change, it just avoids
writing to the destination - "performs no I/O" overstated that.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Document ACL, xattr, sparse-file, and hardlink backup behavior
Extend the README Limitations section with the filesystem-feature
behaviors that weren't documented: ACLs/xattrs/flags are never
preserved by either backend, sparse files are materialized as full
allocated copies, source-side hardlinks are not preserved as
hardlinks in the backup, and ownership is only preserved when
running as root.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Soften README timestamp/ownership claims for external-rsync backend
Nanosecond mtime precision and 'silently logged chown failures' are
Python-backend specifics that don't hold for --external-rsync: rsync's
own timestamp precision depends on the build/filesystem, and it skips
disallowed ownership changes itself rather than going through our
logging path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Add an explicit one-filesystem policy to backups
scantree() gains an opt-in one_filesystem flag: when a subdirectory's
device differs from the walk root's (a mount point), its contents are
excluded from the walk. The mount-point directory itself is still
yielded, so it appears as an empty directory in the backup rather than
vanishing silently - matching rsync's --one-file-system behavior.
Wired into rsync()'s two source-tree walks so backups never silently
pull in content from a bind mount or other filesystem mounted under
the source tree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix one-filesystem policy: symlinked root and external rsync parity
- scantree() computed the walk root's device with os.lstat(), but
scandir() follows the root path if it's a symlink. A symlinked root
would compute the symlink's own device instead of the target
directory's, making every real subdirectory look cross-device and
get its contents wrongly excluded. Use os.stat() for the root only.
- rsync_ext() (the external-rsync backend) didn't pass
--one-file-system, so backups using external_rsync=True still
crossed mount points despite the pure-Python path now refusing to.
Addresses GitHub Copilot review feedback on PR #18.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
test_relative_symlink_to_common_file changed the process working
directory via os.chdir() but never restored it, leaving subsequent
tests running from a tmp_path fixture directory that gets deleted
after the test. Restore cwd in a finally block, matching the pattern
already used in test_pseudo_direntry.py.
Also checks off the now-satisfied xfail-marker item: no xfail markers
remain in the repo.
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
tmp_path nests too deeply for AF_UNIX socket paths (~104-108 byte
limit on macOS). Use a flat tempfile.TemporaryDirectory(), matching
the pattern already used by TestEntryKind::test_socket.
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Implement restore command
Add a restore subcommand alongside the existing backup command:
- curateipsum/restore.py: restore_snapshot() and verify_restored(),
reusing fs.scantree/copy_direntry/rm_direntry so metadata handling
(mode, ownership, timestamps, symlinks) matches backup exactly.
- backup.find_backup(): resolve a snapshot by name or latest.
- cli.py: argparse subcommands (backup/restore), with backup implied
when no subcommand is given so existing invocations keep working.
Restore supports dry-run (creates nothing), partial restore by
snapshot-relative path, an overwrite policy (never/always), a
destination-inside-snapshot boundary check, and manifest-checksum
verification of restored content. It takes the same backups_lock a
backup does, since retention could otherwise delete the snapshot
being restored out from under it.
Closes P1 'Provide a safe restoration workflow' in IMPROVEMENTS.md.
* Address Copilot review: symlink-escape boundary check, docstring fix
- restore_snapshot() and verify_restored() now canonicalize each
destination path and refuse to operate on it if it resolves outside
dest_dir - a symlink planted under dest_dir (by another process, or
left over from a prior run) could otherwise redirect a write, delete,
or read outside the intended destination.
- Fixed restore_snapshot()'s docstring: the skipped-paths list reports
what would be skipped on dry-run too, it isn't always empty.
* Address Copilot suppressed findings: stream plan_restore, restore-specific lock message
- plan_restore() is now a generator instead of building a full list in
memory, matching fs.scantree()'s streaming style everywhere else in
the codebase.
- restore CLI logs a restore-specific message on lock-acquire failure,
since set_backups_lock()'s own log line talks about a 'previous
backup', which read wrong for a restore.
* Fix PATH docs: snapshot-relative means source-basename-prefixed
README and CLI help examples for restore's PATH argument omitted the
source directory's basename, which backup.initiate_backup() always
prefixes onto snapshot-relative paths - the given examples would have
matched nothing in an actual snapshot.
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
* Add immediate safety warning to README
- Remove the production-use claim from Project Status.
- Mark the project experimental until the P0 data-safety checklist
passes, with a prominent warning banner up top.
- Add a Limitations section: warn against modifying hardlinked
snapshots in place, document the lack of a restore command, and
note special-entry exclusion and symlink-preservation behavior.
- Check off the corresponding IMPROVEMENTS.md acceptance criteria.
* Qualify special-entry exclusion limitation to the Python implementation
--external-rsync uses rsync's --archive flag, which copies/recreates
FIFOs, sockets, and device nodes instead of excluding them - the
exclusion behavior only applies to the default Python implementation.
* Qualify device-node recreation reliability under --external-rsync
rsync -a implies -D (devices/specials), but creating device nodes
typically requires root on the destination - without it rsync may
skip or fail on them rather than reliably recreating them.
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
* Handle special filesystem entries explicitly
- Added fs.entry_kind(): classifies via lstat() mode bits alone
("file", "dir", "symlink", "fifo", "socket", "block_device",
"char_device", "unknown") - never opens the entry, so classifying a
FIFO can't block even without a writer.
- rsync() now classifies every source entry before touching it and
yields a new Actions.EXCLUDE for anything outside file/dir/symlink,
instead of falling through to copy_direntry()'s file branch, which
would either block forever opening a writerless FIFO for read, or
raise an OSError that initiate_backup() treats as fatal (aborting the
whole backup for one socket).
- initiate_backup() records exclusions in the completion manifest
(keyed by snapshot-relative path, valued with the excluded kind), and
no longer nest_hardlinks or counts a recurring exclusion as a change
worth keeping a new snapshot for.
- Chose the "recorded exclusion" policy the plan allows, rather than
fail-fast: a FIFO or socket living in a backed-up tree is not itself
a backup failure, and previously turned into one.
Note: rsync_ext() (external rsync) is unaffected by this change and
keeps whatever its own --archive flags do with special files; the P1
item on Python/external implementation parity covers reconciling that.
The pre-existing test_src_is_socket failure (AF_UNIX path length on
macOS tmp_path) is untouched here - it's separately tracked under P1
test fixes - but this PR adds working socket/FIFO/char-device coverage
via a short-path tempdir.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Address Copilot review: fix stale exclude, exclusion-only diffs, test portability
- fs.rsync(): remove the existing dst_entry before yielding EXCLUDE for
a src entry with an unsupported kind, so an entry inherited via
hardlink from a prior snapshot doesn't linger in the new one.
- backup.initiate_backup(): compare this run's excluded set against
the previous snapshot's manifest; a change in exclusions alone now
counts as backup_changed, so the updated manifest gets written.
- tests: skip mkfifo/AF_UNIX-dependent tests on platforms without
support, and use TemporaryDirectory/context managers instead of
manual mkdtemp/unlink for more robust cleanup.
* Remove unused shutil import from test_rsync.py
Left over after switching the socket/fifo tests to
tempfile.TemporaryDirectory(), per Copilot's suppressed comment.
* Fix rm_direntry() no-op on special dst entries
rm_direntry() only handled file/symlink/dir - a FIFO, socket, or
device node at dst (e.g. inherited from an external-rsync snapshot)
silently survived removal, since is_file/is_dir/is_symlink all
return False for it. Any non-directory entry is unlinkable the same
way, so invert the branching: recurse for directories, unlink
everything else.
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Prevent stale, truncated, or changing-file copies
- copy_file() now loops until every byte is actually written (os.write
may write fewer bytes than given) and compares fstat() on the
source's own open descriptor before/after the read loop - this
reflects the underlying inode directly, catching a source file
truncated, grown, or overwritten in place while being read, and isn't
fooled by the source path being renamed away mid-copy. Raises
BackupCreationError (already a fatal, staging-discarding condition in
initiate_backup) on a mismatch.
- copy_file() returns a sha256 hex digest of the copied content;
copy_direntry()/update_direntry() propagate it; rsync() carries it
through the previously-unused msg field for CREATE/REWRITE actions.
- initiate_backup() collects those digests into a checksums dict keyed
by path relative to the snapshot root, recorded in the completion
manifest (already JSON since the topology-validation work).
- Added verify_snapshot(): re-hashes every file with a recorded
checksum and reports any mismatch, satisfying the release gate's
"completed snapshot passes manifest verification" requirement.
- Switched every mtime comparison and os.utime() restore (rsync's
file/directory sync, hardlink_dir's directory metadata, the
hardlink-breaking copy-on-write step) from float seconds to
nanoseconds, so a same-second modification can't be missed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Address Copilot review: hash checksums for external rsync too, fix log/error text
- initiate_backup() now hashes CREATE/REWRITE files itself when msg is
empty (fs.rsync_ext(), the external-rsync path, never yields a
digest). Without this, verify_snapshot() silently became a no-op for
every snapshot created with external_rsync=True.
- Fixed a pre-existing log message that said "different inodes" for
the branch whose actual condition is src/dst sharing an inode.
- verify_snapshot()'s error message now also mentions a corrupt
manifest, not just missing/pre-migration, matching what
_read_manifest() actually distinguishes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Preserve symlinks and old-snapshot immutability
Two real bugs found while implementing this item, both in fs.py:
- copy_direntry() checked entry.is_dir() (default follow_symlinks=True)
before entry.is_symlink(). A symlink pointing at a directory was
misclassified as a directory and an empty real directory was created
in its place, silently destroying the symlink and losing its target
entirely. Fixed by checking is_symlink() first, and is_dir() with
follow_symlinks=False.
- rsync()'s permission/ownership-sync branch called os.chmod/os.chown
on dst_entry.path unconditionally. Two problems: (1) for a symlink
whose target already matched, this had no follow_symlinks=False
guard and could rewrite an external symlink target's permissions;
(2) for a regular file, that path may still be hardlinked to the
same file in the previous, completed snapshot (hardlink_dir() links
unchanged files in at the start of every backup) - mutating it in
place silently corrupted every older snapshot sharing the inode.
Fixed by skipping symlinks entirely (nothing to sync once their
target matches) and adding _break_shared_hardlink(), a copy-on-write
step that gives a file its own inode before any in-place metadata
change.
Also fixed the dir-mtime-restore loop's is_dir() call (same
default-follow_symlinks issue, lower severity since its own operations
were already lstat-based/no-follow).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Address Copilot review: preserve ownership/timestamps when breaking hardlinks, fix test fd leak
- _break_shared_hardlink() now explicitly restores the original file's
uid/gid and atime/mtime on the new private copy before it replaces
the shared one. copy_file() only carries over mode at create time, so
without this a caller that only meant to update permissions would
silently also reset ownership/mtime as a side effect of the
copy-on-write break itself. Also cleans up the temp file on failure
and uses a less collision-prone temp name.
- Fixed a file descriptor leak in the new hardlink-immutability test
(bare open(...).read() without a context manager)
- Strengthened the same test to assert mtime survives the break
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Correct retention before enabling automatic deletion
- Compute each retention tier's kept set independently from its own
name range and union them, instead of carrying a single 'previous
backup' comparison across tier boundaries - that carry-over was the
actual cause of bug #32 (a coarser tier could evict a backup a finer
tier, or the always-keep-latest rule, had already decided to retain)
- Group weekly retention by (ISO year, ISO week) instead of ISO week
number alone, fixing bug #33 (same week number in different years,
or late-December dates that belong to next year's ISO week 1, were
treated as duplicates/collisions)
- Validate the computed retention plan (latest snapshot present, no
phantom entries) before deleting anything
- Fix cli.py to run initiate_backup before cleanup_old_backups, and
skip cleanup entirely if the backup raised - retention must only run
once a new snapshot is durable and complete, and a failed backup must
never trigger deletion of history
- Quarantine (rename) a deletion candidate before recursively removing
it, reusing the already-hardened stale-staging-dir cleanup to finish
the job if the process crashes mid-delete
- Remove both retention xfail markers (bugs #32, #33) now that the
tests they guarded pass; update pre-existing tests whose expectations
encoded the old bug (deleting the latest snapshot when it collided
with the next-newest one in a coarser bucket)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Address Copilot review: replace asserts with runtime checks, don't swallow quarantine-delete failures
- Retention-plan validation now raises BackupFailedError instead of
using assert, which python -O would silently strip ahead of a
destructive operation
- _quarantine_and_delete no longer passes ignore_errors=True to
rmtree; a removal failure is now logged with the quarantine path so
operators can find and retry it, instead of silently leaving a
hidden directory consuming space
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Validate topology and destructive path boundaries
- Canonicalize sources/backups_dir and reject overlap in either direction
- Reject duplicate sources and destination basename collisions
- Reject non-normalized relative paths in nest_hardlink
- Give each backup repository a persistent identity, recorded in every
snapshot's completion manifest
- Verify repo identity, directory type (no symlinks), containment, and
device before any recursive snapshot/staging-dir deletion, refusing to
cross mount points
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Address Copilot review: root-path check, symlink/TOCTOU-safe writes, legacy marker adoption
- _is_same_or_within: drop the redundant, root-mishandling startswith
check; commonpath() alone is already component-aware
- _ensure_repo_id/_write_manifest: write via temp-file + atomic rename
(O_NOFOLLOW, O_EXCL on the temp file) instead of open+write in place,
so writes can never go through a pre-existing symlink and are never
left half-written
- _verify_safe_to_delete: distinguish a legacy empty marker (pre-dates
the manifest format, can only be a pre-migration artifact) from a
non-empty-but-invalid one (corrupt or forged); adopt the former into
the repository instead of refusing to delete it forever
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Replace PID-file process killing with a real fcntl.flock() lock
set_backups_lock()/release_backups_lock() now use an exclusive
fcntl.flock() on the lock file's inode instead of trusting the PID
written inside it. The kernel releases the lock automatically on
normal exit, on any signal, and on a crash - no PID liveness check or
staleness heuristic is needed, and there is nothing left in this code
path that sends a signal to another process.
- The open file descriptor is held for the full operation in a
module-level _held_locks map keyed by the backups directory, and
release_backups_lock() only ever releases a descriptor this process
itself acquired, never one it infers from the lock file's content.
- PID and start time are still written into the lock file, but purely
as diagnostics for whoever reads it (e.g. in a warning log line);
they play no role in acquisition or release decisions anymore.
- --force no longer sends SIGTERM/SIGKILL to the previous holder. It
now blocks on the same flock() until that process releases the lock
on its own (exit, signal, or crash all count).
- release_backups_lock() intentionally stops unlinking the lock file:
since flock() locks an inode rather than a path, unlinking while
another process is blocked opening the same path would let that
process (or a third one) acquire an independent lock on a fresh
inode at the same path, defeating mutual exclusion.
Rewrites TestBackupLock to match: a real second file descriptor with
its own flock() stands in for "another process" in the fast unit
tests, and a new subprocess-based test proves end-to-end that only one
process can hold the lock and that killing the holder with SIGKILL
frees it for the next acquirer without our code ever calling os.kill.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Address Copilot review feedback on flock-based lock
- Make the fcntl import optional (try/except ImportError -> None), so
importing curateipsum.backup no longer crashes on Windows before
cli.py's own "Not supported platform" check gets a chance to run.
- Guard set_backups_lock() against being called twice for a directory
this process already holds: it now fails fast instead of silently
overwriting the tracked fd in _held_locks, which would leak the
original descriptor and leave the lock held even after
release_backups_lock() closes only the newest one.
- Wrap the post-acquire metadata write (ftruncate/write/fsync) in a
try/except: a failure there now releases the flock and closes the fd
before returning False, instead of leaving the lock held with
no caller aware of it.
- Cap _read_lock_metadata()'s read and strip newlines before logging,
so a stale or forged lock file can't inject bogus log lines or force
an unbounded read.
Adds a regression test for the double-acquire guard.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Make snapshot creation transactional
Build each snapshot in a private .incomplete-<name> staging directory
instead of mutating the final backup path directly. Any failure while
building the snapshot (hardlink failure, rsync error, or a single
Actions.ERROR entry) is now fatal: the staging directory is discarded
and BackupFailedError is raised, so a failed or interrupted backup can
never silently look like a success.
On success, the completion marker is written and the directory fsync'd
before an atomic os.rename() into the final snapshot name, so a crash
can never leave a partially-populated directory under a real snapshot
name.
Also:
- cleanup_stale_staging_dirs() removes abandoned staging directories
left by a crashed run before a new backup starts.
- _reserve_snapshot_name() avoids same-second snapshot name collisions.
- cli.py releases the backup lock in finally and returns a nonzero
exit code when BackupFailedError is raised.
Updates the error-recovery tests that previously asserted failures
were swallowed silently to assert BackupFailedError is raised instead,
matching the new fail-loud contract.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Address Copilot review feedback on transactional snapshots
- Remove unused uuid import.
- _reserve_snapshot_name() also checks for a collision with an existing
.incomplete-<name> staging directory, not just the final snapshot
name.
- _fsync_dir() opens with O_DIRECTORY where available.
- Fsync the completion marker file itself, and fsync the parent
backups_dir after the rename, so the marker and the rename survive a
crash/power loss.
- Re-raise an already-BackupFailedError unchanged (preserving its
traceback) instead of wrapping it in a new instance; only
fs.BackupCreationError gets wrapped.
- Wrap the initial os.mkdir(staging.path) failure in BackupFailedError
so every failure path uses the documented exception.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Maxim Shulga <maxim.shulga@alludo.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
