From 49c4d471ff82adccb903c498bbdd86729fc07435 Mon Sep 17 00:00:00 2001 From: kyle Date: Thu, 17 Dec 2020 17:41:36 +0000 Subject: [PATCH 1/3] Added a comment --- ..._d765f93511e0ba446f54f3ad30e91e6f._comment | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_3_d765f93511e0ba446f54f3ad30e91e6f._comment diff --git a/doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_3_d765f93511e0ba446f54f3ad30e91e6f._comment b/doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_3_d765f93511e0ba446f54f3ad30e91e6f._comment new file mode 100644 index 0000000000..5a0372ca92 --- /dev/null +++ b/doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_3_d765f93511e0ba446f54f3ad30e91e6f._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="kyle" + avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" + subject="comment 3" + date="2020-12-17T17:41:35Z" + content=""" +> Is this a case for an rsync remote? (I haven't really figured out +> special remotes yet.) Or is there a typical workflow on the git annex +> side that I could be using to fix this (like `import` rather than +> `add`)? + +I think conceptually that's a good fit. You could set +`importtree=yes` with the special remote and ingest changes with `git +annex import` on beta's side. However, the rsync special remote +doesn't support `importtree` yet. + +https://git-annex.branchable.com/todo/import_tree_from_rsync_special_remote/ + +In your followup comment, you mention unlocked files. That would get +you around the link problem. You could call `rsync` with `--checksum` +to limit what is transferred, though that might be expensive depending +on how big your files are. + +> Are [the annex.hardlink and annex.thin options] annex wide settings? +> (that seems to be the case). Is it possible to apply them at a +> folder level? + +You can set then at the repository level in the repo's .git/config. + +"""]] From c53057caa3e5e2114d6e448f1d1b40abeb85cb53 Mon Sep 17 00:00:00 2001 From: "dscheffy@c203b7661ec8c1ebd53e52627c84536c5f0c9026" Date: Thu, 17 Dec 2020 18:14:16 +0000 Subject: [PATCH 2/3] Added a comment --- ..._62cf96d62228b52bf7dd52eab0b94f67._comment | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_4_62cf96d62228b52bf7dd52eab0b94f67._comment diff --git a/doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_4_62cf96d62228b52bf7dd52eab0b94f67._comment b/doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_4_62cf96d62228b52bf7dd52eab0b94f67._comment new file mode 100644 index 0000000000..f61165767e --- /dev/null +++ b/doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_4_62cf96d62228b52bf7dd52eab0b94f67._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="dscheffy@c203b7661ec8c1ebd53e52627c84536c5f0c9026" + nickname="dscheffy" + avatar="http://cdn.libravatar.org/avatar/62a3a0bf0e203e3746eedbe48fd13f6d" + subject="comment 4" + date="2020-12-17T18:14:16Z" + content=""" +Oh boy -- or should I say, oh \"`man`\"... Now I feel like a bit of an idiot for not checking the actual high level man pages... Thanks for that tip. + +I did some experimenting and noticed another thing that I hadn't noticed -- although my binary version is v6.20180227 my annexes were all using the v5 index. That came as a bit of a surprise. Once I upgraded my annex to v6 the annex.thin settings started working. + +As for rsync, I had tried the `-c` (`--checksum`) option, but it wasn't dereferencing the links on the target side, so the files still registered as different (at least I think I tried this, but I may go back and check again, because I was doing a lot of different things...) Nevermind, I just checked my history and I never actually tried it -- I had been using + +``` +rsync -n -v -r -c --delete ai2/ beta:/media/winston/library/lectures/coursera2/ai2/ +``` + +to try to confirm that the contents of the two folders matched so that I'd know I could safely delete my local copy, but that didn't work because of the links. I didn't add the `-L` option until I reversed the direction and ran the command from the server side with `alpha` as the target. + +Thanks for all the help -- this should work well enough for my stage folder issue, but it also solves a separate problem that I'd been struggling with for making my photos available to a self hosted photo webserver tool that I was trying out (photoprism). It can't currently handle symlinks and my local drive was getting filled up with all the extra copies of my photos directory tree! + +"""]] From 48da7e002d1c2fd7b39f4b27cc7717455d3e2a5b Mon Sep 17 00:00:00 2001 From: kyle Date: Thu, 17 Dec 2020 19:05:13 +0000 Subject: [PATCH 3/3] Added a comment --- ..._40e4e9fe7d301b5e75140136d0132860._comment | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_5_40e4e9fe7d301b5e75140136d0132860._comment diff --git a/doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_5_40e4e9fe7d301b5e75140136d0132860._comment b/doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_5_40e4e9fe7d301b5e75140136d0132860._comment new file mode 100644 index 0000000000..d797576390 --- /dev/null +++ b/doc/forum/rsync_into_annex_without_overwriting_links__63__/comment_5_40e4e9fe7d301b5e75140136d0132860._comment @@ -0,0 +1,26 @@ +[[!comment format=mdwn + username="kyle" + avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" + subject="comment 5" + date="2020-12-17T19:05:12Z" + content=""" +> As for rsync, I had tried the -c (--checksum) option, but it wasn't +> dereferencing the links on the target side + +Just to clarify: My comment was in the context of unlocked files (in +v6+ repos). In that case, symlinks aren't used: the content is kept +in the working tree (and a pointer file is tracked by git). + +Also, since it sounds like you may want all files to be unlocked, you +might want to look into `git annex adjust --unlock` to enter an +adjusted with all files in an unlocked state. + +> it also solves a separate problem that I'd been struggling with for +> making my photos available to a self hosted photo webserver tool +> that I was trying out (photoprism). It can't currently handle +> symlinks [...] + +FWIW, if you don't need importing for this use case, I think using +`git annex export` with an rsync special remote configured with +`exporttree=yes` would work well. +"""]]