From 8ba9aae481533d613961d1485a85b4a9f3215e30 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Mon, 8 Jul 2019 16:30:37 +0000 Subject: [PATCH 1/6] added a request for more extensive retrying of transiently-failing operations --- .../more_extensive_retries_to_mask_transient_failures.mdwn | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 doc/todo/more_extensive_retries_to_mask_transient_failures.mdwn diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures.mdwn b/doc/todo/more_extensive_retries_to_mask_transient_failures.mdwn new file mode 100644 index 0000000000..ebe3348ee8 --- /dev/null +++ b/doc/todo/more_extensive_retries_to_mask_transient_failures.mdwn @@ -0,0 +1,3 @@ +In a number of scenarios (e.g. [[bugs/still_seeing_errors_with_parallel_git-annex-add]], [[bugs/parallel_copy_fails]], [[git-annex-sync/#comment-aceb18109c0a536e04bcdd3aa04bda29]]), `git-annex` operations may fail or hang due to transient conditions. It would help a lot if `git-annex` could be configured to fail timed-out operations, and to retry failed operations after a delay. This would especially help when using `git-annex` in a script or a higher-level tool. I've tried wrapping some retry logic around `git-annex` calls, but it seems `git-annex` itself is in the best position to do that sensibly (e.g. only retrying idempotent operations, or capping retries per remote). This would be a catch-all fix for unusual conditions that are hard to test for. + +`git-annex` already has config options `annex.retry` and `annex.retry-delay`, but it seems that they don't cover all failure types. From fa3524b953276da6f3a4c6fcd16a3ecf4dc8663c Mon Sep 17 00:00:00 2001 From: kyle Date: Mon, 8 Jul 2019 16:37:50 +0000 Subject: [PATCH 2/6] Added a comment --- .../comment_1_034f772098dfb9d3622c6f84976e69a2._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/Regression_in___96__find_--json__96___output/comment_1_034f772098dfb9d3622c6f84976e69a2._comment diff --git a/doc/bugs/Regression_in___96__find_--json__96___output/comment_1_034f772098dfb9d3622c6f84976e69a2._comment b/doc/bugs/Regression_in___96__find_--json__96___output/comment_1_034f772098dfb9d3622c6f84976e69a2._comment new file mode 100644 index 0000000000..5ecb73e01f --- /dev/null +++ b/doc/bugs/Regression_in___96__find_--json__96___output/comment_1_034f772098dfb9d3622c6f84976e69a2._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="kyle" + avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" + subject="comment 1" + date="2019-07-08T16:37:50Z" + content=""" +Thanks for the quick fix. +"""]] From 9d43570ed638121e96838539fbeb37a0277ce45a Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Mon, 8 Jul 2019 19:00:46 +0000 Subject: [PATCH 3/6] initial summary on an old issue of absent benefit from CoW across subvolumes of BTRFS filesystem --- ...n_volumes_of_the_same_BTRFS_partition.mdwn | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 doc/bugs/Uses_rsync_instead_of___96__cp_--reflink__61__auto__96___on_volumes_of_the_same_BTRFS_partition.mdwn diff --git a/doc/bugs/Uses_rsync_instead_of___96__cp_--reflink__61__auto__96___on_volumes_of_the_same_BTRFS_partition.mdwn b/doc/bugs/Uses_rsync_instead_of___96__cp_--reflink__61__auto__96___on_volumes_of_the_same_BTRFS_partition.mdwn new file mode 100644 index 0000000000..a9a4db000c --- /dev/null +++ b/doc/bugs/Uses_rsync_instead_of___96__cp_--reflink__61__auto__96___on_volumes_of_the_same_BTRFS_partition.mdwn @@ -0,0 +1,19 @@ +### Please describe the problem. + +Originally shown/discussed in [local caching](http://git-annex.branchable.com/tips/local_caching_of_annexed_files/#comment-7f214f4eaa629b7731f82014a2e98964) tips page. Decided to give it a separate page for easier tracking etc. + +In my use case I have `/home` and `/mnt/btrfs/` as two subvolumes of the same drive + +[[!format sh """ +/dev/md10 on /mnt/btrfs type btrfs (rw,noatime,compress=lzo,space_cache,subvolid=5,subvol=/) +/dev/md10 on /home type btrfs (rw,noatime,compress=lzo,space_cache,subvolid=257,subvol=/home) +"""]] + +BTRFS's CoW is a great feature for annex, but whenever I try to `annex get` across those two (git annex version was 7.20190322+git133-g59922f1f4-1~ndall+1) - `rsync` is used instead of `cp`, disks got really busy, and I end up with >70GB of additional space utilization, which is "suboptimal". + +As in the original comments thread, I wonder on what is the advantage of using `rsync` over a regular `cp` across devices? (`Device` from `stat` seems to return different ids across volumes, so a bad indicator) + +If there is some generic benefit from `rsync`, could it may be at least be a configuration setting which I would set globally on machines with btrfs to use `cp` instead of `rsync` for local transfers? + + +[[!meta author="yoh"]] From ec62d078832062ea16feff6f7c04c41313a602e0 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Mon, 8 Jul 2019 19:04:03 +0000 Subject: [PATCH 4/6] Added a comment: FTR: a dedicated issue on CoW across subvolumes --- .../comment_21_f7913b3477fb18b1d6eb6b3ab6bef090._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/tips/local_caching_of_annexed_files/comment_21_f7913b3477fb18b1d6eb6b3ab6bef090._comment diff --git a/doc/tips/local_caching_of_annexed_files/comment_21_f7913b3477fb18b1d6eb6b3ab6bef090._comment b/doc/tips/local_caching_of_annexed_files/comment_21_f7913b3477fb18b1d6eb6b3ab6bef090._comment new file mode 100644 index 0000000000..0061b2260d --- /dev/null +++ b/doc/tips/local_caching_of_annexed_files/comment_21_f7913b3477fb18b1d6eb6b3ab6bef090._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="FTR: a dedicated issue on CoW across subvolumes" + date="2019-07-08T19:04:03Z" + content=""" +https://git-annex.branchable.com/bugs/Uses_rsync_instead_of___96__cp_--reflink__61__auto__96___on_volumes_of_the_same_BTRFS_partition/?updated +"""]] From 537c280e75bdd5eade611be37cec4052084d6fcd Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Mon, 8 Jul 2019 19:05:00 +0000 Subject: [PATCH 5/6] removed --- .../comment_21_f7913b3477fb18b1d6eb6b3ab6bef090._comment | 8 -------- 1 file changed, 8 deletions(-) delete mode 100644 doc/tips/local_caching_of_annexed_files/comment_21_f7913b3477fb18b1d6eb6b3ab6bef090._comment diff --git a/doc/tips/local_caching_of_annexed_files/comment_21_f7913b3477fb18b1d6eb6b3ab6bef090._comment b/doc/tips/local_caching_of_annexed_files/comment_21_f7913b3477fb18b1d6eb6b3ab6bef090._comment deleted file mode 100644 index 0061b2260d..0000000000 --- a/doc/tips/local_caching_of_annexed_files/comment_21_f7913b3477fb18b1d6eb6b3ab6bef090._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="FTR: a dedicated issue on CoW across subvolumes" - date="2019-07-08T19:04:03Z" - content=""" -https://git-annex.branchable.com/bugs/Uses_rsync_instead_of___96__cp_--reflink__61__auto__96___on_volumes_of_the_same_BTRFS_partition/?updated -"""]] From ec3c16b51704f0d1247d05a102d776fd3d7765a1 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Mon, 8 Jul 2019 19:05:45 +0000 Subject: [PATCH 6/6] Added a comment: FTR: a dedicated issue on CoW across subvolumes --- .../comment_21_3776a0371faccc673c868323278f3b0d._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/tips/local_caching_of_annexed_files/comment_21_3776a0371faccc673c868323278f3b0d._comment diff --git a/doc/tips/local_caching_of_annexed_files/comment_21_3776a0371faccc673c868323278f3b0d._comment b/doc/tips/local_caching_of_annexed_files/comment_21_3776a0371faccc673c868323278f3b0d._comment new file mode 100644 index 0000000000..d9f2ca4c3a --- /dev/null +++ b/doc/tips/local_caching_of_annexed_files/comment_21_3776a0371faccc673c868323278f3b0d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="FTR: a dedicated issue on CoW across subvolumes" + date="2019-07-08T19:05:45Z" + content=""" +[bugs/Uses_rsync_instead_of___96__cp_--reflink__61__auto__96___on_volumes_of_the_same_BTRFS_partition/](https://git-annex.branchable.com/bugs/Uses_rsync_instead_of___96__cp_--reflink__61__auto__96___on_volumes_of_the_same_BTRFS_partition/) +"""]]