From b11aa063ae0c6183222afd81dca9964553a332e6 Mon Sep 17 00:00:00 2001 From: Lukey Date: Tue, 19 Jan 2021 16:27:07 +0000 Subject: [PATCH 1/4] Added a comment --- ...comment_1_9d812c6dd2fd7b4c255ea88580da4396._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/todo/Incremental_git_annex_sync_--content_--all/comment_1_9d812c6dd2fd7b4c255ea88580da4396._comment diff --git a/doc/todo/Incremental_git_annex_sync_--content_--all/comment_1_9d812c6dd2fd7b4c255ea88580da4396._comment b/doc/todo/Incremental_git_annex_sync_--content_--all/comment_1_9d812c6dd2fd7b4c255ea88580da4396._comment new file mode 100644 index 0000000000..0c7148c739 --- /dev/null +++ b/doc/todo/Incremental_git_annex_sync_--content_--all/comment_1_9d812c6dd2fd7b4c255ea88580da4396._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 1" + date="2021-01-19T16:27:07Z" + content=""" +So I implemented this with an bash script, see [[here|https://gist.github.com/Lukey3332/203ea6f30d48323e7bd1d05c16b5da9c]]. It's a bit hacky, since `git annex sync` doesn't have `--batch` and even if it did, `--batch` can't work with keys. Instead it check out a temporary branch, link the keys to sync in a directory and then uses --content-of= to sync only the keys within that directory. + +I'll use it with my repos and see if it works reliably. +"""]] From da428b152de24e4bd8923c689c7702cec0fd591a Mon Sep 17 00:00:00 2001 From: Lukey Date: Tue, 19 Jan 2021 16:45:31 +0000 Subject: [PATCH 2/4] Added a comment --- ..._1_a38e0ecef49a81a1d2b3d2eaabab65ab._comment | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_1_a38e0ecef49a81a1d2b3d2eaabab65ab._comment diff --git a/doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_1_a38e0ecef49a81a1d2b3d2eaabab65ab._comment b/doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_1_a38e0ecef49a81a1d2b3d2eaabab65ab._comment new file mode 100644 index 0000000000..11ba2c0d15 --- /dev/null +++ b/doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_1_a38e0ecef49a81a1d2b3d2eaabab65ab._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 1" + date="2021-01-19T16:45:30Z" + content=""" +direct-mode doesn't exist anymore, it is replaced with the [[annex.thin config|https://git-annex.branchable.com/tips/unlocked_files]] (See \"using less disk space\"). And yes, it works on NTFS. + +It definitely isn't going to be fast, the numbers you gave suggests that there will be ~1000000 files per repository (For the V*-related dirs). Still you should try it and see if it's fast enough for you. Some tips to improve performance: Don't use `include=`/`exclude=` in preferred-content-expressions and [[Repositories with large number of files|https://git-annex.branchable.com/tips/Repositories_with_large_number_of_files/]]. My experimental script [[here|https://git-annex.branchable.com/todo/Incremental_git_annex_sync_--content_--all/]] might also be worth a try. + +Having the root of the repo on the root of the drive and then excluding everything that shouldn't be in the repo via `.gitignore` can be a vivable approach. But with that many files I'd create one repo per directory. It could also be done with git worktrees. + +Don't use git-annex on top of a network share, in that case run it directly on the server. git-annex is designed to run on local drives/storage. Also, git-annex on windows is way slower than on linux. + +You can donate via [[Patreon|https://www.patreon.com/joeyh]] and [[Liberapay|https://liberapay.com/joeyh/]]. + +"""]] From 40107f1ef1aba8711fc98af083e9f84c89cdd2a9 Mon Sep 17 00:00:00 2001 From: Lukey Date: Tue, 19 Jan 2021 16:49:05 +0000 Subject: [PATCH 3/4] Added a comment --- ..._2_6238b3315dd058b1b8376d06359f23e8._comment | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_2_6238b3315dd058b1b8376d06359f23e8._comment diff --git a/doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_2_6238b3315dd058b1b8376d06359f23e8._comment b/doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_2_6238b3315dd058b1b8376d06359f23e8._comment new file mode 100644 index 0000000000..0753793d2d --- /dev/null +++ b/doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_2_6238b3315dd058b1b8376d06359f23e8._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 2" + date="2021-01-19T16:49:05Z" + content=""" +direct-mode doesn't exist anymore, it is replaced with the [[annex.thin config|https://git-annex.branchable.com/tips/unlocked_files]] (See \"using less disk space\"). And yes, it works on NTFS. + +It definitely isn't going to be fast, the numbers you gave suggests that there will be ~1000000 files per repository (For the V*-related dirs). Still you should try it and see if it's fast enough for you. Some tips to improve performance: Don't use `include=`/`exclude=` in preferred-content-expressions and [[Repositories with large number of files|https://git-annex.branchable.com/tips/Repositories_with_large_number_of_files/]]. My experimental script [[here|https://git-annex.branchable.com/todo/Incremental_git_annex_sync_--content_--all/]] might also be worth a try. + +Having the root of the repo on the root of the drive and then excluding everything that shouldn't be in the repo via `.gitignore` can be a vivable approach. But with that many files I'd create one repo per directory. It could also be done with git worktrees. + +Don't use git-annex on top of a network share, in that case run it directly on the server. git-annex is designed to run on local drives/storage. Also, git-annex on windows is way slower than on linux. + +You can donate via [[Patreon|https://www.patreon.com/joeyh]] and [[Liberapay|https://liberapay.com/joeyh/]]. + +"""]] From 41cc2af2456630ce18020178192c310d5e3abd5d Mon Sep 17 00:00:00 2001 From: Lukey Date: Tue, 19 Jan 2021 16:49:57 +0000 Subject: [PATCH 4/4] removed --- ..._2_6238b3315dd058b1b8376d06359f23e8._comment | 17 ----------------- 1 file changed, 17 deletions(-) delete mode 100644 doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_2_6238b3315dd058b1b8376d06359f23e8._comment diff --git a/doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_2_6238b3315dd058b1b8376d06359f23e8._comment b/doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_2_6238b3315dd058b1b8376d06359f23e8._comment deleted file mode 100644 index 0753793d2d..0000000000 --- a/doc/forum/Large_Windows_annexes_with___42__nix_remotes/comment_2_6238b3315dd058b1b8376d06359f23e8._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="Lukey" - avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" - subject="comment 2" - date="2021-01-19T16:49:05Z" - content=""" -direct-mode doesn't exist anymore, it is replaced with the [[annex.thin config|https://git-annex.branchable.com/tips/unlocked_files]] (See \"using less disk space\"). And yes, it works on NTFS. - -It definitely isn't going to be fast, the numbers you gave suggests that there will be ~1000000 files per repository (For the V*-related dirs). Still you should try it and see if it's fast enough for you. Some tips to improve performance: Don't use `include=`/`exclude=` in preferred-content-expressions and [[Repositories with large number of files|https://git-annex.branchable.com/tips/Repositories_with_large_number_of_files/]]. My experimental script [[here|https://git-annex.branchable.com/todo/Incremental_git_annex_sync_--content_--all/]] might also be worth a try. - -Having the root of the repo on the root of the drive and then excluding everything that shouldn't be in the repo via `.gitignore` can be a vivable approach. But with that many files I'd create one repo per directory. It could also be done with git worktrees. - -Don't use git-annex on top of a network share, in that case run it directly on the server. git-annex is designed to run on local drives/storage. Also, git-annex on windows is way slower than on linux. - -You can donate via [[Patreon|https://www.patreon.com/joeyh]] and [[Liberapay|https://liberapay.com/joeyh/]]. - -"""]]