From e7b0754171a95b42d2c433664c6a7b4bfaf21bf9 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Mon, 4 Jan 2021 12:26:48 -0400 Subject: [PATCH] comment and todo --- ..._a4e2f0e65a9dc92fc4dd85183e8f8090._comment | 21 +++++++++++++++++++ ...speed_up_git_annex_add_of_small_files.mdwn | 16 ++++++++++++++ 2 files changed, 37 insertions(+) create mode 100644 doc/forum/Adding_files_to_git__58___Very_long___34__recording_state_in_git__34___phase/comment_7_a4e2f0e65a9dc92fc4dd85183e8f8090._comment create mode 100644 doc/todo/speed_up_git_annex_add_of_small_files.mdwn diff --git a/doc/forum/Adding_files_to_git__58___Very_long___34__recording_state_in_git__34___phase/comment_7_a4e2f0e65a9dc92fc4dd85183e8f8090._comment b/doc/forum/Adding_files_to_git__58___Very_long___34__recording_state_in_git__34___phase/comment_7_a4e2f0e65a9dc92fc4dd85183e8f8090._comment new file mode 100644 index 0000000000..367202c85a --- /dev/null +++ b/doc/forum/Adding_files_to_git__58___Very_long___34__recording_state_in_git__34___phase/comment_7_a4e2f0e65a9dc92fc4dd85183e8f8090._comment @@ -0,0 +1,21 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 7""" + date="2021-01-04T16:13:36Z" + content=""" +This will avoid the overhead of the smudge filter, when all the files +you're adding are ones you want stored in git, *not* in git-annex. + + git-annex add --force-small + +I do think it would be possible for `git-annex add` to use the same method +whenever it adds non-large files. But it might have unwanted other effects, +since the way that manages to be fast is by avoding using `git add` and +having git-annex hash the file and add it to git itself. Opened +[[todo/speed_up_git_annex_add_of_small_files]] to consider this. + +The only way to speed up `git add` is to disable the smudge filter, but then +all files you `git add` will be stored in git, not in git-annex. And +disabling the smudge filter also will prevent using unlocked annexed files. +(See [[todo/git_smudge_clean_interface_suboptiomal]] for background.) +"""]] diff --git a/doc/todo/speed_up_git_annex_add_of_small_files.mdwn b/doc/todo/speed_up_git_annex_add_of_small_files.mdwn new file mode 100644 index 0000000000..1996e40cc6 --- /dev/null +++ b/doc/todo/speed_up_git_annex_add_of_small_files.mdwn @@ -0,0 +1,16 @@ +When adding a lot of small files to git with `git annex add`, +it is slow because git runs the smudge filter on all files +and [[that_is_slow|todo/git_smudge_clean_interface_suboptiomal]]. + +But `git-annex add --force-small` is much much faster, because that +bypasses git add entirely, hashing the content and staging it in the index +from git-annex. So could that same method be used to speed up the slow case? + +My concern with doing this is that there may be things that `git add` +does that are not done when bypassing it. The only one I can think of is, +if the user has other smudge/clean filters than the git-annex one +installed, they would not be run either. It could be argued that's a bug +with the existing `--force-small` too, but at least that's not the default. + +Possible alternate approach: Unsetting filter.annex.smudge and +filter.annex.clean when running `git add`?