From 647745bb848c1afcb971374164e06d5356ce9b76 Mon Sep 17 00:00:00 2001 From: "rrnewton@63c9faa1997c908b1dc04dfdca33c809660cd158" Date: Mon, 21 May 2018 05:12:20 +0000 Subject: [PATCH 1/3] --- ...s_ls-files_without_--exclude-standard.mdwn | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 doc/bugs/git_proxy_uses_ls-files_without_--exclude-standard.mdwn diff --git a/doc/bugs/git_proxy_uses_ls-files_without_--exclude-standard.mdwn b/doc/bugs/git_proxy_uses_ls-files_without_--exclude-standard.mdwn new file mode 100644 index 0000000000..9e634cf995 --- /dev/null +++ b/doc/bugs/git_proxy_uses_ls-files_without_--exclude-standard.mdwn @@ -0,0 +1,38 @@ +### Please describe the problem. + +I'm starting to slowly migrate my personal data collection (530GB 3.7M files) under git-annex. I'm going piece by piece, and not yet giving up my other synchronization methods (yet); thus I need to stay in direct mode. + +I initially found that `git annex` commands were quite slow (i.e. https://github.com/datalad/datalad/issues/17), but I was able to address that by adding all my "not yet included" directories to `.gitignore` at the root of the working tree. Unfortunately, `git annex proxy` remains super slow, because I notice that it does not include `--exclude-standard` in its calls to ls-files, and thus does not respect `.gitignore`. Here's an example from the `--debug` log: + +``` + read: git ["--git-dir=../../../../../.git","--work-tree=../../../../..","--literal-pathspecs","-c","core.bare=false","ls-files","--others","-z","--","../../../../.."] +``` + +As a result, I was very shocked to learn that (25 min later), `git annex proxy` was still setting up, and had duplicated 140GB of untracked files! + +My end goal is actually just to add files directly to the git repo, bypassing the annex, in spite of being in direct mode. (I can do this with the largefiles attribute, but I'd like to be able to control it directly irrespective of size.) + +### What steps will reproduce the problem? + +``` +git annex proxy --debug -- git commit myfile -m foo +``` + +### What version of git-annex are you using? On what operating system? + +Version 6.20170520 on Mac OS 10.13.4. + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Today is my first day trying it out! It's fabulous so far, but I'm at the beginning of the learning curve. + From 6a0379a056ed85756161b932f6f6617e4787e2e9 Mon Sep 17 00:00:00 2001 From: "rrnewton@63c9faa1997c908b1dc04dfdca33c809660cd158" Date: Mon, 21 May 2018 05:13:10 +0000 Subject: [PATCH 2/3] tweak url --- .../git_proxy_uses_ls-files_without_--exclude-standard.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/bugs/git_proxy_uses_ls-files_without_--exclude-standard.mdwn b/doc/bugs/git_proxy_uses_ls-files_without_--exclude-standard.mdwn index 9e634cf995..dc0e140526 100644 --- a/doc/bugs/git_proxy_uses_ls-files_without_--exclude-standard.mdwn +++ b/doc/bugs/git_proxy_uses_ls-files_without_--exclude-standard.mdwn @@ -2,7 +2,7 @@ I'm starting to slowly migrate my personal data collection (530GB 3.7M files) under git-annex. I'm going piece by piece, and not yet giving up my other synchronization methods (yet); thus I need to stay in direct mode. -I initially found that `git annex` commands were quite slow (i.e. https://github.com/datalad/datalad/issues/17), but I was able to address that by adding all my "not yet included" directories to `.gitignore` at the root of the working tree. Unfortunately, `git annex proxy` remains super slow, because I notice that it does not include `--exclude-standard` in its calls to ls-files, and thus does not respect `.gitignore`. Here's an example from the `--debug` log: +I initially found that `git annex` commands [were quite slow](https://github.com/datalad/datalad/issues/17), but I was able to address that by adding all my "not yet included" directories to `.gitignore` at the root of the working tree. Unfortunately, `git annex proxy` remains super slow, because I notice that it does not include `--exclude-standard` in its calls to ls-files, and thus does not respect `.gitignore`. Here's an example from the `--debug` log: ``` read: git ["--git-dir=../../../../../.git","--work-tree=../../../../..","--literal-pathspecs","-c","core.bare=false","ls-files","--others","-z","--","../../../../.."] From 3a9248385195213806e117843533d33d94eb0104 Mon Sep 17 00:00:00 2001 From: "rrnewton@63c9faa1997c908b1dc04dfdca33c809660cd158" Date: Mon, 21 May 2018 05:29:06 +0000 Subject: [PATCH 3/3] Added a comment: Flag to force same behavior as annex.largefiles attribute? --- ...mment_1_3286fb304f161df9775366db27cf9530._comment | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/git-annex-add/comment_1_3286fb304f161df9775366db27cf9530._comment diff --git a/doc/git-annex-add/comment_1_3286fb304f161df9775366db27cf9530._comment b/doc/git-annex-add/comment_1_3286fb304f161df9775366db27cf9530._comment new file mode 100644 index 0000000000..f752bbbf9f --- /dev/null +++ b/doc/git-annex-add/comment_1_3286fb304f161df9775366db27cf9530._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="rrnewton@63c9faa1997c908b1dc04dfdca33c809660cd158" + nickname="rrnewton" + avatar="http://cdn.libravatar.org/avatar/638acc3e55c2bb09aa0dcca5b5c8acb6" + subject="Flag to force same behavior as annex.largefiles attribute?" + date="2018-05-21T05:29:06Z" + content=""" +When in [direct mode](https://git-annex.branchable.com/direct_mode), the \"add the non-large file directly to the git repository\" behavior described above is very useful, because the option of typing simply `git add foo`, does not exist as it does in [indirect mode](https://git-annex.branchable.com/git-annex-indirect/). + +However, I can't see any combination of flags that trigger this behavior. I suppose it can be accomplished by temporarily setting [annex.largefiles](https://git-annex.branchable.com/tips/largefiles/) to a huge value before executing `git annex add` (i.e. creating a `.gitattributes` and then deleting it). I think I'll try that as a work-around, but it would be great to have a flag that accomplishes this. + +"""]]