From b0684e756738d891621785646a496860fe70ba8d Mon Sep 17 00:00:00 2001 From: jodumont Date: Fri, 19 Feb 2021 04:39:35 +0000 Subject: [PATCH 01/11] Added a comment: Explanation for a noGUI usage --- ...nt_5_8d657646080bf2b99b7792993b877502._comment | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/forum/Starting_assistant_from_CLI/comment_5_8d657646080bf2b99b7792993b877502._comment diff --git a/doc/forum/Starting_assistant_from_CLI/comment_5_8d657646080bf2b99b7792993b877502._comment b/doc/forum/Starting_assistant_from_CLI/comment_5_8d657646080bf2b99b7792993b877502._comment new file mode 100644 index 0000000000..f327bab3be --- /dev/null +++ b/doc/forum/Starting_assistant_from_CLI/comment_5_8d657646080bf2b99b7792993b877502._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="jodumont" + avatar="http://cdn.libravatar.org/avatar/343b334cbc4b2bff019ab6b525a2c9ed" + subject="Explanation for a noGUI usage" + date="2021-02-19T04:39:34Z" + content=""" +I had that issue, git-annex is installed on a noGUI remote machine so the directory and the file **~/.config/git-annex/autostart** didn't exist. +I think it worth to mention than you could create by hand **~/.config/git-annex/autostart** + +> mkdir ~/.config/git-annex/; touch ~/.config/git-annex/autostart + +and in simply add the path of your git-annex you want to start such as **/srv/data** for this example (*which contain a .git/annex/ directory*) then you are ready to go + +> cd /srv/data; git annex assistant --autostart +"""]] From a16d0a5f90068bdf4d3a738d1ec12137642b70c9 Mon Sep 17 00:00:00 2001 From: jodumont Date: Fri, 19 Feb 2021 04:56:47 +0000 Subject: [PATCH 02/11] Added a comment: how to rename/remove the [here] ?? --- ..._a13ef241d6d24b4abee3eedff18032a8._comment | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/forum/Rename_local_repository_for_git-annex-info/comment_2_a13ef241d6d24b4abee3eedff18032a8._comment diff --git a/doc/forum/Rename_local_repository_for_git-annex-info/comment_2_a13ef241d6d24b4abee3eedff18032a8._comment b/doc/forum/Rename_local_repository_for_git-annex-info/comment_2_a13ef241d6d24b4abee3eedff18032a8._comment new file mode 100644 index 0000000000..6080a6fdbe --- /dev/null +++ b/doc/forum/Rename_local_repository_for_git-annex-info/comment_2_a13ef241d6d24b4abee3eedff18032a8._comment @@ -0,0 +1,21 @@ +[[!comment format=mdwn + username="jodumont" + avatar="http://cdn.libravatar.org/avatar/343b334cbc4b2bff019ab6b525a2c9ed" + subject="how to rename/remove the [here] ??" + date="2021-02-19T04:56:46Z" + content=""" +I'm able to change the description of my repo but not the name, which is by default and in that case **[here]** + +> user@machine:/srv/data$ git-annex info +trusted repositories: 0 +semitrusted repositories: 3 +00000000-0000-0000-0000-000000000001 -- web +00000000-0000-0000-0000-000000000002 -- bittorrent +xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx -- sub.domain.tld:/srv/data **[here]** +untrusted repositories: xxx +transfers in progress: xxxx +available local disk space: 2.03 petabytes (+1 gigabyte reserved) + +## How I could edit the name and/or name it properly when I init it ? + +"""]] From 6708bff1a9fbe2d1c9216feb120db3eb4a83a283 Mon Sep 17 00:00:00 2001 From: Lukey Date: Fri, 19 Feb 2021 07:16:27 +0000 Subject: [PATCH 03/11] Added a comment --- .../comment_2_e62b59d5c9cee04a77c51799624a3357._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/git-annex-import/comment_2_e62b59d5c9cee04a77c51799624a3357._comment diff --git a/doc/git-annex-import/comment_2_e62b59d5c9cee04a77c51799624a3357._comment b/doc/git-annex-import/comment_2_e62b59d5c9cee04a77c51799624a3357._comment new file mode 100644 index 0000000000..7eade46253 --- /dev/null +++ b/doc/git-annex-import/comment_2_e62b59d5c9cee04a77c51799624a3357._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 2" + date="2021-02-19T07:16:25Z" + content=""" +Yes, just `cp` or `mv` the files inside the repo and `git annex add` them as usual. +"""]] From 4fc0c58e24633ae3a7e9498bcf2fa1bb1c9f88f3 Mon Sep 17 00:00:00 2001 From: Lukey Date: Fri, 19 Feb 2021 07:28:15 +0000 Subject: [PATCH 04/11] Added a comment --- .../comment_3_c75652b23bc4554d43593d1984c9c5d9._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/Rename_local_repository_for_git-annex-info/comment_3_c75652b23bc4554d43593d1984c9c5d9._comment diff --git a/doc/forum/Rename_local_repository_for_git-annex-info/comment_3_c75652b23bc4554d43593d1984c9c5d9._comment b/doc/forum/Rename_local_repository_for_git-annex-info/comment_3_c75652b23bc4554d43593d1984c9c5d9._comment new file mode 100644 index 0000000000..3d268b811c --- /dev/null +++ b/doc/forum/Rename_local_repository_for_git-annex-info/comment_3_c75652b23bc4554d43593d1984c9c5d9._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 3" + date="2021-02-19T07:28:12Z" + content=""" +AFAIK, the `[here]` can't be changed, it just means \"it is this local repository\". Now, if you add another repository as a git remote, then it'll show `[]` for the git remote. +"""]] From 5e2f2f7d9b04f0629b6336b1768df17295b08bfd Mon Sep 17 00:00:00 2001 From: jwrauch Date: Fri, 19 Feb 2021 14:14:13 +0000 Subject: [PATCH 05/11] --- doc/forum/Help_with_get.mdwn | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/forum/Help_with_get.mdwn diff --git a/doc/forum/Help_with_get.mdwn b/doc/forum/Help_with_get.mdwn new file mode 100644 index 0000000000..997a5baf35 --- /dev/null +++ b/doc/forum/Help_with_get.mdwn @@ -0,0 +1 @@ +I'm new to git annex, but I couldn't find any other posts describing this error. I just set up repositories on two external 1TB, NTFS formatted hard drives and I'm working on a windows 10 os. I followed the walkthrough, setting the repository up on one drive, then cloning it on the other, and now I'm syncing the content (I first sync then get, but the problem occurs when I use the --content flag on sync as well). After a few files are successfully transferred, the process stalls part way through a file saying an "scp error writing 'file': Permission denied"??? Git annex and git processes don't end and I need to forcefully close the terminal and unplug and replug the drive back in before I can resume. I don't know if it is something with using git annex on Windows(I was automatically put on an adjusted branch, which I don't fully understand), or the NTFS drives, or maybe the hard drive is failing. Any advice people can give would be appreciated as this issue is persistent enough (every ~3 files) that it is making git annex unusable. From fbbd1d7cf1fc91da4a1d95bdcd00f8b4c8b8d56b Mon Sep 17 00:00:00 2001 From: Lukey Date: Fri, 19 Feb 2021 15:43:48 +0000 Subject: [PATCH 06/11] Added a comment --- .../comment_1_439c7562d94a969fd841cc44628abedd._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/forum/Help_with_get/comment_1_439c7562d94a969fd841cc44628abedd._comment diff --git a/doc/forum/Help_with_get/comment_1_439c7562d94a969fd841cc44628abedd._comment b/doc/forum/Help_with_get/comment_1_439c7562d94a969fd841cc44628abedd._comment new file mode 100644 index 0000000000..8ac255198f --- /dev/null +++ b/doc/forum/Help_with_get/comment_1_439c7562d94a969fd841cc44628abedd._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 1" + date="2021-02-19T15:43:46Z" + content=""" +Hi, +What version of git-annex are you using (Output of `git annex version`)? And since you are on windows, did you follow the install instructions [[here|install/Windows/]] or are you using wsl or something else? Finally, can you run the failing command with `--debug` and post the output here? +"""]] From dab459787f37aab1ba4c7eaae2b10a35f963fe66 Mon Sep 17 00:00:00 2001 From: jwrauch Date: Fri, 19 Feb 2021 16:10:03 +0000 Subject: [PATCH 07/11] Added a comment --- .../comment_2_137f89ea9c381cdbf57da78308a46078._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/Help_with_get/comment_2_137f89ea9c381cdbf57da78308a46078._comment diff --git a/doc/forum/Help_with_get/comment_2_137f89ea9c381cdbf57da78308a46078._comment b/doc/forum/Help_with_get/comment_2_137f89ea9c381cdbf57da78308a46078._comment new file mode 100644 index 0000000000..13bf43f4ba --- /dev/null +++ b/doc/forum/Help_with_get/comment_2_137f89ea9c381cdbf57da78308a46078._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="jwrauch" + avatar="http://cdn.libravatar.org/avatar/0e83010187ef0c8c587e0c99e2ca5474" + subject="comment 2" + date="2021-02-19T16:10:00Z" + content=""" +I'm using git annex version: 8.20200815-g335aae266. And I followed the instructions the link provided. I looked into wls but decided to try this first. Is that a better approach? +"""]] From 79fb2f499db7b294dc0d078881a662dc048cebc9 Mon Sep 17 00:00:00 2001 From: jwrauch Date: Fri, 19 Feb 2021 16:17:34 +0000 Subject: [PATCH 08/11] Added a comment --- ...t_3_f523e44daca53b3d99350e0d3f7a8874._comment | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/forum/Help_with_get/comment_3_f523e44daca53b3d99350e0d3f7a8874._comment diff --git a/doc/forum/Help_with_get/comment_3_f523e44daca53b3d99350e0d3f7a8874._comment b/doc/forum/Help_with_get/comment_3_f523e44daca53b3d99350e0d3f7a8874._comment new file mode 100644 index 0000000000..ea7c519843 --- /dev/null +++ b/doc/forum/Help_with_get/comment_3_f523e44daca53b3d99350e0d3f7a8874._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="jwrauch" + avatar="http://cdn.libravatar.org/avatar/0e83010187ef0c8c587e0c99e2ca5474" + subject="comment 3" + date="2021-02-19T16:17:33Z" + content=""" +I ran the get command with the debug flag and the output is pretty extensive. It successfully transferred two files before it failed, so I'm posting from the 'ok' of the last success. Here is the command and output: + +$ git annex get . --debug +... +ok +get recorder/2020-philippines/20200906_165152.mp4 (from avi...) +[2021-02-19 11:11:22.1489056] call: cp [\"--reflink=auto\",\"--preserve=timestamps\",\"E:\\video\\.git\\annex\\objects\\d88\\080\\SHA256E-s493726419--95dd6a377dce064935ab2e8c0dda2f3c4d9d2b33e0efaca34fceda200cbee249.mp4\\SHA256E-s493726419--95dd6a377dce064935ab2e8c0dda2f3c4d9d2b33e0efaca34fceda200cbee249.mp4\",\".git\\annex\\tmp\\SHA256E-s493726419--95dd6a377dce064935ab2e8c0dda2f3c4d9d2b33e0efaca34fceda200cbee249.mp4\"] +10% 46.12 MiB 87 MiB/s 4scp: error writing '.git\annex\tmp\SHA256E-s493726419--95dd6a377dce064935ab2e8c0dda2f3c4d9d2b33e0efaca34fceda200cbee249.mp4': Permission denied + +"""]] From 904f25c559bc9c27b7cfc5335734a78a905b2595 Mon Sep 17 00:00:00 2001 From: jwrauch Date: Fri, 19 Feb 2021 16:19:53 +0000 Subject: [PATCH 09/11] Added a comment --- ...nt_4_863d03a8e52335d8a210320d8a02c553._comment | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/forum/Help_with_get/comment_4_863d03a8e52335d8a210320d8a02c553._comment diff --git a/doc/forum/Help_with_get/comment_4_863d03a8e52335d8a210320d8a02c553._comment b/doc/forum/Help_with_get/comment_4_863d03a8e52335d8a210320d8a02c553._comment new file mode 100644 index 0000000000..1183ecc7c0 --- /dev/null +++ b/doc/forum/Help_with_get/comment_4_863d03a8e52335d8a210320d8a02c553._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="jwrauch" + avatar="http://cdn.libravatar.org/avatar/0e83010187ef0c8c587e0c99e2ca5474" + subject="comment 4" + date="2021-02-19T16:19:52Z" + content=""" +Sorry, formatting: + + $ git annex get . --debug + ok + get recorder/2020-philippines/20200906_165152.mp4 (from avi...) + [2021-02-19 11:11:22.1489056] call: cp [\"--reflink=auto\",\"--preserve=timestamps\",\"E:\\video\\.git\\annex\\objects\\d88\\080\\SHA256E-s493726419--95dd6a377dce064935ab2e8c0dda2f3c4d9d2b33e0efaca34fceda200cbee249.mp4\\SHA256E- s493726419--95dd6a377dce064935ab2e8c0dda2f3c4d9d2b33e0efaca34fceda200cbee249.mp4\",\".git\\annex\\tmp\\SHA256E-s493726419--95dd6a377dce064935ab2e8c0dda2f3c4d9d2b33e0efaca34fceda200cbee249.mp4\"] + 10% 46.12 MiB 87 MiB/s 4scp: error writing '.git\annex\tmp\SHA256E-s493726419--95dd6a377dce064935ab2e8c0dda2f3c4d9d2b33e0efaca34fceda200cbee249.mp4': Permission denied + +"""]] From a87688498761a277b6fb80cbf1934c499604166e Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Fri, 19 Feb 2021 17:08:39 +0000 Subject: [PATCH 10/11] initial observation about slow uninit --- ...optimization_needed__58___slow_uninit.mdwn | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 doc/todo/optimization_needed__58___slow_uninit.mdwn diff --git a/doc/todo/optimization_needed__58___slow_uninit.mdwn b/doc/todo/optimization_needed__58___slow_uninit.mdwn new file mode 100644 index 0000000000..0b7e2d8e37 --- /dev/null +++ b/doc/todo/optimization_needed__58___slow_uninit.mdwn @@ -0,0 +1,22 @@ +I am running `git annex uninit` on a quite heavy in number of annexed (text) files repository (can share a tarball if needed -- the product of https://github.com/con/tinuous/ to please those hating CI log navigation UIs ;-) ). + +the box on which it runs is very IO busy ATM, but the fact that I saw no progress from `uninit` for almost 10 minutes made me look into: + +``` +$> px | grep -A7 git-anne[x] +yoh 30339 0.2 0.0 1074123060 38060 pts/4 Sl+ 11:49 0:01 | \_ /home/yoh/miniconda3/envs/tinuous/bin/git-annex uninit +yoh 30396 0.0 0.0 13892 5492 pts/4 S+ 11:49 0:00 | \_ git --git-dir=.git --work-tree=. --literal-pathspecs ls-files --stage -z -- +yoh 30398 0.0 0.0 14720 2988 pts/4 S+ 11:49 0:00 | \_ git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch-check=%(objectname) %(objecttype) %(objectsize) --buffer +yoh 30399 0.0 0.0 14848 4108 pts/4 S+ 11:49 0:00 | \_ git --git-dir=.git --work-tree=. --literal-pathspecs cat-file --batch=%(objectname) %(objecttype) %(objectsize) --buffer +yoh 30417 0.0 0.0 0 0 pts/4 Z+ 11:49 0:00 | \_ [git] +yoh 30421 0.0 0.0 0 0 pts/4 Z+ 11:49 0:00 | \_ [git] +yoh 27761 0.0 0.0 112028 6892 pts/4 D+ 11:51 0:00 | \_ git --git-dir=.git --work-tree=. --literal-pathspecs rm --cached --force --quiet -- 01/21/github/pr/UNK/5f6fe634a863281742c0eb7d238c111da6e2e52e/Test on macOS/758/test (brew)/5_Set up Python 3.6.txt + +``` + +so `git annex` invokes separate `git rm` on each file. Having thousands of those is doomed to make it slow. Unfortunately I have not spotted a `--batch` mode in `git rm --help` but I wondered if may be at least multiple files could be requested to be removed in a single call instead of running that rm per each file? + +git annex 8.20210127-g42239bc + +[[!meta author=yoh]] +[[!tag projects/datalad]] From 9dcb11fc23875b701860d19b436c5c49b17f133e Mon Sep 17 00:00:00 2001 From: jwrauch Date: Fri, 19 Feb 2021 18:58:17 +0000 Subject: [PATCH 11/11] Added a comment --- .../comment_5_92e48d21fd83c0f7e69a5bb7b4c4f4ee._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/forum/Help_with_get/comment_5_92e48d21fd83c0f7e69a5bb7b4c4f4ee._comment diff --git a/doc/forum/Help_with_get/comment_5_92e48d21fd83c0f7e69a5bb7b4c4f4ee._comment b/doc/forum/Help_with_get/comment_5_92e48d21fd83c0f7e69a5bb7b4c4f4ee._comment new file mode 100644 index 0000000000..9acbbbd53b --- /dev/null +++ b/doc/forum/Help_with_get/comment_5_92e48d21fd83c0f7e69a5bb7b4c4f4ee._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="jwrauch" + avatar="http://cdn.libravatar.org/avatar/0e83010187ef0c8c587e0c99e2ca5474" + subject="comment 5" + date="2021-02-19T18:58:14Z" + content=""" +Okay, so it seems like it was my usb port. I tried plugging the drive into a different port and the whole repository copied over. I'd love to know more if anyone has more information about what was happening, but I think the issue is resolved for now. +"""]]