From 649079413b2fdb33204e82021b605aaf8f1ba68e Mon Sep 17 00:00:00 2001 From: Rob Date: Thu, 5 Aug 2021 18:13:36 +0000 Subject: [PATCH 1/6] Added a comment: creating directory special remote "in-place" --- ..._0a6ae6ba0d2835da21fa62b2b376bf3f._comment | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 doc/special_remotes/directory/comment_19_0a6ae6ba0d2835da21fa62b2b376bf3f._comment diff --git a/doc/special_remotes/directory/comment_19_0a6ae6ba0d2835da21fa62b2b376bf3f._comment b/doc/special_remotes/directory/comment_19_0a6ae6ba0d2835da21fa62b2b376bf3f._comment new file mode 100644 index 0000000000..f4f56a0336 --- /dev/null +++ b/doc/special_remotes/directory/comment_19_0a6ae6ba0d2835da21fa62b2b376bf3f._comment @@ -0,0 +1,41 @@ +[[!comment format=mdwn + username="Rob" + avatar="http://cdn.libravatar.org/avatar/1d12b1e7e4d6ef1dd57cd9f3ec848dc7" + subject="creating directory special remote "in-place"" + date="2021-08-05T18:13:35Z" + content=""" +I'm using git-annex for backing up a variety of data, with several different remotes (including a USB drive for backup, rsync for encrypted cloud backup, etc). I have one particular use case that I am trying to figure out how to implement: + +Pictures, Music, and Video are stored on a WD MyCloud under three corresponding folders. They are accessible via NFS, as well as via AFP; I specifically mount them on my laptop via NFS. I want to be able to access the files in the following ways: + +- Store new photos from other sources, including my laptop, using git-annex. +- Have available the list of files on my laptop repository, in git-annex... i.e. like any repository. +- Retain all the file contents on the WD MyCloud... i.e. like a special remote. +- ONLY pull file contents to the laptop (or other repositories) if specifically requested (i.e. I don't want a sync to cause things to get pulled to the laptop). +- When photos are pulled from the special remote, they go into ~/annex/photos (i.e. a different name than the folder on the remote). +- I want to be able to access the files on the WD MyCloud \"normally\" from other computers, TVs, etc. I realize there's some 'danger' here. But I wanted to keep normal file trees for the special remote, not the hash-based filenames with symlinks. + +So what I decided I needed was a directory special remote, with both the exporttree and importtree options. So I created the special remote using something like: + + git annex initremote wdmycloud-pics directory=/net/wdmycloud/nfs/Public/Shared\ Pictures encryption=none exportree=yes importtree=yes + +On my laptop's repository, I executed the following: + + git annex import master:photos --from wdmycloud-pics + git merge wdmycloud-pics/master + +Problem: +- This imports the content from the special remote, into the laptop's repository. I don't want that. + +I do realize that my main git-annex repository on the laptop probably has to actually download all of the files in order to compute their hashes. However, I do not want them to stay on the laptop, as there is nowhere's near enough space for all of them. I did try: + + git-annex wanted wdmycloud-pics \"exclude=photos/*\" + +However, this results in none of the files on the WD MyCloud actually getting processed. + +What am I missing? + +Thanks! +-Rob + +"""]] From 768bcd18a781bbfb553934bcb6d25a100c7eb031 Mon Sep 17 00:00:00 2001 From: Lukey Date: Fri, 6 Aug 2021 06:02:38 +0000 Subject: [PATCH 2/6] Added a comment --- .../comment_20_981eeeaee76f89a59e2dcbb368c14387._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/special_remotes/directory/comment_20_981eeeaee76f89a59e2dcbb368c14387._comment diff --git a/doc/special_remotes/directory/comment_20_981eeeaee76f89a59e2dcbb368c14387._comment b/doc/special_remotes/directory/comment_20_981eeeaee76f89a59e2dcbb368c14387._comment new file mode 100644 index 0000000000..db2bec22c9 --- /dev/null +++ b/doc/special_remotes/directory/comment_20_981eeeaee76f89a59e2dcbb368c14387._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 20" + date="2021-08-06T06:02:38Z" + content=""" +Note that `git-annex-import` has a `--no-content` switch. +"""]] From 29fad2ec553f6de8bd5dfd5c99f4381da467db68 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Fri, 6 Aug 2021 21:53:42 +0000 Subject: [PATCH 3/6] reporting on odds in downloads. --- ...enet.net_do_not_correspond_in_version.mdwn | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 doc/bugs/downloads.kitenet.net_do_not_correspond_in_version.mdwn diff --git a/doc/bugs/downloads.kitenet.net_do_not_correspond_in_version.mdwn b/doc/bugs/downloads.kitenet.net_do_not_correspond_in_version.mdwn new file mode 100644 index 0000000000..888e185668 --- /dev/null +++ b/doc/bugs/downloads.kitenet.net_do_not_correspond_in_version.mdwn @@ -0,0 +1,43 @@ +### Please describe the problem. + +it seems that git-annex/linux/current/git-annex-standalone-amd64.tar.gz which is claimed in the last commit to be 8.20210804 is really a 8.20210715-g6e67ea5d7 + + +### What steps will reproduce the problem? + +``` +(git-annex)lena:~datalad/distributions/downloads.kitenet.net[master]git-annex +$> git log git-annex/linux/current/git-annex-standalone-amd64.tar.gz | head +commit 9670be5007ab9a084e9f7a47c9d315555b477ca0 +Author: Joey Hess +Date: Tue Aug 3 12:30:46 2021 -0400 + + publishing git-annex 8.20210804 + +commit 780e402a5f0fd188e2edef61303db023f492459c +Author: Joey Hess +Date: Wed Jul 14 12:33:03 2021 -0400 + +$> ls -l git-annex/linux/current/git-annex-standalone-amd64.tar.gz +lrwxrwxrwx 1 yoh yoh 215 Aug 6 17:35 git-annex/linux/current/git-annex-standalone-amd64.tar.gz -> ../../../.git/annex/objects/9q/67/SHA256E-s51281961--c2419981d633d598b760c89f5a309d1ec7ec66f5b8a1e734a7239b2d4c7a3d21.tar.gz/SHA256E-s51281961--c2419981d633d598b760c89f5a309d1ec7ec66f5b8a1e734a7239b2d4c7a3d21.tar.gz + +$> tar -xzf git-annex/linux/current/git-annex-standalone-amd64.tar.gz + +$> lst +total 4 +drwx------ 1 yoh yoh 472 Aug 6 17:48 git-annex.linux/ +... + +$> ./git-annex.linux/git-annex version | head +git-annex version: 8.20210715-g6e67ea5d7 +... +``` + +also note that it is "publishing git-annex 8.20210804" (on 0803) and there is no `8.20210804` version but there is + +``` +$> git tag | grep 202108 +8.20210803 +``` + +so it seems odd From 4bebc46ce5bb97edc35993d7073b931a8b86fd6f Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Fri, 6 Aug 2021 22:11:46 +0000 Subject: [PATCH 4/6] get failing to get if with --debug --- ...nly_in_--debug__41___with_rsync_error.mdwn | 81 +++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 doc/bugs/get_fails___40__only_in_--debug__41___with_rsync_error.mdwn diff --git a/doc/bugs/get_fails___40__only_in_--debug__41___with_rsync_error.mdwn b/doc/bugs/get_fails___40__only_in_--debug__41___with_rsync_error.mdwn new file mode 100644 index 0000000000..3bee36a831 --- /dev/null +++ b/doc/bugs/get_fails___40__only_in_--debug__41___with_rsync_error.mdwn @@ -0,0 +1,81 @@ +### Please describe the problem. + +I have been trying to get to the bottom of it but failed + +Seems to relate to the version and/or build of git-annex on remote end, but happens whenever on remote it is a conda-forge build 8.20210621-g91f9aac + +Whenever I can get/drop without `--debug`, with `--debug` it fails: + +``` +(git-annex)lena:/tmp/testds-mit[master]git-annex +$> git annex get 123 +get 123 (from origin...) +ok +(recording state in git...) +1 11897.....................................:Fri 06 Aug 2021 06:09:05 PM EDT:. +(git-annex)lena:/tmp/testds-mit[master]git-annex +$> git annex drop 123 +drop 123 (locking origin...) ok +(recording state in git...) +1 11898.....................................:Fri 06 Aug 2021 06:09:11 PM EDT:. +(git-annex)lena:/tmp/testds-mit[master]git-annex +$> git annex get 123 --debug +[2021-08-06 18:09:22.424837137] (Utility.Process) process [1488170] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","symbolic-ref","-q","HEAD"] +[2021-08-06 18:09:22.437700889] (Utility.Process) process [1488170] done ExitSuccess +[2021-08-06 18:09:22.438748331] (Utility.Process) process [1488171] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","refs/heads/master"] +[2021-08-06 18:09:22.449949392] (Utility.Process) process [1488171] done ExitSuccess +[2021-08-06 18:09:22.451491273] (Utility.Process) process [1488172] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--","123"] +[2021-08-06 18:09:22.45371749] (Utility.Process) process [1488173] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"] +[2021-08-06 18:09:22.455744285] (Utility.Process) process [1488174] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"] +[2021-08-06 18:09:22.458678421] (Utility.Process) process [1488175] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"] +get 123 [2021-08-06 18:09:22.471519462] (Utility.Process) process [1488176] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"] +[2021-08-06 18:09:22.479244675] (Utility.Process) process [1488176] done ExitSuccess +[2021-08-06 18:09:22.480965615] (Utility.Process) process [1488177] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"] +[2021-08-06 18:09:22.494689644] (Utility.Process) process [1488177] done ExitSuccess +[2021-08-06 18:09:22.496859969] (Utility.Process) process [1488178] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..b1854b845d5ab7dbae4f50607ae02b76ae526716","--pretty=%H","-n1"] +[2021-08-06 18:09:22.505499833] (Utility.Process) process [1488178] done ExitSuccess +[2021-08-06 18:09:22.507468513] (Utility.Process) process [1488179] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..83d0262dcb3d3abf7f64ce7079e5d6e4a27c2293","--pretty=%H","-n1"] +[2021-08-06 18:09:22.517489766] (Utility.Process) process [1488179] done ExitSuccess +[2021-08-06 18:09:22.519480403] (Utility.Process) process [1488180] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"] +[2021-08-06 18:09:22.519793644] (Utility.Process) process [1488181] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] +(from origin...) +[2021-08-06 18:09:22.524141345] (Utility.Process) process [1488182] read: ssh ["-o","BatchMode=true","-S",".git/annex/ssh/openmind7.mit.edu","-o","ControlMaster=auto","-o","ControlPersist=yes","-n","-T","openmind7.mit.edu","true"] +[2021-08-06 18:09:24.650559322] (Utility.Process) process [1488182] done ExitSuccess +[2021-08-06 18:09:24.651134161] (Utility.Process) process [1488187] chat: ssh ["openmind7.mit.edu","-S",".git/annex/ssh/openmind7.mit.edu","-o","ControlMaster=auto","-o","ControlPersist=yes","-T","git-annex-shell 'p2pstdio' '/~/tmp/testds' '--debug' '339c2fc7-df7a-44fd-97eb-5a2c41ac8a0a' --uuid f30e2969-4ddf-4edb-ba45-37c455816c9a"] +[2021-08-06 18:09:24.80528113] (P2P.IO) [ssh connection Just 1488187] [ThreadId 4] P2P > ERROR auth failed +[2021-08-06 18:09:24.84393255] (Utility.Process) process [1488187] done ExitSuccess +[2021-08-06 18:09:24.846367147] (Utility.Process) process [1488189] read: rsync ["--progress","--inplace","--perms","-e","'ssh' 'openmind7.mit.edu' '-S' '.git/annex/ssh/openmind7.mit.edu' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=yes' '-T' 'git-annex-shell ''sendkey'' ''/~/tmp/testds'' ''--debug'' ''MD5E-s0--d41d8cd98f00b204e9800998ecf8427e'' --uuid f30e2969-4ddf-4edb-ba45-37c455816c9a ''--'' ''remoteuuid=339c2fc7-df7a-44fd-97eb-5a2c41ac8a0a'' ''unlocked='' ''direct='' ''associatedfile=123'' ''--'''","--","dummy:",".git/annex/tmp/MD5E-s0--d41d8cd98f00b204e9800998ecf8427e"] +protocol version mismatch -- is your shell clean? +(see the rsync man page for an explanation) +rsync error: protocol incompatibility (code 2) at compat.c(178) [Receiver=3.1.3] +[2021-08-06 18:09:24.970785079] (Utility.Process) process [1488189] done ExitFailure 2 +rsync exited 2 + + rsync failed -- run git annex again to resume file transfer + + Transfer failed +[2021-08-06 18:09:25.014462898] (Utility.Process) process [1488180] done ExitSuccess +[2021-08-06 18:09:25.016852848] (Utility.Process) process [1488181] done ExitSuccess + + Unable to access these remotes: origin +[2021-08-06 18:09:25.021878961] (Utility.Process) process [1488191] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"] +[2021-08-06 18:09:25.025537927] (Utility.Process) process [1488192] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] + + No other repository is known to contain the file. +failed +[2021-08-06 18:09:25.038970056] (Utility.Process) process [1488191] done ExitSuccess +[2021-08-06 18:09:25.040408598] (Utility.Process) process [1488192] done ExitSuccess +[2021-08-06 18:09:25.040589952] (Utility.Process) process [1488175] done ExitSuccess +[2021-08-06 18:09:25.040701306] (Utility.Process) process [1488174] done ExitSuccess +[2021-08-06 18:09:25.040787814] (Utility.Process) process [1488173] done ExitSuccess +[2021-08-06 18:09:25.040861945] (Utility.Process) process [1488172] done ExitSuccess +[2021-08-06 18:09:25.041785909] (Utility.Process) process [1488193] read: ssh ["-O","stop","-S","openmind7.mit.edu","-o","ControlMaster=auto","-o","ControlPersist=yes","localhost"] +[2021-08-06 18:09:25.061503656] (Utility.Process) process [1488193] done ExitSuccess +get: 1 failed + +``` + +I just merged freshier update of conda-forge to 8.20210803 -- will try that one later + +[[!meta author=yoh]] +[[!tag projects/datalad]] From 251c24b388881a4693dcd9eabd21afe7fca213ba Mon Sep 17 00:00:00 2001 From: "jgsuess@732b8c62c50d8595d7b1d58eea11e5019c2308b1" Date: Sat, 7 Aug 2021 12:25:02 +0000 Subject: [PATCH 5/6] Added a comment: Automatic watch for the heuristic --- .../comment_3_3309c51dbd251cc46a25557a0d25e946._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/bugs/Continual_space_exhaustion_from_syncing_metadata/comment_3_3309c51dbd251cc46a25557a0d25e946._comment diff --git a/doc/bugs/Continual_space_exhaustion_from_syncing_metadata/comment_3_3309c51dbd251cc46a25557a0d25e946._comment b/doc/bugs/Continual_space_exhaustion_from_syncing_metadata/comment_3_3309c51dbd251cc46a25557a0d25e946._comment new file mode 100644 index 0000000000..55c5122b37 --- /dev/null +++ b/doc/bugs/Continual_space_exhaustion_from_syncing_metadata/comment_3_3309c51dbd251cc46a25557a0d25e946._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="jgsuess@732b8c62c50d8595d7b1d58eea11e5019c2308b1" + nickname="jgsuess" + avatar="http://cdn.libravatar.org/avatar/35a99d1b5df046b206807941ce9795d3" + subject="Automatic watch for the heuristic" + date="2021-08-07T12:25:02Z" + content=""" +It seems there is a good heuristic based on the object size. It is given in this thread. Could this check not be used in annex automatic syncing modes to avoid this problem. I have run into it several times now and recovery, while possible, is time-consuming and messy. +"""]] From 1801400bbb835af67422c6cf504aa965f5e2cd2a Mon Sep 17 00:00:00 2001 From: alex Date: Mon, 9 Aug 2021 04:21:11 +0000 Subject: [PATCH 6/6] --- ...eadDirStream__58___inappropriate_type.mdwn | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 doc/bugs/git-annex__58___readDirStream__58___inappropriate_type.mdwn diff --git a/doc/bugs/git-annex__58___readDirStream__58___inappropriate_type.mdwn b/doc/bugs/git-annex__58___readDirStream__58___inappropriate_type.mdwn new file mode 100644 index 0000000000..58a3dd93fc --- /dev/null +++ b/doc/bugs/git-annex__58___readDirStream__58___inappropriate_type.mdwn @@ -0,0 +1,33 @@ +### Please describe the problem. +I'm not entirely sure what I did. Maybe add files with weird names and I copied a lot of files with cp -rL + +### What steps will reproduce the problem? +No idea. + +### What version of git-annex are you using? On what operating system? +git-annex version: 8.20210803-g9cae7c5bb +Archlinux, up to date. + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + +Basically, everything I do is answered with the error mentioned in the title. + +[alex@proliant bannex]$ git annex fix +[alex@proliant bannex]$ git annex add . +git-annex: readDirStream: inappropriate type (Bad message) +[alex@proliant bannex]$ git annex sync +git-annex: readDirStream: inappropriate type (Bad message) +[alex@proliant bannex]$ git annex fsck +git-annex: readDirStream: inappropriate type (Bad message) + + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Yes. This is my third attempt at getting git-annex running. Conceptually, I think it's the best thing out there. I've had some success at moving around data. But I tend to end up in these dead ends...