From 3e6dba097ea293f7cb40bb335cf8d2249b39e039 Mon Sep 17 00:00:00 2001 From: kolam Date: Mon, 4 Dec 2023 19:32:28 +0000 Subject: [PATCH 1/8] --- ...transfer_repo_not_auto_dropping_files.mdwn | 88 +++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 doc/forum/transfer_repo_not_auto_dropping_files.mdwn diff --git a/doc/forum/transfer_repo_not_auto_dropping_files.mdwn b/doc/forum/transfer_repo_not_auto_dropping_files.mdwn new file mode 100644 index 0000000000..c429099f5e --- /dev/null +++ b/doc/forum/transfer_repo_not_auto_dropping_files.mdwn @@ -0,0 +1,88 @@ +I'm trying to set up two client repositories that don't communicate directly with one another, but sync their data using a transfer repository. + +Here's the script I used for creating a reproducible environment: + +``` +#/usr/bin/env bash + +# Just a way to access the script's directory +cd "$(dirname "$0")" +DIR="$(pwd)" + +# Create the 1st client repository +mkdir $DIR/client1 +cd $DIR/client1 +git init && git annex init + +# Create the 2nd client repository +mkdir $DIR/client2 +cd $DIR/client2 +git init && git annex init + +# Create the transfer repository +mkdir $DIR/share +cd $DIR/share +git init && git annex init + +# Setup the remotes and groups for the transfer repository +cd $DIR/share +git remote add client1 $DIR/client1 +git remote add client2 $DIR/client1 +git annex group . transfer +git annex group client1 client +git annex group client2 client +git co -b main + +# Setup the remotes and groups for the 1st client repository. +cd $DIR/client1 +git remote add share $DIR/share +git annex group . client +git annex group share transfer +git annex config --set annex.addunlocked true +git co -b main + +# Setup the remotes and groups for the 2nd client repository. +cd $DIR/client2 +git remote add share $DIR/share +git annex group . client +git annex group share transfer +git annex config --set annex.addunlocked true +git co -b main + +# Run git-annex assistant for each repository +cd $DIR/client1 && git annex assistant +cd $DIR/client2 && git annex assistant +cd $DIR/share && git annex assistant + +# Add a single file to the 1st client. +cd $DIR/client1 +touch file.txt + +# Need to do this if there are no commits in the 'client2' and 'share' repositories. +# Or else, I'll get the following logs: +# +# merge: refs/remotes/share/main - not something we can merge +# merge: refs/remotes/share/synced/main - not something we can merge +sleep 3; +cd $DIR/share +git pull client1 main +sleep 3; +cd $DIR/client2 +git pull share main + +cd $DIR/client1 +echo "My first line" >> file.txt +``` + +However, after letting `git-annex assistant` do it's thing, `file.txt` seems to never be dropped from the `share` transfer repository. +Event after running `git-annex sync --content`, the `git-annex whereis file.txt` gives: + +``` +whereis file.txt (3 copies) + 274b3417-1bf3-47b3-a75e-53ebe7ca20d8 -- user@server:~/share [share] + 7cc7d6a8-6230-4d2c-8414-1b45b2fc14d8 -- user@server:~/client2 + bdfea36f-d011-4950-8be0-1668f5e56f5c -- user@server:~/client1 [here] +ok +``` + +The documentation says that the file should be automatically dropped, but that doesn't happen here. From b34d9b1405b7831f57f8751191c95629e30e5236 Mon Sep 17 00:00:00 2001 From: "brendan.ward@a2e11ad27f6b2fa2c556aea6811496e0d95dd0da" Date: Tue, 5 Dec 2023 03:24:29 +0000 Subject: [PATCH 2/8] Added a comment --- .../comment_3_81085c1a526ac804edba689043e58944._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/forum/annex.largefile_not_working/comment_3_81085c1a526ac804edba689043e58944._comment diff --git a/doc/forum/annex.largefile_not_working/comment_3_81085c1a526ac804edba689043e58944._comment b/doc/forum/annex.largefile_not_working/comment_3_81085c1a526ac804edba689043e58944._comment new file mode 100644 index 0000000000..d37f196648 --- /dev/null +++ b/doc/forum/annex.largefile_not_working/comment_3_81085c1a526ac804edba689043e58944._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="brendan.ward@a2e11ad27f6b2fa2c556aea6811496e0d95dd0da" + nickname="brendan.ward" + avatar="http://cdn.libravatar.org/avatar/bc7899f1aa523365cdfadc2ba3f7913d" + subject="comment 3" + date="2023-12-05T03:24:29Z" + content=""" +Excellent. Thanks for the help! +"""]] From 8ee111cd8e4d8699d10e2ce47791fcc286241d1a Mon Sep 17 00:00:00 2001 From: "https://esgf-node.llnl.gov/esgf-idp/openid/mvhulten" Date: Tue, 5 Dec 2023 14:05:08 +0000 Subject: [PATCH 3/8] --- doc/forum/dne.mdwn | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 doc/forum/dne.mdwn diff --git a/doc/forum/dne.mdwn b/doc/forum/dne.mdwn new file mode 100644 index 0000000000..d485771ba1 --- /dev/null +++ b/doc/forum/dne.mdwn @@ -0,0 +1,28 @@ +I get a failure in name resolution when running `annex testremote`: + +``` +mvanhult@detekti:~/temp/my_repo$ git annex testremote bbborig | head +testremote bbborig (generating test keys...) Remote Tests + unavailable remote +ssh: Could not resolve hostname !dne!: non-recoverable failure in name resolution + removeKey: OK (0.01s) +ssh: Could not resolve hostname !dne!: non-recoverable failure in name resolution + storeKey: OK +ssh: Could not resolve hostname !dne!: non-recoverable failure in name resolution + checkPresent: OK +ssh: Could not resolve hostname !dne!: non-recoverable failure in name resolution + retrieveKeyFile: OK + retrieveKeyFileCheap: OK + key size Just 1048576; remote chunksize=0 encryption=none +The following modules were not unloaded: + (Use "module --force purge" to unload all): + + 1) Stages/2024 + removeKey when not present: FAIL + Exception: removing content from remote failed +``` + +Where does this `!dne!` come from? +The relevant host of the remote is `juwels` and not `!dne!`. +I have never before this post entered the string `!dne!`. +Is *dne* an abbreviation? From 8c55d65987f35beab9bc592b5107f6f93597eed2 Mon Sep 17 00:00:00 2001 From: "https://esgf-node.llnl.gov/esgf-idp/openid/mvhulten" Date: Tue, 5 Dec 2023 14:07:57 +0000 Subject: [PATCH 4/8] rename forum/dne.mdwn to forum/name_resolution_of___33__dne__33___fails.mdwn --- .../{dne.mdwn => name_resolution_of___33__dne__33___fails.mdwn} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename doc/forum/{dne.mdwn => name_resolution_of___33__dne__33___fails.mdwn} (100%) diff --git a/doc/forum/dne.mdwn b/doc/forum/name_resolution_of___33__dne__33___fails.mdwn similarity index 100% rename from doc/forum/dne.mdwn rename to doc/forum/name_resolution_of___33__dne__33___fails.mdwn From 5d22ccc584f40a5b2c58b14aee962246b8e09960 Mon Sep 17 00:00:00 2001 From: "https://esgf-node.llnl.gov/esgf-idp/openid/mvhulten" Date: Tue, 5 Dec 2023 14:08:52 +0000 Subject: [PATCH 5/8] rename forum/name_resolution_of___33__dne__33___fails.mdwn to forum/name_resolution_of_dne.mdwn --- ..._of___33__dne__33___fails.mdwn => name_resolution_of_dne.mdwn} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename doc/forum/{name_resolution_of___33__dne__33___fails.mdwn => name_resolution_of_dne.mdwn} (100%) diff --git a/doc/forum/name_resolution_of___33__dne__33___fails.mdwn b/doc/forum/name_resolution_of_dne.mdwn similarity index 100% rename from doc/forum/name_resolution_of___33__dne__33___fails.mdwn rename to doc/forum/name_resolution_of_dne.mdwn From 9a9d99efebf5faa9b751741720d6ed60985e6f69 Mon Sep 17 00:00:00 2001 From: "https://esgf-node.llnl.gov/esgf-idp/openid/mvhulten" Date: Tue, 5 Dec 2023 14:10:07 +0000 Subject: [PATCH 6/8] --- doc/forum/name_resolution_of_dne.mdwn | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/forum/name_resolution_of_dne.mdwn b/doc/forum/name_resolution_of_dne.mdwn index d485771ba1..cc3ca101ad 100644 --- a/doc/forum/name_resolution_of_dne.mdwn +++ b/doc/forum/name_resolution_of_dne.mdwn @@ -20,6 +20,9 @@ The following modules were not unloaded: 1) Stages/2024 removeKey when not present: FAIL Exception: removing content from remote failed +mvanhult@detekti:~/temp/my_repo$ git remote -v +bbborig juwels:/p/home/jusers/vanhulten1/juwels/temp/my_repo (fetch) +bbborig juwels:/p/home/jusers/vanhulten1/juwels/temp/my_repo (push) ``` Where does this `!dne!` come from? From f6f4ba3c6ce9d90fa7df44167d14b6fa20bbca2a Mon Sep 17 00:00:00 2001 From: cjmarkie Date: Tue, 5 Dec 2023 14:54:11 +0000 Subject: [PATCH 7/8] --- ...gument_saving_FreeSurfer_file_on_NTFS.mdwn | 93 +++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 doc/bugs/Invalid_argument_saving_FreeSurfer_file_on_NTFS.mdwn diff --git a/doc/bugs/Invalid_argument_saving_FreeSurfer_file_on_NTFS.mdwn b/doc/bugs/Invalid_argument_saving_FreeSurfer_file_on_NTFS.mdwn new file mode 100644 index 0000000000..ff8daac66d --- /dev/null +++ b/doc/bugs/Invalid_argument_saving_FreeSurfer_file_on_NTFS.mdwn @@ -0,0 +1,93 @@ +### Please describe the problem. + +Annexing files can fail with "Invalid argument" error. It appears to be an interaction of filesystem and file name. The file name is `lh.aparc.DKTatlas.annot`/`rh.aparc.DKTatlas.annot`, and the filesystem is: + +[[!format sh """ +$ mount | grep data +/dev/mapper/bitlocker on /data type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096) +$ ntfs-3g +ntfs-3g: No device is specified. + +ntfs-3g 2021.8.22 integrated FUSE 28 - Third Generation NTFS Driver + Configuration type 7, XATTRS are on, POSIX ACLS are on +"""]] + +However, formatting a file as NTFS and mounting as loop to make a minimal reproduction allowed the operation to succeed. + +### What steps will reproduce the problem? + +I am able to reproduce this problem reliably on the filesystem above: + +[[!format sh """ +$ git init test/ +$ cd test +$ git annex init +$ curl -sSL https://file.io/ZtE5ivep7E9V -o lh.aparc.DKTatlas.annot +$ git annex add lh.aparc.DKTatlas.annot +add lh.aparc.DKTatlas.annot + .git/annex/othertmp/: openTempFile template ingest-lh.: invalid argument (Invalid argument) + +failed +add: 1 failed +"""]] + +However, renaming the file to `L.annot`, annexing it, and `git mv`ing it back works, so I am not stuck adding the file contents directly to git. + +I hope the log below will give you a hint of what's going on, as I regret I am unable to reproduce this on other filesystems. + +### What version of git-annex are you using? On what operating system? + +OS: Ubuntu 22.04.3 LTS + +[[!format sh """ +git-annex version: 10.20230408-g5b1e8ba77 +build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV +dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external +operating system: linux x86_64 +supported repository versions: 8 9 10 +upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 +local repository version: 10 +"""]] + +### Please provide any additional information below. +[[!format sh """ +$ git annex add -d lh.aparc.DKTatlas.annot +[2023-12-05 09:40:20.256526552] (Utility.Process) process [1162202] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"] +[2023-12-05 09:40:20.260613884] (Utility.Process) process [1162202] done ExitSuccess +[2023-12-05 09:40:20.260892906] (Utility.Process) process [1162203] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"] +[2023-12-05 09:40:20.265019085] (Utility.Process) process [1162203] done ExitSuccess +[2023-12-05 09:40:20.265446352] (Utility.Process) process [1162204] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..eb539b816feb00e6be3a252d36efaa7426eccca1","--pretty=%H","-n1"] +[2023-12-05 09:40:20.270301207] (Utility.Process) process [1162204] done ExitSuccess +[2023-12-05 09:40:20.271251091] (Utility.Process) process [1162205] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"] +[2023-12-05 09:40:20.275373234] (Utility.Process) process [1162206] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","symbolic-ref","-q","HEAD"] +[2023-12-05 09:40:20.278672692] (Utility.Process) process [1162206] done ExitSuccess +[2023-12-05 09:40:20.278946202] (Utility.Process) process [1162207] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","refs/heads/master"] +[2023-12-05 09:40:20.282475944] (Utility.Process) process [1162207] done ExitFailure 1 +[2023-12-05 09:40:20.282779905] (Utility.Process) process [1162208] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","-z","--others","--exclude-standard","--","lh.aparc.DKTatlas.annot"] +[2023-12-05 09:40:20.286209985] (Utility.Process) process [1162209] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","check-attr","-z","--stdin","annex.backend","annex.largefiles","annex.numcopies","annex.mincopies","--"] +[2023-12-05 09:40:20.28954558] (Utility.Process) process [1162210] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] +add lh.aparc.DKTatlas.annot [2023-12-05 09:40:20.293105736] (Utility.Process) process [1162211] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","symbolic-ref","-q","HEAD"] +[2023-12-05 09:40:20.296109063] (Utility.Process) process [1162211] done ExitSuccess +[2023-12-05 09:40:20.296268327] (Utility.Process) process [1162212] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","refs/heads/master"] +[2023-12-05 09:40:20.299447211] (Utility.Process) process [1162212] done ExitFailure 1 +[2023-12-05 09:40:20.299472144] (Annex.Perms) freezing content lh.aparc.DKTatlas.annot + + .git/annex/othertmp/: openTempFile template ingest-lh.: invalid argument (Invalid argument) + +failed +[2023-12-05 09:40:20.299683526] (Utility.Process) process [1162208] done ExitSuccess +[2023-12-05 09:40:20.299804817] (Utility.Process) process [1162213] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","-z","--modified","--","lh.aparc.DKTatlas.annot"] +[2023-12-05 09:40:20.302582841] (Utility.Process) process [1162213] done ExitSuccess +[2023-12-05 09:40:20.302707305] (Utility.Process) process [1162214] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","diff","--name-only","--diff-filter=T","-z","--cached","--","lh.aparc.DKTatlas.annot"] +[2023-12-05 09:40:20.30608269] (Utility.Process) process [1162214] done ExitSuccess +[2023-12-05 09:40:20.306609675] (Utility.Process) process [1162205] done ExitSuccess +[2023-12-05 09:40:20.306732757] (Utility.Process) process [1162210] done ExitSuccess +[2023-12-05 09:40:20.306873108] (Utility.Process) process [1162209] done ExitSuccess +add: 1 failed +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Yes! I use git annex and datalad all the time for personal and work projects. From 545f3873ca0df5f3a3e9a2c2a88688d28eaaa09e Mon Sep 17 00:00:00 2001 From: cjmarkie Date: Tue, 5 Dec 2023 16:04:46 +0000 Subject: [PATCH 8/8] No change. Just subscribing to comments. --- doc/bugs/Invalid_argument_saving_FreeSurfer_file_on_NTFS.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/bugs/Invalid_argument_saving_FreeSurfer_file_on_NTFS.mdwn b/doc/bugs/Invalid_argument_saving_FreeSurfer_file_on_NTFS.mdwn index ff8daac66d..5b4e7db549 100644 --- a/doc/bugs/Invalid_argument_saving_FreeSurfer_file_on_NTFS.mdwn +++ b/doc/bugs/Invalid_argument_saving_FreeSurfer_file_on_NTFS.mdwn @@ -90,4 +90,4 @@ add: 1 failed ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -Yes! I use git annex and datalad all the time for personal and work projects. +Yes! I use git annex and datalad all the time for personal and work projects.