From ebdf27f0a3f1ba706fa4b469a04969ff64256568 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Sat, 6 Jun 2020 23:03:44 +0000 Subject: [PATCH 1/6] added question about building from source using stack --- doc/forum/building_from_source__58___dependency_versions.mdwn | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/forum/building_from_source__58___dependency_versions.mdwn diff --git a/doc/forum/building_from_source__58___dependency_versions.mdwn b/doc/forum/building_from_source__58___dependency_versions.mdwn new file mode 100644 index 0000000000..aefc199c48 --- /dev/null +++ b/doc/forum/building_from_source__58___dependency_versions.mdwn @@ -0,0 +1 @@ +@joeyh, can you check that this [patch](https://github.com/conda-forge/git-annex-feedstock/blob/master/recipe/0001-extra-deps-01.patch) of `stack.yaml` is safe, in particular that git-annex should work ok with the newer versions of `aws and `hslogger`? From 02ee2d0e47fed16d157153957325bb92b5836d65 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Sun, 7 Jun 2020 19:38:34 +0000 Subject: [PATCH 2/6] --- doc/todo/addurl_video_from_a_Twitter_post.mdwn | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/todo/addurl_video_from_a_Twitter_post.mdwn diff --git a/doc/todo/addurl_video_from_a_Twitter_post.mdwn b/doc/todo/addurl_video_from_a_Twitter_post.mdwn new file mode 100644 index 0000000000..27222a0eb3 --- /dev/null +++ b/doc/todo/addurl_video_from_a_Twitter_post.mdwn @@ -0,0 +1 @@ +Is there already a way to addurl video from a Twitter post. Question came up while proposing git annex as a tech for archival in https://github.com/2020PB/police-brutality/issues/315#issuecomment-640163911 From 2d2a735ce731192ecfc2abc8518640db84d5ddbb Mon Sep 17 00:00:00 2001 From: strmd Date: Mon, 8 Jun 2020 07:11:55 +0000 Subject: [PATCH 3/6] --- doc/forum/Mixed_content_repos_with_import_and_export.mdwn | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 doc/forum/Mixed_content_repos_with_import_and_export.mdwn diff --git a/doc/forum/Mixed_content_repos_with_import_and_export.mdwn b/doc/forum/Mixed_content_repos_with_import_and_export.mdwn new file mode 100644 index 0000000000..02d2e3d6f1 --- /dev/null +++ b/doc/forum/Mixed_content_repos_with_import_and_export.mdwn @@ -0,0 +1,7 @@ +Is it possible to get mixed content repositories working in conjunction with a directory special remote using importtree/exporttree? + +`git annex add` adheres to the `annex.largefiles` setting (`mimetype=text/*`), but importing or syncing from the remote invariably adds text files to the annex, including if the file was originally added to git and then modified on the remote. + +I want `annex.synccontent` set to `true` but have tried it both ways. + +What am I missing? From 634c66ea497c208c46fb3d698f333cde0a0bb3d6 Mon Sep 17 00:00:00 2001 From: "kanakkshetri@9ea0e7639162bddc7bf9f3bb94cc32e93c793b89" Date: Mon, 8 Jun 2020 14:24:37 +0000 Subject: [PATCH 4/6] --- ..._Unable_to_access_these_remotes__34__.mdwn | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn diff --git a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn b/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn new file mode 100644 index 0000000000..0207886edb --- /dev/null +++ b/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn @@ -0,0 +1,76 @@ +### Please describe the problem. + +I am trying to copy files from my computer to an external drive, but git annex get in one annex repo on the drive, keeps failing with "Unable to access these remotes: origin" message. It seems to pull the data, compute a checksum and then give me the error message. The same file passes fsck on the computer, and I was able to "git annex get" it from a different computer without issues. + + +### What steps will reproduce the problem? +- cd into externaldrive/Annex +- git annex get file + + get 8 Simple Rules - [02x01] - Premiere.mp4 (from origin...) + (from origin...) + + Unable to access these remotes: origin + + Try making some of these repositories available: + 1cca049d-83f7-4a78-ad30-650621c8e6f9 -- amnesiac + 3ae7e0b0-f87a-4f5c-9d37-a507273dde61 -- balrog [origin] + bafcea18-6642-4c43-8ae6-8477092c7d02 -- babadook + failed + git-annex: get: 1 failed + + +- The file then shows up in git annex unused. + +- Other commands like "git annex sync origin" work without issues, git annex copy --to origin also work. + +- I tried adding --verbose, --json etc to see if i could get some kind of debugging info out of it, but no luck. + + +### What version of git-annex are you using? On what operating system? + +git annex version + +git-annex version: 8.20200330 +build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite +dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.4 feed-1.1.0.0 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.1 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external +operating system: linux x86_64 +supported repository versions: 8 +upgrade supported from repository versions: 0 1 2 3 4 5 6 7 +local repository version: 8 + +Operating System: + +cat /etc/fedora-release + +Fedora release 32 (Thirty Two) + + +### Please provide any additional information below. + +git annex info --fast + +trusted repositories: 7 + 318863a2-d9eb-467f-8094-f6578e26bd9a -- Melange + 33e44b4c-42f1-4383-90b3-5fcbdbbf3baf -- baldwin + 43d346b9-3c60-4036-a654-4c31f2368206 -- thane + bafcea18-6642-4c43-8ae6-8477092c7d02 -- babadook + d0ad767c-9874-4a3e-8ced-5d6917b212b9 -- zuko + d3ba71fb-64fc-4f41-8e76-7de8af14cee1 -- erebus [here] + d6d1d08e-6ee5-4408-a515-adbc9bebf5b1 -- ocho +semitrusted repositories: 1 + 3ae7e0b0-f87a-4f5c-9d37-a507273dde61 -- balrog [origin] +untrusted repositories: 2 + 1cca049d-83f7-4a78-ad30-650621c8e6f9 -- amnesiac + b3407756-8beb-43c3-877d-ff4fd1061fa9 -- chandrian +transfers in progress: none +available local disk space: 977.01 gigabytes (+1 megabyte reserved) + + + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +YES. Git annex is one of my favorite software. I am managing around 15 terabytes of data in 5 annexes, with data on 3 computers, and around 8 external drives. This is the first time I've had any kind of issue with it. It has been rock solid and completely amazing. Thank you for your work on the software. + From 52412b773b70f56ecbdcf3e3503bebb4c5113dcd Mon Sep 17 00:00:00 2001 From: meribold Date: Mon, 8 Jun 2020 14:58:48 +0000 Subject: [PATCH 5/6] AWS tells me that the Reduced Redundancy storage class is less cost effective than Standard and not recommended --- doc/special_remotes/S3.mdwn | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/special_remotes/S3.mdwn b/doc/special_remotes/S3.mdwn index c8d6a975f2..5b8da9971c 100644 --- a/doc/special_remotes/S3.mdwn +++ b/doc/special_remotes/S3.mdwn @@ -41,15 +41,15 @@ the S3 remote. * `storageclass` - Default is "STANDARD". Consult S3 provider documentation for pricing details and available - storage classes. - - When using Amazon S3, if you have configured git-annex to preserve - multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY" - to save money. + storage classes. For example, the s3cmd(1) man page lists valid storage class names for Amazon S3. - Or, if the remote will be used for backup or archival, - and so its files are Infrequently Accessed, "STANDARD_IA" is also a - good choice to save money. (Requires a git-annex built with aws-0.13.0) + When using Amazon S3, + if the remote will be used for backup or archival, + and so its files are Infrequently Accessed, "STANDARD_IA" is a + good choice to save money (requires a git-annex built with aws-0.13.0). + If you have configured git-annex to preserve + multiple [[copies]], also consider setting this to "ONEZONE_IA" + to save even more money. When using Google Cloud Storage, to make a nearline bucket, set this to "NEARLINE". (Requires a git-annex built with aws-0.13.0) From 34b088b209314d79aeb378a6b645d0058838521c Mon Sep 17 00:00:00 2001 From: "kanakkshetri@9ea0e7639162bddc7bf9f3bb94cc32e93c793b89" Date: Mon, 8 Jun 2020 15:09:20 +0000 Subject: [PATCH 6/6] Output of --debug, Formatting --- ..._Unable_to_access_these_remotes__34__.mdwn | 122 ++++++++++++------ 1 file changed, 86 insertions(+), 36 deletions(-) diff --git a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn b/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn index 0207886edb..f0453b6e47 100644 --- a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn +++ b/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn @@ -7,66 +7,116 @@ I am trying to copy files from my computer to an external drive, but git annex g - cd into externaldrive/Annex - git annex get file - get 8 Simple Rules - [02x01] - Premiere.mp4 (from origin...) - (from origin...) - - Unable to access these remotes: origin - Try making some of these repositories available: - 1cca049d-83f7-4a78-ad30-650621c8e6f9 -- amnesiac - 3ae7e0b0-f87a-4f5c-9d37-a507273dde61 -- balrog [origin] - bafcea18-6642-4c43-8ae6-8477092c7d02 -- babadook - failed - git-annex: get: 1 failed + get 8 Simple Rules - [02x01] - Premiere.mp4 (from origin...) + (from origin...) + + Unable to access these remotes: origin + + Try making some of these repositories available: + 1cca049d-83f7-4a78-ad30-650621c8e6f9 -- amnesiac + 3ae7e0b0-f87a-4f5c-9d37-a507273dde61 -- balrog [origin] + bafcea18-6642-4c43-8ae6-8477092c7d02 -- babadook + failed + git-annex: get: 1 failed - The file then shows up in git annex unused. - Other commands like "git annex sync origin" work without issues, git annex copy --to origin also work. -- I tried adding --verbose, --json etc to see if i could get some kind of debugging info out of it, but no luck. +- I learned about the --debug flag from another post in the forum and here is the output: + + + [2020-06-08 10:02:19.510064296] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","symbolic-ref","-q","HEAD"] + [2020-06-08 10:02:19.511925354] process done ExitSuccess + [2020-06-08 10:02:19.512027344] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","show-ref","refs/heads/master"] + [2020-06-08 10:02:19.513918652] process done ExitSuccess + [2020-06-08 10:02:19.514026372] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","ls-files","--cached","-z","--","8 Simple Rules - [02x01] - Premiere.mp4"] + get 8 Simple Rules - [02x01] - Premiere.mp4 [2020-06-08 10:02:19.520306016] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","show-ref","git-annex"] + [2020-06-08 10:02:19.523513403] process done ExitSuccess + [2020-06-08 10:02:19.523637203] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] + [2020-06-08 10:02:19.525420671] process done ExitSuccess + [2020-06-08 10:02:19.525726091] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","log","refs/heads/git-annex..3faae00aee063e3eafd34c9ad82a25a3455cd699","--pretty=%H","- + n1"] + [2020-06-08 10:02:19.527440579] process done ExitSuccess + [2020-06-08 10:02:19.527526039] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","log","refs/heads/git-annex..7a56b81cf0518b9ffefd8e077e15616fc809c0ee","--pretty=%H","- + n1"] + [2020-06-08 10:02:19.531118595] process done ExitSuccess + [2020-06-08 10:02:19.53722095] chat: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","cat-file","--batch"] + [2020-06-08 10:02:19.537616269] chat: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] + [2020-06-08 10:02:19.541227956] read: git ["config","--null","--list"] + [2020-06-08 10:02:19.542505395] process done ExitSuccess + (from origin...) + [2020-06-08 10:02:19.559535139] read: cp ["--reflink=always","--preserve=timestamps","../../../../../../../home/kanak/Annex/TV/.git/annex/objects/3P/zP/SHA256-s210775700--f9bd3bcd9d6d3c5b90 + 7605c049d6e00fa40734860e4aadcbc8009a175b37ab49/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c049d6e00fa40734860e4aadcbc8009a175b37ab49","../.git/annex/tmp/SHA256-s210775700--f9bd3bcd9d6d3c5b907 + 605c049d6e00fa40734860e4aadcbc8009a175b37ab49"] + [2020-06-08 10:02:19.561441707] process done ExitFailure 1 + [2020-06-08 10:02:19.561515157] read: rsync ["--progress","--inplace","--perms","../../../../../../../home/kanak/Annex/TV/.git/annex/objects/3P/zP/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c + 049d6e00fa40734860e4aadcbc8009a175b37ab49/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c049d6e00fa40734860e4aadcbc8009a175b37ab49","../.git/annex/tmp/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c0 + 49d6e00fa40734860e4aadcbc8009a175b37ab49"] + 100% 201.01 MiB 428 MiB/s 0s [2020-06-08 10:02:21.820358727] process done ExitSuccess + (from origin...) + [2020-06-08 10:02:21.881519879] read: rsync ["--progress","--inplace","--perms","../../../../../../../home/kanak/Annex/TV/.git/annex/objects/3P/zP/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c + 049d6e00fa40734860e4aadcbc8009a175b37ab49/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c049d6e00fa40734860e4aadcbc8009a175b37ab49","../.git/annex/tmp/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c0 + 49d6e00fa40734860e4aadcbc8009a175b37ab49"] + 100% 201.01 MiB 425 MiB/s 0s [2020-06-08 10:02:22.358355859] process done ExitSuccess + + Unable to access these remotes: origin + + Try making some of these repositories available: + 1cca049d-83f7-4a78-ad30-650621c8e6f9 -- amnesiac + 3ae7e0b0-f87a-4f5c-9d37-a507273dde61 -- balrog [origin] + bafcea18-6642-4c43-8ae6-8477092c7d02 -- babadook + failed + [2020-06-08 10:02:22.388873071] process done ExitSuccess + [2020-06-08 10:02:22.38926145] process done ExitSuccess + git-annex: get: 1 failed + ### What version of git-annex are you using? On what operating system? git annex version -git-annex version: 8.20200330 -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.4 feed-1.1.0.0 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.1 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL + + git-annex version: 8.20200330 + build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite + dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.4 feed-1.1.0.0 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.1 + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: linux x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 0 1 2 3 4 5 6 7 -local repository version: 8 + operating system: linux x86_64 + supported repository versions: 8 + upgrade supported from repository versions: 0 1 2 3 4 5 6 7 + local repository version: 8 Operating System: -cat /etc/fedora-release -Fedora release 32 (Thirty Two) + cat /etc/fedora-release + + Fedora release 32 (Thirty Two) ### Please provide any additional information below. git annex info --fast -trusted repositories: 7 - 318863a2-d9eb-467f-8094-f6578e26bd9a -- Melange - 33e44b4c-42f1-4383-90b3-5fcbdbbf3baf -- baldwin - 43d346b9-3c60-4036-a654-4c31f2368206 -- thane - bafcea18-6642-4c43-8ae6-8477092c7d02 -- babadook - d0ad767c-9874-4a3e-8ced-5d6917b212b9 -- zuko - d3ba71fb-64fc-4f41-8e76-7de8af14cee1 -- erebus [here] - d6d1d08e-6ee5-4408-a515-adbc9bebf5b1 -- ocho -semitrusted repositories: 1 - 3ae7e0b0-f87a-4f5c-9d37-a507273dde61 -- balrog [origin] -untrusted repositories: 2 - 1cca049d-83f7-4a78-ad30-650621c8e6f9 -- amnesiac - b3407756-8beb-43c3-877d-ff4fd1061fa9 -- chandrian -transfers in progress: none -available local disk space: 977.01 gigabytes (+1 megabyte reserved) + trusted repositories: 7 + 318863a2-d9eb-467f-8094-f6578e26bd9a -- Melange + 33e44b4c-42f1-4383-90b3-5fcbdbbf3baf -- baldwin + 43d346b9-3c60-4036-a654-4c31f2368206 -- thane + bafcea18-6642-4c43-8ae6-8477092c7d02 -- babadook + d0ad767c-9874-4a3e-8ced-5d6917b212b9 -- zuko + d3ba71fb-64fc-4f41-8e76-7de8af14cee1 -- erebus [here] + d6d1d08e-6ee5-4408-a515-adbc9bebf5b1 -- ocho + semitrusted repositories: 1 + 3ae7e0b0-f87a-4f5c-9d37-a507273dde61 -- balrog [origin] + untrusted repositories: 2 + 1cca049d-83f7-4a78-ad30-650621c8e6f9 -- amnesiac + b3407756-8beb-43c3-877d-ff4fd1061fa9 -- chandrian + transfers in progress: none + available local disk space: 977.01 gigabytes (+1 megabyte reserved)