From 871257ee23d9d3d71223b5648ddf7f18cbb34cef Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Tue, 1 Sep 2020 20:50:02 +0000 Subject: [PATCH 1/8] Added a comment --- .../comment_7_c35df3621c7b6a1b518787fdc8967ee7._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/todo/publicurl_config_for_all_special_remotes/comment_7_c35df3621c7b6a1b518787fdc8967ee7._comment diff --git a/doc/todo/publicurl_config_for_all_special_remotes/comment_7_c35df3621c7b6a1b518787fdc8967ee7._comment b/doc/todo/publicurl_config_for_all_special_remotes/comment_7_c35df3621c7b6a1b518787fdc8967ee7._comment new file mode 100644 index 0000000000..15f30598a9 --- /dev/null +++ b/doc/todo/publicurl_config_for_all_special_remotes/comment_7_c35df3621c7b6a1b518787fdc8967ee7._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 7" + date="2020-09-01T20:49:59Z" + content=""" +Awesome, Thank you! I [still](https://git-annex.branchable.com/todo/generic_readonly_http_remote/#comment-f6e076370814fcfab3a4d203a1adf326) think that it could be named more generically than `http` (e.g. probably could remap to public `rsync://`, `ftp://` or `s3://` to which git-annex knows how to \"talk to\") even if ATM supporting only `http/https`, but I guess time will show if that would be needed. +"""]] From 03cea490425ec0c60e4ccceb0000da45c5c7d32b Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Wed, 2 Sep 2020 02:53:23 +0000 Subject: [PATCH 2/8] Added a comment --- .../comment_2_5bed8f612e182a5c53dd464aea9b505b._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/does_not_handle_youtube_playlists/comment_2_5bed8f612e182a5c53dd464aea9b505b._comment diff --git a/doc/bugs/does_not_handle_youtube_playlists/comment_2_5bed8f612e182a5c53dd464aea9b505b._comment b/doc/bugs/does_not_handle_youtube_playlists/comment_2_5bed8f612e182a5c53dd464aea9b505b._comment new file mode 100644 index 0000000000..04a75db01a --- /dev/null +++ b/doc/bugs/does_not_handle_youtube_playlists/comment_2_5bed8f612e182a5c53dd464aea9b505b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 2" + date="2020-09-02T02:53:21Z" + content=""" +yep, would have been nice if git-annex either asked youtube-dl or just figured out itself to map URL to `https://www.youtube.com/feeds/videos.xml?playlist_id=ID` to get a feed for a copy/pasted playlist URL. +"""]] From b16725a377906c2dededb9b2f4706967d52a1c7b Mon Sep 17 00:00:00 2001 From: "hans.ekbrand@aabd6f2f1f93eab77ef85618d24892ebafc6e20f" Date: Wed, 2 Sep 2020 13:09:04 +0000 Subject: [PATCH 3/8] --- ...ecutable_files_becomes_non-executable.mdwn | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn diff --git a/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn b/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn new file mode 100644 index 0000000000..d89019343a --- /dev/null +++ b/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn @@ -0,0 +1,47 @@ +### Please describe the problem. + + +### What steps will reproduce the problem? + +```touch foo +chmod 755 foo +git-annex add foo +./foo +``` + +### What version of git-annex are you using? On what operating system? + +``` +git-annex version: 7.20190129 +build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite +dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external +operating system: linux x86_64 +supported repository versions: 5 7 +upgrade supported from repository versions: 0 1 2 3 4 5 6 +local repository version: 5 +``` + +### Please provide any additional information below. + +$ touch foo +$ chmod 755 foo +$ ls -l foo +-rwxr-xr-x 1 hans hans 0 sep 2 15:00 foo +$ git-annex add foo +add foo ok +(recording state in git...) +$ ls -lL foo +-rw-rw-rw- 1 hans hans 65 jul 24 2015 foo + +NB: the date is now jul 24 2015, but I created foo 2 sep, so what happened here? (This annex is old, so I might have created a file foo in 2015, but that should not intere with this new file). + +I have my private binaries in ~/annex/bin and this has worked well for many years, but recently it stopped working. + + + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +I've used git-annex since summer 2012, it's great :-) + From 5384b70bdef18da4d8549bc0a84502b1c802bfd7 Mon Sep 17 00:00:00 2001 From: "hans.ekbrand@aabd6f2f1f93eab77ef85618d24892ebafc6e20f" Date: Wed, 2 Sep 2020 13:10:31 +0000 Subject: [PATCH 4/8] --- ...de_bits___40__executable_files_becomes_non-executable.mdwn | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn b/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn index d89019343a..7ff4d136a6 100644 --- a/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn +++ b/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn @@ -6,7 +6,7 @@ ```touch foo chmod 755 foo git-annex add foo -./foo +ls -lL foo ``` ### What version of git-annex are you using? On what operating system? @@ -25,6 +25,7 @@ local repository version: 5 ### Please provide any additional information below. +``` $ touch foo $ chmod 755 foo $ ls -l foo @@ -34,6 +35,7 @@ add foo ok (recording state in git...) $ ls -lL foo -rw-rw-rw- 1 hans hans 65 jul 24 2015 foo +``` NB: the date is now jul 24 2015, but I created foo 2 sep, so what happened here? (This annex is old, so I might have created a file foo in 2015, but that should not intere with this new file). From 6c2db1415ab633c253700bb512f2e0d39d97fbe0 Mon Sep 17 00:00:00 2001 From: "hans.ekbrand@aabd6f2f1f93eab77ef85618d24892ebafc6e20f" Date: Wed, 2 Sep 2020 13:11:03 +0000 Subject: [PATCH 5/8] --- ...mode_bits___40__executable_files_becomes_non-executable.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn b/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn index 7ff4d136a6..f64a70f900 100644 --- a/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn +++ b/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn @@ -37,7 +37,7 @@ $ ls -lL foo -rw-rw-rw- 1 hans hans 65 jul 24 2015 foo ``` -NB: the date is now jul 24 2015, but I created foo 2 sep, so what happened here? (This annex is old, so I might have created a file foo in 2015, but that should not intere with this new file). +NB: the date of the file is now jul 24 2015, but I created foo 2 sep, so what happened here? (This annex is old, so I might have created a file foo in 2015, but that should not intere with this new file). I have my private binaries in ~/annex/bin and this has worked well for many years, but recently it stopped working. From 2225c69fa29ac955bc6ce6118cdf23322f1ab818 Mon Sep 17 00:00:00 2001 From: kyle Date: Wed, 2 Sep 2020 14:02:37 +0000 Subject: [PATCH 6/8] content verification before transfer to special remote --- ...al_remote_verify_source_content__63__.mdwn | 63 +++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__.mdwn diff --git a/doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__.mdwn b/doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__.mdwn new file mode 100644 index 0000000000..42bde18da0 --- /dev/null +++ b/doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__.mdwn @@ -0,0 +1,63 @@ +A [DataLad issue](https://github.com/datalad/datalad/issues/4795) was +raised about users inadvertently corrupting locked files. That led to +an example where a user could copy corrupted content over to a special +remote, and the content isn't flagged until a `get` call. (A slightly +different example, based on a directory remote and without +`exporttree=yes`, is included below.) + +In the case of a regular remote, the `copy` call would fail earlier +with + +``` + verification of content failed + + failed to send content to remote + + + verification of content failed + + failed to send content to remote +failed +git-annex: copy: 1 failed +``` + +Should something similar happen when copying or exporting to a special +remote? Perhaps verification before transfer to a special remote +isn't worth it, but the successful transfer surprised me given the +behavior when transferring to regular remotes. + +[[!format sh """ +cd "$(mktemp -d "${TMPDIR:-/tmp}"/dl-XXXXXXX)" + +mkdir d +git init a +( + cd a + git annex init + + echo one >one + git annex add one + git commit -mone + + one_resolved=$(readlink -f one) + chmod +w $one_resolved + echo more >>$one_resolved + chmod -w $one_resolved + + git annex initremote d type=directory directory="$PWD"/../d encryption=none + git annex copy --to=d + + git annex drop --force one + git annex get one +) +"""]] + +``` +[...] +copy one (to d...) +ok +(recording state in git...) +``` + +[[!meta author=kyle]] +[[!tag projects/datalad]] From 1a6a73eeac7419afb5af02c2eab4d9c8a21a33bc Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Wed, 2 Sep 2020 15:35:22 +0000 Subject: [PATCH 7/8] Added a comment: verifying source contents --- ...ment_1_6de754a926c5ad0e2da6cf6a1c58ce69._comment | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__/comment_1_6de754a926c5ad0e2da6cf6a1c58ce69._comment diff --git a/doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__/comment_1_6de754a926c5ad0e2da6cf6a1c58ce69._comment b/doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__/comment_1_6de754a926c5ad0e2da6cf6a1c58ce69._comment new file mode 100644 index 0000000000..914a5eca88 --- /dev/null +++ b/doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__/comment_1_6de754a926c5ad0e2da6cf6a1c58ce69._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="verifying source contents" + date="2020-09-02T15:35:18Z" + content=""" +This would especially make sense when sending files to a trusted special remote, where the file may be the only copy. +Maybe, set the mtime of all files (and directories) in .git/annex/objects to some sentinel value after they're written, then check if the mtime still has that value before sending the file elsewhere? + +If a special remote [[supports named pipes|todo/let_external_remotes_declare_support_for_named_pipes]], the verification could be done on-the-fly as the file is streamed to the remote. + + +"""]] From 94c4da582dc6daa7d72db5f8937e7a2926b683bd Mon Sep 17 00:00:00 2001 From: kyle Date: Wed, 2 Sep 2020 15:51:49 +0000 Subject: [PATCH 8/8] fix example: meant to prune --force and include rest of output --- ...al_remote_verify_source_content__63__.mdwn | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__.mdwn b/doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__.mdwn index 42bde18da0..091e11c909 100644 --- a/doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__.mdwn +++ b/doc/forum/Should_transfer_to_a_special_remote_verify_source_content__63__.mdwn @@ -46,17 +46,28 @@ git init a git annex initremote d type=directory directory="$PWD"/../d encryption=none git annex copy --to=d - - git annex drop --force one + git annex drop one git annex get one ) """]] ``` [...] -copy one (to d...) -ok +copy one (to d...) +ok (recording state in git...) +drop one ok +(recording state in git...) +get one (from d...) + + verification of content failed + + Unable to access these remotes: d + + Try making some of these repositories available: + e6878750-3a21-4ae9-b9ae-a241f17176a4 -- [d] +failed +git-annex: get: 1 failed ``` [[!meta author=kyle]]