From 975cb5edf13ac15df4ed75e0a38f5c97999a7698 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Fri, 6 Jan 2023 14:21:36 +0000 Subject: [PATCH 1/4] Added a comment: importtree readonly remote folder? --- .../comment_2_7e62dbfc5540a8737d77f25ad2acc4af._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/todo/import_tree_from_rsync_special_remote/comment_2_7e62dbfc5540a8737d77f25ad2acc4af._comment diff --git a/doc/todo/import_tree_from_rsync_special_remote/comment_2_7e62dbfc5540a8737d77f25ad2acc4af._comment b/doc/todo/import_tree_from_rsync_special_remote/comment_2_7e62dbfc5540a8737d77f25ad2acc4af._comment new file mode 100644 index 0000000000..7d4671f077 --- /dev/null +++ b/doc/todo/import_tree_from_rsync_special_remote/comment_2_7e62dbfc5540a8737d77f25ad2acc4af._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="importtree readonly remote folder?" + date="2023-01-06T14:21:35Z" + content=""" +I have ran into the same \"desire\" I think: to be able to `import` the read-only (to me) tree of DICOMs from a remote server reachable over ssh, and so that later on those who have a clone of the repository and have access to that server could easily `annex get` necessary load. Am I right in my importtree-ignorant thinking that if `rsync` special remote supports `importtree` mode -- it would also work fine when I have only read-only access to the remote folder? +"""]] From c7aa99808cbbdb1ee0b2b89a159d5db92170978e Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Fri, 6 Jan 2023 17:27:34 +0000 Subject: [PATCH 2/4] initial todo on prioritization of URLs --- ...ization_WITHIN___40__web__41___remote.mdwn | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 doc/todo/Allow_for_URLs_prioritization_WITHIN___40__web__41___remote.mdwn diff --git a/doc/todo/Allow_for_URLs_prioritization_WITHIN___40__web__41___remote.mdwn b/doc/todo/Allow_for_URLs_prioritization_WITHIN___40__web__41___remote.mdwn new file mode 100644 index 0000000000..06feaf9c8a --- /dev/null +++ b/doc/todo/Allow_for_URLs_prioritization_WITHIN___40__web__41___remote.mdwn @@ -0,0 +1,32 @@ +Per our brief discussion ATM git-annex allows to prioritize URLs only by assigning them to be handled by different special remotes and having different costs for those different remotes. + +This doesn't allow for e.g. prioritization within built-in special "web" remote which is the most frequent use case. Our use case: + + +``` +(base) dandi@drogon:/mnt/backup/dandi/dandizarrs/ea8c43c7-757e-4653-8e4a-a6d356120836$ git annex whereis 0/0/0/3/6/169 +(recording state in git...) +whereis 0/0/0/3/6/169 (2 copies) + 00000000-0000-0000-0000-000000000001 -- web + 86da9d10-da54-4371-8d6f-7559c6a236f5 -- dandi@drogon:/mnt/backup/dandi/dandizarrs/ea8c43c7-757e-4653-8e4a-a6d356120836 [here] + + web: https://api.dandiarchive.org/api/zarr/ea8c43c7-757e-4653-8e4a-a6d356120836.zarr/0/0/0/3/6/169 + web: https://dandiarchive.s3.amazonaws.com/zarr/ea8c43c7-757e-4653-8e4a-a6d356120836/0/0/0/3/6/169?versionId=h3qb0rOswsssHxEdfN8QAWUMoVhddQrY +ok + +``` + +where we have API-server based URL -- we do not want to access unless really really needed (would be the slowest, would bring load to the server etc), and then direct access to public bucket -- fastest (unless some other local remote has it even better). + +Joey envisioned potentially being able to assign priorities via e.g. + + git-annex enableremote web url-priority-1=s3.amazonaws.com/ url-prority-2=/api.dandiarchive.org/ + +but I also wondered if there could just be some way to provide costs (or adjustments to costs) for different URLs so they all become considered while considering costs across remotes? + +E.g. may be I have a URL which is fast (s3 bucket), then I have bunch of average regular remotes with decent speed (e.g. dropbox etc), and then URL to some slow archive (API server). Both urls are served by `web` remote, and there would be no way to "order" all data access schemes/remotes in the optimal sequence of costs unless different URLs could have different costs considered along with different remotes. + +PS somehow I have some odd memory of seeing some config option to provide git-annex a script which would output cost given a URL... I disliked that approach since it would require me to code the script, and thus didn't use it. Did I dream it up? + +[[!meta author=yoh]] +[[!tag projects/dandi]] From a5d3328a6f68e2819b853621b6934671d655d937 Mon Sep 17 00:00:00 2001 From: satra Date: Fri, 6 Jan 2023 19:36:07 +0000 Subject: [PATCH 3/4] Added a comment: use list order for cost --- .../comment_1_f2172080e8e3e324a739d8769dca2ed0._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/todo/Allow_for_URLs_prioritization_WITHIN___40__web__41___remote/comment_1_f2172080e8e3e324a739d8769dca2ed0._comment diff --git a/doc/todo/Allow_for_URLs_prioritization_WITHIN___40__web__41___remote/comment_1_f2172080e8e3e324a739d8769dca2ed0._comment b/doc/todo/Allow_for_URLs_prioritization_WITHIN___40__web__41___remote/comment_1_f2172080e8e3e324a739d8769dca2ed0._comment new file mode 100644 index 0000000000..aae4124bea --- /dev/null +++ b/doc/todo/Allow_for_URLs_prioritization_WITHIN___40__web__41___remote/comment_1_f2172080e8e3e324a739d8769dca2ed0._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="satra" + avatar="http://cdn.libravatar.org/avatar/d3afc58453bc273dc015254e1d9581b3" + subject="use list order for cost" + date="2023-01-06T19:36:07Z" + content=""" +as a starting point is order an implicit notion of cost? so could having s3 before dandi achieve the expected outcome. try s3 first then try api? +"""]] From 31f500bab6435be4f176d4631d9c7ee30367f5bd Mon Sep 17 00:00:00 2001 From: "sb-beryllium@6e2c477eac63b823bd315ef8aaf5f93173c1f15b" Date: Sat, 7 Jan 2023 13:05:24 +0000 Subject: [PATCH 4/4] Added a comment --- ..._82103077cde68c3b6a70dcf70c4ed837._comment | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 doc/bugs/CRLF_breaks_interoperability_Win-Ux__58__..post-receive/comment_4_82103077cde68c3b6a70dcf70c4ed837._comment diff --git a/doc/bugs/CRLF_breaks_interoperability_Win-Ux__58__..post-receive/comment_4_82103077cde68c3b6a70dcf70c4ed837._comment b/doc/bugs/CRLF_breaks_interoperability_Win-Ux__58__..post-receive/comment_4_82103077cde68c3b6a70dcf70c4ed837._comment new file mode 100644 index 0000000000..a54f932f79 --- /dev/null +++ b/doc/bugs/CRLF_breaks_interoperability_Win-Ux__58__..post-receive/comment_4_82103077cde68c3b6a70dcf70c4ed837._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="sb-beryllium@6e2c477eac63b823bd315ef8aaf5f93173c1f15b" + nickname="sb-beryllium" + avatar="http://cdn.libravatar.org/avatar/ef62105380b73ef91d760ec327c14e22" + subject="comment 4" + date="2023-01-07T13:05:23Z" + content=""" +Apologies, this is beryllium (by alias)... due to unforeseen circumstances, I have had to register with a different email address. I am hoping this is a temporary situation. + +Further apologies... I acknowledge that I am vacillating somewhat, but I've reverted to thinking that the hooks should be generated with unix line-endings (LF). + +The reason I say this is, it does appear that the pseudo-standard for files under the .git directory is that they use unix line-endings only (perhaps an actual, documented standard... I'm not sure where to look/ask to confirm it). + +This is the case for files such as .git/config, and .git/refs/** etc... under Git for Windows, and even JGit running with Windows native Java. + +So to me.. it seems to make sense that hook files should have unix line-endings exclusively. + +There are possible other mitigations... but that's where I stand with that finally. I'm still not fussed if no actions is determined to be the best course of action. + +Thanks regardless. + +"""]]