From 593f992e9a9b5504a9a79ff7b3b7f8fcaf8ef73f Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Wed, 13 Nov 2024 17:57:58 +0000 Subject: [PATCH 1/2] Added a comment --- ...ment_5_832824ac8722238ed00fc0604ff394e0._comment | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 doc/bugs/add__58___inconsistently_treats_files_in_dotdirs_as_dotfiles/comment_5_832824ac8722238ed00fc0604ff394e0._comment diff --git a/doc/bugs/add__58___inconsistently_treats_files_in_dotdirs_as_dotfiles/comment_5_832824ac8722238ed00fc0604ff394e0._comment b/doc/bugs/add__58___inconsistently_treats_files_in_dotdirs_as_dotfiles/comment_5_832824ac8722238ed00fc0604ff394e0._comment new file mode 100644 index 0000000000..7fb234fd5a --- /dev/null +++ b/doc/bugs/add__58___inconsistently_treats_files_in_dotdirs_as_dotfiles/comment_5_832824ac8722238ed00fc0604ff394e0._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 5" + date="2024-11-13T17:57:58Z" + content=""" +@lell, would you be so kind to point to me to \"formalization\" you use for your containers/configuration layout? +\"I like containers too\", and in neuro* domain we are formalizing layout of data etc on the filesystem in [BIDS standard](https://bids.neuroimaging.io/). +Moreover we are trying to formalize at the level of the \"entire project\", see e.g. [this issue with examples](https://github.com/bids-standard/bids-specification/pull/1861#issuecomment-2183701293). +I would be interested to learn what/how you do it. +Feel welcome to reach out directly to e.g. `debian AT oneukrainian.com`. +Cheers, +"""]] From 1c8a5dc586c24d5e424809c21a439ee43e285aa3 Mon Sep 17 00:00:00 2001 From: yarikoptic Date: Wed, 13 Nov 2024 18:08:59 +0000 Subject: [PATCH 2/2] Added a comment --- ..._cd7b1bc28a000b262e11c2b89aeed0da._comment | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 doc/bugs/importtree_from_S3_slows_to_halt_even_with_prefix/comment_1_cd7b1bc28a000b262e11c2b89aeed0da._comment diff --git a/doc/bugs/importtree_from_S3_slows_to_halt_even_with_prefix/comment_1_cd7b1bc28a000b262e11c2b89aeed0da._comment b/doc/bugs/importtree_from_S3_slows_to_halt_even_with_prefix/comment_1_cd7b1bc28a000b262e11c2b89aeed0da._comment new file mode 100644 index 0000000000..3c58d06076 --- /dev/null +++ b/doc/bugs/importtree_from_S3_slows_to_halt_even_with_prefix/comment_1_cd7b1bc28a000b262e11c2b89aeed0da._comment @@ -0,0 +1,29 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 1" + date="2024-11-13T18:08:59Z" + content=""" +At the end (after over a day of torturing that poor bucket, whenever it took just few minutes for `s3cmd sync` to get everything including content) it crashed with + +``` +[2024-11-12 22:58:00.366878941] (Remote.S3) Response status: Status {statusCode = 200, statusMessage = \"OK\"} +[2024-11-12 22:58:00.373456754] (Remote.S3) Response header 'x-amz-id-2': 'DGXJztoRJRuHQrcOqs3FtnEUJomRz+53jawFoKoRbKQATcvAppqJcfcAVfR1d8cu7uepkEDvSXo=' +[2024-11-12 22:58:00.384304583] (Remote.S3) Response header 'x-amz-request-id': 'W1PSPV7ZSBKJ7HTT' +[2024-11-12 22:58:00.38437407] (Remote.S3) Response header 'Date': 'Wed, 13 Nov 2024 03:50:18 GMT' +[2024-11-12 22:58:00.384436037] (Remote.S3) Response header 'x-amz-bucket-region': 'us-east-2' +[2024-11-12 22:58:00.384486611] (Remote.S3) Response header 'Content-Type': 'application/xml' +[2024-11-12 22:58:00.384533794] (Remote.S3) Response header 'Transfer-Encoding': 'chunked' +[2024-11-12 22:58:00.384581117] (Remote.S3) Response header 'Server': 'AmazonS3' + +git-annex: Unable to list contents of s3-dandiarchive: Network.Socket.recvBuf: resource vanished (Connection reset by peer) +failed +[2024-11-12 22:58:00.565431711] (Utility.Process) process [3912839] done ExitSuccess +import: 1 failed + +``` + +attesting that it is doing something unnecessary -- either listing full bucket (unlikely) or listing all versions of keys under the prefix (e.g. using [ListObjectVersions](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html) instead of [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html)). + +It would have been useful if logs included the API call involved here. +"""]]