From 97aaf38a3ce34eca789bef83f88fb6b8c50e6ec7 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Fri, 19 Oct 2018 15:52:22 +0000 Subject: [PATCH 1/4] Added a comment --- .../comment_36_f8081ce200700516efef61ec0ac86f86._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/design/external_special_remote_protocol/comment_36_f8081ce200700516efef61ec0ac86f86._comment diff --git a/doc/design/external_special_remote_protocol/comment_36_f8081ce200700516efef61ec0ac86f86._comment b/doc/design/external_special_remote_protocol/comment_36_f8081ce200700516efef61ec0ac86f86._comment new file mode 100644 index 0000000000..ba95e05787 --- /dev/null +++ b/doc/design/external_special_remote_protocol/comment_36_f8081ce200700516efef61ec0ac86f86._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="comment 36" + date="2018-10-19T15:52:21Z" + content=""" +What is the intended use case for CHECKURL-MULTI? Are there examples of external special remote implementations that use this response? +"""]] From 43a8c564a0921b9210c13d6c4532faa8e566fadb Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Fri, 19 Oct 2018 17:54:49 +0000 Subject: [PATCH 2/4] Added a comment --- ...ment_2_915c1c31fcb23455aa11331cd34aa92d._comment | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 doc/todo/option_to_add_user-specified_string_to_key/comment_2_915c1c31fcb23455aa11331cd34aa92d._comment diff --git a/doc/todo/option_to_add_user-specified_string_to_key/comment_2_915c1c31fcb23455aa11331cd34aa92d._comment b/doc/todo/option_to_add_user-specified_string_to_key/comment_2_915c1c31fcb23455aa11331cd34aa92d._comment new file mode 100644 index 0000000000..f0b61e3cf8 --- /dev/null +++ b/doc/todo/option_to_add_user-specified_string_to_key/comment_2_915c1c31fcb23455aa11331cd34aa92d._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="Ilya_Shlyakhter" + avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" + subject="comment 2" + date="2018-10-19T17:54:49Z" + content=""" +Another use case for a user-defined string field in a key is, for URL or WORM keys, to include additional information unique to the object. E.g. for an s3:// URI (handled by an external special remote) could embed the ETag in the key. Or for a dx:// URI to include the DNAnexus file ID . + +From [[internals/key_format]], it seems like adding -uXXXXXX- (with XXXXXX a user-specified string) would not break compatibility? + + + +"""]] From 01f7721d56371b45be30089407391dc77bcac0a5 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Fri, 19 Oct 2018 20:28:12 +0000 Subject: [PATCH 3/4] added suggestion to preserve file extensions in WORM and URL keys --- doc/todo/preserve_file_extensions_in_WORM_and_URL_keys.mdwn | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/todo/preserve_file_extensions_in_WORM_and_URL_keys.mdwn diff --git a/doc/todo/preserve_file_extensions_in_WORM_and_URL_keys.mdwn b/doc/todo/preserve_file_extensions_in_WORM_and_URL_keys.mdwn new file mode 100644 index 0000000000..57cb9333e1 --- /dev/null +++ b/doc/todo/preserve_file_extensions_in_WORM_and_URL_keys.mdwn @@ -0,0 +1 @@ +Right now, when computing a WORM key from a relative path or a URL key from a URL, if the original string is longer than a SHA256 checksum, its tail is replaced with its md5. Unfortunately, this eats up the file extension(s) at the end, causing the issues that *E backends solve. It would be better to keep the tail of the path and replace the start or the middle with the md5, preserving extensions (as configured in annex.maxextensionlength) the same way *E backends do. Maybe also, add a config option for the length beyond which the replacement-with-checksum happens? From c742cc41d1106c3a0bd2303d32005cbd83cfc127 Mon Sep 17 00:00:00 2001 From: Ilya_Shlyakhter Date: Fri, 19 Oct 2018 20:41:52 +0000 Subject: [PATCH 4/4] quoted markdown asterisks --- doc/todo/preserve_file_extensions_in_WORM_and_URL_keys.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/todo/preserve_file_extensions_in_WORM_and_URL_keys.mdwn b/doc/todo/preserve_file_extensions_in_WORM_and_URL_keys.mdwn index 57cb9333e1..c9ff47cded 100644 --- a/doc/todo/preserve_file_extensions_in_WORM_and_URL_keys.mdwn +++ b/doc/todo/preserve_file_extensions_in_WORM_and_URL_keys.mdwn @@ -1 +1 @@ -Right now, when computing a WORM key from a relative path or a URL key from a URL, if the original string is longer than a SHA256 checksum, its tail is replaced with its md5. Unfortunately, this eats up the file extension(s) at the end, causing the issues that *E backends solve. It would be better to keep the tail of the path and replace the start or the middle with the md5, preserving extensions (as configured in annex.maxextensionlength) the same way *E backends do. Maybe also, add a config option for the length beyond which the replacement-with-checksum happens? +Right now, when computing a WORM key from a relative path or a URL key from a URL, if the original string is longer than a SHA256 checksum, its tail is replaced with its md5. Unfortunately, this eats up the file extension(s) at the end, causing the issues that \*E backends solve. It would be better to keep the tail of the path and replace the start or the middle with the md5, preserving extensions (as configured in annex.maxextensionlength) the same way \*E backends do. Maybe also, add a config option for the length beyond which the replacement-with-checksum happens?