From 09c5b7d385afa7ce59e0ead01c19d19f4e2b1b64 Mon Sep 17 00:00:00 2001 From: "http://digiuser.livejournal.com/" Date: Mon, 13 Oct 2014 17:35:40 +0000 Subject: [PATCH 1/5] --- ...oes_not_work_-_chunking_does_not_work.mdwn | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn diff --git a/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn b/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn new file mode 100644 index 0000000000..3661c12516 --- /dev/null +++ b/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn @@ -0,0 +1,45 @@ +I have a ~10 GB file in git-annex. I can't get it to go to S3 whatever I do. + +I added a S3 special remote: + + git annex initremote s3-mybucket type=S3 chunk=1MiB keyid=ABCD1234 bucket=mybucket + +Then, I tried copying files to the remote. Small files worked, but big files don't: + + $ git annex copy bigfile.tgz --to s3-mybucket + copy bigfile.tgz (gpg) + You need a passphrase to unlock the secret key for + user: "user" + 2048-bit RSA key, ID ABCD1234, created 2014-10-13 (main key ID ABCD1234) + + (checking s3-mybucket...) (to s3-mybucket...) + + Your proposed upload exceeds the maximum allowed size + failed + git-annex: copy: 1 failed + +I tried some stuff like this too: + + git annex enableremote s3-oregon chunk=100MiB + git annex enableremote s3-oregon chunksize=100MiB + +It didn't work. Same result. + + $ git annex version + git-annex version: 5.20140717 + build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV FsEvents XMPP DNS Feeds Quvi TDFA CryptoHash + key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL + remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external + local repository version: 5 + supported repository version: 5 + upgrade supported from repository versions: 0 1 2 4 + +The chunk size did seem to bet set properly: + + $ git checkout git-annex + $ cat remote.log + xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx bucket=mybucket chunk=100MiB chunksize=100MiB cipher=.... + +I'm on OSX 10.9.4 and I installed git-annex via homebrew. + +Any ideas? From 6dd45b6f135a9f0a9361e090191d321a80f7a2d8 Mon Sep 17 00:00:00 2001 From: "http://digiuser.livejournal.com/" Date: Mon, 13 Oct 2014 17:43:49 +0000 Subject: [PATCH 2/5] --- .../Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn b/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn index 3661c12516..6b7311b58b 100644 --- a/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn +++ b/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn @@ -20,8 +20,8 @@ Then, I tried copying files to the remote. Small files worked, but big files do I tried some stuff like this too: - git annex enableremote s3-oregon chunk=100MiB - git annex enableremote s3-oregon chunksize=100MiB + git annex enableremote s3-mybucket chunk=100MiB + git annex enableremote s3-mybucket chunksize=100MiB It didn't work. Same result. From 1f50be596963d400837422cb5494f90d16ecb5d7 Mon Sep 17 00:00:00 2001 From: "http://digiuser.livejournal.com/" Date: Mon, 13 Oct 2014 18:27:49 +0000 Subject: [PATCH 3/5] --- .../Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn b/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn index 6b7311b58b..261fe294cb 100644 --- a/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn +++ b/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn @@ -34,7 +34,7 @@ It didn't work. Same result. supported repository version: 5 upgrade supported from repository versions: 0 1 2 4 -The chunk size did seem to bet set properly: +The chunk size did seem to be set properly: $ git checkout git-annex $ cat remote.log From 0b04b28fc3a8dcf4cb37ff64e357bae74871e03b Mon Sep 17 00:00:00 2001 From: "http://digiuser.livejournal.com/" Date: Mon, 13 Oct 2014 18:31:27 +0000 Subject: [PATCH 4/5] --- .../Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn b/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn index 261fe294cb..2865441e78 100644 --- a/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn +++ b/doc/forum/Copying_to_S3_does_not_work_-_chunking_does_not_work.mdwn @@ -1,3 +1,5 @@ +Posting this here because I am not sure if it is a bug or if I am missing something. + I have a ~10 GB file in git-annex. I can't get it to go to S3 whatever I do. I added a S3 special remote: From e6799a5e0cf459ae52f68ffa67c3c2d08e95ef17 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawk9SYh6N-JUMkYkW4aOk55zC3Vr9KonDV4" Date: Mon, 13 Oct 2014 19:27:08 +0000 Subject: [PATCH 5/5] --- doc/forum/Changing_files_during_git_annex_runs.mdwn | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/forum/Changing_files_during_git_annex_runs.mdwn diff --git a/doc/forum/Changing_files_during_git_annex_runs.mdwn b/doc/forum/Changing_files_during_git_annex_runs.mdwn new file mode 100644 index 0000000000..7ff362d1cc --- /dev/null +++ b/doc/forum/Changing_files_during_git_annex_runs.mdwn @@ -0,0 +1,12 @@ +Hello, + +I have my music git annexed, direct mode. It's about 30k files of 429GB size. Some actions take considerable time (sync, add and of course transfer to/from other repos). During this time I don't hear music because of my player changes files. :-( + +When is it a problem when a files changes during git annex operations? + +git annex get gives a wrong checksum I guess and you need to re-transfer later. + +What about git annex add? + +Thx! +Florian