mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc/tips/Internet_Archive_via_S3.mdwn

81 lines
3.5 KiB
Text
Raw Normal View History

added documentation for using the Internet Archive as a remote via S3 Renamed Amazon_S3 page to just S3.
2011-05-16 06:07:59 +00:00
[The Internet Archive](http://www.archive.org/) allows members to upload
collections using an Amazon S3
[compatible API](http://www.archive.org/help/abouts3.txt), and this can
be used with git-annex's [[special_remotes/S3]] support.
cleanup
2011-05-16 15:52:33 +00:00
So, you can locally archive things with git-annex, define remotes that
correspond to "items" at the Internet Archive, and use git-annex to upload
your files to there. Of course, your use of the Internet Archive must
comply with their [terms of service](http://www.archive.org/about/terms.php).
added documentation for using the Internet Archive as a remote via S3 Renamed Amazon_S3 page to just S3.
2011-05-16 06:07:59 +00:00
update
2013-04-26 05:00:58 +00:00
A nice added feature is that whenever git-annex sends a file to the
Internet Archive, it records its url, the same as if you'd run `git annex
addurl`. So any users who can clone your repository can download the files
S3: Allow removing files from IA, but warn about derived versions potentially still existing there. Removal works, only derives are a potential issue, so allow removing with a warning. This way, unexporting a file works, and behavior is consistent with IA remotes whether or not exporttree=yes. Also tested exporting filenames containing unicode, spaces, underscores. All worked, despite the IA's faq saying it doesn't. This commit was sponsored by Trenton Cronholm on Patreon.
2017-09-12 16:33:08 +00:00
from archive.org, without needing any login or password info.
The url to the content in the Internet Archive is also displayed by
`git annex whereis`. This makes the Internet Archive a nice way to
publish the large files associated with a public git repository.
update
2013-04-26 05:00:58 +00:00
show how to set up preferred content for IA like the webapp does
2014-02-28 22:42:31 +00:00
## webapp setup
Just go to "Add Another Repository", pick "Internet Archive",
and you're on your way.
## basic setup
update
2013-04-26 05:00:58 +00:00
added documentation for using the Internet Archive as a remote via S3 Renamed Amazon_S3 page to just S3.
2011-05-16 06:07:59 +00:00
Sign up for an account, and get your access keys here:
<http://www.archive.org/account/s3.php>
# export AWS_ACCESS_KEY_ID=blahblah
# export AWS_SECRET_ACCESS_KEY=xxxxxxx
add a few tweaks to make it easy to use the Internet Archive's variant of S3 In particular, munge key filenames to comply with the IA's filename limits, disable encryption, support their nonstandard way of creating buckets, and allow x-amz-* headers to be specified in initremote to set item metadata. Still TODO: initremote does not handle multiword metadata headers right.
2011-05-16 15:20:30 +00:00
Specify `host=s3.us.archive.org` when doing `initremote` to set up
a remote at the Archive. This will enable a special Internet Archive mode:
Encryption is not allowed; you are required to specify a bucket name
tweak
2011-05-16 16:12:03 +00:00
rather than having git-annex pick a random one; and you can optionally
add a few tweaks to make it easy to use the Internet Archive's variant of S3 In particular, munge key filenames to comply with the IA's filename limits, disable encryption, support their nonstandard way of creating buckets, and allow x-amz-* headers to be specified in initremote to set item metadata. Still TODO: initremote does not handle multiword metadata headers right.
2011-05-16 15:20:30 +00:00
specify `x-archive-meta*` headers to add metadata as explained in their
[documentation](http://www.archive.org/help/abouts3.txt).
cleanup
2011-05-16 15:52:33 +00:00
# git annex initremote archive-panama type=S3 \
host=s3.us.archive.org bucket=panama-canal-lock-blueprints \
add a few tweaks to make it easy to use the Internet Archive's variant of S3 In particular, munge key filenames to comply with the IA's filename limits, disable encryption, support their nonstandard way of creating buckets, and allow x-amz-* headers to be specified in initremote to set item metadata. Still TODO: initremote does not handle multiword metadata headers right.
2011-05-16 15:20:30 +00:00
x-archive-meta-mediatype=texts x-archive-meta-language=eng \
upload to the test collection, giving users an idea of how to change collections as well as forcing people to think before they upload
2018-11-05 16:02:23 +00:00
x-archive-meta-collection=test_collection \
add a few tweaks to make it easy to use the Internet Archive's variant of S3 In particular, munge key filenames to comply with the IA's filename limits, disable encryption, support their nonstandard way of creating buckets, and allow x-amz-* headers to be specified in initremote to set item metadata. Still TODO: initremote does not handle multiword metadata headers right.
2011-05-16 15:20:30 +00:00
x-archive-meta-title="original Panama Canal lock design blueprints"
IA: do not create bucket at initremote time This way, the metadata sent when uploading a file is applied to the bucket then.
2011-05-16 17:10:26 +00:00
initremote archive-panama (Internet Archive mode) ok
heh
2011-05-16 17:33:33 +00:00
# git annex describe archive-panama "a man, a plan, a canal: panama"
added documentation for using the Internet Archive as a remote via S3 Renamed Amazon_S3 page to just S3.
2011-05-16 06:07:59 +00:00
describe archive-panama ok
upload to the test collection, giving users an idea of how to change collections as well as forcing people to think before they upload
2018-11-05 16:02:23 +00:00
The above uploads to the [test collection](https://archive.org/details/test_collection) where items are removed
after thirty days. Uploads can persist by changing to another writable
[collection](https://internetarchive.readthedocs.io/en/latest/metadata.html#collection).
added documentation for using the Internet Archive as a remote via S3 Renamed Amazon_S3 page to just S3.
2011-05-16 06:07:59 +00:00
Then you can annex files and copy them to the remote as usual:
show how to set up preferred content for IA like the webapp does
2014-02-28 22:42:31 +00:00
# git annex add photo1.jpeg --backend=SHA256E
Added filename extension preserving variant backends SHA1E, SHA256E, etc.
2011-05-16 15:46:34 +00:00
add photo1.jpeg (checksum...) ok
add a few tweaks to make it easy to use the Internet Archive's variant of S3 In particular, munge key filenames to comply with the IA's filename limits, disable encryption, support their nonstandard way of creating buckets, and allow x-amz-* headers to be specified in initremote to set item metadata. Still TODO: initremote does not handle multiword metadata headers right.
2011-05-16 15:20:30 +00:00
# git annex copy photo1.jpeg --fast --to archive-panama
copy (to archive-panama...) ok
stop warning about removals from IA In a test, I uploaded a pdf, and several files were derived from it. After removing the pdf, the derived files went away after approximatly half an hour. This window does not seem worth warning about every time. Documented it in the tip.
2017-09-12 16:45:33 +00:00
## update lag
S3: Allow removing files from IA, but warn about derived versions potentially still existing there. Removal works, only derives are a potential issue, so allow removing with a warning. This way, unexporting a file works, and behavior is consistent with IA remotes whether or not exporttree=yes. Also tested exporting filenames containing unicode, spaces, underscores. All worked, despite the IA's faq saying it doesn't. This commit was sponsored by Trenton Cronholm on Patreon.
2017-09-12 16:33:08 +00:00
It may take a while for archive.org to make files publically visible after
they've been uploaded.
While files can be removed from the Internet Archive,
[derived versions](https://archive.org/help/derivatives.php)
stop warning about removals from IA In a test, I uploaded a pdf, and several files were derived from it. After removing the pdf, the derived files went away after approximatly half an hour. This window does not seem worth warning about every time. Documented it in the tip.
2017-09-12 16:45:33 +00:00
of some files may continued to be stored there for a while
after the originals were removed.
S3: Try to ensure bucket name is valid for archive.org.
2013-10-16 20:35:47 +00:00
S3 export (untested) It opens a http connection per file exported, but then so does git annex copy --to s3. Decided not to munge exported filenames for IA. Too large a chance of the munging having confusing results. Instead, export of files not supported by IA, eg with spaces in their name, will fail. This commit was supported by the NSF-funded DataLad project.
2017-09-08 19:41:31 +00:00
## exporting trees
show how to set up preferred content for IA like the webapp does
2014-02-28 22:42:31 +00:00
S3 export (untested) It opens a http connection per file exported, but then so does git annex copy --to s3. Decided not to munge exported filenames for IA. Too large a chance of the munging having confusing results. Instead, export of files not supported by IA, eg with spaces in their name, will fail. This commit was supported by the NSF-funded DataLad project.
2017-09-08 19:41:31 +00:00
By default, files stored in the Internet Archive will show up there named
by their git-annex key, not the original filename. If the filenames
are important, you can run `git annex initremote` with an additional
parameter "exporttree=yes", and then use [[git-annex-export]] to publish
a tree of files to the Internet Archive.
show how to set up preferred content for IA like the webapp does
2014-02-28 22:42:31 +00:00
S3: Allow removing files from IA, but warn about derived versions potentially still existing there. Removal works, only derives are a potential issue, so allow removing with a warning. This way, unexporting a file works, and behavior is consistent with IA remotes whether or not exporttree=yes. Also tested exporting filenames containing unicode, spaces, underscores. All worked, despite the IA's faq saying it doesn't. This commit was sponsored by Trenton Cronholm on Patreon.
2017-09-12 16:33:08 +00:00
Note that the Internet Archive may not support certian characters
in filenames ([see FAQ](http://archive.org/about/faqs.php#1099)).
If exporting a filename fails due to such limitations, you would need
to rename it in your git annex repository in order to export it.
Reference in a new issue Copy permalink