mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2019-03-26 21:10:50 -04:00
commit 77a1e33093
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
8 changed files with 64 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
doc/design/assistant/polls/prioritizing_special_remotes.mdwn
View file

@ -6,7 +6,7 @@ locally paired systems, and remote servers with rsync.
Help me prioritize my work: What special remote would you most like
to use with the git-annex assistant?
[[!poll open=yes 18 "Amazon S3 (done)" 13 "Amazon Glacier (done)" 10 "Box.com (done)" 79 "My phone (or MP3 player)" 29 "Tahoe-LAFS" 17 "OpenStack SWIFT" 37 "Google Drive"]]
[[!poll open=yes 18 "Amazon S3 (done)" 13 "Amazon Glacier (done)" 10 "Box.com (done)" 79 "My phone (or MP3 player)" 29 "Tahoe-LAFS" 18 "OpenStack SWIFT" 37 "Google Drive"]]
This poll is ordered with the options I consider easiest to build
listed first. Mostly because git-annex already supports them and they

20
doc/forum/Getting__58___hGetChar__58___illegal_operation___40__handle_is_closed__41__/comment_2_7c88ebd9d7cdf4a009aa2580cc55787d._comment Normal file
View file

@ -0,0 +1,20 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="The issue persists with recent git-annex"
date="2019-03-24T23:00:30Z"
content="""
The same with annex locally 7.20190219+git191-g2d6a364d4-1~ndall+1 and on remote 7.20190129+git78-g3fa6be1fe-1~ndall+1, while copying to the remote host a LOT (59k) of files, 3 \"failed\" and I get datalad print me a summary of those 3 error messages
[[!format sh \"\"\"
git-annex: <stdin>: hGetChar: illegal operation (handle is closed)git-annex-shell: p2pstdio: 1 failed Lost connection (fd:20: hGetChar: end of file)
This could have failed because --fast is enabled.
git-annex: <stdin>: hGetChar: illegal operation (handle is closed)git-annex-shell: p2pstdio: 1 failed Lost connection (fd:19: hGetChar: end of file)
This could have failed because --fast is enabled.
git-annex: <stdin>: hGetChar: illegal operation (handle is closed)git-annex-shell: p2pstdio: 1 failed Lost connection (fd:19: hGetChar: end of file)
This could have failed because --fast is enabled.
git-annex: copy: 3 failed
\"\"\"]]
I guess it is because the files are actually pointing to the same key(s), which get transferred in some other parallel batch.
"""]]

8
doc/forum/Getting__58___hGetChar__58___illegal_operation___40__handle_is_closed__41__/comment_3_976cb63e97774859023123536bdc9d99._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 3"
date="2019-03-24T23:49:48Z"
content="""
actually not sure if that is a matter of multiple files with the same key since with a subsequent call to `datalad publish` 3 files did get published, so I guess it was something else.
"""]]

8
doc/tips/unlocked_files/comment_14_e3425d219001fc34470f0cd55fc7241e._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="annex.thin without hardlinks would be useful for non-crippled systems as well"
date="2019-03-25T18:57:36Z"
content="""
One of the concerns with use of git-annex on HPC and other heavily loaded systems is the >3x consumption of inodes due to all the symlinks etc. In some cases it could be completely avoided probably, if repository is instructed to be installed just for consumption (to access data) only. In this way it would be nice if we could get the same \"annex.thin without hardlinks\" that there would not be even any `.git/annex/objects` (`superthin`?) and files just get installed in-place.
"""]]

1
doc/todo/document_git-annex_dependencies.mdwn Normal file
View file

@ -0,0 +1 @@
It would help to document, in one place, the external programs and libraries on which git-annex depends for various functionalities, including optional ones. Ones I know: curl, gpg, bup. But there are also references in places to lsof, rsync, nocache. For reliable packaging, it would be useful to have an authoritative list of dependencies and which functionality each supports.

2
doc/todo/simpler__44___trusted_export_remotes.mdwn
View file

@ -1,3 +1,3 @@
Currently, some issues impede the use of export remotes: (1) they're untrusted, except for versioned ones -- and from those keys cannot be dropped; (2) using them is different than using normal remotes: one can't just copy or move keys to them, one has to first make a tree-ish. Maybe this could be fixed, as follows. To copy a key to a external remote, if the key is not yet present in it, put it under .keys/aaa/bbb/keyname on the remote. That is, take the tree-ish currently on the remote, merge .keys/aaa/bbb/keyname with it, and put that on the remote. To drop a key from an external remote, take the tree-ish currently on the remote, drop all instances of the key from it, and push the changed tree-ish to the remote. To git-annex-export add an option --add , which will add the tree-ish to the tree-ish currently on the remote, without losing any keys currently on the remote: take the tree-ish currently on the remote; overlay on it the treeish being exported; for any files that would be overwritten, if no copies of that key would be left, move it to .keys/aaa/bbb/keyname in the tree-ish that is then pushed to the remote.
Currently, some issues impede the use of export remotes: (1) they're untrusted, except for versioned ones -- and from those keys cannot be dropped; (2) using them is different than using normal remotes: one can't just copy or move keys to them, one has to first make a tree-ish. Maybe this could be fixed, as follows. To copy a key to an export remote, if the key is not yet present in it, put it under .keys/aaa/bbb/keyname on the remote. That is, take the tree-ish currently on the remote, merge .keys/aaa/bbb/keyname with it, and put that on the remote. To drop a key from an external remote, take the tree-ish currently on the remote, drop all instances of the key from it, and push the changed tree-ish to the remote. To git-annex-export add an option --add , which will add the tree-ish to the tree-ish currently on the remote, without losing any keys currently on the remote: take the tree-ish currently on the remote; overlay on it the treeish being exported; for any files that would be overwritten, if no copies of that key would be left, move it to .keys/aaa/bbb/keyname in the tree-ish that is then pushed to the remote.
This way, can always just copy any tree to the remote, without worrying about losing data.

14
doc/todo/simpler__44___trusted_export_remotes/comment_4_be525962211e8c4da681279aba132c71._comment Normal file
View file

@ -0,0 +1,14 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="simplifying the interface"
date="2019-03-26T17:40:32Z"
content="""
\"I'm doubtful that this would actually let the interface be simplified\" -- I only meant that the minimum required interface would be simplified, in that git-annex could provide a default implementation of key-value remote methods in terms of the export remote interface; but any given remote could provide a more efficient implementation of these methods, overriding the default ones.
But the main benefit would be to simplify the user-facing interface: as far as the user is concerned, all special remotes could be trusted, and accessed with the same basic commands, whether configured as export or not.
\"if a S3 bucket is read-only, then an import can't re-upload.\" -- if the special remote is configured as read-only, then git-annex itself would not attempt to upload things there, no?
\"not all users are going to want export remotes to store past versions of files\" -- maybe, there could be an option to store past versions encrypted, while storing current versions plain?
"""]]

11
doc/todo/simpler__44___trusted_export_remotes/comment_5_aeefb4803340ff993a1d6673c8e5e97b._comment Normal file
View file

@ -0,0 +1,11 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="comment 5"
date="2019-03-26T18:27:04Z"
content="""
Another approach would be to let a key-value remote and an export remote be combined into one \"combo\" remote. To the user, this would look like one trusted, versioned remote supporting both key-value and export operations. Keys overwritten in the export remote would be stored in the key-value remote. Either keys or trees could be copied to the combo remote, keys going to the key-value remote and trees to the export remote. The downside is that files could not be moved directly between the backing remotes. But the inefficiency might not always matter. Also, TRANSFER and TRANSFEREXPORT could be extended to optionally accept URIs in lieu of content, and to do the transfer in the cloud.
More generically, maybe repository groups could be treated as special remotes? You'd configure the minimum number of copies of a key in a group. You could then put a key-value remote and an export remote in a group. When copying a tree to the group, if this would cause old keys to be overwritten, git-annex would first copy them to a key-value remote in the group, to preserve the per-group minimum number of copies constraint.
"""]]