diff --git a/doc/bugs/warning_about_ssh_caching_keeps_showing.mdwn b/doc/bugs/warning_about_ssh_caching_keeps_showing.mdwn new file mode 100644 index 0000000000..370be161a7 --- /dev/null +++ b/doc/bugs/warning_about_ssh_caching_keeps_showing.mdwn @@ -0,0 +1,57 @@ +### Please describe the problem. +I keep getting the warning about ssh caching being disabled, even when I explicitly enable it. + +### What steps will reproduce the problem? +See log below + +### What version of git-annex are you using? On what operating system? +7.20200204 on Amazon Linux 2 + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + +(just-git-annex-env) 13:00 [viral-ngs-benchmarks] $ git annex sync -c annex.sshcaching=true +On branch is-devel +Your branch is up to date with 'origin/is-devel'. + + +It took 8.50 seconds to enumerate untracked files. 'status -uno' +may speed it up, but you have to be careful not to forget to add +new files yourself (see 'git help status'). +nothing to commit, working tree clean +commit ok +pull origin + You have enabled concurrency, but ssh connection caching is not enabled. This may result in multiple ssh processes prompting for pas\ +swords at the same time. +ok + +(just-git-annex-env) 13:00 [viral-ngs-benchmarks] $ uname -a +Linux ip-172-31-86-201.ec2.internal 4.14.165-131.185.amzn2.x86_64 #1 SMP Wed Jan 15 14:19:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux +(just-git-annex-env) 13:02 [viral-ngs-benchmarks] $ git annex version +git-annex version: 7.20200204-g4db801d +build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite +dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sql\ +ite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_2\ +24 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE\ +2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224\ + BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external +operating system: linux x86_64 +supported repository versions: 7 +upgrade supported from repository versions: 0 1 2 3 4 5 6 +local repository version: 7 + + + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +I've been using git-annex for 1.5 years to manage bioinformatics analyses. It's a very versatile and well-designed tool. I've been able to adapt it to many use cases; +the ability to easily write your own external backends has been especially helpful for that. The amount of work and thought that has gone into designing/building git-annex is +enormous, and very much appreciated. diff --git a/doc/contribute/comment_3_e7219514f52207fcfb97aeec03241e8d._comment b/doc/contribute/comment_3_e7219514f52207fcfb97aeec03241e8d._comment new file mode 100644 index 0000000000..513f6e5ae5 --- /dev/null +++ b/doc/contribute/comment_3_e7219514f52207fcfb97aeec03241e8d._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="Dan" + avatar="http://cdn.libravatar.org/avatar/986de9e060699ae70ff7c31342393adc" + subject="Editing Comments?" + date="2020-02-14T01:22:46Z" + content=""" +Is it possible to edit comments on the branchable wiki? I realized there was a sentence I failed to finish when posting [this comment](https://git-annex.branchable.com/todo/sync_--branches__to_sync_only_specified_branches___40__e.g._git-annex__41__/#comment-19feab1afb2e0b33315a8368a7cdebf7) and I'd love to go back and finish the thought. The \"Edit\" button at the top of the page lets me edit the content of the page, but not any of the comments. + +I tried cloning the wiki, editing the file corresponding to my comment, and then pushing, but the push was rejected (the changes were in doc tree so I expected it to be accepted, but perhaps comments are more locked down). +"""]] diff --git a/doc/forum/Transparent_compression_of_files.mdwn b/doc/forum/Transparent_compression_of_files.mdwn new file mode 100644 index 0000000000..2b3a28a35f --- /dev/null +++ b/doc/forum/Transparent_compression_of_files.mdwn @@ -0,0 +1,5 @@ +Hi, + +I have a lot of files which are around 80MB and can be easily compressed down to ~55MB. I did some tests with brotli and decompression was reasonable fast, at least fast enough that I would probably not notice given my current transfer speeds. In order to save disk space I would like to able to transparently compress my files. That means, a file is stored compressed in git-annex's blob store and decompressed when I `get` it. + +I understand that gpg does compression, but I don't want to deal with encryption, all my repos are local. I've looked at the code and from what I could see the Hash-Backends are rather simple. However, that's probably not the right place. Is this a planned feature? Would it be hard to implement? Of course, ideally the compression algorithm should be configurable. E.g. by just doing a syscall to `brotli` or `gzip`. diff --git a/doc/forum/Transparent_compression_of_files/comment_1_7242325defa000572ca8e78e29012451._comment b/doc/forum/Transparent_compression_of_files/comment_1_7242325defa000572ca8e78e29012451._comment new file mode 100644 index 0000000000..a2a96f1b77 --- /dev/null +++ b/doc/forum/Transparent_compression_of_files/comment_1_7242325defa000572ca8e78e29012451._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="lykos@d125a37d89b1cfac20829f12911656c40cb70018" + nickname="lykos" + avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2" + subject="comment 1" + date="2020-02-13T12:29:40Z" + content=""" +It's quite easy to implement an [external special remote](https://git-annex.branchable.com/special_remotes/external/) that does transparent compression. You can use the example implementations of the directory remote as a starting point (see [Bash](https://git-annex.branchable.com/special_remotes/external/example.sh/) or [Python](https://github.com/Lykos153/AnnexRemote/blob/master/examples/git-annex-remote-directory)). Then just modify the dostore() and doretrieve() functions to your liking. You probably don't want to support exporting when compression is enabled, though. +"""]] diff --git a/doc/forum/Transparent_compression_of_files/comment_2_a75e32b0825de0b405c45de14b9711c0._comment b/doc/forum/Transparent_compression_of_files/comment_2_a75e32b0825de0b405c45de14b9711c0._comment new file mode 100644 index 0000000000..a1ba8b4d4b --- /dev/null +++ b/doc/forum/Transparent_compression_of_files/comment_2_a75e32b0825de0b405c45de14b9711c0._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="jochen.keil@38b1f86ab65128dab3e62e726403ceee4f5141bf" + nickname="jochen.keil" + avatar="http://cdn.libravatar.org/avatar/a1329c0b3a262017553cc5497aa12c18" + subject="comment 2" + date="2020-02-14T12:04:53Z" + content=""" +Thanks for your hint, I appretiate that and I think it could be done that way. + +However, on closer thought I was wondering if git-annex is the right tool for job. I had the impression that my idea came more from a hammer and nail situation. So, FUSE came to my mind. I popped it into google and found this: https://github.com/FS-make-simple/fusecompress +Unfortunately this does not look very active though. + +Now, since I'm already at the FS layer I can look into ZFS compression. My repos are already on ZFS but I haven't looked at the built-in compression yet. I think I'll evaluate that first. If none of that is satisfactory I'll turn to git-annex again :) +"""]] diff --git a/doc/forum/git-annex-sync_without_syncing_master/comment_3_233c20435643d2701c94a2cf30ca6483._comment b/doc/forum/git-annex-sync_without_syncing_master/comment_3_233c20435643d2701c94a2cf30ca6483._comment new file mode 100644 index 0000000000..2b1fd79229 --- /dev/null +++ b/doc/forum/git-annex-sync_without_syncing_master/comment_3_233c20435643d2701c94a2cf30ca6483._comment @@ -0,0 +1,25 @@ +[[!comment format=mdwn + username="Nick_P" + avatar="http://cdn.libravatar.org/avatar/abf8aa3ac1a976a6a292416b9c604581" + subject="comment 3" + date="2020-02-12T10:50:06Z" + content=""" +Relatedly, my team is struggling to identify the process for \"add and share a file\" - e.g., add the hash to git, and copy the content to a central git-annex remote, and push the git-annex branch information that the file is now in the central remote. + +We used 'git annex sync' and 'git annex sync --content' for a while, but this creates the synced/ branches which introduced confusion; and it's unclear why both --content and with args are needed; and if I remember, it also committed or pushed changes we didn't want it to. + +Now we do like this: + + git checkout features/your-new-branch + git annex add my-new-file + git commit -m \".....\" + git push # pushes your \"features\" branch + git annex copy --to SharedRemote --jobs=5 + git fetch origin git-annex:refs/remotes/origin/git-annex + git annex merge + git push origin git-annex + +I've read through , , + +Maybe the OP has the same confusion, that there seems to be a missing git-annex-sync that does only (a) copy to a remote and (b) the git-annex branch work - Anyone able to help out direct us to how it's intended to be done? +"""]] diff --git a/doc/special_remotes/S3/comment_33_36788d742259b3b078e02d7b2a251ce5._comment b/doc/special_remotes/S3/comment_33_36788d742259b3b078e02d7b2a251ce5._comment new file mode 100644 index 0000000000..2c47361014 --- /dev/null +++ b/doc/special_remotes/S3/comment_33_36788d742259b3b078e02d7b2a251ce5._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="cnjr2" + avatar="http://cdn.libravatar.org/avatar/f7e9654cc967c8815947b19829bfd746" + subject="comment 33" + date="2020-02-12T10:37:24Z" + content=""" +First of all thanks for `git-annex`! I have recently discovered it and was so pleased to see how it solves problems that I had not anticipated! + +Related to [comment 26](http://git-annex.branchable.com/special_remotes/S3/#comment-e2000bfcb5d4c2d1a017f52c1c02a1ba) above, I want to create a bucket in Frankfurt. It seems like the aws library used by git-annex is from [aristidb/aws](https://github.com/aristidb/aws) where the topic of `V4` signing was raised in [this issue](https://github.com/aristidb/aws/issues/167) ([comment](https://github.com/aristidb/aws/issues/167#issuecomment-258864661) by you joey?). + +Are there workarounds here? I can only use Frankfurt. + +Thanks again! + + +"""]] diff --git a/doc/todo/provide_machine_readable___40__--json__63____41___version_of_initremote_--whatelse.mdwn b/doc/todo/provide_machine_readable___40__--json__63____41___version_of_initremote_--whatelse.mdwn new file mode 100644 index 0000000000..8e32ba0d48 --- /dev/null +++ b/doc/todo/provide_machine_readable___40__--json__63____41___version_of_initremote_--whatelse.mdwn @@ -0,0 +1,59 @@ +
+ATM it is a formatted text (click to expand) + +```shell +$> git annex initremote myrsync type=rsync --whatelse +shellescape + avoid usual shell escaping (not recommended) + (yes or no) +rsyncurl + (required) url or hostname:/directory for rsync to use +chunk + size of chunks (eg, 1MiB) +encryption + how to encrypt data stored in the special remote + (hybrid or none or pubkey or shared or sharedpubkey) +embedcreds + embed credentials into git repository + (yes or no) +mac + how to encrypt filenames used on the remote + (HMACSHA1 or HMACSHA224 or HMACSHA256 or HMACSHA384 or HMACSHA512) +keyid + gpg key id +keyid+ + add additional gpg key +keyid- + remove gpg key +exporttree + export trees of files to this remote + (yes or no) +importtree + import trees of files from this remote + (yes or no) +``` +
+ +which would make it necessary to establish a possibly fragile parsing by any tool which would like to programmatically obtain/use/expose those options. + +It would be great if there was a way to trigger such listing be output in more friendly for machines form? e.g. a json dictionary alike + +```json +{ + "rsyncurl": { + "required": True, + "description": "url or hostname:/directory for rsync to use" + }, + "shellescape": { + "description": "avoid usual shell escaping (not recommended)", + "choices": ["yes", "no"] + }, + ... +} +``` + +Looking at the [protocol](https://git-annex.branchable.com/design/external_special_remote_protocol/) I see no indication of "required" or "choices" to be actually explicitly provided by the remote fields, so I guess just supposed to be included in the text, so may be given current state of things, aforementioned dictionary would be simply ```{"NAME": "DESCRIPTION"}```, which someone makes this proposed TODO less valuable. + + +[[!meta author=yoh]] +[[!tag projects/datalad]] diff --git a/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type/comment_2_96fae68472f7f9ef720c27149556f7a0._comment b/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type/comment_2_96fae68472f7f9ef720c27149556f7a0._comment new file mode 100644 index 0000000000..47fd2773dc --- /dev/null +++ b/doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type/comment_2_96fae68472f7f9ef720c27149556f7a0._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="comment 2" + date="2020-02-14T17:20:27Z" + content=""" +Sweet, thanks for the feature! + +I wondered, why `--whatelse` which has no semantic relation to \"parameter\" or \"config\", and not something like `--help-params`, `--help-configs`, or `--listconfigs` to relate it to either PARAMS (as --help reports configs) or LISTCONFIGS protocol command? +"""]] diff --git a/doc/todo/sync_--branches__to_sync_only_specified_branches___40__e.g._git-annex__41__/comment_5_8f19d38c815dd3051004301a15657cf6._comment b/doc/todo/sync_--branches__to_sync_only_specified_branches___40__e.g._git-annex__41__/comment_5_8f19d38c815dd3051004301a15657cf6._comment new file mode 100644 index 0000000000..47e2061fd4 --- /dev/null +++ b/doc/todo/sync_--branches__to_sync_only_specified_branches___40__e.g._git-annex__41__/comment_5_8f19d38c815dd3051004301a15657cf6._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="Dan" + avatar="http://cdn.libravatar.org/avatar/986de9e060699ae70ff7c31342393adc" + subject="Still wanted (update with example)" + date="2020-02-13T20:08:45Z" + content=""" +I see this page recently was edited (when todo's were tagged) and so I wanted to chime in that this is still a feature I'm looking for, and I have a much less hypothetical use case for it. + +I'm a PhD student working on a research project where I supervise several undergraduates. We have a git repository that manages all of our code, and I let git-annex manage the large datafiles (also in the same repository) on which we run our code. The main repository is hosted on GitHub, and my students have read-only access to it. They've each made forks to which they have write access. We use a special remote that we all have write-access to, with wanted set to standard and group set to archive, so that it gets all of the content and distributes it as needed (the data is massive so git-annex is vital here since the student laptops can't realistically download it all at any one time). + +They use pull requests to the main GitHub repository to integrate their code changes, but we need a way to get the content of the git-annex branches in their forks (which are pushed to from their local repos) into the git-annex branch in the main GitHub repository. The natural solution seems for me (who has read/write access to the main repo and the fork) to do this, essentially pulling in git-annex branches from their forks to my local repo, and then pushing it to the main repo on GitHub. It'd also be nice if I can then push this back to all of their forks, too. I can do this manually, but I think I'd need to actually check out the git-annex branch (or stuff it in another worktree) and then do lots of work manually (or automate it in a script). + +First I tried `git annex sync --no-commit --no-push --no-pull` which (somewhat to my surprise) *did* pull the git-annex branches from their forks into my local repo, but didn't push `git-annex` back anywhere, and it neither pushed nor pulled `master`. So this was a good start, but I wanted to also push *only* the git-annex branch to the main repo (and ideally to their forks, too). So then I (foolishly) started dropping flags, and ended up in inadvertently pulling their work-in-progress `master` branches into the mainline and pushing this super-merged thing back to all of them. I was able to do some reseting and quick force-pushes before anyone noticed, but I should've known better :) + +Throughout this process I'm trying to teach them how to use git-annex (it's pretty clearly the right tool for the job :) but need to be really careful with what `git annex sync` commands I encourage them to run since I don't want the, + +I'd love it if there was like a `--git-annex-branch-only` option that I could pass that would then do all the pushing/pulling goodness of `git annex sync` but *without* touching `master` (or whatever branch happens to be checked out). I could then teach the students to always use this flag to avoid actually introducing changes to their master branch (they're still learning git, too, so they'd have a hard time recovering from this). Even better if this was configurable, and something I could stick in the `git-annex-config` options so that when they clone the repo this setting would propagate to them along with the git-annex branch. + +Is something like this in the pipeline? Also, is there a simpler workaround I can do for now that doesn't involve tons of (manual) merges and pushes? + +Thanks so much for such an excellent tool; if we didn't have this, we'd essentially just give up on version control for our scientific data, which would be a real bummer. +"""]]