Merge branch 'master' of ssh://git-annex.branchable.com
This commit is contained in:
Joey Hess
commit
a9f078da43
9 changed files with 258 additions and 9 deletions
|
|
@ -0,0 +1,73 @@
|
|||
### Please describe the problem.
|
||||
|
||||
I have syncing set up between two peers via tor. Sometimes when I change a (non-annexed) file in one peer, the other peer receives the change, but then within a second or two adds a commit which removes the entire file, and within another second another commit which adds it back.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
This is not 100% reproducible. I haven't figured out any pattern yet as to when it happens or doesn't happen.
|
||||
|
||||
1. Set up a fresh repo
|
||||
1. `echo '* annex.largefiles=nothing' >.gitattributes` and commit
|
||||
1. Add another (text) file and commit
|
||||
1. `scp` the repo to another machine
|
||||
1. Run `enable-tor`, `remotedaemon`, and `p2p --pair` on both
|
||||
1. Modify the text file
|
||||
1. Observe that the remote peer ends up with three new commits on `master` rather than one; from oldest to newest:
|
||||
1. The commit received via tor from the source side (4bd9c99 in the below example)
|
||||
2. A local commit removing the text file (1ef9061)
|
||||
3. A local commit adding it back with identical contents (e662a33)
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
7.20190912-g05bc37910 on both openSUSE nodes (sender is Leap 15.0, receiver is Tumbleweed)
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
# Relevant output from .git/annex/daemon.log on the sending side
|
||||
[2019-09-29 23:44:13.781640446] Committer: Committing changes to git
|
||||
(recording state in git...)
|
||||
[2019-09-29 23:44:15.105628185] Committer: Committing changes to git
|
||||
|
||||
Updating 4bd9c99..1ef9061
|
||||
Fast-forward
|
||||
TODO.org | 10919 --------------------------------------------------------------------------------------------------------------------------------------------
|
||||
1 file changed, 10919 deletions(-)
|
||||
delete mode 100644 TODO.org
|
||||
[2019-09-29 23:44:17.668876311] Committer: Committing changes to git
|
||||
(recording state in git...)
|
||||
|
||||
Updating 1ef9061..e662a33
|
||||
Fast-forward
|
||||
TODO.org | 10919 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
1 file changed, 10919 insertions(+)
|
||||
create mode 100644 TODO.org
|
||||
|
||||
|
||||
# Relevant output from .git/annex/daemon.log on the receiving side
|
||||
[2019-09-29 23:44:14.293079441] RemoteControl: Syncing with peer1
|
||||
From tor-annex::rpxkm55i6gxlirqohcjup4g4orlgzyval2n2yymah36ekipjcppmrvid.onion:42294
|
||||
28aa4b9..4bd9c99 master -> peer1/master
|
||||
28aa4b9..4bd9c99 synced/master -> peer1/synced/master
|
||||
|
||||
Updating 28aa4b9..4bd9c99
|
||||
Fast-forward
|
||||
TODO.org | 2 +-
|
||||
1 file changed, 1 insertion(+), 1 deletion(-)
|
||||
[2019-09-29 23:44:16.054093704] Committer: Committing changes to git
|
||||
(recording state in git...)
|
||||
[2019-09-29 23:44:16.169742975] Pusher: Syncing with peer1
|
||||
[2019-09-29 23:44:17.169984217] Committer: Committing changes to git
|
||||
(recording state in git...)
|
||||
To tor-annex::rpxkm55i6gxlirqohcjup4g4orlgzyval2n2yymah36ekipjcppmrvid.onion:42294
|
||||
4bd9c99..1ef9061 master -> synced/master
|
||||
remote: error: refusing to update checked out branch: refs/heads/master
|
||||
To tor-annex::rpxkm55i6gxlirqohcjup4g4orlgzyval2n2yymah36ekipjcppmrvid.onion:42294
|
||||
! [remote rejected] master -> master (branch is currently checked out)
|
||||
error: failed to push some refs to 'tor-annex::rpxkm55i6gxlirqohcjup4g4orlgzyval2n2yymah36ekipjcppmrvid.onion:42294'
|
||||
[2019-09-29 23:44:23.623525612] Pusher: Syncing with peer1
|
||||
To tor-annex::rpxkm55i6gxlirqohcjup4g4orlgzyval2n2yymah36ekipjcppmrvid.onion:42294
|
||||
1ef9061..e662a33 master -> synced/master
|
||||
|
||||
# End of transcript or log.
|
||||
"""]]
|
||||
80
doc/bugs/enable-tor_fails_with_quoting_issue.mdwn
Normal file
80
doc/bugs/enable-tor_fails_with_quoting_issue.mdwn
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
### Please describe the problem.
|
||||
|
||||
git annex enable-tor fails with an error in the KDE su dialog window:
|
||||
|
||||
Cannot execute command ' ''\''/home/adam/software/scm/git-annex.static/git-annex'\'' '\''enable-tor'\'' '\''1000'\''''.
|
||||
|
||||
The console shows the following:
|
||||
|
||||
```
|
||||
enable-tor
|
||||
You may be prompted for a password
|
||||
org.kde.kdesud: priority set to 50
|
||||
org.kde.kdesud: Scheduler set to 0
|
||||
|
||||
git-annex: Failed to run as root: /home/adam/software/scm/git-annex.static/git-annex enable-tor 1000
|
||||
failed
|
||||
git-annex: enable-tor: 1 failed
|
||||
```
|
||||
|
||||
Interestingly this happens on my openSUSE Tumbleweed system running XFCE4 (*not* KDE), but not on the peer I'm trying to pair with, which is an older openSUSE Leap 15.0 box. On the latter, for some reason that uses `gksu`, even though both systems are running XFCE4:
|
||||
|
||||
```
|
||||
enable-tor
|
||||
You may be prompted for a password
|
||||
gksu-run: gksu/'|home|adam|software|scm|git-annex.static|git-annex' 'enable-tor' '1000'/9140-0-pacific_TIME0
|
||||
gksu-run: e680f4b50a49c297709613029db91b7a
|
||||
|
||||
Tor hidden service is configured. Checking connection to it. This may take a few minutes.
|
||||
|
||||
Tor hidden service is working.
|
||||
ok
|
||||
```
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
Simply run `git-annex enable-tor`
|
||||
|
||||
### Workaround
|
||||
|
||||
I found that a valid workaround is to `su` to `root`, `cd` to the same repo, and run `git annex enable-tor 1000`.
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
The identical statically compiled 7.20190912-g05bc37910 is running on both the failing Tumbleweed system and the succeeding Leap 15.0 system.
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
Here is the relevant fragment of the process tree:
|
||||
|
||||
```
|
||||
| |-zsh,9829
|
||||
| | |-git,21554 annex enable-tor
|
||||
| | | `-git-annex,21555 --library-path /home/adam/software/scm/git-annex.static//usr/lib/x86_64-linux-gnu/gconv:/home/adam/software/scm/git-annex.static//usr/lib/x86_64-linux-gnu/audit>
|
||||
| | | |-kdesu,21584 '/home/adam/software/scm/git-annex.static/git-annex' 'enable-tor' '1000'
|
||||
| | | | |-{kdesu},21585
|
||||
| | | | |-{kdesu},21586
|
||||
| | | | |-{kdesu},21588
|
||||
| | | | |-{kdesu},21589
|
||||
| | | | |-{kdesu},21591
|
||||
| | | | `-{kdesu},21595
|
||||
| | | |-{git-annex},21579
|
||||
| | | |-{git-annex},21580
|
||||
| | | |-{git-annex},21581
|
||||
| | | `-{git-annex},21583
|
||||
```
|
||||
|
||||
You can clearly see that the arguments to `kdesu` have extra quotes, which I'm guessing is part of the problem:
|
||||
|
||||
```
|
||||
$ xxd /proc/21584/cmdline
|
||||
00000000: 6b64 6573 7500 272f 686f 6d65 2f61 6461 kdesu.'/home/ada
|
||||
00000010: 6d2f 736f 6674 7761 7265 2f73 636d 2f67 m/software/scm/g
|
||||
00000020: 6974 2d61 6e6e 6578 2e73 7461 7469 632f it-annex.static/
|
||||
00000030: 6769 742d 616e 6e65 7827 2027 656e 6162 git-annex' 'enab
|
||||
00000040: 6c65 2d74 6f72 2720 2731 3030 3027 00 le-tor' '1000'.
|
||||
```
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Yes, lots and lots for many years :-)
|
||||
24
doc/bugs/remotedaemon_--stop_doesn__39__t_work.mdwn
Normal file
24
doc/bugs/remotedaemon_--stop_doesn__39__t_work.mdwn
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
### Please describe the problem.
|
||||
|
||||
`git annex remotedaemon --help` claims that `--stop` is a valid option for stopping the remote daemon, but it results in:
|
||||
|
||||
```
|
||||
git annex remotedaemon --stop
|
||||
git-annex: --stop not implemented for remotedaemon
|
||||
CallStack (from HasCallStack):
|
||||
error, called at ./Command/RemoteDaemon.hs:24:32 in main:Command.RemoteDaemon
|
||||
```
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
`git annex remotedaemon --stop`
|
||||
|
||||
Just `kill`ing it seems to do the job, however :-)
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
7.20190912-g05bc37910 on openSUSE Tumbleweed and Leap 15.0.
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Yes for many years, just finally getting round to exploring the more advanced features :-)
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e"
|
||||
nickname="branchable"
|
||||
avatar="http://cdn.libravatar.org/avatar/ae41dba34ee6000056f00793c695be75"
|
||||
subject="Commits could be rate-limited too"
|
||||
date="2019-09-29T23:03:47Z"
|
||||
content="""
|
||||
See [todo/wishlist: disable_automatic_commits](/todo/wishlist__58___disable_automatic_commits) for another kind of rate limiting, which would also help reduce I/O between remotes in some cases simply by reducing \"churn\" within any given repo.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e"
|
||||
nickname="branchable"
|
||||
avatar="http://cdn.libravatar.org/avatar/ae41dba34ee6000056f00793c695be75"
|
||||
subject="Can conversion to/from annexed be made easier?"
|
||||
date="2019-09-29T22:04:32Z"
|
||||
content="""
|
||||
IMHO it's somewhat user-unfriendly and error-prone to have to remember a sequence of three commands to convert an unannexed file to annexed, or vice-versa. So it would be nice if there were git-annex commands to do this in one go.
|
||||
|
||||
In fact, I would expect `git annex add` to handle adding to the annex, and `git annex unannex` to do the opposite. Are there good reasons why they should not be made to do that? Even if there are, aren't those subcommands already doing very similar things? In which case maybe the solution would be to add extra option to each to enable this behaviour? For example `--force` or something like that in order to tell it to ignore `annex.largefiles` (although for `add` that is already used to allow adding ignored files, so we probably shouldn't conflate the two cases under one option).
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e"
|
||||
nickname="branchable"
|
||||
avatar="http://cdn.libravatar.org/avatar/ae41dba34ee6000056f00793c695be75"
|
||||
subject="Follow-up thought"
|
||||
date="2019-09-29T22:08:29Z"
|
||||
content="""
|
||||
Afterthought on the immediately preceding comment - if the intention of `unannex` is to reverse `add`, then IMHO `unadd` or `undo-add` seem like more logical choices. Renaming to one of these would free up `unannex` for the conversion use case. But I appreciate that this is potentially confusing for users already used to `unannex`'s current behaviour, so I expect the idea will be rejected.
|
||||
"""]]
|
||||
|
|
@ -10,12 +10,21 @@ To use this, you need to get Tor installed and running. See
|
|||
|
||||
sudo apt-get install tor
|
||||
|
||||
You also need to install [Magic Wormhole](https://github.com/warner/magic-wormhole).
|
||||
You also need to install [Magic Wormhole](https://github.com/warner/magic-wormhole) -
|
||||
here are [the installation instructions](https://magic-wormhole.readthedocs.io/en/latest/welcome.html#installation).
|
||||
|
||||
sudo apt-get install magic-wormhole
|
||||
*Important:*
|
||||
|
||||
Important: You need git-annex version 6.20180705. Older versions of git-annex
|
||||
unfortunately had a bug that prevents this process from working correctly.
|
||||
* At the time of writing, you need to install Magic Wormhole under Python 2,
|
||||
because [Tor support is only available under python2.7](https://magic-wormhole.readthedocs.io/en/latest/tor.html).
|
||||
|
||||
* The installation process must make a `wormhole` executable available
|
||||
somewhere on your `$PATH`. Some distributions may only install executables
|
||||
which reference the Python version, e.g. `wormhole-2.7`, in which case you
|
||||
will need to manually create a symlink (and maybe file a bug with your distribution).
|
||||
|
||||
* You need git-annex version 6.20180705. Older versions of git-annex
|
||||
unfortunately had a bug that prevents this process from working correctly.
|
||||
|
||||
## pairing two repositories
|
||||
|
||||
|
|
@ -26,7 +35,7 @@ to accomplish this.
|
|||
|
||||
(The instructions below use the command line. If you or your friend would
|
||||
rather avoid using the command line, follow the
|
||||
[[webapp_walktrough|assistant/share_with_a_friend_walkthrough]]. It's fine
|
||||
[[webapp_walkthrough|assistant/share_with_a_friend_walkthrough]]. It's fine
|
||||
for one person to use the command line and the other to use the webapp.)
|
||||
|
||||
In each git-annex repository, run these commands:
|
||||
|
|
@ -162,8 +171,8 @@ how it works.
|
|||
git-annex's Tor support uses onion address as the address of a git remote.
|
||||
You can `git pull`, push, etc with those onion addresses:
|
||||
|
||||
git pull tor-annnex::eeaytkuhaupbarfi.onion:4412
|
||||
git remote add peer1 tor-annnex::eeaytkuhaupbarfi.onion:4412
|
||||
git pull tor-annex::eeaytkuhaupbarfi.onion:4412
|
||||
git remote add peer1 tor-annex::eeaytkuhaupbarfi.onion:4412
|
||||
|
||||
Onion addresses are semi-public. When you add a remote, they appear in your
|
||||
`.git/config` file. For security, there's a second level of authentication
|
||||
|
|
@ -171,10 +180,10 @@ that git-annex uses to make sure that only people you want to can access
|
|||
your repository over Tor. That takes the form of a long string of numbers
|
||||
and letters, like "7f53c5b65b8957ef626fd461ceaae8056e3dbc459ae715e4".
|
||||
|
||||
The addresses generated by `git annex peer --gen-addresses`
|
||||
The addresses generated by `git annex p2p --gen-addresses`
|
||||
combine the onion address with the authentication data.
|
||||
|
||||
When you run `git annex peer --link`, it sets up a git remote using
|
||||
When you run `git annex p2p --link`, it sets up a git remote using
|
||||
the onion address, and it stashes the authentication data away in a file in
|
||||
`.git/annex/creds/`
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e"
|
||||
nickname="branchable"
|
||||
avatar="http://cdn.libravatar.org/avatar/ae41dba34ee6000056f00793c695be75"
|
||||
subject="Issue on openSUSE with Tor's requirement for Python 2.7 "
|
||||
date="2019-09-29T16:41:15Z"
|
||||
content="""
|
||||
I have updated the instructions to note two extra dependencies, the combination of which break this setup process out of the box on openSUSE. I've filed [a bug with openSUSE](https://bugzilla.opensuse.org/show_bug.cgi?id=1152403) to improve their python-magic-wormhole package. Hopefully I can work with the package maintainers to make this a non-issue.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
[[!comment format=mdwn
|
||||
username="branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e"
|
||||
nickname="branchable"
|
||||
avatar="http://cdn.libravatar.org/avatar/ae41dba34ee6000056f00793c695be75"
|
||||
subject=".gitattributes could solve this"
|
||||
date="2019-09-29T22:59:48Z"
|
||||
content="""
|
||||
Revisiting this topic 3.5 years later ...
|
||||
|
||||
The assistant will still commit a change as soon as it notices it. This obviously has the advantage of synchronising changes to peers quicker, but it also has downsides:
|
||||
|
||||
- It pollutes the git history with every single (saved) edit.
|
||||
- Consequently it bloats the git object store.
|
||||
|
||||
I find this undesirable even when editing my TODO list, but it could be particularly problematic with large files, e.g. using `ffmpeg` to transcode a video between formats would presumably capture lots of intermediate states of the unfinished transcoding process. Similarly for rendering from a video editor.
|
||||
|
||||
So it would be helpful if there was a configurable option to determine how often changes get committed. Ideally this would be configurable via `.gitattributes`, e.g.
|
||||
|
||||
```
|
||||
* annex.autocommit=anything
|
||||
*.webm annex.autocommit=(mtimebefore=10mins)
|
||||
```
|
||||
|
||||
would autocommit most stuff immediately, but would only autocommit webm files if they haven't changed within the last 10 minutes.
|
||||
"""]]
|
||||
Loading…
Add table
Reference in a new issue