Merge branch 'master' into forget

This commit is contained in:
Joey Hess 2013-08-29 13:23:02 -04:00
commit 336d5ec349
15 changed files with 574 additions and 13 deletions

View file

@ -19,7 +19,7 @@ import Utility.Base64
- the UUID of the repo that will be pushing it, and possibly with other
- information.
-
- Pushing to branches on the remote that have out uuid in them is ugly,
- Pushing to branches on the remote that have our uuid in them is ugly,
- but it reserves those branches for pushing by us, and so our pushes will
- never conflict with other pushes.
-

View file

@ -0,0 +1,379 @@
### Please describe the problem.
I try to share some files between a windows machine and a linux machine in direct mode, with a remote repository between the two. I can't seem to reproduce the problem in indirect mode...
I add a file on the windows machine then copy it to the remote repository. I then get it on the linux machine and edit it there. If I `git annex sync` and copy the file's data to the remote repository, a `git annex sync` on windows will fail like this:
[[!format sh """
$ git annex sync
commit
ok
pull origin
remote: Counting objects: 41, done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 31 (delta 9), reused 0 (delta 0)
Unpacking objects: 100% (31/31), done.
From ssh://192.168.1.29:/home/raz/work/test
b23cb44..9073cc3 git-annex -> origin/git-annex
7d4f0e2..87438fc master -> origin/master
57a661c..b33b025 synced/git-annex -> origin/synced/git-annex
7d4f0e2..87438fc synced/master -> origin/synced/master
ok
(merging origin/git-annex origin/synced/git-annex into git-annex...)
(Recording state in git...)
push origin
Counting objects: 17, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 565 bytes | 0 bytes/s, done.
Total 5 (delta 2), reused 0 (delta 0)
To ssh://192.168.1.29:/home/raz/work/test.git
b33b025..dcfaf23 git-annex -> synced/git-annex
! [rejected] master -> synced/master (non-fast-forward)
error: failed to push some refs to 'ssh://192.168.1.29:/home/raz/work/test.git'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. Check out this branch and merge the remote changes
hint: (e.g. 'git pull') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
failed
git-annex: sync: 1 failed
"""]]
A `git log` tells me the master branch is not properly merged so I assume the pull failed silently (I paste the log in the transcript).
I can then try to manually fix the windows working copy and sometimes it kind of work, but sometimes I have a hard time recovering and I am better off cloning a new version and moving data's around...
### What steps will reproduce the problem?
Create a repository on a windows.
Add a file.
`git clone --bare` the repository and put it on a remote machine.
Clone the remote repository to a linux machine.
Get the file's data.
Edit the file on linux.
Commit it and copy the data's to the remote machine.
`git clone sync` on the windows machine --> Error.
### What version of git-annex are you using? On what operating system?
On Windows:
[[!format sh """
git-annex version: 4.20130827-g4f18612
build flags: Pairing Testsuite S3 WebDAV DNS
local repository version: 4
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 2
"""]]
On Linux
[[!format sh """
git-annex version: 4.20130815
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP
local repository version: 4
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2
"""]]
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
### On Windows
$ mkdir test_windows
$ cd test_windows/
$ git init
Initialized empty Git repository in c:/Users/raz/test_windows/.git/
$ git annex init windows
init windows
Detected a crippled filesystem.
Enabling direct mode.
Detected a filesystem without fifo support.
Disabling ssh connection caching.
ok
(Recording state in git...)
$ echo "test 1" > test.txt
$ git annex add test.txt
add test.txt (checksum...) ok
(Recording state in git...)
$ git annex sync
commit
ok
git-annex: no branch is checked out
$ cd ..
$ git clone --bare test_windows/ test.git
Cloning into bare repository 'test.git'...
done.
$ scp -r test.git 192.168.1.29:/home/raz/work/
config 100% 183 0.2KB/s 00:00
description 100% 73 0.1KB/s 00:00
HEAD 100% 23 0.0KB/s 00:00
applypatch-msg.sample 100% 452 0.4KB/s 00:00
commit-msg.sample 100% 896 0.9KB/s 00:00
post-commit.sample 100% 160 0.2KB/s 00:00
post-receive.sample 100% 552 0.5KB/s 00:00
post-update.sample 100% 189 0.2KB/s 00:00
pre-applypatch.sample 100% 398 0.4KB/s 00:00
pre-commit.sample 100% 1704 1.7KB/s 00:00
pre-push.sample 100% 1348 1.3KB/s 00:00
pre-rebase.sample 100% 4951 4.8KB/s 00:00
prepare-commit-msg.sample 100% 1239 1.2KB/s 00:00
update.sample 100% 3611 3.5KB/s 00:00
exclude 100% 240 0.2KB/s 00:00
825dc642cb6eb9a060e54bf8d69288fbee4904 100% 15 0.0KB/s 00:00
a31b7a55380d79248658d3fffff6d2dc41a726 100% 116 0.1KB/s 00:00
dcabbf728abe62b6e2bcc06b8306eb3aa9a497 100% 176 0.2KB/s 00:00
332ecbfe923879df51a7a3f9bb86ebdfb64273 100% 45 0.0KB/s 00:00
4f0e20d38d6dc757340a8c569270b5a857fc67 100% 160 0.2KB/s 00:00
62956bcf45c63b24a637b22a742db0d9458248 100% 86 0.1KB/s 00:00
7ec519eb1b257a85ae8f9373d0835dea9ddc04 100% 82 0.1KB/s 00:00
1d1559a050b4a2df2e653b1e7ac15bdf7d56b1 100% 51 0.1KB/s 00:00
b74dbedfac4d6a0a90580cff2088a5d61f6675 100% 130 0.1KB/s 00:00
0ca042f439ef997f04a25a449aa1f539b93cf4 100% 53 0.1KB/s 00:00
bbce5e51e5a90ffd26900de3546fc4f9704181 100% 151 0.2KB/s 00:00
18bd66316298fcc5c06af9865c30a2fe4d2476 100% 176 0.2KB/s 00:00
edf8b5cc207e16ca0173785a2ba569c5d4208c 100% 71 0.1KB/s 00:00
packed-refs 100% 160 0.2KB/s 00:00
$ cd test_windows/
$ git remote add origin ssh://192.168.1.29:/home/raz/work/test.git
$ git annex copy --to origin
copy test.txt (checking origin...) (to origin...)
test.txt
7 100% 0.00kB/s 0:00:00 (xfer#1, to-check=0/1)
sent 79 bytes received 31 bytes 220.00 bytes/sec
total size is 7 speedup is 0.06
ok
(Recording state in git...)
$ git annex sync
commit
ok
pull origin
remote: Counting objects: 11, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 6 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.
From ssh://192.168.1.29:/home/raz/work/test
* [new branch] git-annex -> origin/git-annex
* [new branch] master -> origin/master
ok
(merging origin/git-annex into git-annex...)
(Recording state in git...)
push origin
Counting objects: 18, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (10/10), 874 bytes | 0 bytes/s, done.
Total 10 (delta 4), reused 0 (delta 0)
To ssh://192.168.1.29:/home/raz/work/test.git
* [new branch] git-annex -> synced/git-annex
* [new branch] master -> synced/master
ok
### On Linux
$ git clone test.git test_linux
Cloning into 'test_linux'...
done.
$ cd test_linux
$ git annex init linux
init linux ok
(Recording state in git...)
$ git annex sync
(merging origin/git-annex origin/synced/git-annex into git-annex...)
(Recording state in git...)
commit
ok
pull origin
ok
push origin
Counting objects: 11, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 775 bytes | 0 bytes/s, done.
Total 8 (delta 3), reused 0 (delta 0)
To /home/raz/work/test.git
0b4f175..41fba1d git-annex -> synced/git-annex
ok
$ git annex get
get test.txt (from origin...) ok
(Recording state in git...)
$ git annex direct
commit
# On branch master
nothing to commit, working directory clean
ok
direct test.txt ok
direct ok
$ git annex sync
commit
ok
pull origin
ok
push origin
Counting objects: 9, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 492 bytes | 0 bytes/s, done.
Total 5 (delta 1), reused 0 (delta 0)
To /home/raz/work/test.git
41fba1d..57a661c git-annex -> synced/git-annex
ok
$ cat test.txt
test 1
$ echo "test 2" > test.txt
$ git annex sync
add test.txt (checksum...) ok
(Recording state in git...)
commit
(Recording state in git...)
ok
pull origin
ok
push origin
Counting objects: 22, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (16/16), 1.40 KiB | 0 bytes/s, done.
Total 16 (delta 4), reused 0 (delta 0)
To /home/raz/work/test.git
57a661c..4883cad git-annex -> synced/git-annex
7d4f0e2..87438fc master -> synced/master
ok
$ git annex copy --to origin
copy test.txt (to origin...) ok
(Recording state in git...)
$ git annex sync
commit
ok
pull origin
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (5/5), done.
From /home/raz/work/test
b23cb44..9073cc3 git-annex -> origin/git-annex
ok
(merging origin/git-annex into git-annex...)
(Recording state in git...)
push origin
Counting objects: 18, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (10/10), 924 bytes | 0 bytes/s, done.
Total 10 (delta 3), reused 0 (delta 0)
To /home/raz/work/test.git
4883cad..b33b025 git-annex -> synced/git-annex
ok
$
### On Windows
$ git annex sync
commit
ok
pull origin
remote: Counting objects: 41, done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 31 (delta 9), reused 0 (delta 0)
Unpacking objects: 100% (31/31), done.
From ssh://192.168.1.29:/home/raz/work/test
b23cb44..9073cc3 git-annex -> origin/git-annex
7d4f0e2..87438fc master -> origin/master
57a661c..b33b025 synced/git-annex -> origin/synced/git-annex
7d4f0e2..87438fc synced/master -> origin/synced/master
ok
(merging origin/git-annex origin/synced/git-annex into git-annex...)
(Recording state in git...)
push origin
Counting objects: 17, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 565 bytes | 0 bytes/s, done.
Total 5 (delta 2), reused 0 (delta 0)
To ssh://192.168.1.29:/home/raz/work/test.git
b33b025..dcfaf23 git-annex -> synced/git-annex
! [rejected] master -> synced/master (non-fast-forward)
error: failed to push some refs to 'ssh://192.168.1.29:/home/raz/work/test.git'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. Check out this branch and merge the remote changes
hint: (e.g. 'git pull') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
failed
git-annex: sync: 1 failed
$ git log --graph --oneline --decorate --all
* 87438fc (origin/synced/master, origin/master) git-annex automatic sync
* 7d4f0e2 (HEAD, synced/master, master) git-annex automatic sync
*-. dcfaf23 (origin/synced/git-annex, git-annex) merging origin/git-annex orig
|\ \
| | * b33b025 merging origin/git-annex into git-annex
| | |\
| | |/
| |/|
| * | 9073cc3 (origin/git-annex) update
| | * 73518bc update
| | * 4883cad update
| | * 1b447f5 update
| |/
|/|
* | 57a661c update
| |
| \
*-. \ 41fba1d merging origin/git-annex origin/synced/git-annex into git-annex
|\ \ \
| | |/
| |/|
| | * 0b4f175 merging origin/git-annex into git-annex
| | |\
| | |/
| |/|
| * | b23cb44 update
| | * b5755a2 update
| |/
| * 6adcabb update
| * df18bd6 update
| * dcbbce5 branch created
* 0d138eb update
* fe6bccc branch created
$
# End of transcript or log.
"""]]

View file

@ -0,0 +1,14 @@
[[!comment format=mdwn
username="http://edheil.wordpress.com/"
ip="173.162.44.162"
subject="comment 2"
date="2013-08-28T13:54:00Z"
content="""
On a stock Nexus 7 running the latest OS (4.3), this is still the case as of release 4.20130827.
Falling back to hardcoded app location; cannot find expected files in /data/app-lib
git annex webapp
u0_a18@grouper:/sdcard/git-annex.home $ git annex webapp
CANNOT LINK EXECUTABLE: git-annex invalid R_ARM_COPY relocation against DT_SYMBOLIC shared library libc.so (built with -Bsymbolic?)
1|u0_a18@grouper:/sdcard/git-annex.home $
"""]]

View file

@ -1,10 +1,11 @@
Work on the git-annex assistant is [crowdfunded](https://campaign.joeyh.name/).
I'll be blogging about my progress here on a semi-daily basis.
This blog was updated on a semi-daily basis by Joey during the year of work
concentrating on the git-annex assistant that was funded by his kickstarter
campaign.
[[!sidebar content="""
[[!calendar type="month" pages="page(design/assistant/blog/*)"]]
[[!calendar type="month" month="-1" pages="page(design/assistant/blog/*)"]]
[[!calendar type="month" month="-2" pages="page(design/assistant/blog/*)"]]
"""]]
Post-kickstarter work will instead appear on the [[devblog]]. However,
this page's RSS feed will continue to work, so you don't have to migrate
your RSS reader.
[[!inline pages="page(design/assistant/blog/*)" show=0]]
[[!inline pages="page(devblog/*)" show=-1 feedshow=7]]
[[!inline pages="page(design/assistant/blog/*)" show=0 feeds=no]]

10
doc/devblog.mdwn Normal file
View file

@ -0,0 +1,10 @@
Work on git-annex is [crowdfunded](https://campaign.joeyh.name/).
Joey blogs about his progress here on a semi-daily basis.
[[!sidebar content="""
[[!calendar type="month" pages="page(devblog/*)"]]
[[!calendar type="month" month="-1" pages="page(devblog/*)"]]
[[!calendar type="month" month="-2" pages="page(devblog/*)"]]
"""]]
[[!inline pages="page(devblog/*)" show=0]]

View file

@ -0,0 +1,80 @@
Yesterday I spent making a release, and shopping for a new laptop, since
this one is dying. (Soon I'll be able to compile git-annex fast-ish! Yay!)
And thinking about [[todo/wishlist:_dropping_git-annex_history]].
Today, I added the `git annex forget` command. It's currently been lightly
tested, seems to work, and is living in the `forget` branch until I gain
confidence with it. It should be perfectly safe to use, even if it's buggy,
because you can use `git reflog git-annex` to pull out and revert to an old
version of your git-annex branch. So if you're been wanting this feature,
please beta test!
----
I actually implemented something more generic than just forgetting git
history. There's now a whole mechanism for git-annex doing distributed
transitions of whatever sort is needed.
There were several subtleties involved in distributed transitions:
First is how to tell when a given transition has already been done on a
branch. At first I was thinking that the transition log should include the
sha of the first commit on the old branch that got rewritten. However, that
would mean that after a single transition had been done, every git-annex
branch merge would need to look up the first commit of the current branch,
to see if it's done the transition yet. That's slow! Instead, transitions
are logged with a timestamp, and as long as a branch contains a transition
with the same timestamp, it's been done.
A really tricky problem is what to do if the local repository has
transitioned, but a remote has not, and changes keep being made to the
remote. What it does so far is incorporate the changes from the remote into
the index, and re-run the transition code over the whole thing to yeild a
single new commit. This might not be very efficient (once I write the more
full-featured transition code), but it lets the local repo keep up with
what's going on in the remote, without directly merging with it (which
would revert the transition). And once the remote repository has its
git-annex upgraded to one that knows about transitions, it will finish up
the transition on its side automatically, and the two branches will once
again merge.
Related to the previous problem, we don't want to keep trying to merge
from a remote branch when it's not yet transitioned. So a blacklist is
used, of untransitioned commits that have already been integrated.
One really subtle thing is that when the user does a transition more
complicated than `git annex forget`, like the `git annex forget --dead`
that I need to implement to forget dead remotes, they're not just telling
git-annex to forget whatever dead remotes it knows right now. They're
actually telling git-annex to perform the transition one time on every
existing clone of the repository, at some point in the future. Repositories
with unfinished transitions could hang around for years, and at some future
point when git-annex runs in the repository again, it would merge in the
current state of the world, and re-do the transition. So you might tell it
to forget dead remotes today, and then the very repository you ran that in
later becomes dead, and a long-slumbering repo wakes up and forgets about
the repo that started the whole process! I hope users don't find this
massively confusing, but that's how the implementation works right now.
----
I think I have at least two more days of work to do to finish up this
feature.
* I still need to add some extra features like forgetting about dead remotes,
and forgetting about keys that are no longer present on any remote.
* After `git annex forget`, `git annex sync`
will fail to push the synced/annex branch to remotes, since the branch
is no longer a fast-forward of the old one. I will probably fix this by
making `git annex sync` do a fallback push of a unique branch in this case,
like the assistant already does. Although I may need to adjust that code
to handle this case, too..
* For some reason the automatic transitioning code triggers
a "(recovery from race)" commit. This is certianly a bug somewhere,
because you can't have a race with only 1 participant.
----
Today's work was sponsored by Richard Hartmann.

View file

@ -0,0 +1,5 @@
I've started a new page for my devblog, since I'm not focusing extensively
on the assistant and so keeping the blog [[here|design/assistant/blog]]
increasingly felt wrong. Also, my new year of
[crowdfunded development](https://campaign.joeyh.name)
formally starts in September, so a new blog seemed good.

View file

@ -2,6 +2,6 @@
[[!inline pages="news/* and !*/Discussion" archive=yes show=2 feeds=no]]
### [[Dev blog|design/assistant/blog]]
### [[devblog|devblog]]
[[!inline pages="design/assistant/blog/* and !*/Discussion" archive=yes show=5 feeds=no]]
[[!inline pages="devblog/* and !*/Discussion" archive=yes show=5 feeds=no]]

View file

@ -0,0 +1,15 @@
Hello,
I have a Samsung ARM Chromebook running Ubuntu with crouton (chroot in chromeos). I want to use git-annex, and add the chromebook as a second "device" to my already set up git-annex repo on my main pc. That repo is one folder set up with the assistant, and it has one encrypted ssh(rsync) remote.
I've read that the webapp is not available on ARM, and I've also found a topic with a simple script to replicate git-annex-assistant steps: http://git-annex.branchable.com/forum/Running_assistant_steps_manually/. So, I've used the `cabal` steps to install git-annex manually, which is working.
I want to use git-annex in a dropbox-like fashion, folder and file sync between these two devices (and maybe later on another encrypted ssh remote). However I'm in need of some guidance regarding the command line usage and such.
My questions are:
- How do I "add" the chromebook repo to the existing git-annex setup?
- The machines are not on at the same time. One is my main workstation and the chromebook is my "mobile friend". The encrypted ssh remote is on a VPS, that is always on.
- How do I configure the syncing between the main machine, the chromebook and the encrypted ssh remote on the vps? If possible, bi-directional.
If it is possible, both explanation and command line examples would help a lot.

View file

@ -0,0 +1,3 @@
I have a backups repository with a few files in it, that at one point had some huge filesets erroneously added to it. As a result, even though there are only 23,334 annexed files, the number of non-dangling Git objects in the repository comes to 593,584.
Normally I would use `git filter-branch` to clear out the deadwood in situations like this, since it is a completely private repository. What I'm wondering is, is any such thing possible with git-annex, or is the best option just to start over, copy all the files into the new repository, and then `git-add` them all?

View file

@ -0,0 +1,20 @@
[[!comment format=mdwn
username="http://a-or-b.myopenid.com/"
ip="203.45.2.230"
subject="comment 3"
date="2013-08-28T01:32:38Z"
content="""
One thing I do slightly differently is put comments in my ```feeds``` file so I know what feed is what...
# NPR - Wait Wait... Don't Tell Me!
http://www.npr.org/rss/podcast.php?id=35
# NPR - Car Talk
http://www.npr.org/rss/podcast.php?id=510208
Then I use
grep -v '^#' feeds| xargs git-annex importfeed --relaxed
I reckon that makes maintenance easier.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmTNrhkVQ26GBLaLD5-zNuEiR8syTj4mI8"
nickname="Juan"
subject="This is an awesome feature"
date="2013-08-28T13:40:23Z"
content="""
Thanks. I have quite a lot of papers in PDF formats. Now I'm saving space, have them controlled, synchronized with many devices and found more than 200 duplicates.
Is there a way to donate to the project? You really deserve it.
Thanks.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="4.153.8.7"
subject="comment 9"
date="2013-08-28T20:25:20Z"
content="""
@Juan the best thing to do is tell people about git-annex, help them use it, and file bug reports. Just generally be part of the git-annex community.
(If you really want to donate to me, <http://campaign.joeyh.name/> is still open.)
"""]]

View file

@ -0,0 +1,14 @@
It would be nice if a couple of additional environment variables to be set for hook uses.
In particular:
GIT_ANNEX_DIRECT=`git config annex.direct`
and
GIT_TOP_LEVEL=`git rev-parse --show-toplevel`
I've made some changes to flickrannex to allow the sub-directories above the uploaded image to be added as tags. This change has been merged into trunk: [[https://github.com/TobiasTheViking/flickrannex]]
What I needed was both the environment variables mentioned above. One is set as part of the annex-hook and the other I guestimate from the file path. If it was set in git-annex it would be much cleaner (and accurate). So...I think this info would be useful for other hook.

View file

@ -6,8 +6,8 @@
content="""
If starting commit id _and_ commit id from when history is being dropped are documented, you could potentially drop more data.
* Don't have any commits in common? Full merge?
* Only share the starting ids? Reduce local history as much as possible and then merge.
* Don't have any commits in common? Full merge.
* Only share the starting ids? Reduce local history as much as possible, and then merge.
* Share both starting id and have the last id somewhere in history? Take history from last id up to current, reduce that, and merge.
-- RichiH