Commit graph

795 commits

Author SHA1 Message Date
Joey Hess
46e3319995
assistant: Deal with upcoming git's refusal to merge unrelated histories by default
git 2.8.1 (or perhaps 2.9.0) is going to prevent git merge from merging in
unrelated branches. Since the webapp's pairing etc features often combine
together repositories with unrelated histories, work around this behavior
change by setting GIT_MERGE_ALLOW_UNRELATED_HISTORIES when the assistant
merges.

Note though that this is not done for git annex sync's merges, so
it will follow git's default or configured behavior.
2016-04-22 14:26:44 -04:00
Joey Hess
56dee9af10
fix build with ghc 7.6.3 2016-04-08 16:09:00 -04:00
Joey Hess
251405eca2
avoid withWorkTreeRelated affecting annex symlink calculation 2016-04-08 14:24:00 -04:00
Joey Hess
887ef93a7f
run out of tree merge with --no-ff
This is how direct mode does it too, and somehow, for reasons that
currently escape me, this makes git merge not care if it's run with an
empty work tree.
2016-04-06 18:40:28 -04:00
Joey Hess
eb9ac8d6d7
sync: Show output of git commit.
Rationalle: User might have hook scripts whose output they want to see.
Also, git commit output may tell the user they forgot to add a file.
The output is not too ugly when there's nothing to commit.
2016-04-05 16:22:21 -04:00
Joey Hess
c3e0859846
Upgrading a direct mode repository to v6 has changed to enter an adjusted unlocked branch.
This makes the direct mode to v6 upgrade able to be performed in one clone
of a repository without affecting other clones, which can continue using v5
and direct mode.
2016-04-04 13:17:24 -04:00
Joey Hess
11935c4d6f
fix parsing of commit with no parents 2016-03-31 17:12:01 -04:00
Joey Hess
a585731935
add reflog messages 2016-03-31 12:27:48 -04:00
Joey Hess
70e8d6860e
Merge branch 'master' into adjustedbranch 2016-03-29 11:07:40 -04:00
Joey Hess
2d234de781
Sped up git-annex merge by using git hash-object --batch.
This does mean that it has to write out temp files containing updated
objects for the merge. So may use more disk space, and disk IO, but that
should generally win out over needing to launch N separate
git hash-object processes.
2016-03-14 16:23:22 -04:00
Joey Hess
6c023e14ef
grafting new items into existing tree 2016-03-11 19:29:43 -04:00
Joey Hess
ad04550055
refactor 2016-03-11 16:45:40 -04:00
Joey Hess
f3b9c48a09
fixme 2016-03-11 16:37:31 -04:00
Joey Hess
ba1ef156a2
fix deletion of files in adjustTree 2016-03-11 16:30:06 -04:00
Joey Hess
b9184f69a7
improve propigation of commits from adjusted branches
Only reverse adjust the changes in the commit, which means that adjustments
do not need to be generally cleanly reversable.

For example, an adjustment can unlock all locked files, but does not need
to worry about files that were originally unlocked when reversing, because
it will only ever be run on files that have been changed. So, it's ok
if it locks all files when reversed, or even leaves all files as-is when
reversed.
2016-03-11 16:05:06 -04:00
Joey Hess
3c4ad3eeca
indent 2016-03-11 14:46:54 -04:00
Joey Hess
ec8eba18ad
fix warning 2016-03-11 14:33:38 -04:00
Joey Hess
fed8fcb99f
allow adding new items via adjustTree 2016-03-11 14:08:06 -04:00
Joey Hess
8d124beba8
add commitDiff, and clean up partial function 2016-03-11 13:15:49 -04:00
Joey Hess
fbf4d89e82
extract commit parent(s) 2016-03-11 12:47:14 -04:00
Joey Hess
cf24e9b892
working toward adjusted commit propigation 2016-03-03 16:19:09 -04:00
Joey Hess
730b249477
replicate git's message about an existing lock file 2016-03-03 13:06:39 -04:00
Joey Hess
de4bd97c9d
support for git-style lock files, on unix and windows 2016-03-03 12:49:54 -04:00
Joey Hess
7c20bf6e7a
make sync aware of adjusted branches
So, it will pull and push the original branch, not the adjusted one.

And, for merging, it will use updateAdjustedBranch (not implemented yet).

Note that remaining uses of Git.Branch.current need to be checked too;
for things that should act on the original branch, and not the adjusted
branch.
2016-02-29 15:23:08 -04:00
Joey Hess
955ab3a973
fix android build 2016-02-29 11:43:22 -04:00
Joey Hess
facc50d965
forgot to use sfile 2016-02-26 16:12:40 -04:00
Joey Hess
3b74dc8be8
add fromBlobType 2016-02-25 15:34:22 -04:00
Joey Hess
7b2496508f
factor out commitTree 2016-02-25 15:33:50 -04:00
Joey Hess
1f91d1d0b7
add catCommit, with commit object parser 2016-02-25 15:14:47 -04:00
Joey Hess
be2e9427ad
refactor 2016-02-25 13:46:31 -04:00
Joey Hess
804aeca5d2
parse strictly
This reduces memory use, because it avoids thunks that buffer parts of the
ls-tree output that are not needed.
2016-02-23 23:08:41 -04:00
Joey Hess
e5dd91b189
better encapsulation 2016-02-23 22:22:22 -04:00
Joey Hess
4ea36b8c63
few strictness improvemnets 2016-02-23 22:03:47 -04:00
Joey Hess
85b05a29df
refactor 2016-02-23 21:56:08 -04:00
Joey Hess
e08bebf0eb
add adjustTree (low-level) interface that avoids buffering much in memory
Using getTree and recordTree in my big repo takes 594 mb ram.
Using adjustTree takes 73 mb.
2016-02-23 21:35:16 -04:00
Joey Hess
9519af25f3
remove support for network older than 2.4
debian stable has 2.4
2016-02-23 20:35:32 -04:00
Joey Hess
123f823ef7
no streaming
extractTree has to parse the whole input list in order to generate a tree,
so convert interface to non-streaming.

Some quick memory benchmarks in a repo with 60k files
don't look too bad despite not streaming.

To stream, without building up a whole tree object, one way would
be a new interface:

adjustTree :: MonadIO m :: (TreeItem -> m (Maybe TreeItem)) -> Ref -> Repo -> m Sha

This would only need to buffer tree objects from the current one down
to the root, in order to update trees when a TreeItem is changed.

But, while it supports changing items in the tree, and removing items,
it does not support adding new items, or moving items from one directory to
another.
2016-02-23 20:25:31 -04:00
Joey Hess
e266a6ec78
use getSha 2016-02-23 18:30:11 -04:00
Joey Hess
fc072699b7
minor improvements 2016-02-23 17:21:42 -04:00
Joey Hess
ae76cfde7d
add mktree interface 2016-02-23 16:36:38 -04:00
Joey Hess
a49d5d30fe
fix handling of unspecified attributes (particularly for annex.largefiles) 2016-02-05 18:41:23 -04:00
Joey Hess
d37fe6a547
annex.largefiles can be configured in .gitattributes too
This is particulary useful for v6 repositories, since the .gitattributes
configuration will apply in all clones of the repository.
2016-02-02 15:18:17 -04:00
Joey Hess
b52cf5697b
immediate queue flushing when annex.queuesize=1
Previously, it only flushed when the queue got larger than 1.

Also, make the queue auto-flush when items are added, rather than needing
to be flushed as a separate step. This simplifies the code and make it more
efficient too, as it avoids needing to read the queue out of the state to
check if it should be flushed.
2016-01-13 14:55:01 -04:00
Joey Hess
3320870bad
optimise
03cb2c8ece put a cat-file into the fast
bloomfilter generation path. Instead, add another bloom filter which diffs
from the work tree to the index.

Also, pull the sha of the changed object out of the diffs, and cat that
object directly, rather than indirecting through the filename.

Finally, removed some hacks that are unncessary thanks to the worktree to
index diff.
2016-01-06 20:38:02 -04:00
Joey Hess
aa4f353e5d
clarify absPathFrom
The repo path is typically relative, not absolute, so
providing it to absPathFrom doesn't yield an absolute path.
This is not a bug, just unclear documentation.

Indeed, there seem to be no reason to simplifyPath here, which absPathFrom
does, so instead just combine the repo path and the TopFilePath.

Also, removed an export of the TopFilePath constructor; asTopFilePath
is provided to construct one as-is.
2016-01-05 17:33:48 -04:00
Joey Hess
b3d60ca285
use TopFilePath for associated files
Fixes several bugs with updates of pointer files. When eg, running
git annex drop --from localremote
it was updating the pointer file in the local repository, not the remote.
Also, fixes drop ../foo when run in a subdir, and probably lots of other
problems. Test suite drops from ~30 to 11 failures now.

TopFilePath is used to force thinking about what the filepath is relative
to.

The data stored in the sqlite db is still just a plain string, and
TopFilePath is a newtype, so there's no overhead involved in using it in
DataBase.Keys.
2016-01-05 17:22:19 -04:00
Joey Hess
ec28151722
improve data type 2016-01-01 15:56:24 -04:00
Joey Hess
f7256842cc
wait for git lstree to exit 2016-01-01 15:51:29 -04:00
Joey Hess
70fee8208c
remove old TODO 2016-01-01 15:43:13 -04:00
Joey Hess
b0626230b7
fix use of hifalutin terminology 2015-11-16 14:37:31 -04:00
Joey Hess
53db9d0b5c
work around git check-ignore --batch bad exit status bug, and bring back import -J 2015-11-06 15:39:51 -04:00
Joey Hess
31472161e4
merge git command queue when joining with concurrent thread 2015-11-05 18:21:48 -04:00
Joey Hess
ef5496b8de
Catch up with current git behavior when both repo and repo.git exist; it seems it now prefers repo in this case, although historically it may have preferred repo.git. 2015-10-26 15:35:55 -04:00
Joey Hess
b0e5c09408
fix various build warnings, mostly on Windows
And some when S3 is disabled
2015-10-13 13:24:44 -04:00
Joey Hess
f2b6ebd502 status: Show added but not yet committed files.
Seems easy, but git ls-files can't list the right subset of files.
So, I wrote a whole new parser for git status output, and converted the
status command to use that.

There are a few other small behavior changes. The order changed. Unlocked
files show as T. In indirect mode, deleted files were not shown before, and
that's fixed. Regular files checked directly into git and modified
were not shown before, and are now.
2015-09-22 17:32:28 -04:00
Joey Hess
6158036e23 Switched to using git for Windows, rather than msysgit.
Using msysgit with git-annex is no longer supported.

At the same time, I'm updating the rsync.exe in my downloads repository
with the one from msys2.

Note that rsync is currently still being ldded and installed in Git/cmd/
like the other cygwin programs. The ldd fails and this failure is ignored.
It would be better to special case it to go in Git/usr/bin/, so that the
user can't run rsync in a dos prompt window, which doesn't work, as it needs
additional libs. However, as far as git-annex running rsync running ssh,
it works ok in this location.

Removed the ssh.cmd and ssh-keygen.cmd; these are not needed with git for
windows. Keeping them would let ssh be run manually from a dos prompt
window, but that's not really a goal.
2015-09-10 19:16:30 -04:00
Joey Hess
06dac3bfed avoid nul-truncation
This might be a little slower, but it's safer, in the event that a
union-merged file contains a NUL.

AFAIK, no files in the git-annex branch do.
2015-08-11 18:46:10 -04:00
Joey Hess
24800b1bf1 Only look at reflogs for relevant branches, not for git-annex branches
This speeds it up quite a bit.. May still be too slow in large repos.
2015-07-07 17:36:30 -04:00
Joey Hess
b11d2f5a8a unused: --used-refspec can now be configured to look at refs in the reflog. This provides a way to not consider old versions of files to be unused after they have reached a specified age, when the old refs in the reflog expire.
May be slow.
2015-07-07 17:13:50 -04:00
Joey Hess
dbd41159e0 Support git's undocumented core.sharedRepository=2 value, which is equivilant to "world". 2015-07-06 15:33:44 -04:00
Joey Hess
f7dc20595e refactor ls-tree params
All in one place to avoid bugs like 174da80ddc
2015-07-06 14:21:43 -04:00
Joey Hess
eb33569f9d remove Params constructor from Utility.SafeCommand
This removes a bit of complexity, and should make things faster
(avoids tokenizing Params string), and probably involve less garbage
collection.

In a few places, it was useful to use Params to avoid needing a list,
but that is easily avoided.

Problems noticed while doing this conversion:

	* Some uses of Params "oneword" which was entirely unnecessary
	  overhead.
	* A few places that built up a list of parameters with ++
	  and then used Params to split it!

Test suite passes.
2015-06-01 13:52:23 -04:00
Joey Hess
03667a162a couple of AMP warnings I missed before 2015-05-10 16:51:03 -04:00
Joey Hess
5c7cdbae46 more {-# OPTIONS_GHC -fno-warn-tabs #-} ... Forcing people who have what is merely a difference of opinion to you to do this is a bit of an asshole move. Just saying. 2015-05-10 16:38:49 -04:00
Joey Hess
c3dd133257 desc 2015-04-19 08:18:17 -04:00
Joey Hess
addc82dab7 removed all uses of undefined from code base
It's a code smell, can lead to hard to diagnose error messages.
2015-04-19 00:38:29 -04:00
Joey Hess
d3d92abf95 spotted a few more places where diff-tree needed --
None of these are very likely at all to ever be ambiguous, since tree
refs almost never have symbolic names and the sha is very unlikely
to be in the work tree.. But, let's get it right!
2015-04-09 21:22:35 -04:00
Joey Hess
2879adc551 fix union merge to call diff-index with -- after the ref
Otherwise, if there's a file in the repo with a name matching the ref,
git could get confused and the merge not work.
2015-04-09 21:13:28 -04:00
Joey Hess
0f740fd198 This fixes a bug in the assistant introduced by the literal pathspec changes in version 5.20150406.
git-checkignore refuses to work if any pathspec options are set. Urgh.

I audited the rest of git, and no other commands used by git-annex have
such limitations. Indeed, AFAICS, *all* other commands support
--literal-pathspecs. So, worked around this where git-checkignore is
called.
2015-04-09 13:37:06 -04:00
Joey Hess
15d45186cc use --literal-pathspecs globally, as a better way to avoid globbing
This might be overkill; I only know I need it in ls-files, but other git
commands can also do their own globbing, it turns out, and I am pretty sure
I never want them too when git-annex is using them as plumbing.

Test suite still passes and it looks ok.
2015-03-30 19:44:13 -04:00
Joey Hess
f933898cd0 workaround git ls-files bug in handling slash-escaped wildcards
There's no good solution for git-annex here; I can't escape or un-escape
and avoid breaking in some cases, so I've chosen the combo least likely
to result in breakage.

Git really needs to fix its behavior here.

The only other thing git-annex could do is treat this as a feature,
and don't try to escape at all. Ugh.
2015-03-30 19:00:04 -04:00
Joey Hess
f35d0bf4b2 Prevent git-ls-files from double-expanding wildcards when an unexpanded wildcard is passed to a git-annex command like add or find.
Note that previously, `git annex find *.jpg` would find eg, foo/bar.jpg.
That was never intended or documented behavior, so I'm going to change it.
But this is potentially a behavior change if someone discovered that
behavior and relied on it despite it being accidental. Oh well.. can't make
an omlette w/o breaking some eggs.
2015-03-27 16:45:50 -04:00
Joey Hess
cedca095b9 promote forum request to todo item so it is not lost 2015-03-27 16:38:51 -04:00
Joey Hess
3af4691978 Improve error message when --in @date is used and there is no reflog for the git-annex branch. 2015-03-26 11:15:15 -04:00
Joey Hess
a6db10d565 sync: Fix committing when in a direct mode repo that has no HEAD ref.
Seen for example, a newly checked out git submodule. In this case,
.git/HEAD is a raw sha, rather than the usual reference to a ref.

Removed currentSha in passing, since it was a more roundabout way of
doing what headSha does, and headSha is more robust.
2015-03-04 15:25:35 -04:00
Joey Hess
e322826e33 Submodules are now supported by git-annex!
Seems to work, but still experimental until it's been tested more.

When repositories are on filesystems not supporting symlinks, the .git dir
symlink trick cannot be used. Since we're going to be in direct mode
anyway, the .git dir symlink is not strictly needed.

However, I have not fixed the code that creates new annex symlinks to
handle this case -- the committed symlinks will be wrong.

git annex sync happens to currently fail in a submodule using direct mode,
because there's no HEAD ref. That also needs to be dealt with to get
this fully working in crippled filesystems.

Leaving http://github.com/datalad/datalad/issues/44 open until these issues
are dealt with.
2015-03-02 16:43:44 -04:00
Joey Hess
5169999b07 add -q to git symbolic-ref call
Avoids a warning message from git when HEAD doesn't exist. Which it won't
when eg, git-annex is used in a submodule just cloned with
git clone --recursive. In this case, a specific ref is checked out and
there's no HEAD yet.

The code already returned Nothing in this case, so no behavior change other
than not showing the warning. And git-annex operates fine in this
situation.
2015-03-02 15:56:37 -04:00
Joey Hess
52e40970c8 avoid unncessary IO 2015-02-12 15:33:44 -04:00
Joey Hess
afc5153157 update my email address and homepage url 2015-01-21 12:50:09 -04:00
Joey Hess
6035f94666 Windows: Fix running of the pre-commit-annex hook. 2015-01-20 14:48:16 -04:00
Joey Hess
f4de021a54 convert parentDir to be based on takeDirectory, but fixed for trailing / 2015-01-09 14:26:52 -04:00
Joey Hess
3bab5dfb1d revert parentDir change
Reverts 965e106f24

Unfortunately, this caused breakage on Windows, and possibly elsewhere,
because parentDir and takeDirectory do not behave the same when there is a
trailing directory separator.
2015-01-09 13:11:56 -04:00
Joey Hess
676ef32547 Merge branch 'master' into relativepaths 2015-01-06 21:41:25 -04:00
Joey Hess
adefcf189a Bugfix: A file named HEAD in the work tree could confuse some git commands run by git-annex. 2015-01-06 21:41:21 -04:00
Joey Hess
858d776352 Merge branch 'master' into relativepaths
Conflicts:
	Locations.hs
	debian/changelog
2015-01-06 19:00:01 -04:00
Joey Hess
965e106f24 made parentDir return a Maybe FilePath; removed most uses of it
parentDir is less safe than takeDirectory, especially when working
with relative FilePaths. It's really only useful in loops that
want to terminate at /

This commit was sponsored by Audric SCHILTKNECHT.
2015-01-06 18:55:56 -04:00
Joey Hess
d44b28437d git-hash-object needs absolute files (git bug)
A relative path to a file makes it fail. I am pretty sure this is a git
bug; workaround it.
2015-01-06 17:33:29 -04:00
Joey Hess
82f667e7f2 git repo path may be relative, so don't assume absolute any more
Fixes 6 test failures.
2015-01-06 16:32:44 -04:00
Joey Hess
cd865c3b8f Switch to using relative paths to the git repository.
This allows the git repository to be moved while git-annex is running in
it, with fewer problems.

On Windows, this avoids some of the problems with the absurdly small
MAX_PATH of 260 bytes. In particular, git-annex repositories should
work in deeper/longer directory structures than before. See
http://git-annex.branchable.com/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/

There are several possible ways this change could break git-annex:

1. If it changes its working directory while it's running, that would
   be Bad News. Good news everyone! git-annex never does so. It would also
   break thread safety, so all such things were stomped out long ago.

2. parentDir "." -> "" which is not a valid path. I had to fix one
   instace of this, and I should probably wipe all calls to parentDir out
   of the git-annex code base; it was never a good idea.

3. Things like relPathDirToFile require absolute input paths,
   and code assumes that the git repo path is absolute and passes it to it
   as-is. In the case of relPathDirToFile, I converted it to not make
   this assumption.

Currently, the test suite has 16 failures.
2015-01-06 16:19:41 -04:00
Joey Hess
4d786ebe4a Check git version at runtime, rather than assuming it will be the same as the git version used at build time when running git-checkattr and git-branch remove.
It's ok to probe every time for git-branch remove because that's
run quite rarely. For git-checkattr, it's run only once, when
starting the --batch mode, and so again the overhead is pretty minimal.

This leaves 2 places where the build version is still used.
git merge might be interactive or fail if one skews, and --no-gpg-sign
might not be pased, or might be passed to a git that doesn't understand it
if the other skews. It seems a little expensive to check the git version
each time these are used.

This doesn't seem likely to cause many problems, at least compared with
check-attr hanging on skew.
2015-01-05 15:54:52 -04:00
Joey Hess
db27ad26bf split out DiffTreeItem
This makes github-backup happier when it reuses this library.
2014-12-22 15:32:51 -04:00
Joey Hess
c64ede23cd Use wget -q --show-progress for less verbose wget output, when built with wget 1.16. 2014-12-16 14:04:40 -04:00
Joey Hess
13260ccc3a undo command
This commit was sponsored by Andrew Cant.
2014-11-14 14:41:07 -04:00
Joey Hess
c5ca0dc543 simplify 2014-11-12 15:57:38 -04:00
Joey Hess
864086a956 proxy: for all your direct mode repository munging needs
This allows bypassing the direct mode guard in a safe way to do all sorts
of things including git revert, git mv, git checkout ...

This commit was sponsored by the WikiMedia Foundation.
2014-11-12 15:51:46 -04:00
Joey Hess
bf2b029c49 comment typo 2014-11-10 15:38:31 -04:00
Joey Hess
adc5ca70a8 pre-commit: Block partial commit of unlocked annexed file, since that left a typechange staged in index
I had hoped that the git devs could change git's handling of partial
commits to not use a false index file, but seems not.

So, this relies on some git internals to detect that case. The test suite
has a test case added to catch it if changes to git break it.

This commit was sponsored by Paul Tagliamonte.
2014-11-10 15:36:24 -04:00
Joey Hess
20a497b181 move remote removal into separate module
This allows using Git.Remote w/o needing to have Git.BuildVersion, which
requires configure. It will simplify github-backup when these libraries are
used there.
2014-10-27 11:28:58 -04:00
Joey Hess
1e59df083d Use haskell setenv library to clean up several ugly workarounds for inability to manipulate the environment on windows.
Didn't know that this library existed!

This includes making git-annex not re-exec itself on start on windows, and
making the test suite on Windows run tests without forking.
2014-10-15 20:33:52 -04:00
Joey Hess
c6e9125c61 repair: Prevent auto gc from happening when fetching from a remote. 2014-10-12 14:27:46 -04:00
Joey Hess
9fd95d9025 indent with tabs not spaces
Found these with:
git grep "^  " $(find -type  f -name \*.hs) |grep -v ':  where'

Unfortunately there is some inline hamlet that cannot use tabs for
indentation.

Also, Assistant/WebApp/Bootstrap3.hs is a copy of a module and so I'm
leaving it as-is.
2014-10-09 15:09:26 -04:00
Joey Hess
7b50b3c057 fix some mixed space+tab indentation
This fixes all instances of " \t" in the code base. Most common case
seems to be after a "where" line; probably vim copied the two space layout
of that line.

Done as a background task while listening to episode 2 of the Type Theory
podcast.
2014-10-09 15:09:11 -04:00
Joey Hess
11f111bf1a Fix parsing of ipv6 address in git remote address when it was not formatted as an url. 2014-09-10 14:17:02 -04:00
Joey Hess
b874f84086 New annex.hardlink setting. Closes: #758593
* New annex.hardlink setting. Closes: #758593
* init: Automatically detect when a repository was cloned with --shared,
  and set annex.hardlink=true, as well as marking the repository as
  untrusted.

Had to reorganize Logs.Trust a bit to avoid a cycle between it and
Annex.Init.
2014-09-05 13:44:09 -04:00
Joey Hess
4405650828 Fix handing of autocorrection when running outside a git repository.
Old behavior was to take the first fuzzy match. Now, it checks the globa
git config, and runs the normal fuzzy handling, including failing to run a
semi-random command by default.
2014-08-23 16:51:33 -07:00
Joey Hess
c784ef4586 unify exception handling into Utility.Exception
Removed old extensible-exceptions, only needed for very old ghc.

Made webdav use Utility.Exception, to work after some changes in DAV's
exception handling.

Removed Annex.Exception. Mostly this was trivial, but note that
tryAnnex is replaced with tryNonAsync and catchAnnex replaced with
catchNonAsync. In theory that could be a behavior change, since the former
caught all exceptions, and the latter don't catch async exceptions.

However, in practice, nothing in the Annex monad uses async exceptions.
Grepping for throwTo and killThread only find stuff in the assistant,
which does not seem related.

Command.Add.undo is changed to accept a SomeException, and things
that use it for rollback now catch non-async exceptions, rather than
only IOExceptions.
2014-08-07 22:03:29 -04:00
Joey Hess
000dd42ac4 improve repair of bad branches
The repair code assumed that if fsck found no broken objects, after
removing bad objects and possibly pulling replacements from remote, all was
well.. but this is not really true. Removing bad objects could leave some
branches broken. fsck doesn't report any missing objects in this case,
and its messages about broken branches are ignored by the fsck output
parser.

To deal with this, added a separate scan of all refs to find broken ones
and remove them when --forced. This will also let anyone who ran into this
bug run repair again to fix up the incomplete repair done before.

This commit was sponsored by Aaron Whitehouse.
2014-07-21 18:42:58 -04:00
Joey Hess
ec5ed2af9d Set gcrypt-publish-participants when setting up a gcrypt repository, to avoid unncessary passphrase prompts.
This is a security/usability tradeoff. To avoid exposing the gpg key ids
who can decrypt the repository, users can unset
gcrypt-publish-participants.

The gcrypt-publish-participants option is available in my fork of
git-remote-gcrypt.

This commit was sponsored by Christopher Kernahan.
2014-07-15 17:33:14 -04:00
Joey Hess
eef8e8c51a Fix git version that supported --no-gpg-sign.
This is weird, git describe said the commit landed in 1.8.5, but 1.9.3 does
not have it on OSX. Assume 2.0.0.
2014-07-08 12:46:15 -04:00
Joey Hess
1c1f463c3a
avoid using --no-gpg-sign with old versions of git
and refactor some
2014-07-04 13:49:12 -04:00
Joey Hess
fc67925fd7
reorg
avoid Git.Command needing Utility.Batch which needs async

For github-backup etc
2014-07-04 12:18:49 -04:00
Joey Hess
d41849bc23
support commit.gpgsign
Support users who have set commit.gpgsign, by disabling gpg signatures for
git-annex branch commits and commits made by the assistant.

The thinking here is that a user sets commit.gpgsign intending the commits
that they manually initiate to be gpg signed. But not commits made in the
background, whether by a deamon or implicitly to the git-annex branch.
gpg signing those would be at best a waste of CPU and at worst would fail,
or flood the user with gpg passphrase prompts, or put their signature on
changes they did not directly do.

See Debian bug #753720.

Also makes all commits done by git-annex go through a few central control
points, to make such changes easier in future.

Also disables commit.gpgsign in the test suite.

This commit was sponsored by Antoine Boegli.
2014-07-04 11:53:51 -04:00
Joey Hess
986bf1d6f6 Fix bug in annex.queuesize calculation that caused much more queue flushing than necessary.
The bug caused the size of the queue to be miscalculted; it was doubled
each time an item was added. Commands run after approx 140 items rather
than the intended 10240!
2014-06-18 17:23:36 -04:00
Joey Hess
fbd5a67cba fix a test suite reversion on Windows
Forgot to pass gitEnv when running commands in the git queue on windows.
2014-06-12 18:37:12 -04:00
Joey Hess
a44fd2c019 export CreateProcess fields from Utility.Process
update code to avoid cwd and env redefinition warnings
2014-06-10 19:20:14 -04:00
Joey Hess
d6711800ad avoid bad commits after interrupted direct mode sync (or merge)
It was possible for a interrupted sync or merge in direct mode to
leave the work tree out of sync with the last recorded commit.
This would result in the next commit seeing files missing from the work
tree, and committing their removal.

Now, a direct mode merge happens not only in a throwaway work tree, but using
a temporary index file, and without any commits or index changes
being made until the real work tree has been updated. If the merge is
interrupted, the work tree may have some updated files, but worst case a
commit will redundantly commit changes that come from the merge.

This commit was sponsored by Tony Cantor.
2014-06-09 19:40:28 -04:00
Joey Hess
138d25518d Merge branch 'master' into remotecontrol
Conflicts:
	doc/devblog/day_152__more_ssh_connection_caching.mdwn
2014-04-14 13:38:35 -04:00
Joey Hess
e53a85743e
adjust to not use cpp in modules used by configure 2014-04-14 13:37:12 -04:00
Joey Hess
f67d5abc41 support gcrypt remotes (assuming them to be over ssh transport) 2014-04-08 16:16:46 -04:00
Joey Hess
43909723b3 added git-annex remotedaemon
So far, handling connecting to git-annex-shell notifychanges, and
pulling immediately when a change is pushed to a remote.

A little bit buggy (crashes after the first pull), but it already works!

This commit was sponsored by Mark Sheppard.
2014-04-06 19:10:23 -04:00
Joey Hess
1052eeface Windows: Fix some filename encoding bugs.
http://git-annex.branchable.com/bugs/Unicode_file_names_ignored_on_Windows/

Not a complete fix yet.
2014-03-19 15:57:56 -04:00
Joey Hess
67f09bca6d fully fix fsck memory use by iterative fscking
Not very well tested, but I'm sure it doesn't eg, loop forever.
2014-03-12 15:18:43 -04:00
Joey Hess
475bf70af6 read stdout and stderr concurrently
Avoids any buffering-related blocking.
2014-03-12 13:54:29 -04:00
Joey Hess
85d13b4302 better streaming when cleaning up corrupt objects
A repo with a lot of objects will now stream them through, rather than
buffering a list of them all in memory.
2014-03-10 16:36:18 -04:00
Joey Hess
0e0d396b27 Improve memory usage when git fsck finds a great many broken objects.
From 1.7 gb to 900 mb on 300 thousand unique reported shas.

When shas are not unique, this streams much better than before, so won't
buffer the full list before putting them into the Set and throwing away
dups. And when fsck output includes ignorable lines, especially
dangling object lines, they won't be buffered in memory at all.
2014-03-10 15:14:09 -04:00
Joey Hess
8496d8aa63
improved direct mode dir/file conflicted merge resultion, using tree grafting 2014-03-04 15:00:19 -04:00
Joey Hess
1192d98721 sync: Fix bug in direct mode that caused a file not checked into git to be deleted when merging with a remote that added a file by the same name. (Thanks, jkt) 2014-03-03 14:57:16 -04:00
Joey Hess
d0fce426c4 pre-commit-annex hook script to automatically extract metadata from lots of types of files
Using the extract(1) program to do the heavy lifting.

Decided to make git-annex run pre-commit-annex when committing. Since
git-annex pre-commit also runs it, it'll be run when git commit is run too,
via the pre-commit hook. This basically gives back the pre-commit hook
that git-annex took away. The implementation avoids repeatedly looking
for the hook script when the assistant is running and committing
repeatedly; only checks if the hook is available once.

To make the script simpler, made git-annex metadata -s field?=value
only set a field when it's not already got a value.

This commit was sponsored by bak.
2014-03-02 20:11:58 -04:00
Joey Hess
f8cfcd4e44 couple more warning fixes 2014-02-25 14:53:43 -04:00
Joey Hess
3f6e4b8c7c fix all remaining -Wall warnings on Windows 2014-02-25 14:48:50 -04:00
Joey Hess
46cc39f1a4 repair: Optimise unpacking of pack files, and avoid repeated error messages about corrupt pack files. 2014-02-24 19:36:58 -04:00
Joey Hess
4e0be2792b remove Read instance for Ref
Removed instance, got it all to build using fromRef. (With a few things
that really need to show something using a ref for debugging stubbed out.)

Then added back Read instance, and made Logs.View use it for serialization.
This changes the view log format.
2014-02-19 01:19:57 -04:00
Joey Hess
67fd06af76 add git annex view command
(And a vpop command, which is still a bit buggy.)

Still need to do vadd and vrm, though this also adds their documentation.

Currently not very happy with the view log data serialization. I had to
lose the TDFA regexps temporarily, so I can have Read/Show instances of
View. I expect the view log format will change in some incompatable way
later, probably adding last known refs for the parent branch to View
or something like that.

Anyway, it basically works, although it's a bit slow looking up the
metadata. The actual git branch construction is about as fast as it can be
using the current git plumbing.

This commit was sponsored by Peter Hogg.
2014-02-18 18:22:20 -04:00
Joey Hess
9633c67842 filter branches (incomplete)
Promosing work toward metadata driven filter branches. A few methods
to construct them are stubbed out; all the data types and pure code
seems good.

This commit was sponsored by Walter Somerville.
2014-02-16 17:39:54 -04:00
Joey Hess
61ecf76644 unbreak the build 2014-02-12 14:34:01 -04:00
Joey Hess
029a1c431a
remove windows --git-dir unix style path hack
This is no longer necessary, at least with msysgit 1.8.5.2.msysgit.0.
Its root cause may have been fixed by other recent git path fixes.
It was causing the webapp to fail to make repos on other drives.
2014-02-11 16:12:22 -04:00
Joey Hess
c95d0cf7a8 Windows: Fix handling of absolute unix-style git repository paths.
Note that on Windows a remote with a path like /home/foo/bar
is interpreted by git as being some screwy relative path (relative to what
exactly seems ill-defined -- it seemed relative to C:\Program Files\Git\ in
my tests!) So no attempt has been made to handle such a path sanely, just not
to crash when encountering it.

Note that "C:\\foo" </> "/home/foo/bar" yields /home/foo/bar even though
that is not absolute! I don't know what to make of all this,
except that I will be very happy when this crock of **** vanishes from
the face of the earth.
2014-02-08 15:39:04 -04:00
Joey Hess
92edee0b04 remove workaround
This was needed when absNormPath was not being used on Windows, since path
normalization includes removing ./
2014-02-08 14:47:57 -04:00
Joey Hess
a44e01c29c --in can now refer to files that were located in a repository at some past date. For example, --in="here@{yesterday}" 2014-02-06 12:43:56 -04:00
Joey Hess
ed7c61914c assistant: Run the periodic git gc in batch mode. 2014-01-22 17:11:41 -04:00
Joey Hess
78ead70ea4 repair: Check git version at run time. 2014-01-21 13:22:48 -04:00
Joey Hess
4e19e87921 repair: Fix bug in packed refs file exploding code that caused a .gitrefs directory to be created instead of .git/refs 2014-01-15 16:34:18 -04:00
Joey Hess
5e6e89f423 repair: Support old git versions from before git fsck --no-dangling was implemented. 2014-01-13 18:10:45 -04:00
Joey Hess
858eb26303 Avoid looping if long-running git cat-file or git hash-object crashes and keeps crashing when restarted. 2014-01-01 21:42:25 -04:00
Joey Hess
49aad120b9 Windows: Fix bug in direct mode merge code that could cause files in subdirectories to go missing. 2013-12-31 16:39:11 -04:00
Richard Hartmann
974fe009bf Another round of s/amoung/among/ 2013-12-19 12:30:53 -04:00
Joey Hess
c99d6a8151 assistant: Fix OSX-specific bug that caused the startup scan to try to follow symlinks to other directories, and add their contents to the annex. 2013-12-18 15:05:29 -04:00
Joey Hess
625076f9a5 status: Ignore new files that are gitignored. 2013-12-12 14:01:24 -04:00
Joey Hess
e6c4f550d8 repair: Remove damaged git-annex sync branches. 2013-12-10 16:17:49 -04:00
Joey Hess
b37323d857 update 2013-12-10 15:48:24 -04:00
Joey Hess
c0ce3269e9 accidentially committed wrong version of file 2013-12-10 15:45:22 -04:00
Joey Hess
ce045a51af Improve repair of git-annex index file.
Fixes a test case I received where a corrupted repo was repaired, but the
git-annex branch was not. The root of the problem was that the
MissingObject returned by the repair code was not necessarily a complete
set of all objects that might have been deleted during the repair.

So, stop trying to return that at all, and instead make the index file
checking code explicitly verify that each object the index uses is present.
2013-12-10 15:40:01 -04:00
Joey Hess
c717905d15 work around msysgit very strange behavior on ./ or .\ at start of path
Seems that verify_path() rejects such a path on Windows, but I cannot see
why. Git bug?
2013-12-04 23:49:18 -04:00
Joey Hess
4882a611e5 assistant: Batch jobs are now run with ionice and nocache, when those commands are available. 2013-12-01 14:53:15 -04:00
Joey Hess
03932212ec Avoid using git commit in direct mode, since in some situations it will read the full contents of files in the tree.
The assistant's commit code also always avoids git commit, for simplicity.
Indirect mode sync still does a git commit -a to catch unstaged changes.

Note that this means that direct mode sync no longer runs the pre-commit
hook or any other hooks git commit might call. The git annex pre-commit
hook action for direct mode is however explicitly run. (The assistant
already ran git commit with hooks disabled, so no change there.)
2013-12-01 13:59:45 -04:00
Joey Hess
6edac746f0 merge improved fsck types from git-repair and some associated changes 2013-11-30 14:29:11 -04:00
Joey Hess
0980f3dae6 Fix bug that broke switching between local repositories in the webapp when they use the new guarded direct mode.
git treats eg ~/annex as a bare git repository located in ~/.annex/.git
if ~/annex/.git/config has core.bare=true.
2013-11-22 23:27:15 -04:00
Joey Hess
d490bbb891 make runRepairOf run preRepair
This may be a little late, since a fsck has already been done,
but it can't hurt.
2013-11-21 20:13:55 -04:00
Joey Hess
7d682dd844 merge from git-repair 2013-11-21 20:07:44 -04:00
Joey Hess
ff2b0a9df6 merge from git-repair 2013-11-21 00:43:30 -04:00
Joey Hess
8217e97d88 merge from git-repair 2013-11-20 19:34:30 -04:00
Joey Hess
e80d935b53 merge from git-repair 2013-11-20 19:16:42 -04:00
Joey Hess
8a466247ed merge from git-repair 2013-11-20 18:45:22 -04:00
Joey Hess
7dbb702edd merge from git-repair 2013-11-20 18:31:00 -04:00
Joey Hess
ef34316c45 fix repair failure that occurred when index was corrupted, and other objects too
In this case, the index problem prevented fsck from finding the other
problems.
2013-11-19 17:16:33 -04:00
Joey Hess
b1ed98636b merge with git-repair 2013-11-19 17:08:57 -04:00
Joey Hess
b245aa40df moving git-repair to its own package 2013-11-18 13:24:55 -04:00
Joey Hess
eab4470440 better handling of missing index file 2013-11-13 14:39:26 -04:00
Joey Hess
13108b7196 assistant: Notice on startup when the index file is corrupt, and auto-repair. 2013-11-13 14:27:17 -04:00
Joey Hess
5e7e0c7dc0 repair: Handle case where index file is corrupt, but all objects are ok. 2013-11-13 13:41:02 -04:00
Joey Hess
958312885f webapp: Improve UI around remote that have no annex.uuid set, either because setup of them is incomplete, or because the remote git repository is not a git-annex repository.
Complicated by such repositories potentially being repos that should have
an annex.uuid, but it failed to be gotten, perhaps due to the past ssh repo
setup bugs. This is handled now by an Upgrade Repository button.
2013-11-07 18:02:00 -04:00
Joey Hess
59ecc804cd add new status command
This works for both direct and indirect mode.

It may need some performance tuning.

Note that unlike git status, it only shows the status of the work tree, not
the status of the index. So only one status letter, not two .. and since
files that have been added and not yet committed do not differ between the
work tree and the index, they are not shown. Might want to add display of
the index vs the last commit eventually.

This commit was sponsored by an unknown bitcoin contributor, whose
contribution as been going up lately! ;)
2013-11-07 14:07:25 -04:00
Joey Hess
3802f2f270 work around lack of receive.denyCurrentBranch in direct mode
Now that direct mode sets core.bare=true, git's normal prohibition about
pushing into the currently checked out branch doesn't work.

A simple fix for this would be an update hook which blocks the pushes..
but git hooks must be executable, and git-annex needs to be usable on eg,
FAT, which lacks x bits.

Instead, enabling direct mode switches the branch (eg master) to a special
purpose branch (eg annex/direct/master). This branch is not pushed when
syncing; instead any changes that git annex sync commits get written to
master, and it's pushed (along with synced/master) to the remote.

Note that initialization has been changed to always call setDirect,
even if it's just setDirect False for indirect mode. This is needed because
if the user has just cloned a direct mode repo, that nothing has synced
with before, it may have no master branch, and only a annex/direct/master.
Resulting in that branch being checked out locally too. Calling setDirect False
for indirect mode moves back out of this branch, to a new master branch,
and ensures that a manual "git push" doesn't push changes directly to
the annex/direct/master of the remote. (It's possible that the user
makes a commit w/o using git-annex and pushes it, but nothing I can do
about that really.)

This commit was sponsored by Jonathan Harrington.
2013-11-05 21:08:31 -04:00
Joey Hess
cf34e59c8c factor out update 2013-11-05 18:20:52 -04:00
Joey Hess
4510819215 v5 for direct mode, with automatic upgrade
This includes storing the current state of the HEAD ref, which git annex
sync is going to need, but does not make sync use it.
2013-11-05 17:05:03 -04:00
Joey Hess
04768e44b2 automatically set and unset core.bare when switching to/from direct mode 2013-11-05 15:41:24 -04:00
Joey Hess
0edd9ec03a refactored hook setup 2013-11-05 15:29:56 -04:00
Joey Hess
c2862d9585 pass -c option on to all git commands run
The -c option now not only modifies the git configuration seen by
git-annex, but it is passed along to every git command git-annex runs.

This was easy to plumb through because gitCommandLine is already used to
construct every git command line, to add --git-dir and --work-tree
2013-11-05 13:38:37 -04:00
Joey Hess
58db042033 map: Work when there are gcrypt remotes. 2013-11-04 14:14:44 -04:00
Joey Hess
7ed8e87a34 assistant: Support repairing git remotes that are locally accessible
(eg, on removable drives)

gcrypt remotes are not yet handled.

This commit was sponsored by Sören Brunk.
2013-10-27 15:38:59 -04:00
Joey Hess
0036139b33 wire git repair into webapp 2013-10-23 14:43:58 -04:00
Joey Hess
1ab2ad86c7 minor 2013-10-23 13:19:37 -04:00
Joey Hess
435ea52f3c repair command: add handling of git-annex branch and index 2013-10-23 13:00:45 -04:00
Joey Hess
d5eb85acf4 add repair command 2013-10-23 12:21:59 -04:00
Joey Hess
d345e5b52f add git fsck to cronner, and UI for repository repair (not yet wired up) 2013-10-22 16:02:52 -04:00
Joey Hess
44bb9a808f clean warnings 2013-10-22 14:52:17 -04:00
Joey Hess
ff3f654cbe make git fsck batch-capable 2013-10-22 14:49:41 -04:00
Joey Hess
3e61749d08 index file recovery 2013-10-22 12:58:04 -04:00
Joey Hess
2fb08acda5 add reflog 2013-10-21 16:41:46 -04:00
Joey Hess
18487c779f corrupt branch resetting (but not yet reflog walking) 2013-10-21 16:20:54 -04:00
Joey Hess
fcd91be6f0 implemented removal of corrupt tracking branches
Oh, git, you made this so hard. Not determining if a branch pointed to some
corrupt object, that was easy, but dealing with corrupt branches using git
plumbing is a PITA.
2013-10-21 15:28:06 -04:00
Joey Hess
6d8250c255 avoid redundant fsck when no changes are made 2013-10-20 19:42:17 -04:00
Joey Hess
4f871f89ba git-recover-repository 1/2 done 2013-10-20 17:50:51 -04:00
Joey Hess
f482de1b76 remove workaround for bug in git 1.8.4r0 2013-10-20 15:23:06 -04:00
Joey Hess
edbf177628 fix lsTreeFiles to use --full-tree
This makes it show the full tree, not just the current directory,
and enables --full-name, which yields TopFilePaths.
2013-10-18 15:50:26 -04:00
Joey Hess
c979e0ea62 fix 2013-10-17 19:51:16 -04:00
Joey Hess
c116383b5d fix 2013-10-17 19:49:44 -04:00
Joey Hess
81c4259a0d fix 2013-10-17 19:41:00 -04:00
Joey Hess
16243b9972 missing import 2013-10-17 19:39:22 -04:00
Joey Hess
e93206e294 Windows: Deal with strange msysgit 1.8.4 behavior of not understanding DOS formatted paths for --git-dir and --work-tree. 2013-10-17 19:35:57 -04:00
Joey Hess
aff125ddab try working around windows xargs problem 2013-10-17 15:56:56 -04:00
Joey Hess
d785432f78 use TopFilePath for DiffTree and LsTree 2013-10-17 14:51:19 -04:00
Joey Hess
82ff37520f fix off-by-one 2013-10-16 12:14:14 -04:00
Joey Hess
bac078742d Deal with git check-attr -z output format change in git 1.8.5.
I have not actually tested with 1.8.5, which is not yet relesaed, but
git.git commit f7cd8c50b9ab83e084e8f52653ecc8d90665eef2 changes -z
to also apply to output, without regards to back-compat. (But with pretty
good reasons.)

New code should work with both versions, by fingerprinting for NULs and
newlines.
2013-10-15 16:05:27 -04:00
Joey Hess
f1295b5141 fix windows build 2013-10-02 20:26:00 -04:00
Joey Hess
1536ebfe47 Disable receive.denyNonFastForwards when setting up a gcrypt special remote
gcrypt needs to be able to fast-forward the master branch. If a git
repository is set up with git init --shared --bare, it gets that set, and
pushing to it will then fail, even when it's up-to-date.
2013-10-01 15:23:48 -04:00
Joey Hess
57d49a6d04 remove *>=> and >=*> ; use <$$> instead
I forgot I had <$$> hidden away in Utility.Applicative.
It allows doing the same kind of currying as does >=*>
and I found using it made the code more readable for me.

(*>=> was not used)
2013-09-27 19:58:48 -04:00
Joey Hess
e864c8d033 blind enabling gcrypt repos on rsync.net
This pulls off quite a nice trick: When given a path on rsync.net, it
determines if it is an encrypted git repository that the user has
the key to decrypt, and merges with it. This is works even when
the local repository had no idea that the gcrypt remote exists!

(As previously done with local drives.)

This commit sponsored by Pedro Côrte-Real
2013-09-27 16:21:56 -04:00
Joey Hess
1550759220 enabling rsync.net gcrypt repos
Still need to detect when the user is trying to create a repo
that already exists, and jump to the enabling code.
2013-09-26 23:47:30 -04:00
Joey Hess
735ed3b822 prep for enabling remotre gcrypt repos in webapp 2013-09-26 17:26:13 -04:00
Joey Hess
3192b059b5 add back lost check that git-annex-shell supports gcrypt 2013-09-24 17:51:12 -04:00
Joey Hess
7390f08ef9 Use cryptohash rather than SHA for hashing.
This is a massive win on OSX, which doesn't have a sha256sum normally.

Only use external hash commands when the file is > 1 mb,
since cryptohash is quite close to them in speed.

SHA is still used to calculate HMACs. I don't quite understand
cryptohash's API for those.

Used the following benchmark to arrive at the 1 mb number.

1 mb file:

benchmarking sha256/internal
mean: 13.86696 ms, lb 13.83010 ms, ub 13.93453 ms, ci 0.950
std dev: 249.3235 us, lb 162.0448 us, ub 458.1744 us, ci 0.950
found 5 outliers among 100 samples (5.0%)
  4 (4.0%) high mild
  1 (1.0%) high severe
variance introduced by outliers: 10.415%
variance is moderately inflated by outliers

benchmarking sha256/external
mean: 14.20670 ms, lb 14.17237 ms, ub 14.27004 ms, ci 0.950
std dev: 230.5448 us, lb 150.7310 us, ub 427.6068 us, ci 0.950
found 3 outliers among 100 samples (3.0%)
  2 (2.0%) high mild
  1 (1.0%) high severe

2 mb file:

benchmarking sha256/internal
mean: 26.44270 ms, lb 26.23701 ms, ub 26.63414 ms, ci 0.950
std dev: 1.012303 ms, lb 925.8921 us, ub 1.122267 ms, ci 0.950
variance introduced by outliers: 35.540%
variance is moderately inflated by outliers

benchmarking sha256/external
mean: 26.84521 ms, lb 26.77644 ms, ub 26.91433 ms, ci 0.950
std dev: 347.7867 us, lb 210.6283 us, ub 571.3351 us, ci 0.950
found 6 outliers among 100 samples (6.0%)

import Crypto.Hash
import Data.ByteString.Lazy as L
import Criterion.Main
import Common

testfile :: FilePath
testfile = "/run/shm/data" -- on ram disk

main = defaultMain
        [ bgroup "sha256"
                [ bench "internal" $ whnfIO internal
                , bench "external" $ whnfIO external
                ]
        ]

sha256 :: L.ByteString -> Digest SHA256
sha256 = hashlazy

internal :: IO String
internal = show . sha256 <$> L.readFile testfile

external :: IO String
external = do
	s <- readProcess "sha256sum" [testfile]
        return $ fst $ separate (== ' ') s
2013-09-22 20:06:02 -04:00
Joey Hess
006cf7976f more completely solve catKey memory leak
Done using a mode witness, which ensures it's fixed everywhere.

Fixing catFileKey was a bear, because git cat-file does not provide a
nice way to query for the mode of a file and there is no other efficient
way to do it. Oh, for libgit2..

Note that I am looking at tree objects from HEAD, rather than the index.
Because I cat-file cannot show a tree object for the index.
So this fix is technically incomplete. The only cases where it matters
are:

1. A new large file has been directly staged in git, but not committed.
2. A file that was committed to HEAD as a symlink has been staged
   directly in the index.

This could be fixed a lot better using libgit2.
2013-09-19 16:41:21 -04:00
Joey Hess
f26c996dc6 interface to parse git tree objects 2013-09-19 15:58:35 -04:00
Joey Hess
eb42bde19a sync, pre-commit, indirect: Avoid unnecessarily catting non-symlink files from git, which can be so large it runs out of memory. 2013-09-19 14:48:42 -04:00
Joey Hess
e8e209f4e5 better probing for gcrypt repositories using new --check option
Now can tell if a repo uses gcrypt or not, and whether it's decryptable
with the current gpg keys.

This closes the hole that undecryptable gcrypt repos could have before been
combined into the repo in encrypted mode.
2013-09-19 12:53:24 -04:00
Joey Hess
8062f6337f webapp: support adding existing gcrypt special remotes from removable drives
When adding a removable drive, it's now detected if the drive contains
a gcrypt special remote, and that's all handled nicely. This includes
fetching the git-annex branch from the gcrypt repo in order to find
out how to set up the special remote.

Note that gcrypt repos that are not git-annex special remotes are not
supported. It will attempt to detect such a gcrypt repo and refuse
to use it. (But this is hard to do any may fail; see
https://github.com/blake2-ppc/git-remote-gcrypt/issues/6)

The problem with supporting regular gcrypt repos is that we don't know
what the gcrypt.participants setting is intended to be for the repo.
So even if we can decrypt it, if we push changes to it they might not be
visible to other participants.

Anyway, encrypted sneakernet (or mailnet) is now fully possible with the
git-annex assistant! Assuming that the gpg key distribution is handled
somehow, which the assistant doesn't yet help with.

This commit was sponsored by Navishkar Rao.
2013-09-18 15:55:31 -04:00
Joey Hess
6c35038643 gcrypt: Ensure that signing key is set to one of the participants keys.
Otherwise gcrypt will fail to pull, since it requires this to be the case.

This needs a patched gcrypt, which is in my forked version.
2013-09-17 16:06:29 -04:00
Joey Hess
ab9dd6d8a0 sync: Fix bug that caused direct mode mappings to not be updated when merging files into the tree on Windows. 2013-09-13 13:49:28 -04:00
Joey Hess
7c1a9cdeb9 partially complete gcrypt remote (local send done; rest not)
This is a git-remote-gcrypt encrypted special remote. Only sending files
in to the remote works, and only for local repositories.

Most of the work so far has involved making initremote work. A particular
problem is that remote setup in this case needs to generate its own uuid,
derivied from the gcrypt-id. That required some larger changes in the code
to support.

For ssh remotes, this will probably just reuse Remote.Rsync's code, so
should be easy enough. And for downloading from a web remote, I will need
to factor out the part of Remote.Git that does that.

One particular thing that will need work is supporting hot-swapping a local
gcrypt remote. I think it needs to store the gcrypt-id in the git config of the
local remote, so that it can check it every time, and compare with the
cached annex-uuid for the remote. If there is a mismatch, it can change
both the cached annex-uuid and the gcrypt-id. That should work, and I laid
some groundwork for it by already reading the remote's config when it's
local. (Also needed for other reasons.)

This commit was sponsored by Daniel Callahan.
2013-09-07 18:38:00 -04:00
Joey Hess
dad34e0ea8 add getParticipantList
Note that it needs to look at global git config, since git-remote-gcrypt
will see any setting there as a fallback.
2013-09-05 16:34:13 -04:00
Joey Hess
a48a4e2f8a automatically derive an annex-uuid from a gcrypt-uuids 2013-09-05 16:02:39 -04:00
Joey Hess
6cdac3a003 sync, assistant: Force push of the git-annex branch.
Necessary to ensure it gets pushed to remotes after being rewritten by forget.
See inline rationalles for why I think this is safe!
2013-08-29 14:27:53 -04:00
guilhem
f754779c02 Unused: bugfix
Detect staged files that are not in the working tree.
2013-08-26 13:50:09 -04:00
guilhem
f15fda60ed Speed up the 'unused' command.
Instead of populating the second-level Bloom filter with every key
referenced in every Git reference, consider only those which differ
from what's referenced in the index.

Incidentaly, unlike with its old behavior, staged
modifications/deletion/... will now be detected by 'unused'.

Credits to joeyh for the algorithm. :-)
2013-08-25 21:02:13 -04:00
guilhem
b4a32c7506 Unescape characters in 'file://...' URIs.
That allows, in Git remotes, such URIs to contain spaces or UTF-8
characters. Closes http://git-annex.branchable.com/bugs/Unable_to_use_remotes_with_space_in_the_path/ .
2013-08-22 11:33:16 -04:00
Joey Hess
6fd2935a5a unused: Pay attention to symlinks that are not yet staged in the index. 2013-08-22 10:20:03 -04:00
Joey Hess
a3224ce35b avoid more build warnings on Windows 2013-08-04 14:05:36 -04:00
Joey Hess
b191d5c595 gitignore support for the assistant and watcher
Requires git 1.8.4 or newer. When it's installed, a background
git check-ignore process is run, and used to efficiently check ignores
whenever a new file is added.

Thanks to Adam Spiers, for getting the necessary support into git for this.

A complication is what to do about files that are gitignored but have
been checked into git anyway. git commands assume the ignore has been
overridden in this case, and not need any more overriding to commit a
changed version.

However, for the assistant to do the same, it would have to run git ls-files
to check if the ignored file is in git. This is somewhat expensive. Or it
could use the running git-cat-file process to query the file that way,
but that requires transferring the whole file content over a pipe, so it
can be quite expensive too, for files that are not git-annex
symlinks.

Now imagine if the user knows that a file or directory tree will be getting
frequent changes, and doesn't want the assistant to sync it, so gitignores
it. The assistant could overload the system with repeated ls-files checks!

So, I've decided that the assistant will not automatically commit changes
to files that are gitignored. This is a tradeoff. Hopefully it won't be a
problem to adjust .gitignore settings to not ignore files you want the
assistant to autocommit, or to manually git annex add files that are listed
in .gitignore.

(This could be revisited if git-annex gets access to an interface to check
the content of the index w/o forking a git command. This could be libgit2,
or perhaps a separate git cat-file --batch-check process, so it wouldn't
need to ship over the whole file content.)

This commit was sponsored by Francois Marier. Thanks!
2013-08-02 20:37:03 -04:00
Joey Hess
672cfc3923 better git version checking 2013-08-02 18:32:26 -04:00
Joey Hess
93f2371e09 get rid of __WINDOWS__, use mingw32_HOST_OS
The latter is harder for me to remember, but avoids build failures in code
used by the configure program.
2013-08-02 12:27:32 -04:00
Joey Hess
d16114d024 Slow and ugly work around for bug #718517 in git, which broke git-cat-file --batch for filenames containing spaces.
This runs git-cat-file in non-batch mode for all files with spaces.
If a directory tree has a lot of them, and is in direct mode, even "git
annex add" when there are few new files will need a *lot* of forks!

The only reason buffering the whole file content to get the sha is not a
memory leak is that git-annex only ever uses this on symlinks.

This needs to be reverted as soon as a fix is available in git!
2013-08-01 17:30:47 -04:00
Joey Hess
ebd778c519 Escape ':' in file/directory names to avoid it being treated as a pathspec by some git commands
A git pathspec is a filename, except when it starts with ':', it's taken
to refer to a branch, etc. Rather than special case ':', any filename
starting with anything unusual is prefixed with "./"

This could have been a real mess to deal with, but luckily SafeCommand
is already extensively used and so we know at the type level the difference
between parameters that are files, and parameters that are command options.

Testing did show that Git.Queue was not using SafeCommand on
filenames fed to xargs. (Filenames starting with '-' worked before only
because -- was used to separate filenames from options when calling eg git
add.)

The test suite now passes with filenames starting with ':'. However, I did
not keep that change to it, because such filenames are probably not legal
on windows, and I have enough ugly windows ifdefs in there as it is.

This commit was sponsored by Otavio Salvador. Thanks!
2013-08-01 15:15:49 -04:00
Joey Hess
7e66d260ea importfeed: git-annex becomes a podcatcher in 150 LOC 2013-07-28 16:55:42 -04:00
Joey Hess
4e2fab90d5 avoid newline translation when writing to git hash-object
They're like mushrooms, just keep popping up.
2013-06-18 15:08:51 -04:00
Joey Hess
02c51266ec missed another hash-object call, disable filtering there too 2013-06-18 14:48:15 -04:00
Joey Hess
a1f8771d2b avoid filtering object being hashed
This avoids newline conversion being done on it in Windows.
2013-06-18 13:42:16 -04:00
Joey Hess
077ca355d0 Revert "flush stream after each write to update-index, to possibly avoid buffering issues on Windows"
Didn't help.
2013-06-14 14:34:24 -04:00
Joey Hess
b97a9ea786 flush stream after each write to update-index, to possibly avoid buffering issues on Windows 2013-06-14 14:25:17 -04:00
Joey Hess
91c4dcfc69 Can now restart certain long-running git processes if they crash, and continue working.
Fuzz tests have shown that git cat-file --batch sometimes stops running.
It's not yet known why (no error message; repo seems ok). But this is
something we can deal with in the CoProcess framework, since all 3 types of
long-running git processes should be restartable if they fail.

Note that, as implemented, only IO errors are caught. So an error thrown
by the reveiver, when it sees something that is not valid output from
git cat-file (etc) will not cause a restart. I don't want it to retry
if git commands change their output or are just outputting garbage.
This does mean that if the command did a partial output and crashed in the
middle, it would still not be restarted.

There is currently no guard against restarting a command repeatedly, if,
for example, it crashes repeatedly on startup.
2013-05-31 12:42:13 -04:00
Joey Hess
a600471a23 include HEAD in CanPush shas 2013-05-21 20:04:38 -04:00
Joey Hess
08c03b2af3 XMPP: Avoid redundant and unncessary pushes. Note that this breaks compatibility with previous versions of git-annex, which will refuse to accept any XMPP pushes from this version. 2013-05-21 18:24:29 -04:00
Joey Hess
25dba9da24 fix windows build 2013-05-21 13:07:43 -04:00
Joey Hess
369fb69fe7 fix warning 2013-05-20 18:01:27 -04:00
Joey Hess
25cb9a48da fix the day's Windows permissions damage 2013-05-14 20:15:14 -04:00
Joey Hess
959536ef03 fill in a few windows stubs 2013-05-14 16:32:03 -05:00
Joey Hess
306a36260f typo 2013-05-14 15:44:49 -04:00
Joey Hess
7b92ffc3a1 more leaning toothpick fixes 2013-05-14 15:43:23 -04:00
Joey Hess
dc66b1f27d Merge branch 'master' into windows
Conflicts:
	Annex/Environment.hs
	Build/Configure.hs
	Git/Construct.hs
	Utility/FileMode.hs
2013-05-14 15:37:24 -04:00
Joey Hess
81cded2b9d detect local urls on DOS 2013-05-14 15:27:39 -04:00
Joey Hess
03e8594369 fix the day's windows permissions damage 2013-05-12 19:09:48 -04:00
Joey Hess
73d2f8b280 deal with git using / internally, even on DOS 2013-05-12 17:29:49 -05:00
Joey Hess
06551ad86b set raw mode for git check-attr 2013-05-12 16:37:06 -05:00
Joey Hess
abe8d549df fix permission damage (thanks, Windows) 2013-05-11 23:54:25 -04:00
Joey Hess
5e1458152f refactoring 2013-05-11 23:11:56 -04:00
Joey Hess
1e2ddcb68a use setCurrentDirectory
On POSIX, this just calls changeWorkingDirectory.
2013-05-11 19:14:30 -04:00
Joey Hess
18bdff3fae clean up from windows porting 2013-05-11 18:23:41 -04:00
Joey Hess
dc22549ab3 git annex init works on Windows!
git hash-object and cat-file both only use \n at ends of line, even on Windows.
2013-05-11 16:02:35 -05:00
Joey Hess
c45a723876 catFile expects no \r, even on Windows 2013-05-11 15:32:34 -05:00
Joey Hess
3c7e30a295 git-annex now builds on Windows (doesn't work) 2013-05-11 15:03:00 -05:00
Joey Hess
763cbda14f fixup #if 0 stubs to use #ifndef mingw32_HOST_OS
That's needed in files used to build the configure program.
For the other files, I'm keeping my __WINDOWS__ define, as I find that much easier to type.
I may search and replace it to use the mingw32_HOST_OS thing later.
2013-05-10 16:57:21 -05:00
Joey Hess
6c74a42cc6 stub out POSIX stuff 2013-05-10 16:29:59 -05:00
Joey Hess
8a2d1988d3 expose Control.Monad.join
I think I've been looking for that function for some time.
Ie, I remember wanting to collapse Just Nothing to Nothing.
2013-04-22 20:24:53 -04:00
Joey Hess
a5dded0401 assistant: The ConfigMonitor left one zombie behind each time it checked for changes, now fixed. 2013-03-18 22:09:51 -04:00
Joey Hess
2c05c85437 webapp: DTRT when told to create a git repo that already exists. 2013-03-12 08:09:31 -04:00
Joey Hess
ea672b7c77 Bugfix: git annex add, when ran without any file or directory specified, should add files in the current directory, but not act on unlocked files elsewhere in the tree. 2013-03-07 19:03:06 -04:00
Joey Hess
82f639c70f fix type introduced in 0c13d306
Doubled command name broke show-ref, which broke git annex sync.
Re-read all of 0c13d306 to check for other problems.
2013-03-07 11:09:30 -04:00
Joey Hess
0c13d3065e git subcommand cleanup
Pass subcommand as a regular param, which allows passing git parameters
like -c before it. This was already done in the pipeing set of functions,
but not the command running set.
2013-03-03 13:39:07 -04:00
Joey Hess
4d33423067 assistant: Avoid noise in logs from git commit about typechanged files in direct mode repositories. 2013-03-01 16:21:29 -04:00
Joey Hess
8d9c2afd89 Additional GIT_DIR support bugfixes. May actually work now.
Two fixes. First, and most importantly, relax the isLinkToAnnex check
to only look for /annex/objects/, not [^|/].git/annex/objects. If
GIT_DIR is used with a detached work tree, the git directory is
not necessarily named .git.

There are important caveats with doing that at all, since git-annex will
make symlinks that point at GIT_DIR, which means that the relative path
between GIT_DIR and GIT_WORK_TREE needs to remain stable across all clones
of the repository.

----

The other fix is just fixing crazy and wrong code that, when GIT_DIR is
set, expects to still find a git repository in the path below the work
tree, and uses some of its configuration, and some of GIT_DIR. What was I
thinking, and why can't I seem to get this code right?
2013-02-23 12:41:22 -04:00
Joey Hess
52902c0945 make adding modified files work on crippled filesystems 2013-02-20 14:12:55 -04:00
Joey Hess
547d7745fb pre-commit: Update direct mode mappings.
Making the pre-commit hook look at git diff-index to find changed direct
mode files and update the mappings works pretty well.

One case where it does not work is when a file is git annex added, and then
git rmed, and then this is committed. That's a no-op commit, so the hook
probably doesn't even run, and it certianly never notices that the file
was deleted, so the mapping will still have the original filename in it.

For this and other reasons, it's important that the mappings still be
treated as possibly inconsistent.

Also, the assistant now allows the pre-commit hook to run when in direct
mode, so the mappings also get updated there.
2013-02-06 12:44:19 -04:00
Joey Hess
5cd152b8a9 annex.autocommit
New setting, can be used to disable autocommit of changed files by the
assistant, while it still does data syncing and other tasks.

Also wired into webapp UI
2013-01-27 22:43:05 +11:00
Joey Hess
0214e0fb17 union merge bugfix
Union merges involving two or more repositories could sometimes result in
data from one repository getting lost. This could result in the location
log data becoming wrong, and fsck being needed to fix it.

NB: I audited for any other occurrences of this problem. There are other
places than union merge where multiple changes are fed into update-index
in a stream, but they all involve working copy files being staged, or their
deletion being staged, and in this case it's fine for the later changes
to override the earlier ones.
2013-01-16 21:31:06 -04:00
Joey Hess
95db595e91 make startup scan for deleted files work in direct mode
git add --update cannot be used, because it'll stage typechanged direct
mode files. Intead, use ls-files to find deleted files, and stage them
ourselves.

It seems that no commit was made before when the scan staged deleted files.
(Probably masked since if files were added, a commit happened then..)
Now that I'm doing the staging, I was also able to fix that bug.
2012-12-24 14:24:13 -04:00
Joey Hess
92bd889e61 unused 2012-12-18 17:15:11 -04:00
Joey Hess
53dbcce645 direct mode merging works!
Automatic merge resoltion code needs to be fixed to preserve objects from
direct mode files.
2012-12-18 15:04:44 -04:00
Joey Hess
ffdd08fd2e Merge branch 'master' into desymlink 2012-12-13 00:46:10 -04:00
Joey Hess
0d50a6105b whitespace fixes 2012-12-13 00:45:27 -04:00
Joey Hess
b080a58b76 Merge branch 'master' into desymlink
Conflicts:
	Annex/CatFile.hs
	Annex/Content.hs
	Git/LsFiles.hs
	Git/LsTree.hs
2012-12-13 00:29:06 -04:00
Joey Hess
f87a781aa6 finished where indentation changes 2012-12-13 00:24:19 -04:00
Joey Hess
e7b8cb0063 direct mode committing 2012-12-12 19:20:38 -04:00
Joey Hess
b0c5cbfde2 add notStaged 2012-12-12 13:25:26 -04:00
Joey Hess
e8a74e9493 where indentation 2012-12-12 13:20:58 -04:00
Joey Hess
0714b0bd03 remove unused function 2012-12-12 13:17:41 -04:00
Joey Hess
715c67a3e5 git diff-tree interface 2012-12-10 14:36:57 -04:00
Joey Hess
444e984727 don't treat foo::bar as a ssh url
It's a git-remote-helper location, and will be stored as just an url.
2012-11-09 13:50:23 -04:00
Joey Hess
39e82b1af8 webapp: Generate better git remote names.
Wrote a better git remote name sanitizer. Git blows up on lots of weird
stuff, especially if it starts the remote name, but I managed to get
some common punctuation working.
2012-10-31 15:26:19 -04:00
Joey Hess
7ee0ffaeb9 Use USER and HOME environment when set, and only fall back to getpwent, which doesn't work with LDAP or NIS. 2012-10-25 18:17:54 -04:00
Joey Hess
c7c2015435 add ConfigMonitor thread
Monitors git-annex branch for changes, which are noticed by the Merger
thread whenever the branch ref is changed (either due to an incoming push,
or a local change), and refreshes cached config values for modified config
files.

Rate limited to run no more often than once per minute. This is important
because frequent git-annex branch changes happen when files are being
added, or transferred, etc.

A primary use case is that, when preferred content changes are made,
and get pushed to remotes, the remotes start honoring those settings.
Other use cases include propigating repository description and trust
changes to remotes, and learning when a remote has added a new special
remote, so the webapp can present the GUI to enable that special remote
locally.

Also added a uuid.log cache. All other config files already had caches.
2012-10-20 16:43:35 -04:00
Joey Hess
b281584422 remove some more !! 2012-10-20 16:21:43 -04:00
Joey Hess
e6b1f36e1d Fix handling of GIT_DIR when it refers to a git submodule.
The old code was just wrong in taking fromPath of GIT_DIR -- that made an
localUnknown location with the GIT_DIR in it, which only worked by
accident, and failed in submodules.
2012-10-17 14:28:05 -04:00
Joey Hess
919fec85cd better fix for zombie problem, which turns out to be a zombie ssh started by rsync
When rsyncProgress pipes rsync's stdout, this turns out to cause a ssh
process started by rsync to be left behind as a zombie. I don't know why,
but my recent zombie reaping cleanup was correct, it's just that this other
zombie, that's not directly started by git-annex, was no longer reaped
due to changes in the cleanup. Make rsyncProgress reap the zombie started
by rsync, as a workaround.

FWIW, the process tree looks like this. It seems like the rsync child
is for some reason starting but not waiting on this extra ssh process.
Ssh connection caching may be involved -- disabling it seemed to change
the shape of the tree, but did not eliminate the zombie.

 9378 pts/14   S+     0:00  |           \_ rsync -p --progress --inplace -4 -e 'ssh' '-S' ...
 9379 pts/14   S+     0:00  |           |   \_ ssh ...
 9380 pts/14   S+     0:00  |           |   \_ rsync -p --progress --inplace -4 -e 'ssh' '-S' ...
 9381 pts/14   Z+     0:00  |           \_ [ssh] <defunct>
2012-10-17 00:47:52 -04:00
Joey Hess
4f95cc8ef1 ensure that gitdir is absolute
calcGitLink turns out to need it to be absolute, and it normally is,
but not if it's read from a .git file in a submodule, or perhaps from
GIT_DIR.

I should look into dropping this invariant.
2012-10-16 16:25:45 -04:00
Joey Hess
8fec62d299 A relative core.worktree is relative to the gitdir.
Now that this is handled correctly, git-annex can be used in git submodules.

Also, fixed infelicity where Git.CurrentRepo and Git.Config.updateLocation
were both dealing with core.worktree. Now updateLocation handles it for
Local as well as for LocalUnknown repos.
2012-10-16 00:08:39 -04:00
Joey Hess
148d9f0088 simplify 2012-10-15 23:12:50 -04:00
Joey Hess
429b77844e drop old config when rereading repo config
Before, the new config was merged into the old, so if eg, a remote was
renamed, it would have both the new and the old remote name.
2012-10-14 17:23:40 -04:00
Joey Hess
06831e7754 fix slightly incorrect comment 2012-10-12 12:20:45 -04:00
Joey Hess
e05c21cb73 Fix a crash when merging files in the git-annex branch that contain invalid utf8.
The crash actually occurred when writing out the file, which was done to a
handle that had not had fileSystemEncoding applied to it.
2012-10-12 12:19:30 -04:00
Joey Hess
47314c0fad fix last zombies in the assistant
Made Git.LsFiles return cleanup actions, and everything waits on
processes now, except of course for Seek.
2012-10-04 19:56:32 -04:00
Joey Hess
f7f1d25df8 bugfix 2012-10-04 19:41:58 -04:00
Joey Hess
de3ea4adb6 remove now-unnecessary manual reaps 2012-10-04 18:58:57 -04:00
Joey Hess
5594bf0643 more zombie fighting
I'm down to 9 places in the code that can produce unwaited for zombies.

Most of these are pretty innocuous, at least for now, are only
used in short-running commands, or commands that run a set of
actions and explicitly reap zombies after each one.

The one from Annex.Branch.files could be trouble later,
since both Command.Fsck and Command.Unused can trigger it,
and the assistant will be doing those eventally. Ditto the one in
Git.LsTree.lsTree, which Command.Unused uses.

The only ones currently affecting the assistant though, are
in Git.LsFiles. Several threads use several of those.

(And yeah, using pipes or ResourceT would be a less ad-hoc approach,
but I don't really feel like ripping my entire code base apart right
now to change a foundation monad. Maybe one of these days..)
2012-10-04 18:47:31 -04:00
Joey Hess
f67b54e5e3 make a pipeReadStrict, that properly waits on the process
Nearly everything that's reading from git is operating on a small
amount of output and has been switched to use that. Only pipeNullSplit
stuff continues using the lazy version that yields zombies.
2012-10-04 18:04:09 -04:00
Joey Hess
582316f66f avoid webapp crash on startup when there's no ~/.gitconfig
git config --list --global exits nonzero when there's no global config
2012-09-23 12:43:14 -04:00
Joey Hess
e8188ea611 flip catchDefaultIO 2012-09-17 00:18:07 -04:00
Joey Hess
ba744c84a4 better name for fallback sync refs
Don't expose these as branches in refs/heads/. Instead hide them away in
refs/synced/ where only show-ref will find them.

Make unused only look at branches and tags, not these other things,
so it won't care if some stale sync ref used to use a file.

This means they don't need to be deleted, which could have
led to an incoming sync being missed.
2012-09-16 23:09:08 -04:00
Joey Hess
6cddda4143 make the merger merge any equivilant sync branch into the current branch
Not just synced/master, but synced/UUID/master, for example
2012-09-16 19:41:26 -04:00
Joey Hess
da63b7e96c Support repositories created with --separate-git-dir. Closes: #684405 2012-09-15 22:40:04 -04:00
Joey Hess
ca45cea113 Revert "add catFileIndex"
This interface is not a good idea, because a running git cat-file --batch
does not notice when existing files in the index are changed.
2012-09-15 18:30:53 -04:00
Joey Hess
0b63ee6cd5 run git coprocesses with gitEnv 2012-09-15 17:43:37 -04:00
Joey Hess
e1baf48d88 add catFileIndex 2012-09-15 17:06:10 -04:00
Joey Hess
c9b3b8829d thread safe git-annex index file use 2012-08-24 20:50:39 -04:00
Joey Hess
fb4b19deed make the webapp honor the web.browser git config 2012-08-08 13:15:35 -04:00
Joey Hess
5ae1f75a39 handle case of adding populated drive to just created repo
The just created repo has no master branch commits yet. This is now
handled, merging in the master branch from the populated drive.
2012-08-05 16:35:30 -04:00
Joey Hess
34fc0d358e fix crashes when run in a git repo that has been initted but has no master branch yet 2012-08-05 15:53:47 -04:00
Joey Hess
9fc94d780b better readProcess 2012-07-19 00:57:40 -04:00
Joey Hess
1db7d27a45 add back debug logging
Make Utility.Process wrap the parts of System.Process that I use,
and add debug logging to them.

Also wrote some higher-level code that allows running an action
with handles to a processes stdin or stdout (or both), and checking
its exit status, all in a single function call.

As a bonus, the debug logging now indicates whether the process
is being run to read from it, feed it data, chat with it (writing and
reading), or just call it for its side effect.
2012-07-19 00:46:52 -04:00
Joey Hess
d1da9cf221 switch from System.Cmd.Utils to System.Process
Test suite now passes with -threaded!

I traced back all the hangs with -threaded to System.Cmd.Utils. It seems
it's just crappy/unsafe/outdated, and should not be used. System.Process
seems to be the cool new thing, so converted all the code to use it
instead.

In the process, --debug stopped printing commands it runs. I may try to
bring that back later.

Note that even SafeSystem was switched to use System.Process. Since that
was a modified version of code from System.Cmd.Utils, it needed to be
converted too. I also got rid of nearly all calls to forkProcess,
and all calls to executeFile, which I'm also doubtful about working
well with -threaded.
2012-07-18 18:00:24 -04:00
Joey Hess
fc5652c811 Merge branch 'master' into threaded 2012-07-18 13:31:28 -04:00
Joey Hess
05310538ef more debugging 2012-07-18 13:31:00 -04:00
Joey Hess
0962d50ad2 typo 2012-07-17 14:51:42 -04:00
Joey Hess
4db09814e4 avoid --no-edit with older git versions 2012-07-17 14:50:37 -04:00
Joey Hess
182526ff68 add debugging 2012-07-17 14:40:05 -04:00
Joey Hess
048b64024a sync: Automatically resolves merge conflicts.
untested, but it compiles :)
2012-06-27 13:08:32 -04:00
Joey Hess
051c68041b properly handle deleted files when processing ls-files --unmerged 2012-06-27 12:11:03 -04:00
Joey Hess
8e8439a519 add ls-files --unmerged support 2012-06-27 09:27:59 -04:00
Joey Hess
6f45827fe0 git-config fileEncoding
Accept arbitrarily encoded repository filepaths etc when reading git config
output. This fixes support for remotes with unusual characters in their
names.

For example, a remote with a url of /tmp/çüş was previously
skipped, because the filename wasn't encoded right so it didn't think it
was available. And when setting the annex-uuid of a remote named "çüş",
it used to add it under a mis-encoded form of the remote's name. Both these
cases now work ok in my testing.
2012-06-26 23:07:11 -04:00
Joey Hess
1093d82f6b Got rid of the last place that did utf8 decoding.
Probably fixes bugs/git-annex:_Cannot_decode_byte___39____92__xfc__39__/
although I don't know how to reproduce that bug.
2012-06-26 22:58:44 -04:00
Joey Hess
c79e3b67e9 sync: Avoid recent git's interactive merge. 2012-06-23 10:22:56 -04:00
Joey Hess
75b6ee81f9 avoid ByteString.Char8 where not needed
Its truncation behavior is a red flag, so avoid using it in these places
where only raw ByteStrings are used, without looking at the data inside.
2012-06-20 13:13:40 -04:00
Joey Hess
da62edb42a optimisation and memory leak fix 2012-06-12 21:13:15 -04:00
Joey Hess
ca9ee21bd7 crazy optimisation
Crazy like a fox..
2012-06-10 19:58:34 -04:00
Joey Hess
c5707c84d3 queue size fix
Increase queue size for update-index actions, because otherwise they'll
never be flushed.
2012-06-10 13:56:04 -04:00
Joey Hess
5308b51ec0 stage deletions directly using update-index
no need to run git-rm separately
2012-06-10 13:05:58 -04:00
Joey Hess
7f39415600 force thunk for precalculated value 2012-06-10 12:50:15 -04:00
Joey Hess
d45a9a7831 refactor and function name cleanup
(oops, I had a calcMerge and a calc_merge!)
2012-06-08 00:29:39 -04:00
Joey Hess
20f425be19 make watch use the queue
May not work. Certianly needs to flush the queue from time to time
when only symlink changes are being made.
2012-06-07 15:40:44 -04:00
Joey Hess
0a11b35d89 extend Git.Queue to be able to queue more than simple git commands
While I was in there, I noticed and fixed a bug in the queue size
calculations. It was never encountered only because Queue.add was
only ever run with 1 file in the list.
2012-06-07 15:19:44 -04:00
Joey Hess
91db540769 add support for staging other types of blobs, like symlinks, into the index
Also added a utility TopFilePath type, which could stand to be used more
widely.
2012-06-06 14:26:15 -04:00
Joey Hess
4b32ea793d Merge branch 'master' into watch 2012-06-06 12:52:21 -04:00
Joey Hess
f596084a59 move hashObject to HashObject library and generalize it to support all git object types 2012-06-06 02:31:31 -04:00
Joey Hess
27cfeca4ea Merge branch 'master' into watch 2012-06-06 02:16:21 -04:00
Joey Hess
f1bd72ea54 factor out generic update-index code from unionmerge code 2012-06-06 00:10:34 -04:00
Joey Hess
7a6fb8ae4e flush the git queue when a new type of action is being added to it
This allows the queue to be used in a single process for multiple possibly
conflicting commands, like add and rm, without running them out of order.

This assumes that running the same git subcommand with different parameters
cannot itself conflict.
2012-06-04 20:41:22 -04:00
Joey Hess
ebbd24e5ed more worktree improvements
Avoid more expensive code path when no core.worktree is configured.

Don't change worktree when reading config if one is already set.
This could happen if GIT_CORE_WORKTREE is set, and the repo also has
core.worktree, and the config is reread. Now GIT_CORE_WORKTREE will
prevail.
2012-05-19 11:08:50 -04:00
Joey Hess
9d98144776 avoid chdir when already inside worktree 2012-05-19 10:37:28 -04:00
Joey Hess
0093a456e8 test suite saved my bacon
git config reading memoization shouldn't be used when changing config
2012-05-19 10:22:43 -04:00
Joey Hess
a1885bd116 make GIT_DIR, GIT_WORK_TREE absolute
GIT_DIR is set to something relative, like ".git" in the pre-commit hook.
But internally all the directories are assumed to be absolute.
2012-05-18 18:32:19 -04:00
Joey Hess
eb6cb1b87f Add support for core.worktree, and fix support for GIT_WORK_TREE and GIT_DIR.
The environment needs to override git-config. Changed when git config is
read, and avoid rereading it once it's been read.

chdir for both worktree settings.
2012-05-18 18:20:53 -04:00
Joey Hess
bb4f31a0ee Clean up handling of git directory and git worktree.
Baked into the code was an assumption that a repository's git directory
could be determined by adding ".git" to its work tree (or nothing for bare
repos). That fails when core.worktree, or GIT_DIR and GIT_WORK_TREE are
used to separate the two.

This was attacked at the type level, by storing the gitdir and worktree
separately, so Nothing for the worktree means a bare repo.

A complication arose because we don't learn where a repository is bare
until its configuration is read. So another Location type handles
repositories that have not had their config read yet. I am not entirely
happy with this being a Location type, rather than representing them
entirely separate from the Git type. The new code is not worse than the
old, but better types could enforce more safety.

Added support for core.worktree. Overriding it with -c isn't supported
because it's not really clear what to do if a git repo's config is read, is
not bare, and is then overridden to bare. What is the right git directory
in this case? I will worry about this if/when someone has a use case for
overriding core.worktree with -c. (See Git.Config.updateLocation)

Also removed and renamed some functions like gitDir and workTree that
misused git's terminology.

One minor regression is known: git annex add in a bare repository does not
print a nice error message, but runs git ls-files in a way that fails
earlier with a less nice error message. This is because before --work-tree
was always passed to git commands, even in a bare repo, while now it's not.
2012-05-18 17:03:12 -04:00
Joey Hess
84ac8c58db Add annex.httpheaders and annex.httpheader-command config settings
Allow custom headers to be sent with all HTTP requests.

(Requested by the Internet Archive)
2012-04-22 01:13:09 -04:00
Joey Hess
ed79596b75 noop 2012-04-21 23:32:33 -04:00
Joey Hess
b4a5e39ee6 Support git's core.sharedRepository configuration
This is incomplete, it does not honor it yet for hash directories
and other annex bookkeeping files. Some of that is not needed for a bare
repo; some of it may be.
2012-04-21 15:36:52 -04:00
Joey Hess
70538dac84 compute distance in correct direction 2012-04-14 16:01:08 -04:00
Joey Hess
52a158a7c6 autocorrection
git-annex (but not git-annex-shell) supports the git help.autocorrect
configuration setting, doing fuzzy matching using the restricted
Damerau-Levenshtein edit distance, just as git does. This adds a build
dependency on the haskell edit-distance library.
2012-04-12 15:37:21 -04:00
Joey Hess
c924542e61 bup: Properly handle key names with spaces or other things that are not legal git refs.
Continue using the key name as bup ref name, to preserve backwards
compatability, unless it is an illegal git ref. In that case, use a sha256
of the key name instead.
2012-04-11 12:45:49 -04:00
Joey Hess
378f61d0ef nicer style; also empty refs are implicitly not allowed 2012-04-11 12:29:31 -04:00
Joey Hess
0be6ebb0aa added a git ref legality checker
git-check-ref-format is .. wow. Good design on one level, but what a mess.
2012-04-11 12:21:54 -04:00
Joey Hess
184a69171d removed another 10 lines via ifM 2012-03-16 01:59:07 -04:00
Joey Hess
00d814aecc fix filename encoding for git cat-file
The filename sent to git cat-file needs to be sent on a File encoded handle.

Also set the read handle to use the File encoding, so that any error
message mentioning the filename is received properly.

The actual file content is read using Data.ByteString.Char8, which
will ignore the read handle's encoding, so this won't change that.
(Whether that is entirely correct remains to be seen.)
2012-02-26 14:11:50 -04:00
Joey Hess
cac130b205 cleanup 2012-02-21 00:16:24 -04:00
Joey Hess
6c0155efb7 refactor 2012-02-20 15:22:21 -04:00
Joey Hess
f0f07db01d reorder prams and put -- after atrributes, for compatability with old git
(cherry picked from commit c8ec0e233e)
2012-02-15 14:01:06 -04:00
Joey Hess
52c5b164d8 Added a annex.queuesize setting
useful when adding hundreds of thousands of files on a system with plenty
of memory.

git add gets quite slow in such a large repository, so if the system has
more than the ~32 mb of memory the queue can use by default, it's a useful
optimisation to increase the queue size, in order to decrease the number
of times git add is run.
2012-02-15 11:14:19 -04:00
Joey Hess
7ebd98d8d8 fix memory leak when staging the journal
The list of files had to be retained until the end so it could be deleted.
Also, a list of update-index lines was generated and only then fed into it.
Now everything streams in constant space.
2012-02-14 14:37:59 -04:00
Joey Hess
a40ec5e03e Fixed a memory leak due to excessive strictness when committing journal files.
When hashing the files, the entire list of shas was read strictly.
That was entirely unnecessary, since there's a cleanup action run
after they're consumed.
2012-02-14 11:20:34 -04:00
Joey Hess
8f76d66f32 set fileEncoding on CheckAttr handles
Seemed to work without it, but this is correct.
2012-02-14 04:31:39 -04:00
Joey Hess
a2f241d503 fix LsFiles.typeChanged paths
Passing absolute paths to Command.Add used to work, but after recent
changes doesn't. All LsFiles should use relative paths anyway, so fix it
there.
2012-02-14 00:22:42 -04:00
Joey Hess
cbaebf538a rework git check-attr interface
Now gitattributes are looked up, efficiently, in only the places that
really need them, using the same approach used for cat-file.

The old CheckAttr code seemed very fragile, in the way it streamed files
through git check-attr.
I actually found that cad8824852
was still deadlocking with ghc 7.4, at the end of adding a lot of files.
This should fix that problem, and avoid future ones.

The best part is that this removes withAttrFilesInGit and withNumCopies,
which were complicated Seek methods, as well as simplfying the types
for several other Seek methods that had a Backend tupled in.
2012-02-13 23:52:21 -04:00
Joey Hess
d35a8d85b5 another place hGetBoth was used without a writer thread 2012-02-13 20:23:45 -04:00
Joey Hess
cad8824852 thinko
I removed the now unnecessary forkProcess, but forgot to change back to
pipeBoth, so there was no writer thread.
2012-02-13 20:01:37 -04:00
Joey Hess
3ac2677e00 comment typo 2012-02-13 16:58:26 -04:00
Joey Hess
e4d0923544 wording 2012-02-09 17:35:36 -04:00
Joey Hess
dc682e53a2 use fileEncoding for git-update-index input handle 2012-02-04 13:03:33 -04:00
Joey Hess
586be39952 fix file encoding of HashObject 2012-02-04 13:01:00 -04:00
Joey Hess
d8fb97806c support all filename encodings with ghc 7.4
Under ghc 7.4, this seems to be able to handle all filename encodings
again. Including filename encodings that do not match the LANG setting.
I think this will not work with earlier versions of ghc, it uses some ghc
internals.

Turns out that ghc 7.4 has a special filesystem encoding that it uses when
reading/writing filenames (as FilePaths). This encoding is documented
to allow  "arbitrary undecodable bytes to be round-tripped through it".

So, to get FilePaths from eg, git ls-files, set the Handle that is reading
from git to use this encoding. Then things basically just work.

However, I have not found a way to make Text read using this encoding.
Text really does assume unicode. So I had to switch back to using String
when reading/writing data to git. Which is a pity, because it's some
percent slower, but at least it works.

Note that stdout and stderr also have to be set to this encoding, or
printing out filenames that contain undecodable bytes causes a crash.
IMHO this is a misfeature in ghc, that the user can pass you a filename,
which you can readFile, etc, but that default, putStr of filename may
cause a crash!

Git.CheckAttr gave me special trouble, because the filenames I got back
from git, after feeding them in, had further encoding breakage.
Rather than try to deal with that, I just zip up the input filenames
with the attributes. Which must be returned in the same order queried
for this to work.

Also of note is an apparent GHC bug I worked around in Git.CheckAttr. It
used to forkProcess and feed git from the child process.  Unfortunatly,
after this forkProcess, accessing the `files` variable from the parent
returns []. Not the value that was passed into the function. This screams
of a bad bug, that's clobbering a variable, but for now I just avoid
forkProcess there to work around it. That forkProcess was itself only added
because of a ghc bug, #624389. I've confirmed that the test case for that
bug doesn't reproduce it with ghc 7.4. So that's ok, except for the new ghc
bug I have not isolated and reported. Why does this simple bit of code
magnet the ghc bugs? :)

Also, the symlink touching code is currently broken, when used on utf-8
filenames in a non-utf-8 locale, or probably on any filename containing
undecodable bytes, and I temporarily commented it out.
2012-02-03 16:23:20 -04:00
Joey Hess
3d49258e5b attempt at a quick, utf-8 only fix to the ghc 7.4 problem
If you have only utf-8 filenames, and need to build git-annex with ghc 7.4,
this will work. But, it will crash on non-utf-8 filenames.
2012-02-01 16:16:08 -04:00
Joey Hess
a964012fc3 switch to the strict state monad
I had not realized what a memory leak the lazy state monad could be,
although I have not seen much evidence of actual leaking in git-annex.
However, if running git-annex on a great many files, this could matter.

The additional Utility.State.changeState adds even more strictness,
avoiding a problem I saw in github-backup where repeatedly modifying
state built up a huge pile of thunks.
2012-01-29 22:55:06 -04:00
Joey Hess
97209ac08d fix error message 2012-01-25 20:43:01 -04:00
Joey Hess
3ca7cf5db1 export fromPath
Not used in git-annex, but I am using it in git-backup
2012-01-25 20:42:05 -04:00
Joey Hess
ce5637498f remove Utility.Conditional and use IfElse
This drops the >>! and >>? with the nice low fixity. IfElse does have
undocumented >>=>>! and >>=>>? operators, but I deem that too fishy.
Anyway, using whenM and unlessM is easier; I sometimes mixed the operators
up.
2012-01-24 16:22:07 -04:00
Joey Hess
ba6088b249 rename readMaybe to readish
a stricter (but also partial) readMaybe is getting added to base
2012-01-23 17:00:10 -04:00
Joey Hess
8c87293b48 avoid unnecessary stats when traversing to parent 2012-01-14 11:48:10 -04:00
Joey Hess
92a4af8b20 avoid unnecessary chdir 2012-01-14 11:42:51 -04:00
Joey Hess
1f66af2b53 optimize away 3 stats 2012-01-14 11:28:49 -04:00
Joey Hess
ff5703ce77 tweak 2012-01-13 21:06:00 -04:00
Joey Hess
66aac77467 support relative GIT_DIR 2012-01-13 14:40:36 -04:00
Joey Hess
1ae780ee79 git-annex, git-union-merge: Support GIT_DIR and GIT_WORK_TREE.
Note that GIT_WORK_TREE cannot influence GIT_DIR; that is necessary for
git-fake-bare and vcsh type things to work.
2012-01-13 12:52:09 -04:00
Joey Hess
0d5c402210 Add annex-trustlevel configuration settings, which can be used to override the trust level of a remote.
This overrides the trust.log, and is overridden by the command-line trust
parameters.

It would have been nicer to have Logs.Trust.trustMap just look up the
configuration for all remotes, but a dependency loop prevented that
(Remotes depends on Logs.Trust in several ways). So instead, look up
the configuration when building remotes, storing it in the same forcetrust
field used for the command-line trust parameters.
2012-01-09 23:31:44 -04:00
Joey Hess
9fb5f3edc7 log --after=date 2012-01-06 17:24:03 -04:00
Joey Hess
0b27e6baa0 Support unescaped repository urls, like git does.
Turns out that git will accept a .git/config containing an url with eg,
spaces in its name. Handle this by escaping the url if it's not valid.

This also fixes support for urls containing escaped characters like %20
for space. Before, the path from the url was not unescaped properly.
2012-01-05 14:32:20 -04:00
Joey Hess
f0957426c5 skip local remotes that are not available (ie, not mounted)
With --fast, unavailable local remotes are filtered out of the fast set.
This way, if there are local remotes, --fast always acts only on them,
and if none are mounted, acts on nothing. This consistency is better
than --fast acting on different remotes depending on what's mounted.
2011-12-31 04:50:39 -04:00
Joey Hess
a2ec2d3760 refactor and check for a detached HEAD 2011-12-31 03:38:58 -04:00
Joey Hess
52104dae6f refactor 2011-12-30 18:36:40 -04:00
Joey Hess
26040d6419 add base, under
The describe function was only intended to generate a human-visible
description of a branch, but taking the base of a branch is a useful
operation to be able to do no matter the human-visible representation.

Converting a branch like refs/heads/master to refs/heads/origin/master
is also a useful operation, and under can do that.
2011-12-30 16:48:26 -04:00
Joey Hess
5287d1dc3f fixed behavior when multiple insteadOf configs are provided for the same url base
Consider this git config --list case:

url.git+ssh://git@example.com/.insteadOf=gl
url.git+ssh://git@example.com/.insteadOf=shared

Since config is stored in a Map, only the last of the values for this key
was stored and available for use by the insteadOf code. But that
is wrong; git allows either "gl" or "shared" to be used in an url and
the insteadOf value to be substituted in.

To support this, it seems best to keep the existing config map as-is,
and add a second map that accumulates a list of multiple values for
config keys. This new fullconfig map can be used in the rare places where
multiple values for a key make sense, without needing to complicate
everything else.

Haskell's laziness and data sharing keep the overhead of adding
this second map low.
2011-12-30 14:07:46 -04:00
Joey Hess
cba3ce08df handle C-style escapes in Format
I was happily able to repurpose some code from Git.Filename to handle this.

I remember writing that code... a whole afternoon at a coffee shop, after
which I felt I'd struggled with Haskell and git, and sorta lost, in needing
to write this nasty peice of code. But was also pleased at the use of a
pair of functions and quickcheck that allowed me to get it 100% right.
So, turns out I not only got it right, but the code wasn't as special-purpose
as I'd feared. Yay!
2011-12-23 01:05:16 -04:00
Joey Hess
5a275a3f5d Can now be built with older git versions (before 1.7.7); the resulting binary should only be used with old git.
Remove git old version check from configure, and use the git version
it was built against in the git check-attr code.
2011-12-22 15:01:13 -04:00
Joey Hess
6bffe509d7 Add --include, which is the same as --not --exclude. 2011-12-22 14:00:17 -04:00