Commit graph

754 commits

Author SHA1 Message Date
Joey Hess
0b27e6baa0 Support unescaped repository urls, like git does.
Turns out that git will accept a .git/config containing an url with eg,
spaces in its name. Handle this by escaping the url if it's not valid.

This also fixes support for urls containing escaped characters like %20
for space. Before, the path from the url was not unescaped properly.
2012-01-05 14:32:20 -04:00
Joey Hess
f0957426c5 skip local remotes that are not available (ie, not mounted)
With --fast, unavailable local remotes are filtered out of the fast set.
This way, if there are local remotes, --fast always acts only on them,
and if none are mounted, acts on nothing. This consistency is better
than --fast acting on different remotes depending on what's mounted.
2011-12-31 04:50:39 -04:00
Joey Hess
a2ec2d3760 refactor and check for a detached HEAD 2011-12-31 03:38:58 -04:00
Joey Hess
52104dae6f refactor 2011-12-30 18:36:40 -04:00
Joey Hess
26040d6419 add base, under
The describe function was only intended to generate a human-visible
description of a branch, but taking the base of a branch is a useful
operation to be able to do no matter the human-visible representation.

Converting a branch like refs/heads/master to refs/heads/origin/master
is also a useful operation, and under can do that.
2011-12-30 16:48:26 -04:00
Joey Hess
5287d1dc3f fixed behavior when multiple insteadOf configs are provided for the same url base
Consider this git config --list case:

url.git+ssh://git@example.com/.insteadOf=gl
url.git+ssh://git@example.com/.insteadOf=shared

Since config is stored in a Map, only the last of the values for this key
was stored and available for use by the insteadOf code. But that
is wrong; git allows either "gl" or "shared" to be used in an url and
the insteadOf value to be substituted in.

To support this, it seems best to keep the existing config map as-is,
and add a second map that accumulates a list of multiple values for
config keys. This new fullconfig map can be used in the rare places where
multiple values for a key make sense, without needing to complicate
everything else.

Haskell's laziness and data sharing keep the overhead of adding
this second map low.
2011-12-30 14:07:46 -04:00
Joey Hess
cba3ce08df handle C-style escapes in Format
I was happily able to repurpose some code from Git.Filename to handle this.

I remember writing that code... a whole afternoon at a coffee shop, after
which I felt I'd struggled with Haskell and git, and sorta lost, in needing
to write this nasty peice of code. But was also pleased at the use of a
pair of functions and quickcheck that allowed me to get it 100% right.
So, turns out I not only got it right, but the code wasn't as special-purpose
as I'd feared. Yay!
2011-12-23 01:05:16 -04:00
Joey Hess
5a275a3f5d Can now be built with older git versions (before 1.7.7); the resulting binary should only be used with old git.
Remove git old version check from configure, and use the git version
it was built against in the git check-attr code.
2011-12-22 15:01:13 -04:00
Joey Hess
6bffe509d7 Add --include, which is the same as --not --exclude. 2011-12-22 14:00:17 -04:00
Joey Hess
ee3b5b2a42 use Common in a few more modules 2011-12-20 14:37:53 -04:00
Joey Hess
95d2391f58 more partial function removal
Left a few Prelude.head's in where it was checked not null and too hard to
remove, etc.
2011-12-15 18:19:36 -04:00
Joey Hess
fbc3d32f7d avoid partial function, and parse git-ref output better
It's possible that a ref name might contain a space, this properly
preserves the space.
2011-12-15 16:58:04 -04:00
Joey Hess
eb132a854e avoid partial head function
(although it was used safely)
2011-12-15 16:04:08 -04:00
Joey Hess
111b6937ec avoid partial functions, and added check for correct sha content 2011-12-15 15:57:47 -04:00
Joey Hess
a8643ca44c refactor 2011-12-15 13:05:47 -04:00
Joey Hess
09cd042775 Properly handle multiline git config values.
A crash on parsing was fixed a while ago. This adds support for fully
correctly parsing multiline git config values, using git config --null.

Since git-annex-shell configlist uses normal git config output, I left in
support for that too; the two forms of config output can be easily
identified by the parser. Since configlist only prints the annex.uuid
config, there's no risk of multiline values there, so no need to change it.
2011-12-15 12:48:27 -04:00
Joey Hess
ef28b3fef7 split out Git/Command.hs 2011-12-14 15:56:11 -04:00
Joey Hess
02f1bd2bf4 split more stuff out of Git.hs 2011-12-14 15:43:13 -04:00
Joey Hess
9db8ec210f split out two more Git modules 2011-12-13 15:24:23 -04:00
Joey Hess
25b2cc4148 move commit to Git.Branch 2011-12-13 15:08:44 -04:00
Joey Hess
13fff71f20 split out three modules from Git
Constructors and configuration make sense in separate modules.
A separate Git.Types is needed to avoid cycles.
2011-12-13 15:06:49 -04:00
Joey Hess
46588674b0 avoid closing pipe before all the shas are read from it
Could have just used hGetContentsStrict here, but that would require
storing all the shas in memory. Since this is called at the end of a
git-annex run, it may have created a *lot* of shas, so I avoid that memory
use and stream them out like before.
2011-12-12 21:41:37 -04:00
Joey Hess
0e45b762a0 broke out Git/HashObject.hs 2011-12-12 21:24:55 -04:00
Joey Hess
31a0c07ee9 broke out Git/Branch.hs and reorganized 2011-12-12 21:12:51 -04:00
Joey Hess
543d0d2501 split out Git/Ref.hs 2011-12-12 18:30:33 -04:00
Joey Hess
acd7a52dfd always find optimal merge
Testing b9ac585454, it didn't find the
optimal union merge, the second sha was the one to use, at least in
the case I tried. Let's just try all shas to see if any can be reused.

I stopped using the expensive nub, so despite the use of sets to
sort/uniq file contents, this is probably as fast or faster than it
was before.
2011-12-12 01:59:29 -04:00
Joey Hess
0cbab5de65 refactor 2011-12-12 00:48:25 -04:00
Joey Hess
b9ac585454 more efficient union merges
Tries to avoid generating a new object when the merged content has the same
lines that were in the old object.

I've noticed some merge commits that only move lines around, like this:

- 1323478057.181191s 1 be23c3ac-0ee5-11e0-b185-3b0f9b5b00c5
  1323204972.062151s 1 87e06c7a-7388-11e0-ba07-03cdf300bd87
++1323478057.181191s 1 be23c3ac-0ee5-11e0-b185-3b0f9b5b00c5

Unsure if this will really save anything in practice, since it only looks
at one of the two old objects, and maybe I didn't pick the best one.
2011-12-11 23:02:25 -04:00
Joey Hess
d64132a43a hslint 2011-12-09 01:57:13 -04:00
Joey Hess
9290095fc2 improve type signatures with a Ref newtype
In git, a Ref can be a Sha, or a Branch, or a Tag. I added type aliases for
those. Note that this does not prevent mixing up of eg, refs and branches
at the type level. Since git really doesn't care, except rare cases like
git update-ref, or git tag -d, that seems ok for now.

There's also a tree-ish, but let's just use Ref for it. A given Sha or Ref
may or may not be a tree-ish, depending on the object type, so there seems
no point in trying to represent it at the type level.
2011-11-16 02:41:46 -04:00
Joey Hess
272a67921c better name 2011-11-16 01:46:46 -04:00
Joey Hess
e83b966eb5 cleanup 2011-11-15 23:51:24 -04:00
Joey Hess
21a925dcf1 merge: Now runs in constant space.
Before, a merge was first calculated, by running various actions that
called git and built up a list of lines, which were at the end sent
to git update-index. This necessarily used space proportional to the size
of the diff between the trees being merged.

Now, lines are streamed into git update-index from each of the actions in
turn.

Runtime size of git-annex merge when merging 50000 location log files
drops from around 100 mb to a constant 4 mb.

Presumably it runs quite a lot faster, too.
2011-11-15 23:28:01 -04:00
Joey Hess
922e9af528 cleanup 2011-11-15 22:40:40 -04:00
Joey Hess
b76dc2d210 avoid space leak writing merge
This reduces the memory use of a merge by 1/3rd. The space leak was
apparently because the whole update-index input was generated strictly, not
lazily.

I wondered if the change to ByteStrings contributed to this, due to the
need to convert with L.pack here. But going back to the old code, I still
see a much similar leak, and worse performance besides due to it not using
ByteStrings.

The fix is to just hPutStr the lines repeatedly. (Note the \0 is written
separately, to avoid allocation overheads in adding it to the string.)
The Git.pipeWrite interface is probably just wrong for any large inputs to
git. This was the only place using it for input of any size.

There is still at least one other space leak in the merge code.
2011-11-15 22:19:12 -04:00
Joey Hess
04edae6791 Optimised union merging; now only runs git cat-file once. 2011-11-12 17:45:12 -04:00
Joey Hess
637b5feb45 lint 2011-11-11 01:52:58 -04:00
Joey Hess
bf460a0a98 reorder repo parameters last
Many functions took the repo as their first parameter. Changing it
consistently to be the last parameter allows doing some useful things with
currying, that reduce boilerplate.

In particular, g <- gitRepo is almost never needed now, instead
use inRepo to run an IO action in the repo, and fromRepo to get
a value from the repo.

This also provides more opportunities to use monadic and applicative
combinators.
2011-11-08 16:27:20 -04:00
Joey Hess
3acdba3995 faster union merge of multiple branches into index
only write index once
2011-10-07 13:36:48 -04:00
Joey Hess
7ff89ccfee convert all git read/write functions to use ByteStrings
This yields a second or so speedup in unused, find, etc. Seems that even
when the ByteString is immediately split and then converted to Strings,
it's faster.

I may try to push ByteStrings out into more of git-annex gradually,
although I suspect most of the time-critical parts are already covered
now, and many of the rest rely on libraries that only support Strings.
2011-09-29 23:48:57 -04:00
Joey Hess
949ef94d5e layout 2011-09-29 22:31:20 -04:00
Joey Hess
67f2b7cb3e use ByteStrings when reading content of files
didn't bother to benchmark this
2011-09-29 19:19:28 -04:00
Joey Hess
a91c8a15d5 Sped up unused.
Added Git.ByteString which replaces Git IO methods with ones using lazy
ByteStrings. This can be more efficient when large quantities of data are
being read from git.

In Git.LsTree, parse git ls-tree output more efficiently, thanks
to ByteString. This benchmarks 25% faster, in a benchmark that includes
(probably predominately) the run time for git ls-tree itself.

In real world numbers, this makes git annex unused 2 seconds faster for
each branch it needs to check, in my usual large repo.
2011-09-29 19:04:24 -04:00
Joey Hess
297bc648b9 make unused check branches and tags too
needs time and space optimisation
2011-09-28 16:43:10 -04:00
Joey Hess
ad245a6375 refactor catfile code
split into generic IO code, and a thin Annex wrapper
2011-09-28 15:17:36 -04:00
Joey Hess
a3cb5c47e5 use FileMode 2011-09-28 14:14:52 -04:00
Joey Hess
93807564d0 add ls-tree interface
This parser should be fast. I hope.
2011-09-28 14:03:59 -04:00
Joey Hess
7724f895a8 tweak 2011-09-25 14:37:13 -04:00
Joey Hess
203148363f split groups of related functions out of Utility 2011-08-22 16:14:12 -04:00
Joey Hess
e784757376 hlint tweaks
Did all sources except Remotes/* and Command/*
2011-07-15 03:12:05 -04:00
Joey Hess
ded2591124 unannex: Clean up use of git commit -a.
This was more complex than would be expected. unannex has to use git commit -a
since it's removing files from git; git commit filelist won't do.

Allow commands to be added to the Git queue that have no associated files,
and run such commands once.
2011-07-14 17:15:37 -04:00
Joey Hess
896726cde4 rename GitUnionMerge to Git.UnionMerge
Also, moved commit function into Git proper, it's not union merge specific.
2011-06-30 13:32:47 -04:00
Joey Hess
f0497312a7 rename GitQueue to Git.Queue 2011-06-30 13:25:37 -04:00
Joey Hess
f6063a094e renamed GitRepo to Git
It was always imported qualified as Git anyway
2011-06-30 13:21:39 -04:00