Commit graph

85 commits

Author SHA1 Message Date
Joey Hess
d55f3c0716 Fix teardown of stale cached ssh connections. 2012-02-09 21:49:46 -04:00
Joey Hess
146c36ca54 IO exception rework
ghc 7.4 comaplains about use of System.IO.Error to catch exceptions.
Ok, use Control.Exception, with variants specialized to only catch IO
exceptions.
2012-02-03 16:47:24 -04:00
Joey Hess
b81d662cbf Avoid repeated location log commits when a remote is receiving files.
Done by adding a oneshot mode, in which location log changes are written to
the journal, but not committed. Taking advantage of git-annex's existing
ability to recover in this situation.

This is used by git-annex-shell and other places where changes are made to
a remote's location log.
2012-01-28 15:41:52 -04:00
Joey Hess
ba6088b249 rename readMaybe to readish
a stricter (but also partial) readMaybe is getting added to base
2012-01-23 17:00:10 -04:00
Joey Hess
eb9001044f order user provided params after connection caching params
So the user can override them.
2012-01-20 17:32:32 -04:00
Joey Hess
6ef82665de add annex.sshcaching config setting 2012-01-20 17:15:46 -04:00
Joey Hess
47250a153a ssh connection caching
Ssh connection caching is now enabled automatically by git-annex. Only one
ssh connection is made to each host per git-annex run, which can speed some
things up a lot, as well as avoiding repeated password prompts. Concurrent
git-annex processes also share ssh connections. Cached ssh connections are
shut down when git-annex exits.

Note: The rsync special remote does not yet participate in the ssh
connection caching.
2012-01-20 17:14:56 -04:00
Joey Hess
61dbad505d fsck --from remote --fast
Avoids expensive file transfers, at the expense of checking file size
and/or contents.

Required some reworking of the remote code.
2012-01-20 13:23:11 -04:00
Joey Hess
effaa298fa optimise fsck --from normal git remotes
For a local git remote, can symlink the file.
For a git remote using rsync, can preseed any local content.

There are a few reasons to use fsck --from on a normal git remote.
One is if it's using gitosis or similar, and you don't have shell access
to run git annex locally. Another reason could be if you just want to
fsck certian files of a bare remote.
2012-01-19 17:10:44 -04:00
Joey Hess
81856c3175 add a configure check for StatFS
This way, the build log will indicate whether StatFS can be relied on.
I've tested all the failing architectures now, and on all of them,
the StatFS code now returns Nothing, rather than Just nonsense.

Also, if annex.diskreserve is set on a platform where StatFS is not
working, git-annex will complain.

Also, the Makefile was missing the sources target used when building with
cabal.
2012-01-15 13:49:32 -04:00
Joey Hess
a3d97e0c85 tweak 2012-01-14 14:31:16 -04:00
Joey Hess
5e2b4e16ba avoid multiple unnecessary stats of the index file
Up to one per file processed.
2012-01-14 12:07:36 -04:00
Joey Hess
abdacf58ed tweaks 2012-01-11 00:06:54 -04:00
Joey Hess
16e7178f20 reorg 2012-01-10 15:29:10 -04:00
Joey Hess
a3a9f87047 log: New command that displays the location log for file, showing each repository they were added to and removed from.
This needs to run git log on the location log files to get at all past
versions of the file, which tends to be a bit slow.

It would be possible to make a version optimised for showing the location
logs for every key. That would only need to run git log once, so would be
faster, but it would need to process an enormous amount of data, so
would not speed up the individual file case.

In the future it would be nice to support log --format. log --json also
doesn't work right yet.
2012-01-06 15:40:07 -04:00
Joey Hess
aa0882691b Added remote.name.annex-web-options configuration setting, which can be used to provide parameters to whichever of wget or curl git-annex uses (depends on which is available, but most of their important options suitable for use here are the same). 2012-01-02 14:20:20 -04:00
Joey Hess
252376d639 Merge branch 'master' into autosync 2011-12-30 20:38:59 -04:00
Joey Hess
52104dae6f refactor 2011-12-30 18:36:40 -04:00
Joey Hess
925b6390aa add forceUpdate
This code is picked from my tweak-fetch branch, which already did the needed
refactoring.
2011-12-30 15:57:28 -04:00
Joey Hess
6d4382a89e Merge branch 'new-monad-control' 2011-12-24 23:02:42 -04:00
Joey Hess
ee3b5b2a42 use Common in a few more modules 2011-12-20 14:37:53 -04:00
Joey Hess
ef28b3fef7 split out Git/Command.hs 2011-12-14 15:56:11 -04:00
Joey Hess
02f1bd2bf4 split more stuff out of Git.hs 2011-12-14 15:43:13 -04:00
Joey Hess
25b2cc4148 move commit to Git.Branch 2011-12-13 15:08:44 -04:00
Joey Hess
13fff71f20 split out three modules from Git
Constructors and configuration make sense in separate modules.
A separate Git.Types is needed to avoid cycles.
2011-12-13 15:06:49 -04:00
Joey Hess
46588674b0 avoid closing pipe before all the shas are read from it
Could have just used hGetContentsStrict here, but that would require
storing all the shas in memory. Since this is called at the end of a
git-annex run, it may have created a *lot* of shas, so I avoid that memory
use and stream them out like before.
2011-12-12 21:41:37 -04:00
Joey Hess
0e45b762a0 broke out Git/HashObject.hs 2011-12-12 21:24:55 -04:00
Joey Hess
31a0c07ee9 broke out Git/Branch.hs and reorganized 2011-12-12 21:12:51 -04:00
Joey Hess
543d0d2501 split out Git/Ref.hs 2011-12-12 18:30:33 -04:00
Joey Hess
da95cbadca split out Annex/Journal.hs 2011-12-12 18:03:28 -04:00
Joey Hess
98dfc0c9b0 split out Annex/BranchState.hs 2011-12-12 17:38:46 -04:00
Joey Hess
b2f934e07a update comment 2011-12-12 17:24:12 -04:00
Joey Hess
79345ad5fc optimisation
avoids a redundant call to git show-ref
2011-12-12 03:30:47 -04:00
Joey Hess
f9cd3f6ad1 optimisation
avoids a useless diff from git-annex..refs/heads/git-annex
2011-12-12 02:31:07 -04:00
Joey Hess
2332afb4bc cleanup 2011-12-12 02:04:48 -04:00
Joey Hess
29b88ad657 avoid redundant call to updateIndex
commitBranch calls updateIndex
2011-12-11 21:46:21 -04:00
Joey Hess
c4c965d602 detect and recover from branch push/commit race
Dealing with a race without using locking is exceedingly difficult and tricky.
Fully tested, I hope.

There are three places left where the branch can be updated, that are not
covered by the race recovery code. Let's prove they're all immune to the
race:

1. tryFastForwardTo checks to see if a fast-forward can be done,
   and then does git-update-ref on the branch to fast-forward it.

   If a push comes in before the check, then either no fast-forward
   will be done (ok), or the push set the branch to a ref that can
   still be fast-forwarded (also ok)

   If a push comes in after the check, the git-update-ref will
   undo the ref change made by the push. It's as if the push did not come
   in, and the next git-push will see this, and try to re-do it.
   (acceptable)

2. When creating the branch for the very first time, an empty index
   is created, and a commit of it made to the branch. The commit's ref
   is recorded as the current state of the index. If a push came in
   during that, it will be noticed the next time a commit is made to the
   branch, since the branch will have changed. (ok)

3. Creating the branch from an existing remote branch involves making
   the branch, and then getting its ref, and recording that the index
   reflects that ref.

   If a push creates the branch first, git-branch will fail (ok).

   If the branch is created and a racing push is then able to change it
   (highly unlikely!) we're still ok, because it first records the ref into
   the index.lck, and then updating the index. The race can cause the
   index.lck to have the old branch ref, while the index has the newly pushed
   branch merged into it, but that only results in an unnecessary update of
   the index file later on.
2011-12-11 20:41:35 -04:00
Joey Hess
e04852c8af Merge branch 'master' into new-monad-control
Conflicts:
	git-annex.cabal
2011-12-11 16:55:36 -04:00
Joey Hess
cfbbda99f4 optimize index updating
The last branch ref that the index was updated to is stored in
.git/annex/index.lck, and the index only updated when the current
branch ref differs.

(The .lck file should later be used for locking too.)

Some more optimization is still needed, since there is some redundancy in
calls to git show-ref.
2011-12-11 16:14:59 -04:00
Joey Hess
8680c415de slow, stupid, and safe index updating
Always merge the git-annex branch into .git/annex/index before making a
commit from the index.

This ensures that, when the branch has been changed in any way
(by a push being received, or changes pulled directly into it, or
even by the user checking it out, and committing a change), the index
reflects those changes.

This is much too slow; it needs to be optimised to only update the
index when the branch has really changed, not every time.

Also, there is an unhandled race, when a change is made to the branch
right after the index gets updated. I left it in for now because it's
unlikely and I didn't want to complicate things with additional locking
yet.
2011-12-11 15:05:53 -04:00
Joey Hess
0ba4b1de18 move a file location to Locations.hs 2011-12-11 14:14:28 -04:00
Joey Hess
eecaf42485 no need to show, it's a string 2011-12-10 12:30:31 -04:00
Joey Hess
d64132a43a hslint 2011-12-09 01:57:13 -04:00
Joey Hess
f3a2f60abc adjust to build with monad-control-0.3
I had to, I hope temporarily, lose my nice Annex newtype, and use a type
synonym. This because I cannot find a way to derive a MonadBaseControl
instance of the Annex newtype. I've emailed Bas van Dijk in hope he can
help get the newtype back.

Otherwise appears to build & work.
2011-12-05 22:51:37 -04:00
Joey Hess
598eb2e2da cleanup 2011-11-30 12:01:15 -04:00
Joey Hess
da9cd315be add support for using hashDirLower in addition to hashDirMixed
Supporting multiple directory hash types will allow converting to a
different one, without a flag day.

gitAnnexLocation now checks which of the possible locations have a file.
This means more statting of files. Several places currently use
gitAnnexLocation and immediately check if the returned file exists;
those need to be optimised.
2011-11-28 22:43:51 -04:00
Joey Hess
6869e6023e support .git/annex on a different disk than the rest of the repo
The only fully supported thing is to have the main repository on one disk,
and .git/annex on another. Only commands that move data in/out of the annex
will need to copy it across devices.

There is only partial support for putting arbitrary subdirectories of
.git/annex on different devices. For one thing, but this can require more
copies to be done. For example, when .git/annex/tmp is on one device, and
.git/annex/journal on another, every journal write involves a call to
mv(1). Also, there are a few places that make hard links between various
subdirectories of .git/annex with createLink, that are not handled.

In the common case without cross-device, the new moveFile is actually
faster than renameFile, avoiding an unncessary stat to check that a file
(not a directory) is being moved. Of course if a cross-device move is
needed, it is as slow as mv(1) of the data.
2011-11-28 16:17:55 -04:00
Joey Hess
128b4bd015 tweaks 2011-11-19 15:57:08 -04:00
Joey Hess
0fa1d136dc tweak 2011-11-19 15:40:40 -04:00
Joey Hess
1ffd54ef78 ensure branch exists before trying to update it
The branch may not exist, if .git/annex has been copied over from another
repo (or a corrupted repo). I suppose it could also have gotten deleted
somehow. Without this, there is a confusing failure.
2011-11-16 18:56:06 -04:00