Commit graph

1600 commits

Author SHA1 Message Date
Joey Hess
766c31c95c watch git-annex program file to detect upgrades
Not yet wired up to restart the assistant on upgrade; that needs careful
sanity checking to wait until the upgrade is done before restarting.

Used the DirWatcher here, so it gets events for any changes to the
directory containing the program file. (But not subdirs.) This is necessary
in order to detect when the file is renamed as part of the upgrade, which
an inotify on a single file would not detect. (Also, I have DirWatcher code,
but not FileWatcher code.)

Note that upgrades that remove or rename a whole directory tree containing
the executable will *not* trigger this code. So eg, deleting and replacing
the whole standalone tarball dir tree won't work -- but untarring it
over top will. So should dpkg package upgrades.

Added programPath, using a new GHC feature to find the full path to the
executable. The fallback code for old GHC or unsupported OS is less good;
its worst failure mode would be either failing to find the program, and so
not checking for upgrades, or finding a git-annex that's in PATH, but is
not the one running.

This commit was sponsored by John Roepke.
2013-11-22 18:46:45 -04:00
Joey Hess
be069bd962 update 2013-11-22 15:44:39 -04:00
Joey Hess
6d23786d96 Avoid misbehavior when addurl is used with quvi 0.9.
In 0.9, -v shows version, rather than controlling verbosity.

Still need to port to 0.9, this just avoids massively confusing addurl when
quvi prints its version and exits successfully, on urls that it cannot be
used with.
2013-11-22 14:12:44 -04:00
Joey Hess
7dbb702edd merge from git-repair 2013-11-20 18:31:00 -04:00
Joey Hess
e9b88f8db1 releasing package git-annex version 5.20131120 2013-11-20 13:10:11 -04:00
Joey Hess
ef2d3a7999 changelog 2013-11-19 17:18:52 -04:00
Joey Hess
211bcc9456 Oops, I forgot that the test suite would be run even though the tasty test framework is not out of NEW and not in the build dependencies.
The best fix is to put in an override_dh_auto_test that avoids running
make test.
2013-11-19 13:44:07 -04:00
Joey Hess
b876df6fdb Ensure that core.sharedrepository is honored when creating the .git/annex directory. 2013-11-18 18:20:20 -04:00
Joey Hess
310c549b5a Ensure execute bit is set on directories when core.sharedrepsitory is set. 2013-11-18 18:13:09 -04:00
Joey Hess
0d0e21ea57 dropunused, addunused: Allow "all" instead of a range to act on all unused data. 2013-11-18 17:24:18 -04:00
Joey Hess
187b844b1e releasing package git-annex version 5.20131118 2013-11-18 11:01:00 -04:00
Joey Hess
9c20185f55 webapp: Check annex.version. 2013-11-17 14:58:35 -04:00
Joey Hess
f35cc2d4ba Android: Adjust default .gitignore to ignore .thumbnails at any location in the tree, not just at its top. 2013-11-16 18:48:16 -04:00
Joey Hess
493a989d06 Switched to the tasty test framework. 2013-11-15 16:02:12 -04:00
Joey Hess
c20f3c3cba update 2013-11-15 14:52:39 -04:00
Joey Hess
d48b00ebed Direct mode .git/annex/objects directories are no longer left writable
Because that allowed writing to symlinks of files that are not present,
which followed the link and put bad content in an object location.

fsck: Fix up .git/annex/object directory permissions.

This commit was sponsored by an anonymous bitcoin donor.
2013-11-15 14:52:03 -04:00
Joey Hess
b0f85b3e22 Fix direct mode merge bug when a direct mode file was deleted and replaced with a directory. An ordering problem caused the directory to not get created in this case. Thanks to Tim for the test cases. 2013-11-15 13:40:12 -04:00
Joey Hess
2a297dd6d1 on second thought, don't change sync to exit nonzero on successful conflicted merge resolution
For one thing, that breaks the test suite. May revisit later.
2013-11-14 14:36:48 -04:00
Joey Hess
13108b7196 assistant: Notice on startup when the index file is corrupt, and auto-repair. 2013-11-13 14:27:17 -04:00
Joey Hess
5e7e0c7dc0 repair: Handle case where index file is corrupt, but all objects are ok. 2013-11-13 13:41:02 -04:00
Joey Hess
98ac1f4fd5 depend on git 1.8.4 2013-11-12 15:14:35 -04:00
Joey Hess
16fa980689 separate android 4.0 and 4.3 builds needed
Also, parameterize the abdroid abi version
2013-11-12 14:24:53 -04:00
Joey Hess
46cf00ffd8 Fix bug that caused bad information to be written to the git-annex branch when running describe or other commands with a remote that has no uuid.
Still need to fix crash caused by the bad info.
2013-11-09 13:37:30 -04:00
Joey Hess
f3c714293a fixup 2013-11-08 17:04:36 -04:00
Joey Hess
28a64c8fd8 Include ssh-keygen in standalone bundle. 2013-11-08 14:29:47 -04:00
Joey Hess
958312885f webapp: Improve UI around remote that have no annex.uuid set, either because setup of them is incomplete, or because the remote git repository is not a git-annex repository.
Complicated by such repositories potentially being repos that should have
an annex.uuid, but it failed to be gotten, perhaps due to the past ssh repo
setup bugs. This is handled now by an Upgrade Repository button.
2013-11-07 18:02:00 -04:00
Joey Hess
b7c15f3b60 webapp: Avoid encoding problems when displaying the daemon log file. 2013-11-07 15:29:20 -04:00
Joey Hess
9ff229a798 watcher: Avoid loop when adding a file owned by someone else fails in indirect mode because its permissions cannot be modified.
Adding the file moved it to the annex, and then tried to set the mode.
Error unwind then moved the file back, and so the watcher saw the file get
deleted and then added back, and so tried again..
2013-11-07 15:18:54 -04:00
Joey Hess
4f373503f2 sync: When there are merge conflicts that get automatically resolved, exit nonzero at the end. 2013-11-07 15:02:11 -04:00
Joey Hess
d99bdbbb84 Merge branch 'directguard' 2013-11-07 14:12:13 -04:00
Joey Hess
eed2ed4fdb rename status to info, and update docs 2013-11-07 12:45:59 -04:00
Joey Hess
691850ca39 Allow optionally configuring git-annex with -fEKG to enable awesome remote monitoring interfaceat http://localhost:4242/ 2013-11-06 16:33:45 -04:00
Joey Hess
4f3f918abe fix merge 2013-11-06 13:03:23 -04:00
Joey Hess
00c91816fb Merge branch 'master' into directguard 2013-11-06 13:02:35 -04:00
Joey Hess
ee23be55fd Fix exception handling bug that could cause .git/annex/index to be used for git commits outside the git-annex branch. Known to affect git-annex when used with the git shipped with Ubuntu 13.10. 2013-11-06 12:21:50 -04:00
Joey Hess
a74018ea98 prep release 2013-11-06 11:37:06 -04:00
Joey Hess
4510819215 v5 for direct mode, with automatic upgrade
This includes storing the current state of the HEAD ref, which git annex
sync is going to need, but does not make sync use it.
2013-11-05 17:05:03 -04:00
Joey Hess
04768e44b2 automatically set and unset core.bare when switching to/from direct mode 2013-11-05 15:41:24 -04:00
Joey Hess
0e31234e8e support direct mode repositories with core.bare=true (not yet default)
Direct mode repositories can now have core.bare=true set, to prevent
accidentally running git commands that try to operate on the work tree,
and so do the wrong thing.

This is not yet the default, and it causes known problems for git-annex sync
due to receive.denyCurrentBranch not working in bare repositories.

This commit was sponsored by Richard Hartmann.
2013-11-05 14:24:28 -04:00
Joey Hess
c2862d9585 pass -c option on to all git commands run
The -c option now not only modifies the git configuration seen by
git-annex, but it is passed along to every git command git-annex runs.

This was easy to plumb through because gitCommandLine is already used to
construct every git command line, to add --git-dir and --work-tree
2013-11-05 13:38:37 -04:00
Joey Hess
58db042033 map: Work when there are gcrypt remotes. 2013-11-04 14:14:44 -04:00
Joey Hess
1f39aef893 Fix zombie process that occurred when switching between repository views in the webapp. 2013-11-02 22:25:08 -04:00
Joey Hess
dff303aa43 Fix enabling of gcrypt repository accessed over ssh; git-annex-shell gcryptsetup had a bug that caused it to fail with permission denied. 2013-11-02 20:15:40 -04:00
Joey Hess
7be69a2491 gcrypt, bup: Fix bug that prevented using these special remotes with encryption=pubkey.
I think both of these are all that's affected, but I went ahead and fixed
all the remotes that set their config to M.empty to instead store the
actual config. Who knows what will expect it to be actually present in
future, the Remote instance of getGpgEncParams came to..
2013-11-02 16:37:28 -04:00
Joey Hess
bc18cde325 sync: Work even when the local git repository is new and empty, with no master branch. 2013-11-02 15:29:38 -04:00
Joey Hess
7e7e765cba Improve local pairing behavior when two computers both try to start the pairing process separately.
I was able to reproduce something very like this bug by starting
pairing separately on both computers under poor network conditions (ie,
weak wifi on my front porch). Neither computer showed an alert for the
PairReq messages it was seeing (intermittently) from the other.

So, I've made a new PairReq message that has not been seen before
always make the alert pop up, even if the assistant thinks it is
in the middle of its own pairing process (or even another pairing
process with a different box on the LAN).

(This shouldn't cause a rogue PairAck to disrupt a pairing process part
way through.)
2013-11-02 15:10:29 -04:00
Joey Hess
3b76a141ea prep release 2013-11-01 11:57:47 -04:00
Joey Hess
0eff0dd910 unannex: New, much slower, but more safe behavior
Copies files out of the annex. This avoids an unannex of one file breaking
other files that link to the same content. Also, it means that the content
remains in the annex using up space until cleaned up with  "git annex
unused".

(The behavior of unannex --fast has not changed; it still hard
links to content in the annex. --fast was not made the default because it
is potentially unsafe; editing such a hard linked file can unexpectedly
change content stored in the annex.)
2013-10-28 16:56:01 -04:00
Joey Hess
230bfa9688 add --want-get and --want-drop options
New --want-get and --want-drop options which can be used to test preferred
content settings. For example, "git annex find --in . --want-drop"
2013-10-28 14:50:17 -04:00
Joey Hess
afddbfd7e9 The "git annex content" command is renamed to "git annex wanted". 2013-10-28 14:08:38 -04:00
Joey Hess
780a24374f add: Fix reversion in 4.20130827 when adding unlocked files that have not yet been committed.
catKeyFileHEAD is still checked too, because when doing a git commit with
unlocked files, the file gets staged to the index, so is not typechanged
there.

(This is also why git annex add foo; git annex unlock foo; git commit -a
does not re-annex foo, because there is no indication left that it was
added.)
2013-10-28 13:46:26 -04:00
Joey Hess
7ed8e87a34 assistant: Support repairing git remotes that are locally accessible
(eg, on removable drives)

gcrypt remotes are not yet handled.

This commit was sponsored by Sören Brunk.
2013-10-27 15:38:59 -04:00
Joey Hess
b48aaa22d0 assistant: Automatically repair damanged git repository, if it can be done without losing data. 2013-10-26 17:16:29 -04:00
Joey Hess
bcd77e65c2 changelog 2013-10-26 16:12:36 -04:00
Joey Hess
5756636486 directory, webdav: Fix bug introduced in version 4.20131002 that caused the chunkcount file to not be written. Work around repositories without such a file, so files can still be retreived from them. 2013-10-26 15:03:12 -04:00
Joey Hess
0dfe604ddc webapp: When setting up a bare shared repository, enable non-fast-forward pushes. 2013-10-26 13:06:43 -04:00
Joey Hess
2233ddd5a2 assistant: When autostarted, wait 5 seconds before running the startup scan, to avoid contending with the user's desktop login process. 2013-10-26 12:42:58 -04:00
Joey Hess
f51327f44e releasing package git-annex version 4.20131024 2013-10-24 13:14:54 -04:00
Joey Hess
e32b62b50e update 2013-10-23 15:05:57 -04:00
Joey Hess
d5eb85acf4 add repair command 2013-10-23 12:21:59 -04:00
Joey Hess
00932eda06 webapp: Fix bug when adding a remote and git-remote-gcrypt is not installed. 2013-10-22 13:32:10 -04:00
Joey Hess
b7800eab24 webapp: Move sidebar to the right hand side of the screen. 2013-10-21 18:05:52 -04:00
Joey Hess
4f871f89ba git-recover-repository 1/2 done 2013-10-20 17:50:51 -04:00
Joey Hess
f482de1b76 remove workaround for bug in git 1.8.4r0 2013-10-20 15:23:06 -04:00
Joey Hess
e93206e294 Windows: Deal with strange msysgit 1.8.4 behavior of not understanding DOS formatted paths for --git-dir and --work-tree. 2013-10-17 19:35:57 -04:00
Joey Hess
19816bca41 update for DiffTree type change (which fixes assistant in subdir confusion bug) 2013-10-17 15:11:21 -04:00
Joey Hess
c76c94a0da S3: Try to ensure bucket name is valid for archive.org. 2013-10-16 16:35:47 -04:00
Joey Hess
e227e8f683 sync: Fix automatic resolution of merge conflicts where one side is an annexed file, and the other side is a non-annexed file, or a directory.
Note that this case is only fully automatically resolved in direct mode.
In indirect mode, git merge moves the file to file~HEAD, and replaces it
with the directory, and leaves the file in unmerged state, and sync doesn't
yet change that.
2013-10-16 14:56:40 -04:00
Joey Hess
ecb4a30548 Work around sed output difference that led to version containing a newline on OSX. 2013-10-16 10:28:13 -04:00
Joey Hess
bac078742d Deal with git check-attr -z output format change in git 1.8.5.
I have not actually tested with 1.8.5, which is not yet relesaed, but
git.git commit f7cd8c50b9ab83e084e8f52653ecc8d90665eef2 changes -z
to also apply to output, without regards to back-compat. (But with pretty
good reasons.)

New code should work with both versions, by fingerprinting for NULs and
newlines.
2013-10-15 16:05:27 -04:00
Joey Hess
296e21b381 add schedule command
Mostly because it gives me an excuse and a hook to document the schedule
expression format.
2013-10-13 15:40:38 -04:00
Joey Hess
68518a9b3d status: Fix a crash if a temp file went away while its size was being checked for status. 2013-10-13 13:30:24 -04:00
Joey Hess
747f5b123c url size fixes
addurl: Improve message when adding url with wrong size to existing file.
Before the message suggested the url didn't exist.

Fixed handling of URL keys that have no recorded size. Before, if the key
has no size, the url also had to not declare any size, which was unlikely
and wrong, or it was taken to not exist. This probably would mostly affect
keys that were added to the annex with addurl --relaxed.
2013-10-11 13:05:00 -04:00
Joey Hess
2943592f51 Remove bogus runshell loop check.
git-annex.linux/git-annex can legitimately try to run itself -- this
happens when the programfile is used. So this check was bogus.
2013-10-11 01:24:13 -04:00
Joey Hess
45aed381df import: Skip .git directories. 2013-10-07 13:03:05 -04:00
Joey Hess
6622875cf8 Revert "use vector in local status", which was not an improvement
This reverts commit eb3ce3581a.
2013-10-07 04:06:10 -04:00
Joey Hess
eb3ce3581a use vector in local status
Thought was that this would be faster than a map, since a vector can be
updated more efficiently. It turns out to not seem to matter; runtime and
memory usage are basically identical.
2013-10-07 04:05:14 -04:00
Joey Hess
1200788859 status: Fix space leak in local mode, introduced in version 4.20130920.
Actually fixed 2 leaks, the tuple leak may have been older.
2013-10-07 03:59:14 -04:00
Joey Hess
6f38426cb8 work around ssh brain-damange
The control socket path passed to ssh needs to be 17 characters shorter
than the maximum unix domain socket length, because ssh appends stuff to it
to make a temporary filename. Closes: #725512

Also, take the shorter of the relative and the absolute paths to the
socket. Typically the relative path will be a lot shorter (unless
deep inside a subdirectory of the repository), and so using it will
avoid flirting with the maximum safe socket lenghts in more situations,
and so lead to less breakage if all my attempts at fixing this are
still buggy.
2013-10-06 20:59:36 -04:00
Joey Hess
635c9a1549 assistant: Detect stale git lock files at startup time, and remove them.
Extends the index.lock handling to other git lock files. I surveyed
all lock files used by git, and found more than I expected. All are
handled the same in git; it leaves them open while doing the operation,
possibly writing the new file content to the lock file, and then closes
them when done.

The gc.pid file is excluded because it won't affect the normal operation
of the assistant, and waiting for a gc to finish on startup wouldn't be
good.

All threads except the webapp thread wait on the new startup sanity checker
thread to complete, so they won't try to do things with git that fail
due to stale lock files. The webapp thread mostly avoids doing that kind of
thing itself. A few configurators might fail on lock files, but only if the
user is explicitly trying to run them. The webapp needs to start
immediately when the user has opened it, even if there are stale lock
files.

Arranging for the threads to wait on the startup sanity checker was a bit
of a bear. Have to get all the NotificationHandles set up before the
startup sanity checker runs, or they won't see its signal. Perhaps
the NotificationBroadcaster is not the best interface to have used for
this. Oh well, it works.

This commit was sponsored by Michael Jakl
2013-10-05 17:04:21 -04:00
Joey Hess
1be4d281d6 Better sanitization of problem characters when generating URL and WORM keys.
FAT has a lot of characters it does not allow in filenames, like ? and *
It's probably the worst offender, but other filesystems also have
limitiations.

In 2011, I made keyFile escape : to handle FAT, but missed the other
characters. It also turns out that when I did that, I was also living
dangerously; any existing keys that contained a : had their object
location change. Oops.

So, adding new characters to escape to keyFile is out. Well, it would be
possible to make keyFile behave differently on a per-filesystem basis, but
this would be a real nightmare to get right. Consider that a rsync special
remote uses keyFile to determine the filenames to use, and we don't know
the underlying filesystem on the rsync server..

Instead, I have gone for a solution that is backwards compatable and
simple. Its only downside is that already generated URL and WORM keys
might not be able to be stored on FAT or some other filesystem that
dislikes a character used in the key. (In this case, the user can just
migrate the problem keys to a checksumming backend. If this became a big
problem, fsck could be made to detect these and suggest a migration.)

Going forward, new keys that are created will escape all characters that
are likely to cause problems. And if some filesystem comes along that's
even worse than FAT (seems unlikely, but here it is 2013, and people are
still using FAT!), additional characters can be added to the set that are
escaped without difficulty.

(Also, made WORM limit the part of the filename that is embedded in the key,
to deal with filesystem filename length limits. This could have already
been a problem, but is more likely now, since the escaping of the filename
can make it longer.)

This commit was sponsored by Ian Downes
2013-10-05 15:01:49 -04:00
Joey Hess
478eeea02e addurl: Better sanitization of generated filenames.
Use sanitizeFilePath rather than rolling our own sanitizer.
2013-10-05 13:30:13 -04:00
Joey Hess
49ccf56d55 add back 2013-10-04 13:09:36 -04:00
Joey Hess
3d5fe9b794 add news item for git-annex 4.20131002 2013-10-04 13:09:23 -04:00
Joey Hess
93dbb7842e watcher: Detect at startup time when there is a stale .git/lock, and remove it so it does not interfere with the automatic commits of changed files. 2013-10-03 16:57:21 -04:00
Joey Hess
f8880c4fe4 Automatically and safely detect and recover from dangling .git/annex/index.lock files, which would prevent git from committing to the git-annex branch, eg after a crash. 2013-10-03 15:43:08 -04:00
Joey Hess
c07aaec323 prep release 2013-10-02 16:13:45 -04:00
Joey Hess
6b727839d6 prep release 2013-10-02 16:01:07 -04:00
Joey Hess
29385dc393 Moved list of backends and remote types from status to version command. 2013-10-01 20:50:46 -04:00
Joey Hess
a05b763b01 Added SKEIN256 and SKEIN512 backends
SHA3 is still waiting for final standardization.
Although this is looking less likely given
https://www.cdt.org/blogs/joseph-lorenzo-hall/2409-nist-sha-3

In the meantime, cryptohash implements skein, and it's used by some of the
haskell ecosystem (for yesod sessions, IIRC), so this implementation is
likely to continue working. Also, I've talked with the cryprohash author
and he's a reasonable guy.

It makes sense to have an alternate high security hash, in case some
horrible attack is found against SHA2 tomorrow, or in case SHA3 comes out
and worst fears are realized.

I'd also like to support using skein for HMAC. But no hurry there and
a new version of cryptohash has much nicer HMAC code, so I will probably
wait until I can use that version.
2013-10-01 20:34:36 -04:00
Joey Hess
202a932323 changelog 2013-10-01 19:16:56 -04:00
Joey Hess
1536ebfe47 Disable receive.denyNonFastForwards when setting up a gcrypt special remote
gcrypt needs to be able to fast-forward the master branch. If a git
repository is set up with git init --shared --bare, it gets that set, and
pushing to it will then fail, even when it's up-to-date.
2013-10-01 15:23:48 -04:00
Joey Hess
6b37fcffd8 assistant: More robust inotify handling; avoid crashing if a directory cannot be read. 2013-09-30 13:11:26 -04:00
Joey Hess
7f7dcd315b fix direct mode switch permissions problem
Similar to how a similar problem with indirect was earlier fixed.
2013-09-30 12:48:40 -04:00
Joey Hess
3f0ea53fc8 finally sorted out the OSX gpg mess 2013-09-29 16:30:49 -04:00
Joey Hess
44e1524be5 webapp: Fixed a bug where when a new remote is added, one file may fail to sync to or from it
This happened because the transferrer process did not know about the new
remote. remoteFromUUID crashed, which crashed the transferrer. When it was
restarted, the new one knew about the new remote so all further files would
transfer, but the one file would temporarily not be, until transfers retried.

Fixed by making remoteFromUUID not crash, and try reloading the remote list
if it does not know about a remote.

Note that this means that remoteFromUUID does not only return Nothing anymore
when the UUID is the UUID of the local repository. So had to change some code
that dependend on that assumption.
2013-09-29 14:51:49 -04:00
Joey Hess
12f6b9693a Send a git-annex user-agent when downloading urls.
Overridable with --user-agent option.

Not yet done for S3 or WebDAV due to limitations of libraries used --
nether allows a user-agent header to be specified.

This commit sponsored by Michael Zehrer.
2013-09-28 14:35:21 -04:00
Joey Hess
588494cbce webapp: Support storing encrypted git repositories on rsync.net.
Does not yet support re-enabling such a repository though.

This commit was sponsored by Jan Pieper.
2013-09-26 16:43:00 -04:00
Joey Hess
98fc7e8a19 add, import, assistant: Better preserve the mtime of symlinks, when when adding content that gets deduplicated.
Note that this turned out to remove a syscall, not add any expense.
Otherwise, I would not have done it.
2013-09-25 16:07:11 -04:00
Joey Hess
c45f5fbdb3 indirect: Better behavior when a file in direct mode is not owned by the user running the conversion. 2013-09-25 15:29:56 -04:00
Joey Hess
c923c981b9 import: Preserve top-level directory structure. 2013-09-25 13:16:55 -04:00
Joey Hess
4dc4a9a385 assistant: Clear the list of failed transfers when doing a full transfer scan. This prevents repeated retries to download files that are not available, or are not referenced by the current git tree.
This is motivated by a user report that the assistant was repeatedly
retrying transfers of files that had been deleted (in direct mode, so
removing the only copy).

Note that the glacier code retries failed transfers after a while to retry
downloads that have aged long enough to be available. This is ok; if we're
doing a full transfer scan we'll retry on every file that is still in the
git tree.

Also note that this makes the assistant less likely to get every file
referenced by old revs of the git tree. Not something the assistant tries
to ensure anyway, so I feel this is acceptable.
2013-09-25 11:46:17 -04:00
Joey Hess
4c954661a1 git-annex-shell: Added support for operating inside gcrypt repositories.
* Note that the layout of gcrypt repositories has changed, and
  if you created one you must manually upgrade it.
  See http://git-annex.branchable.com/upgrades/gcrypt/
2013-09-24 17:25:47 -04:00
Joey Hess
96b8224b63 pin term 2013-09-22 22:47:20 -04:00
Joey Hess
d6c2fa5199 explicit cryptohash dep 2013-09-22 21:53:01 -04:00
Joey Hess
7390f08ef9 Use cryptohash rather than SHA for hashing.
This is a massive win on OSX, which doesn't have a sha256sum normally.

Only use external hash commands when the file is > 1 mb,
since cryptohash is quite close to them in speed.

SHA is still used to calculate HMACs. I don't quite understand
cryptohash's API for those.

Used the following benchmark to arrive at the 1 mb number.

1 mb file:

benchmarking sha256/internal
mean: 13.86696 ms, lb 13.83010 ms, ub 13.93453 ms, ci 0.950
std dev: 249.3235 us, lb 162.0448 us, ub 458.1744 us, ci 0.950
found 5 outliers among 100 samples (5.0%)
  4 (4.0%) high mild
  1 (1.0%) high severe
variance introduced by outliers: 10.415%
variance is moderately inflated by outliers

benchmarking sha256/external
mean: 14.20670 ms, lb 14.17237 ms, ub 14.27004 ms, ci 0.950
std dev: 230.5448 us, lb 150.7310 us, ub 427.6068 us, ci 0.950
found 3 outliers among 100 samples (3.0%)
  2 (2.0%) high mild
  1 (1.0%) high severe

2 mb file:

benchmarking sha256/internal
mean: 26.44270 ms, lb 26.23701 ms, ub 26.63414 ms, ci 0.950
std dev: 1.012303 ms, lb 925.8921 us, ub 1.122267 ms, ci 0.950
variance introduced by outliers: 35.540%
variance is moderately inflated by outliers

benchmarking sha256/external
mean: 26.84521 ms, lb 26.77644 ms, ub 26.91433 ms, ci 0.950
std dev: 347.7867 us, lb 210.6283 us, ub 571.3351 us, ci 0.950
found 6 outliers among 100 samples (6.0%)

import Crypto.Hash
import Data.ByteString.Lazy as L
import Criterion.Main
import Common

testfile :: FilePath
testfile = "/run/shm/data" -- on ram disk

main = defaultMain
        [ bgroup "sha256"
                [ bench "internal" $ whnfIO internal
                , bench "external" $ whnfIO external
                ]
        ]

sha256 :: L.ByteString -> Digest SHA256
sha256 = hashlazy

internal :: IO String
internal = show . sha256 <$> L.readFile testfile

external :: IO String
external = do
	s <- readProcess "sha256sum" [testfile]
        return $ fst $ separate (== ' ') s
2013-09-22 20:06:02 -04:00
Joey Hess
70f205717c release 2013-09-20 10:43:08 -04:00
Joey Hess
006cf7976f more completely solve catKey memory leak
Done using a mode witness, which ensures it's fixed everywhere.

Fixing catFileKey was a bear, because git cat-file does not provide a
nice way to query for the mode of a file and there is no other efficient
way to do it. Oh, for libgit2..

Note that I am looking at tree objects from HEAD, rather than the index.
Because I cat-file cannot show a tree object for the index.
So this fix is technically incomplete. The only cases where it matters
are:

1. A new large file has been directly staged in git, but not committed.
2. A file that was committed to HEAD as a symlink has been staged
   directly in the index.

This could be fixed a lot better using libgit2.
2013-09-19 16:41:21 -04:00
Joey Hess
eb42bde19a sync, pre-commit, indirect: Avoid unnecessarily catting non-symlink files from git, which can be so large it runs out of memory. 2013-09-19 14:48:42 -04:00
Antoine Beaupré
f4e8b70bba rename remotes to list 2013-09-19 14:16:55 -04:00
Joey Hess
e8e209f4e5 better probing for gcrypt repositories using new --check option
Now can tell if a repo uses gcrypt or not, and whether it's decryptable
with the current gpg keys.

This closes the hole that undecryptable gcrypt repos could have before been
combined into the repo in encrypted mode.
2013-09-19 12:53:24 -04:00
Joey Hess
3d88559e58 webapp: Show encryption information when editing a remote. 2013-09-17 20:02:42 -04:00
Joey Hess
6c35038643 gcrypt: Ensure that signing key is set to one of the participants keys.
Otherwise gcrypt will fail to pull, since it requires this to be the case.

This needs a patched gcrypt, which is in my forked version.
2013-09-17 16:06:29 -04:00
Joey Hess
b37aad6c06 webapp: Initial support for setting up encrypted removable drives.
No support yet for generating new gpg keys.
No support yet for adding existing encrypted repos from removable drives.
2013-09-16 16:07:27 -04:00
Joey Hess
9d366dc638 make --fast disable the numcopies stats
Looking up the location log for every key is not the fastest operation..
2013-09-15 19:17:56 -04:00
Joey Hess
a3bbda5bed status: In local mode, displays information about variance from configured numcopies levels. 2013-09-15 19:10:38 -04:00
Joey Hess
5c71dc1087 addurl: Fix quvi audodetection, broken in last release. 2013-09-15 16:37:03 -04:00
Joey Hess
2407170eaf sync: Don't fail if the directory it is run in gets removed by the sync. 2013-09-13 14:55:55 -04:00
Joey Hess
ab9dd6d8a0 sync: Fix bug that caused direct mode mappings to not be updated when merging files into the tree on Windows. 2013-09-13 13:49:28 -04:00
Joey Hess
65fe2314be fsck: Fix detection and fixing of present direct mode files that are wrongly represented as standin symlinks on crippled filesystems. 2013-09-13 12:50:29 -04:00
Joey Hess
5fe49b98f8 Support hot-swapping of removable drives containing gcrypt repositories.
To support this, a core.gcrypt-id is stored by git-annex inside the git
config of a local gcrypt repository, when setting it up.

That is compared with the remote's cached gcrypt-id. When different, a
drive has been changed. git-annex then looks up the remote config for
the uuid mapped from the core.gcrypt-id, and tweaks the configuration
appropriately. When there is no known config for the uuid, it will refuse to
use the remote.
2013-09-12 15:54:35 -04:00
Joey Hess
82759b6a5d remotes: New command, displays a compact table of remotes that contain files. (Thanks, anarcat for display code and mastensg for inspiration.)
Note that it would be possible to extend the display to show all
repositories. But there can be a lot of repositories that are not set up as
remotes, and it would significantly clutter the display to show them all.

Since we're not showing all repositories, it's not worth trying to show
numcopies count either.

I decided to embrace these limitations and call the command remotes.
2013-09-12 12:21:21 -04:00
Joey Hess
e5d4f4fb0c prep release 2013-09-11 13:02:22 -04:00
Joey Hess
e4d0b2f180 Fix problem with test suite in non-unicode locale. 2013-09-11 12:07:59 -04:00
Joey Hess
b64f5baf2d sync: support gcrypt 2013-09-09 10:02:15 -04:00
Joey Hess
813db0863d recommend git-remote-gcrypt (package in incoming) 2013-09-09 08:31:21 -04:00
Joey Hess
ecbb326e9d Allow building without quvi support. 2013-09-09 02:16:22 -04:00
Joey Hess
b5678d74a2 webapp: Improve javascript's handling of longpolling connection failures, by reloading the current page in this case. Works around chromium behavior where ajax connections to urls that were already accessed are denied after navigating back to a previous page. 2013-09-09 01:24:20 -04:00
Joey Hess
3e079cdcd1 gcrypt: now supports rsync
Use rsync for gcrypt remotes that are not local to the disk.
(Note that I have punted on supporting http transport for now, it doesn't
seem likely to be very useful.)

This was mostly quite easy, it just uses the rsync special remote to handle
the transfers. The git repository url is converted to a RsyncOptions
structure, which required parsing it separately, since the rsync special
remote only supports rsync urls, which use a different format.

Note that annexed objects are now stored at the top of the gcrypt repo,
rather than inside annex/objects. This simplified the rsync suport,
since it doesn't have to arrange to create that directory. And git-annex
is not going to be run directly within gcrypt repos -- or if in some
strance scenario it was, it would make sense for it to not see the
encrypted objects.

This commit was sponsored by Sheila Miguez
2013-09-08 14:54:28 -04:00
Joey Hess
0a2f5f3993 gpg: Force --no-textmode in case the user has it turned on in config. 2013-09-07 13:06:36 -04:00
Joey Hess
dce7389dd0 Remind user when annex-ignore is set for some remotes, if unable to get or drop a file, possibly because it's on an ignored remote. 2013-09-06 16:54:31 -04:00
Joey Hess
3da8cc7008 Fix Feeds display in build flags. 2013-09-05 12:20:39 -04:00
Joey Hess
5667553064 fix 2013-09-05 11:15:06 -04:00
Joey Hess
6883c17d62 highlight the syntax change 2013-09-05 00:10:43 -04:00
Joey Hess
6e09893160 reword changelog 2013-09-04 19:31:36 -04:00
Joey Hess
6f531d903d detailed changelog for the encryption changes 2013-09-04 18:15:38 -04:00
Joey Hess
2fcae0348f Merge branch 'master' into encryption 2013-09-04 18:08:47 -04:00
Joey Hess
db83cc82d6 Merge branch 'forget'
Conflicts:
	debian/changelog
2013-09-03 14:36:00 -04:00
guilhem
8293ed619f Allow public-key encryption of file content.
With the initremote parameters "encryption=pubkey keyid=788A3F4C".

/!\ Adding or removing a key has NO effect on files that have already
been copied to the remote. Hence using keyid+= and keyid-= with such
remotes should be used with care, and make little sense unless the point
is to replace a (sub-)key by another. /!\

Also, a test case has been added to ensure that the cipher and file
contents are encrypted as specified by the chosen encryption scheme.
2013-09-03 14:34:16 -04:00
Joey Hess
d1bacccff4 importfeed: Also ignore transient problems with downloading content from feeds. 2013-09-03 14:32:26 -04:00
Joey Hess
67fda9e669 Honor core.sharedrepository when receiving and adding files in direct mode. 2013-09-03 13:35:49 -04:00
Joey Hess
0831e18372 forget --drop-dead: Completely removes mentions of repositories that have been marked as dead from the git-annex branch.
Wrote nice pure transition calculator, and ugly code to stage its results
into the git-annex branch. Also had to split up several Log modules
that Annex.Branch needed to use, but that themselves used Annex.Branch.

The transition calculator is limited to looking at and changing one file at
a time. While this made the implementation relatively easy, it precludes
transitions that do stuff like deleting old url log files for keys that are
being removed because they are no longer present anywhere.
2013-08-31 17:51:13 -04:00
Joey Hess
6cdac3a003 sync, assistant: Force push of the git-annex branch.
Necessary to ensure it gets pushed to remotes after being rewritten by forget.
See inline rationalles for why I think this is safe!
2013-08-29 14:27:53 -04:00
Joey Hess
a8d74d39c1 prep release 2013-08-27 13:31:41 -04:00
Joey Hess
9e8521910d Android: Fix bug in terminal app that caused it to spin using much CPU and battery. This problem was introduced in version 4.20130601. 2013-08-26 22:00:16 -04:00
Joey Hess
45e10dbe42 changelog 2013-08-26 13:54:22 -04:00
Joey Hess
bb228fa46d remove a bug that didn't get fixed yet 2013-08-25 21:35:20 -04:00
Joey Hess
604c6d5038 document insanely good unused change 2013-08-25 21:09:22 -04:00
Joey Hess
46b6d75274 Youtube support! (And 53 other video hosts)
When quvi is installed, git-annex addurl automatically uses it to detect
when an page is a video, and downloads the video file.

web special remote: Also support using quvi, for getting files,
or checking if files exist in the web.

This commit was sponsored by Mark Hepburn. Thanks!
2013-08-22 18:50:43 -04:00
Joey Hess
412dcb8017 Fix bug that caused typechanged symlinks to be assumed to be unlocked files, so they were added to the annex by the pre-commit hook. 2013-08-22 13:57:07 -04:00
Joey Hess
74034ec781 Better error message when trying to use a git remote that has annex.ignore set. 2013-08-22 12:01:53 -04:00
Joey Hess
d1ea1f2474 Unescape characters in 'file://...' URIs. (Thanks, guilhem for the patch.) 2013-08-22 11:36:48 -04:00
Joey Hess
6fd2935a5a unused: Pay attention to symlinks that are not yet staged in the index. 2013-08-22 10:20:03 -04:00
Joey Hess
d603f536bd Set --clobber when running wget to ensure resuming works properly. 2013-08-21 18:19:01 -04:00
Joey Hess
0912e752b5 Revert "Delete empty downloaded file when wget fails, to work around reported resume failure."
This reverts commit 98886e3fbf.

Better fix forthcoming
2013-08-21 18:17:48 -04:00
Joey Hess
98886e3fbf Delete empty downloaded file when wget fails, to work around reported resume failure.
<RichiH> i richih@eudyptes (git)-[master] ~git/debconf-share/debconf13/photos/chrysn % rm /home/richih/work/git/debconf-share/.git/annex/tmp/SHA256E-s3044235--693b74fcb12db06b5e79a8b99d03e2418923866506ee62d24a4e9ae8c5236758.JPG
<RichiH> richih@eudyptes (git)-[master] ~git/debconf-share/debconf13/photos/chrysn % git annex get P8060008.JPG
<RichiH> get P8060008.JPG (from website...) --2013-08-21 21:42:45--  http://annex.debconf.org/debconf-share/.git//annex/objects/1a4/67d/SHA256E-s3044235--693b74fcb12db06b5e79a8b99d03e2418923866506ee62d24a4e9ae8c5236758.JPG/SHA256E-s3044235--693b74fcb12db06b5e79a8b99d03e2418923866506ee62d24a4e9ae8c5236758.JPG
<RichiH> Resolving annex.debconf.org (annex.debconf.org)... 5.153.231.227, 2001:41c8:1000:19::227:2
<RichiH> Connecting to annex.debconf.org (annex.debconf.org)|5.153.231.227|:80... connected.
<RichiH> HTTP request sent, awaiting response... 404 Not Found
<RichiH> 2013-08-21 21:42:45 ERROR 404: Not Found.
<RichiH> File `/home/richih/work/git/debconf-share/.git/annex/tmp/SHA256E-s3044235--693b74fcb12db06b5e79a8b99d03e2418923866506ee62d24a4e9ae8c5236758.JPG' already there; not retrieving.
<RichiH>   Unable to access these remotes: website
<RichiH>   Try making some of these repositories available:
<RichiH>    3e0356ac-0743-11e3-83a5-1be63124a102 -- website (annex.debconf.org)
<RichiH>     a7495021-9f2d-474e-80c7-34d29d09fec6 -- chrysn@hephaistos:~/data/projects/debconf13/debconf-share
<RichiH>     eb8990f7-84cd-4e6b-b486-a5e71efbd073 -- joeyh passport usb drive
<RichiH>     f415f118-f428-4c68-be66-c91501da3a93 -- joeyh laptop
<RichiH> failed
<RichiH> git-annex: get: 1 failed
<RichiH> richih@eudyptes (git)-[master] ~git/debconf-share/debconf13/photos/chrysn %

I was not able to reproduce the failure, but I did reproduce that
wget -O http://404/ results in an empty file being written.
2013-08-21 16:05:51 -04:00
Joey Hess
0f921307e7 mirror: New command, makes two repositories contain the same set of files.
This is a simple approach for setting up a mirroring repository.

It will work with any type of remotes.

Mirror --from is more expensive than mirror --to in general.
OTOH, mirror --from will get the file from any remote that has it, not only
the named mirror remote. And if the named mirror remote is not the fastest
available remote with a file, that can speed things up.

It would be possible to make the assistant or watch command do a more
dynamic mirroring, that didn't need to scan every time.
2013-08-20 15:46:35 -04:00
Joey Hess
e240cb99f7 Merge branch 'duplicate'
Conflicts:
	debian/changelog
2013-08-20 10:27:24 -04:00
Joey Hess
a6a047192e sync, merge: Bug fix: Don't try to merge into master when in a bare repo. 2013-08-17 21:29:44 +02:00
Joey Hess
641b14b124 Debian: Recommend ssh-askpass, which ssh will use when the assistant is run w/o a tty. Closes: #719832 2013-08-16 10:07:44 +02:00
Joey Hess
59f9781a23 Debian: Run the builtin test suite as an autopkgtest. 2013-08-15 15:49:19 +02:00
Joey Hess
572d6fde14 releasing version 4.20130815 2013-08-15 10:46:33 +02:00
Joey Hess
36693b3e86 releasing version 4.20130815 2013-08-15 10:09:55 +02:00
Joey Hess
38022f4f49 Windows: Fixed permissions problem that prevented removing files from directory special remote.
Directory special remotes now fully usable.
2013-08-04 13:43:48 -04:00
Joey Hess
a837ed47f7 Windows: Added support for encrypted special remotes. 2013-08-04 13:03:34 -04:00
Joey Hess
b28023cb52 importfeed: Fix handling of dots in extensions. 2013-08-03 02:36:38 -04:00
Joey Hess
24c8a6042b importfeed: Ignores transient problems with feeds. Only exits nonzero when a feed has repeatedly had a problems for at least 1 day. 2013-08-03 01:40:21 -04:00
Joey Hess
b191d5c595 gitignore support for the assistant and watcher
Requires git 1.8.4 or newer. When it's installed, a background
git check-ignore process is run, and used to efficiently check ignores
whenever a new file is added.

Thanks to Adam Spiers, for getting the necessary support into git for this.

A complication is what to do about files that are gitignored but have
been checked into git anyway. git commands assume the ignore has been
overridden in this case, and not need any more overriding to commit a
changed version.

However, for the assistant to do the same, it would have to run git ls-files
to check if the ignored file is in git. This is somewhat expensive. Or it
could use the running git-cat-file process to query the file that way,
but that requires transferring the whole file content over a pipe, so it
can be quite expensive too, for files that are not git-annex
symlinks.

Now imagine if the user knows that a file or directory tree will be getting
frequent changes, and doesn't want the assistant to sync it, so gitignores
it. The assistant could overload the system with repeated ls-files checks!

So, I've decided that the assistant will not automatically commit changes
to files that are gitignored. This is a tradeoff. Hopefully it won't be a
problem to adjust .gitignore settings to not ignore files you want the
assistant to autocommit, or to manually git annex add files that are listed
in .gitignore.

(This could be revisited if git-annex gets access to an interface to check
the content of the index w/o forking a git command. This could be libgit2,
or perhaps a separate git cat-file --batch-check process, so it wouldn't
need to ship over the whole file content.)

This commit was sponsored by Francois Marier. Thanks!
2013-08-02 20:37:03 -04:00
Joey Hess
7fdf9ea5dd releasing version 4.20130802 2013-08-02 13:38:18 -04:00
Joey Hess
d16114d024 Slow and ugly work around for bug #718517 in git, which broke git-cat-file --batch for filenames containing spaces.
This runs git-cat-file in non-batch mode for all files with spaces.
If a directory tree has a lot of them, and is in direct mode, even "git
annex add" when there are few new files will need a *lot* of forks!

The only reason buffering the whole file content to get the sha is not a
memory leak is that git-annex only ever uses this on symlinks.

This needs to be reverted as soon as a fix is available in git!
2013-08-01 17:30:47 -04:00
Joey Hess
ebd778c519 Escape ':' in file/directory names to avoid it being treated as a pathspec by some git commands
A git pathspec is a filename, except when it starts with ':', it's taken
to refer to a branch, etc. Rather than special case ':', any filename
starting with anything unusual is prefixed with "./"

This could have been a real mess to deal with, but luckily SafeCommand
is already extensively used and so we know at the type level the difference
between parameters that are files, and parameters that are command options.

Testing did show that Git.Queue was not using SafeCommand on
filenames fed to xargs. (Filenames starting with '-' worked before only
because -- was used to separate filenames from options when calling eg git
add.)

The test suite now passes with filenames starting with ':'. However, I did
not keep that change to it, because such filenames are probably not legal
on windows, and I have enough ugly windows ifdefs in there as it is.

This commit was sponsored by Otavio Salvador. Thanks!
2013-08-01 15:15:49 -04:00
Joey Hess
d1ed337035 webapp: Improve handling of remotes whose setup has stalled.
This includes recovery from the ssh-agent problem that led to many reporting
http://git-annex.branchable.com/bugs/Internal_Server_Error:_Unknown_UUID/
(Including fixing up .ssh/config to set IdentitiesOnly.)

Remotes that have no known uuid are now displayed in the webapp as
"unfinished". There's a link to check their status, and if the remote
has been set annex-ignore, a retry button can be used to unset that and
try again to set up the remote.

As this bug has shown, the process of adding a ssh remote has some failure
modes that are not really ideal. It would certianly be better if, when
setting up a ssh remote it would detect if it's failed to get the UUID,
and handle that in the remote setup process, rather than waiting until
later and handling it this way.

However, that's hard to do, particularly for local pairing, since the
PairListener runs as a background thread. The best it could do is pop up an
alert if there's a problem. This solution is not much different.

Also, this solution handles cases where the user has gotten their repo into
a mess manually and let's the assistant help with cleaning it up.

This commit was sponsored by Chia Shee Liang. Thanks!
2013-07-31 16:36:29 -04:00
Joey Hess
cbfdf3ab21 set IdentitiesOnly
When setting up a dedicated ssh key to access the annex on a host,
set IdentitiesOnly to prevent the ssh-agent from forcing use of a different
ssh key.

That behavior could result in unncessary password prompts. I remember
getting a message or two from people who got deluged with password
prompts and I couldn't at the time see why.

Also, it would prevent git-annex-shell from being run on the remote host,
when git-annex was installed there by unpacking the standalone tarball,
since the authorized_keys line for the dedicated ssh key, which sets
up calling git-annex-shell when it's not in path, wouldn't be used.

This fixes
http://git-annex.branchable.com/bugs/Internal_Server_Error:_Unknown_UUID
but I've not closed that bug yet since I should still:

1. Investigate why the ssh remote got set up despite being so broken.
2. Make the webapp not handle the NoUUID state in such an ugly way.
3. Possibly add code to fix up systems that encountered the problem.
   Although since it requires changes to .ssh/config this may be one for
   the release notes.

Thanks to TJ for pointing me in the right direction to understand what
was happening here.
2013-07-31 13:30:49 -04:00
Joey Hess
9476355bc3 find: Avoid polluting stdout with progress messages. Closes: #718186 2013-07-30 20:24:27 -04:00
Joey Hess
ddd46db09a Fix a few bugs involving filenames that are at or near the filesystem's maximum filename length limit.
Started with a problem when running addurl on a really long url,
because the whole url is munged into the filename. Ended up doing
a fairly extensive review for places where filenames could get too large,
although it's hard to say I'm not missed any..

Backend.Url had a 128 character limit, which is fine when the limit is 255,
but not if it's a lot shorter on some systems. So check the pathconf()
limit. Note that this could result in fromUrl creating different keys
for the same url, if run on systems with different limits. I don't see
this is likely to cause any problems. That can already happen when using
addurl --fast, or if the content of an url changes.

Both Command.AddUrl and Backend.Url assumed that urls don't contain a
lot of multi-byte unicode, and would fail to truncate an url that did
properly.

A few places use a filename as the template to make a temp file.
While that's nice in that the temp file name can be easily related back to
the original filename, it could lead to `git annex add` failing to add a
filename that was at or close to the maximum length.

Note that in Command.Add.lockdown, the template is still derived from the
filename, just with enough space left to turn it into a temp file.
This is an important optimisation, because the assistant may lock down
a bunch of files all at once, and using the same template for all of them
would cause openTempFile to iterate through the same set of names,
looking for an unused temp file. I'm not very happy with the relatedTemplate
hack, but it avoids that slowdown.

Backend.WORM does not limit the filename stored in the key.
I have not tried to change that; so git annex add will fail on really long
filenames when using the WORM backend. It seems better to preserve the
invariant that a WORM key always contains the complete filename, since
the filename is the only unique material in the key, other than mtime and
size. Since nobody has complained about add failing (I think I saw it
once?) on WORM, probably it's ok, or nobody but me uses it.

There may be compatability problems if using git annex addurl --fast
or the WORM backend on a system with the 255 limit and then trying to use
that repo in a system with a smaller limit. I have not tried to deal with
those.

This commit was sponsored by Alexander Brem. Thanks!
2013-07-30 19:18:29 -04:00
Joey Hess
147a9f8882 Improve test suite on Windows; now tests git annex sync. 2013-07-30 17:03:32 -04:00
Joey Hess
7b0970b340 Fix inverted logic in last release's fix for data loss bug, that caused git-annex sync on FAT or other crippled filesystems to add symlink standin files to the annex. 2013-07-30 16:08:09 -04:00
Joey Hess
f55dc1b64f OSX: Make git-annex-webapp run in the background, so that the app icon can be clicked on the open a new webapp when the assistant is already running. 2013-07-30 15:04:31 -04:00
Joey Hess
10f297d989 typo 2013-07-28 17:04:46 -04:00
Joey Hess
7e66d260ea importfeed: git-annex becomes a podcatcher in 150 LOC 2013-07-28 16:55:42 -04:00
Joey Hess
869c638b82 assistant: Fix bug that caused it to stall when adding a very large number of files at once (around 5 thousand).
This bug was introduced in 82a6db8fe8,
which improved handling of adding very large numbers of files by ensuring
that a minimum number of max size commits (5000 files each) were done.

I accidentially made it wait for another change to appear after such a max
size commit, even if a lot of queued changes were already accumulated.
That resulted in a stall when it got to the end. Now fixed to not wait
any longer than necessary to ensure the watcher has had time to wake back
up after the max size commit.

This commit was sponsored by Michael Linksvayer. Thanks!
2013-07-27 17:42:18 -04:00
Joey Hess
ec4d974dcf assistant: Fix deadlock that could occur when adding a lot of files at once in indirect mode.
This is a laziness problem. Despite the bang pattern on newfiles, the list
was not being fully evaluated before cleanup was called. Moving cleanup out
to after the list is actually used fixes this.

More evidence that I should be using ResourceT or pipes, if any was needed.
2013-07-26 18:42:22 -04:00
Joey Hess
97f3aecb17 assistant: Fix NetWatcher to not sync with remotes that have remote.<name>.annex-sync set to false.
This affected both the hourly NetWatcherFallback thread and the syncing
when network connection is detected.

It was a reversion of sorts, introduced in
8861e270be, when annex-ignore was changed to
not control git syncing. I forgot to make it check annex-sync at that
point.
2013-07-26 16:54:20 -04:00
Joey Hess
2311f30e4c correct changelog; git annex get --unused doesn't make sense 2013-07-25 19:53:52 -04:00
Joey Hess
c6100aa5cc unused: No longer shows as unused tmp files that are actively being transferred. 2013-07-25 19:51:08 -04:00
Joey Hess
822918089e dropunused behavior change: Now refuses to drop the last copy of a file, unless you use the --force.
This was the last place in git-annex that could remove data referred to by
the git history, without being forced.

Like drop, dropunused checks remotes, and honors the global annex.numcopies
setting. (However, .gitattributes settings cannot apply to unused files.)
2013-07-25 19:50:44 -04:00
Joey Hess
fc1a79835b Always build with -threaded, to avoid a deadlock when communicating with gpg. 2013-07-25 13:57:53 -04:00
Joey Hess
5c82c99c76 webapp: When creating a repository on a removable drive, set core.fsyncobjectfiles, to help prevent data loss when the drive is yanked. 2013-07-23 13:38:05 -04:00
Joey Hess
381637e4c8 Add status message to XMPP presence tag, to identify to others that the client is a git-annex client.
I only added this to the presense messages that are really intended for
presence. The ones used for tunneling git etc don't have the tag, because
that would waste bandwidth.
2013-07-23 12:41:41 -04:00
Joey Hess
ba4e0b4878 releasing version 4.20130723 2013-07-23 11:41:16 -04:00
Joey Hess
5e3a404d4f Support import in direct mode. 2013-07-22 20:18:00 -04:00
Joey Hess
f353f13c9d Support unannex and uninit in direct mode.
In direct mode, it's best to whenever possible not move direct mode files
out of the way, and so I made unannex avoid touching the direct mode file at
all.

That actually turns out to be easy, because in direct mode, unlike indirect
mode, the pre-commit hook won't get confused if the unannexed file later
gets added back by git add. So there's no need to commit the unannex right
away; it can be staged for the user to commit later. This also means that
unannex in direct mode is a lot faster than in indirect mode!

Another subtle bit is the bookkeeping that is done when unannexing a direct
mode file. The inode cache needs to be removed so that when uninit runs
getKeysPresent, it doesn't see the cache and think the key is still
present and crash when it's not.

This commit is sponsored by Douglas Butts. Thanks!
2013-07-22 17:28:53 -04:00
Joey Hess
6ae2637eb1 For long hostnames, use a hash of the hostname to generate the socket file for ssh connection caching.
This is ok to do now that the socket filename never needs to be mapped back
to a hostname.

Short hostnames will still appear in the clear, which is less obfuscated.
So this cannot possibly make ssh connection caching fail for a hostname it
used to work for.
2013-07-22 15:09:41 -04:00
Joey Hess
002de3e547 point to android icons too 2013-07-21 13:28:35 -04:00
Joey Hess
9fb5eaa179 ref debian bug 2013-07-20 21:32:52 -04:00
Joey Hess
780efc775c When an XMPP server has SRV records, try them, but don't then fall back to the regular host if they all fail.
gmail.com has some XMPP SRV records, but does not itself respond to XMPP
traffic, although it does accept connections on port 5222. So if a user
entered the wrong password, it would try all the SRVs and fall back to
trying gmail, and hang at that point.

This seems the right thing to do, not just a workaround.
2013-07-20 21:18:55 -04:00
Joey Hess
6f526518eb fixed nasty data loss bug
I wanted to try to guard against it in Command.Add too, but it's a case of
garbage in, garbage out. Once Command.Add has been told it's dealing with a
dummy symlink, it goes and deletes it, and even though the object it
thinks it points to is not present in the annex, it's Command.Add is still
doing the right thing to go ahead and add the broken symlink. So the two
fixes I was able to put in will have to do.
2013-07-20 19:37:12 -04:00
Joey Hess
9fc1448947 webapp: Differentiate between creating a new S3/Glacier/WebDav remote, and initializing an existing remote. When creating a new remote, avoid conflicts with other existing (or deleted) remotes with the same name. 2013-07-20 18:15:16 -04:00
Joey Hess
ca9ac8770f directory special remote: Fix checking that there is enough disk space to hold an object, was broken when using encryption. 2013-07-20 16:30:49 -04:00
Joey Hess
6cc5b5d604 changelog for 8b644e74fa now that I understand it better 2013-07-20 14:34:35 -04:00
Joey Hess
432569c4b9 New improved version of the git-annex logo, contributed by John Lawrence. 2013-07-20 13:16:47 -04:00
Joey Hess
647a938f6a Display byte sizes with more precision. 2013-07-19 12:21:44 -04:00
Joey Hess
d2f40d3d76 Fix checking when content is present in a non-bare repository accessed via http.
I thought at first this was a Windows specific problem, but it's not;
this affects checking any non-bare repository exported via http. Which is
a potentially important use case!

The actual bug was the case where Right False was returned by the first url
short-curcuited later checks. But the whole method used felt like code
I'd no longer write, and the use of undefined was particularly disgusting.
So I rewrote it.

Also added an action display.

This commit was sponsored by Eric Hanchrow. Thanks!
2013-07-18 14:20:57 -04:00
Joey Hess
7afd92d083 When a transfer is already being run by another process, proceed on to the next file, rather than dying. 2013-07-17 15:54:01 -04:00
Joey Hess
1d7d3ac325 uninit: Preserve .git/annex/objects at the end, if it still has content, so that old versions of files and deleted files are not deleted. Print a message with some suggested actions. 2013-07-16 15:00:25 -04:00
Joey Hess
6402f2081e Revert "avoid pulling in unneeded dependencies when the assistant is disabled"
Cabal does not seem to have a way to check if flag A is set and then, if
flag B is set, add a dep. Instead, it makes flag B get unset if the
dep is not available.
2013-07-16 11:29:43 -04:00
Joey Hess
4b9fa37b72 avoid pulling in unneeded dependencies when the assistant is disabled 2013-07-12 15:45:34 -04:00
Joey Hess
988692e145 update webapp for rsync.net discount
Note that've told me:

We'll see how it goes, but I think this could be a permanent offer for
your userbase.  People using git-annex are clueful and won't be a big
support burden for us, so it's a win-win.
2013-07-11 14:15:09 -04:00
Joey Hess
c936384164 fix: Preserve the original mtime of fixed symlinks. 2013-07-11 11:39:42 -04:00
Joey Hess
dba1e29949 webapp: Better display of added files. 2013-07-10 15:37:40 -04:00
Joey Hess
19b8bcbe30 Install XDG desktop icon files.
The icon files will be installed when running make install or cabal
install. Did not try to run update-icon-caches, since I think it's debian
specific, and dh_icons will take care of that for the Debian package.

Using the favicon as a 16x16 icon. At 24x24 the svg displays pretty well,
although the dotted lines are rather faint. The svg is ok at all higher
resolutions.

The standalone linux build auto-installs the desktop and autostart files
when run. I have not made it auto-install the icon file too, because
a) that would take more work to include them in the tarball and find them
b) it would need to be an install to ~/.icons/, and I don't know if that
   really works!
2013-07-09 19:56:30 -04:00
Joey Hess
a2269eef9e releasing version 4.20130709 2013-07-09 15:58:01 -04:00
Joey Hess
b3db88252c Make --numcopies override annex.numcopies set in .gitattributes. 2013-07-09 12:05:56 -04:00
Joey Hess
c1307b1388 fsck: Don't claim to fix direct mode when run on a symlink whose content is not present. 2013-07-08 17:29:42 -04:00
Joey Hess
207c9f3c4a dropunused, addunused: Complain when asked to operate on a number that does not correspond to any unused key. 2013-07-08 16:47:34 -04:00
Joey Hess
00f52d9e53 Android: gpg is built without --enable-minimal, so it interoperates better with other gpg builds that may default to using other algorithms for encryption. 2013-07-08 16:00:40 -04:00
Joey Hess
4657c980fb OSX Mountain Lion: Fixed gpg bundled in dmg to not fail due to a missing gpg-agent. 2013-07-08 13:16:12 -04:00
Joey Hess
57f5c9f41a webapp: Fix authorized_keys line added when setting up a rsync remote on a server that also supports git-annex, to not force running git-annex-shell. 2013-07-08 12:51:07 -04:00
Joey Hess
7e7b2daddf Windows: Fix url to object when using a http remote.
annexLocations uses OS-native directory separators, but for an url,
it needs to use / even on Windows.

This is an ugly workaround. Could parameterize a lot of stuff in
annexLocations to fix it better. I suspect this is probably the only place
it's needed though.
2013-07-07 13:35:56 -04:00
Joey Hess
79e1a0c571 Pass -f to curl when downloading a file with it, so it propigates failure. 2013-07-06 00:55:00 -04:00
Joey Hess
86b845ff8d Windows: Look for .exe extension when searching for a command in path. 2013-07-06 00:48:47 -04:00
Joey Hess
74ad3072e4 addurl --pathdepth: Fix failure when the pathdepth specified is deeper than the urls's path. 2013-07-05 12:46:38 -04:00
Joey Hess
d84a000e92 detect system with no dot in FQDN, where git commit will fail, and workaround
Sigh, git is so *fragile*. Or rather, across the set of systems that use
git-annex, where are no many horribly broken systems..
2013-07-05 12:24:28 -04:00
Joey Hess
980e9a15e0 merge: Now also merges synced/master or similar branches, which makes it useful to put in a post-receive hook to make a repository automatically update its working copy when git annex sync or the assistant sync with it. 2013-07-03 15:42:56 -04:00
Joey Hess
04d07f2c1f --unused: New switch that makes git-annex operate on all data found by the last run of git annex unused. Supported by fsck, get, move, copy. 2013-07-03 15:26:59 -04:00
Joey Hess
c0a5b669ea no migrate --all for now
migrate wants to know the associated filename, in order to look up
the new backend. Can't do that with --all

migrate --all --backend=newvalue could be useful to support, in the future.
2013-07-03 14:07:09 -04:00
Joey Hess
ebfd6fc2fe drop --all cannot check numcopies from .gitattributes, so don't implement it!
I spent a long time worrying about this problem with --all, that it cannot
check .gitattributes files for numcopies settings, and so would not be
entirely safe to use. The solution turns out to be simple, just don't
implement `git annex drop --all`. drop is the only command that needs to
check numcopies (move can also reduce the number of copies, but explicitly
bypasses numcopies settings).

Use cases that might need a drop --all are probably better served by using
unused and dropunused, which already work in a bare repository.
2013-07-03 14:01:31 -04:00
Joey Hess
def7cb706f Add --all option, and support it for fsck 2013-07-03 13:12:53 -04:00
Joey Hess
1b395dcb76 webapp: Fix ssh setup with nonstandard port, broken in last release. 2013-07-02 15:46:02 -04:00
Joey Hess
0292e82eb1 releasing version 4.20130627 2013-06-27 14:58:40 -04:00
Joey Hess
b44c978e2c webapp: Fix bug that caused the webapp to hang when built with yesod 1.2. 2013-06-27 00:01:31 -04:00
Joey Hess
b717823402 Temporarily revert back to building with yesod before 1.2. Version 1.2 of yesod, or the new version of warp seem to have a bug that causes the webapp to hang. 2013-06-26 17:52:39 -04:00
Joey Hess
579446aed4 assistant: Fix bug that prevented adding files written by gnucash, and more generally support adding hard links to files. However, other operations on hard links are still unsupported. 2013-06-26 12:30:37 -04:00
Joey Hess
f1fe8edbbb webapp: Fix bug setting up ssh repo if the user enters "~/" at the start of the path. 2013-06-25 15:43:46 -04:00
Joey Hess
c8e6947693 webapp: Ensure that ssh keys generated for different directories on a server are always different. 2013-06-25 15:31:04 -04:00
Joey Hess
10072157cb webapp: Fix bug when setting up a remote ssh repo repeatedly on the same server.
The ssh setup first runs ssh to the real hostname, to probe if a ssh key is
needed. If one is, it generates a mangled hostname that uses a key. This
mangled hostname was being used to ssh into the server to set up the key.
But if the server already had the key set up, and it was locked down, the
setup would fail. This changes it to use the real hostname when sshing in
to set up the key, which avoids the problem.

Note that it will redundantly set up the key on the ssh server. But it's
the same key; the ssh key generation code uses the key if it already
exists.
2013-06-25 14:46:20 -04:00
Joey Hess
a35bdcb3f2 fsck: Ensures that direct mode is used for files when it's enabled.
A common failure mode for direct mode has been for files to end up still
stored in indirect mode. While I hope that doesn't happen anymore, fsck
should deal with it.
2013-06-24 16:26:00 -04:00
Joey Hess
1f7f9ede3a bup: Handle /~/ in bup remote paths. Thanks, Oliver Matthews 2013-06-23 12:39:37 -04:00
Joey Hess
5e48aa4d4b assistant: Daily sanity check thread is run niced. 2013-06-21 13:29:42 -04:00
Joey Hess
d901ba1781 assistant --autostart: Automatically ionices the daemons it starts. 2013-06-21 13:23:20 -04:00
Joey Hess
62ef805a54 releasing version 4.20130621 2013-06-21 12:12:43 -04:00
Joey Hess
e95e9e2020 Enable assistant and WebDAV support on powerpc and sparc architectures, which now have the necessary dependencies built. 2013-06-21 01:50:00 -04:00
Joey Hess
6e309b63f8 assistant: On Linux, the expensive transfer scan is run niced.
This is a compromise. I would like to nice every thread except for the
webapp thread, but it's not practical to do so. That would need every
thread to run as a bound thread, which could add significant overhead.
And any forkIO would escape the nice level.
2013-06-20 22:25:41 -04:00
Joey Hess
8b3427c52e Windows: The test suite now passes on Windows (a few broken parts are disabled).
Better to have a working test suite that doesn't test a few things
than no working test suite.

Most of the disabled stuff is because for some reason "git annex sync"
doesn't work when run inside the test suite. Looks like PATH problems.

The directory and rsync special remotes seem broken on Windows, or
maybe the tests are. Pretty sure the hook special remote test is broken.
2013-06-18 16:16:25 -04:00
Joey Hess
4e2fab90d5 avoid newline translation when writing to git hash-object
They're like mushrooms, just keep popping up.
2013-06-18 15:08:51 -04:00
Joey Hess
1312cffad0 Revert "Windows: Ssh connection caching is now supported."
Yeah, that didn't actually work. Got error messages like it couldn't read
from the control socket, so probably ssh doesn't really support that on
Windows, at least the cygwin ssh build I'm using.
2013-06-17 22:13:28 -04:00
Joey Hess
07a17f58b7 Windows: Ssh connection caching is now supported.
Turns out the socket stuff just works on windows.
2013-06-17 22:05:49 -04:00
Joey Hess
9ef09587dc fsck: Avoid getting confused by Windows path separators 2013-06-17 21:18:43 -04:00
Joey Hess
fa1c1e0f65 annex.debug can now be set to enable debug logging by default. The webapp's debugging check box does this. 2013-06-17 20:41:27 -04:00
Joey Hess
9200e9a5f9 update standards-version 2013-06-17 13:59:17 -04:00