Commit graph

465 commits

Author SHA1 Message Date
Joey Hess
070ed4a766 change a few renameFile's to rename
AFAIK, none of these ever operate on directories, but nor do I want to
explicitly check if they're files and fail if not.
2014-01-29 15:21:02 -04:00
Joey Hess
891c85cd88 use locking on Windows
This is all the easy cases, where there was already a separate lock file.
2014-01-28 14:42:03 -04:00
Joey Hess
74b101d1dd reorg 2014-01-26 16:36:31 -04:00
Joey Hess
1ca111620d reorg 2014-01-26 16:32:55 -04:00
Joey Hess
5fc2d760ea Optimise non-bare http remotes; no longer does a 404 to the wrong url every time before trying the right url. Needs annex-bare to be set to false, which is done when initially probing the uuid of a http remote. 2014-01-26 13:03:25 -04:00
Joey Hess
b40df4f0d0 reorganize numcopies code (no behavior changes)
Move stuff into Logs.NumCopies. Add a NumCopies newtype.

Better names for various serialization classes that are specific to one
thing or another.
2014-01-21 16:08:59 -04:00
Joey Hess
b6ba0bd556 sync --content: New option that makes the content of annexed files be transferred.
Similar to the assistant, this honors any configured preferred content
expressions.

I am not entirely happpy with the implementation. It would be nicer if
the seek function returned a list of actions which included the individual
file gets and copies and drops, rather than the current list of calls to
syncContent. This would allow getting rid of the somewhat reundant display
of "sync file [ok|failed]" after the get/put display.

But, do that, withFilesInGit would need to somehow be able to construct
such a mixed action list. And it would be less efficient than the current
implementation, which is able to reuse several values between eg get and
drop.

Note that currently this does not try to satisfy numcopies when
getting/putting files (numcopies are of course checked when dropping
files!) This makes it like the assistant, and unlike get --auto
and copy --auto, which do duplicate files when numcopies is not yet
satisfied. I don't know if this is the right decision; it only seemed to
make sense to have this parallel the assistant as far as possible to start
with, since I know the assistant works.

This commit was sponsored by Øyvind Andersen Holm.
2014-01-19 17:49:54 -04:00
Joey Hess
0d544649d0 catch exception checking if url exists when network is disconnected
Leads to better failure message (or possibly fallback to another remote).
2014-01-16 21:24:17 -04:00
Joey Hess
207ac67aaa avoid needing a build-dep on hxt for Data.AssocList 2014-01-14 16:42:10 -04:00
Joey Hess
d07f2d7865 Fix a long-standing bug that could cause the wrong index file to be used when committing to the git-annex branch, if GIT_INDEX_FILE is set in the environment. This typically resulted in git-annex branch log files being committed to the master branch and later showing up in the work tree. (These log files can be safely removed.) 2014-01-14 15:36:33 -04:00
Joey Hess
c20f31a1ad add GETAVAILABILITY to external special remote protocol
And some reworking of types, and added an annex-availability git config
setting.
2014-01-13 14:41:10 -04:00
Joey Hess
57edce8ad9 external special remote protocol: Added GETGITDIR. 2014-01-13 14:00:09 -04:00
Joey Hess
d8fb366cf7 forgot to delay for 1 second in busy wait loop 2014-01-08 19:58:47 -04:00
Joey Hess
215ea66471 only run tahoe start once 2014-01-08 19:17:18 -04:00
Joey Hess
93161d0dea copyright year 2014-01-08 16:29:15 -04:00
Joey Hess
85272d8a98 Added tahoe special remote.
Known problems:

1. Tries to tahoe start when daemon is already running.

2. If multiple tahoe remotes are set up on the same computer,
   they will have the same node.url configured by default,
   and this confuses tahoe commands.

This commit was sponsored by LeastAuthority.com
2014-01-08 16:14:41 -04:00
Joey Hess
5e23dfabd6 add DEBUG 2014-01-07 13:23:58 -04:00
Joey Hess
614986b19a show PATH on failure 2014-01-07 12:59:26 -04:00
Joey Hess
3e68c1c2fd add remote state logs
This allows a remote to store a piece of arbitrary state associated with a
key. This is needed to support Tahoe, where the file-cap is calculated from
the data stored in it, and used to retrieve a key later. Glacier also would
be much improved by using this.

GETSTATE and SETSTATE are added to the external special remote protocol.

Note that the state is left as-is even when a key is removed from a remote.
It's up to the remote to decide when it wants to clear the state.

The remote state log, $KEY.log.rmt, is a UUID-based log. However,
rather than using the old UUID-based log format, I created a new variant
of that format. The new varient is more space efficient (since it lacks the
"timestamp=" hack, and easier to parse (and the parser doesn't mess with
whitespace in the value), and avoids compatability cruft in the old one.

This seemed worth cleaning up for these new files, since there could be a
lot of them, while before UUID-based logs were only used for a few log
files at the top of the git-annex branch. The transition code has also
been updated to handle these new UUID-based logs.

This commit was sponsored by Daniel Hofer.
2014-01-03 16:35:57 -04:00
Joey Hess
f7727d2df1 Remotes can now be made read-only, by setting remote.<name>.annex-readonly 2014-01-02 13:12:32 -04:00
Joey Hess
8e3032df2d added GETWANTED, SETWANTED for Tobias's flickr remote
This was unexpectedly difficult because of a depdenency cycle. To parse a
preferred content expression involves several things that need to operate
on the list of remotes. Which needs Remote.External. The only way to avoid
this cycle (I tried breaking it at several points) was to skip parsing the
expression in SETWANTED.

That's sorta ok, because git-annex already has to deal with unparsable
preferred content expressions being stored, in order to handle eg,
upgrades. But I'm still not very happy that I cannot check it.

I feel this is a strong indication that I need to beware of further
bloating the special remote protocol interface.
2014-01-01 20:12:20 -04:00
Joey Hess
ed1fcab6d7 external special remote protocol: Added GETUUID. 2013-12-31 13:50:18 -04:00
Joey Hess
054e4f17e2 implement PREPARE-FAILURE for Tobias 2013-12-29 13:39:25 -04:00
Joey Hess
aa97a33dde better error messages when external special remote exits unexpectedly or is not in PATH 2013-12-27 17:14:44 -04:00
Joey Hess
445b7b41b9 add credential storage support for external special remotes & update example 2013-12-27 16:01:43 -04:00
Joey Hess
551573570f better protocol error message, indicate if the command was able to be parsed or was misplaced 2013-12-27 14:03:35 -04:00
Joey Hess
21342cae63 flush handle after writing message 2013-12-27 13:22:06 -04:00
Joey Hess
fa6f404a5f fix deadlock when state TMVar is empty 2013-12-27 13:17:22 -04:00
Joey Hess
9125a25738 defer SETSTATE and GETSTATE for now
TAHOE-LAFS may use these eventually, but that's TBD and none of git-annex's
own special remotes need that, except for the web special remote's urls.
2013-12-27 13:07:56 -04:00
Joey Hess
a7f3724e21 implement GETCONFIG and SETCONFIG
Changed protocol spec to make SETCONFIG only store it persistently when run
during INITREMOTE. I see no reason to support storing it persistently at
other times, and doing so would unnecessarily complicate the code.

Also, letting that be done would probably result in use for storing data that
doesn't really belong there, and special remote authors who don't
understand how the union merging works would probably be surprised the
results.
2013-12-27 12:37:23 -04:00
Joey Hess
91c9e98168 support encryption 2013-12-27 12:21:55 -04:00
Joey Hess
5d8ff64dc1 make --debug show transcript of special remote protocol messages 2013-12-27 03:10:00 -04:00
Joey Hess
3289155e28 don't send PREPARE before INITREMOTE
That complicated special remote programs, because they had to avoid making
PREPARE fail if some configuration is missing, because the remote might not
be initialized yet. Instead, complicate git-annex slightly by only sending
PREPARE immediately before some other request other than INITREMOTE (or
PREPARE of course).
2013-12-27 02:49:10 -04:00
Joey Hess
6d504b57e7 make some requests optional, simplify and future-proof protocol more 2013-12-27 02:11:06 -04:00
Joey Hess
6c565ec905 external special remotes mostly implemented (untested)
This has not been tested at all. It compiles!

The only known missing things are support for encryption, and for get/set
of special remote configuration, and of key state. (The latter needs
separate work to add a new per-key log file to store that state.)

Only thing I don't much like is that initremote needs to be passed both
type=external and externaltype=foo. It would be better to have just
type=foo

Most of this is quite straightforward code, that largely wrote itself given
the types. The only tricky parts were:

* Need to lock the remote when using it to eg make a request, because
  in theory git-annex could have multiple threads that each try to use
  a remote at the same time. I don't think that git-annex ever does
  that currently, but better safe than sorry.

* Rather than starting up every external special remote program when
  git-annex starts, they are started only on demand, when first used.
  This will avoid slowdown, especially when running fast git-annex query
  commands. Once started, they keep running until git-annex stops, currently,
  which may not be ideal, but it's hard to know a better time to stop them.

* Bit of a chicken and egg problem with caching the cost of the remote,
  because setting annex-cost in the git config needs the remote to already
  be set up. Managed to finesse that.

This commit was sponsored by Lukas Anzinger.
2013-12-26 18:23:13 -04:00
Joey Hess
8803e36814 future-proofing 2013-12-25 20:04:31 -04:00
Joey Hess
1dc930063a basic data types and serialization for external special remote protocol
This is mostly straightforward, but did turn out quite nicely stronly
typed, and with a quite nice automatic tokenization and parsing of received
messages.

Made a few minor changes to the protocol to clear up ambiguities and make
it easier to parse. Note particularly that setting remote configuration
is moved to a separate command, which allows a remote to set arbitrary data.
2013-12-25 17:54:57 -04:00
Joey Hess
011b8bc7ec pull in Win32-extras, to be able to get current process id in Windows
Fixed up a number of things that had worked around there not being a way to
get that.

Most notably, transfer info files on windows now include the process id,
since no locking is currently done. This means the file format varies
between windows and unix.
2013-12-11 00:15:10 -04:00
Joey Hess
e425a966ed Deal with box.com changing the url of their webdav endpoint.
Use new url when making new remotes.

Transparently rewrite old url to new for existing remotes.
2013-12-02 16:01:20 -04:00
Joey Hess
0a63ed563f rsync special remote: Fix fallback mode for rsync remotes that use hashDirMixed. Closes: #731142 2013-12-02 12:53:39 -04:00
Joey Hess
58db042033 map: Work when there are gcrypt remotes. 2013-11-04 14:14:44 -04:00
Joey Hess
2203690822 really fix gcrypt for 7be69a2491
Fixed all the other ones, but forgot to fix gcrypt!
2013-11-02 20:10:54 -04:00
Joey Hess
b2cca95d1c clean import list 2013-11-02 19:55:18 -04:00
Joey Hess
a04fe350b8 fix build 2013-11-02 19:54:59 -04:00
Joey Hess
7be69a2491 gcrypt, bup: Fix bug that prevented using these special remotes with encryption=pubkey.
I think both of these are all that's affected, but I went ahead and fixed
all the remotes that set their config to M.empty to instead store the
actual config. Who knows what will expect it to be actually present in
future, the Remote instance of getGpgEncParams came to..
2013-11-02 16:37:28 -04:00
Joey Hess
7ed8e87a34 assistant: Support repairing git remotes that are locally accessible
(eg, on removable drives)

gcrypt remotes are not yet handled.

This commit was sponsored by Sören Brunk.
2013-10-27 15:38:59 -04:00
Joey Hess
5756636486 directory, webdav: Fix bug introduced in version 4.20131002 that caused the chunkcount file to not be written. Work around repositories without such a file, so files can still be retreived from them. 2013-10-26 15:03:12 -04:00
Joey Hess
06ea92282f fix inverted logic when determining whether to write a chunkcount file
late-night hlint bit me on this one..
Reviewed c1990702e9 and
the rest of it seems ok
2013-10-26 14:08:29 -04:00
Joey Hess
c76c94a0da S3: Try to ensure bucket name is valid for archive.org. 2013-10-16 16:35:47 -04:00
Joey Hess
a6e9386d39 fix remote fsck to run in remote 2013-10-14 15:05:29 -04:00