Commit graph

308 commits

Author SHA1 Message Date
Joey Hess
3e839ab327
temporary hack to get windows build working
Code for terminating processes on Windows is not linking anymore;
made a warning be displayed instead. This breaks restarting the
assistant and git annex assistant --stop.

I hope to see the code added to the Win32 library, where it should fit
better and should avoid whatever problem is making the linker not like it
when included in git-annex. I opened an issue requesting its addition,
here: https://github.com/haskell/win32/issues/91

This commit was sponsored by Thomas Hochstein on Patreon.
2017-10-24 13:16:40 -04:00
Joey Hess
bb20b1ed35
releasing package git-annex version 6.20171018 2017-10-18 15:41:08 -04:00
Joey Hess
e0966bbb16
reorder 2017-10-18 13:36:43 -04:00
Joey Hess
0c1c55efea
changelog 2017-10-18 13:34:53 -04:00
Joey Hess
e1ac299ad0
better dup key with -J fix
This avoids all the complication about redundant work discussed in
the previous try at fixing this. At the expense of needing each command
that could have the problem to be patched to simply wrap the action in
onlyActionOn once the key is known. But there do not seem to be many
such commands.

onlyActionOn' should not be used with a CommandStart (or CommandPerform),
although the types do allow it. onlyActionOn handles running the whole
CommandStart chain. I couldn't immediately see a way to avoid mistken
use of onlyActionOn'.

This commit was supported by the NSF-funded DataLad project.
2017-10-17 18:48:53 -04:00
Joey Hess
68a49adcda
Improve behavior when -J transfers multiple files that point to the same key
After a false start, I found a fairly non-intrusive way to deal with it.
Although it only handles transfers -- there may be issues with eg
concurrent dropping of the same key, or other operations.

There is no added overhead when -J is not used, other than an added
inAnnex check. When -J is used, it has to maintain and check a small
Set, which should be negligible overhead.

It could output some message saying that the transfer is being done by
another thread. Or it could even display the same progress info for both
files that are being downloaded since they have the same content. But I
opted to keep it simple, since this is rather an edge case, so it just
doesn't say anything about the transfer of the file until the other
thread finishes.

Since the deferred transfer action still runs, actions that do more than
transfer content will still get a chance to do their other work. (An
example of something that needs to do such other work is P2P.Annex,
where the download always needs to receive the content from the peer.)
And, if the first thread fails to complete a transfer, the second thread
can resume it.

But, this unfortunately means that there's a risk of redundant work
being done to transfer a key that just got transferred.
That's not ideal, but should never cause breakage; the same
thing can occur when running two separate git-annex processes.

The get/move/copy/mirror --from commands had extra inAnnex checks added,
inside the download actions. Without those checks, the first thread
downloaded the content, and then the second thread woke up and
downloaded the same content redundantly.

move/copy/mirror --to is left doing redundant uploads for now. It
would need a second checkPresent of the remote inside the upload
to avoid them, which would be expensive. A better way to avoid
redundant work needs to be found..

This commit was supported by the NSF-funded DataLad project.
2017-10-17 17:10:50 -04:00
Joey Hess
193b5f4ca4
new lts 2017-10-17 13:00:22 -04:00
Joey Hess
0c7fee832a
close bug 2017-10-16 14:50:30 -04:00
Joey Hess
85ed38a574
Avoid repeated checking that files passed on the command line exist.
git annex add, git annex lock etc make multiple seek passes,
and each seek pass checked that files existed. That was unncessary
redundant work.

Fixed by adding a new WorkTreeItem type, make seek actions use it,
and check that the files exist when constructing it.

This commit was supported by the NSF-funded DataLad project.
2017-10-16 14:10:20 -04:00
Joey Hess
a461cf2ce6
add: Replace work tree file atomically.
Before, there was a window where interrupting an add could result in the
file being moved into the annex, with no symlink yet created.

This commit was supported by the NSF-funded DataLad project.
2017-10-16 12:57:28 -04:00
Joey Hess
9aaf7e2b52
webdav: Avoid unncessisarily creating the collection at the top of the repo
when storing files there, since that collection is created by initremote.
(This seems to work around some brokenness of the box.com webdav server
which was entering a redirect loop.)

Note that the fix makes locationParent return Nothing instead of "."
when there's no parent directory between the path and the top of the webdav
repo.

This commit was sponsored by André Pereira on Patreon.
2017-10-11 11:10:33 -04:00
Joey Hess
61dccecad7
Fix build with aws-0.17.
This commit was sponsored by Denis Dzyubenko on Patreon.
2017-10-11 10:57:20 -04:00
Joey Hess
34bb350724
webdav: Make --debug show all webdav operations. 2017-10-07 14:11:32 -04:00
Joey Hess
e7a9ed330f
releasing package git-annex version 6.20171003 2017-10-03 13:18:47 -04:00
Joey Hess
3cd47f9978
info: Improve cleanup of stale transfer info files.
In my git-annex repos, I found some stale transfer info files
without lock files.

Pass a mode to tryLockExclusive, so it will create the lock file if
not present, and so not fail to clean up such transfer info files.

Normally, transfer info files are accompanied by a lock file.
But, when alwaysRunTransfer is used, the locking can fail
and it will still write the transfer info file. Perhaps there are other
cases too? Note that mkProgressUpdater's meter
writes to the transfer info file too, and it might be possible for
that meter to fire after runTransfer has cleaned up.

This commit was sponsored by andrea rota.
2017-10-02 13:55:26 -04:00
Joey Hess
5c32196a37
fix process and FD leak
Fix process and file descriptor leak that was exposed when git-annex was
built with ghc 8.2.1. Apparently ghc has changed its behavior of GC
of open file handles that are pipes to running processes. That
broke git-annex test on OSX due to running out of FDs.

Audited for all uses of Annex.new and made stopCoProcesses be called
once it's done with the state. Fixed several places that might have
leaked in other situations than running the test suite.

This commit was sponsored by Ewen McNeill.
2017-09-29 22:36:08 -04:00
Joey Hess
f403c23bc6
copy, move: Behave same with --fast when sending to remotes located on a local disk as when sending to other remotes.
Let --fast override use of hasKey even when hasKeyCheap.
2017-09-29 16:30:43 -04:00
Joey Hess
f84e34883c
test: Fix reversion that made it only run inside a git repository.
Using annexeval to run probeCrippledFileSystem' caused Git.CurrentRepo.get
to be run. Fixed easily since probeCrippledFileSystem' had no need to use
the Annex monad.

This commit was sponsored by Ethan Aubin.
2017-09-29 15:08:18 -04:00
Joey Hess
e9e5613e94
external crash fixes
When the external special remote program crashed, a newline
could be output, which messed up the expected output for --batch mode.

Avoid checking EXPORTSUPPORTED for special remotes that are
not configured to use exports. The datalad special remote apparently is/was
buggy and crashed on EXPORTSUPPORTED. Anyway, there's no need to send
it when the configuration doesn't need it.

This commit was supported by the NSF-funded DataLad project.
2017-09-28 15:44:45 -04:00
Joey Hess
e8c9a5c515
sync: Added --cleanup, which removes local and remote synced/ branches.
Also deletes any tagged pushes that the assistant might have done,
since those would also prevent resetting a branch back.

This commit was sponsored by andrea rota.
2017-09-28 14:58:48 -04:00
Joey Hess
4d0e522b72
Warn when metadata is inherited from a previous version of a file
to avoid the user being surprised in cases where that behavior is not desired or expected

This commit was supported by the NSF-funded DataLad project.
2017-09-28 12:56:35 -04:00
Joey Hess
812d90022b
metadata: Added --remove-all.
Motivation is to remove all metadata when it gets copied from a previous
version of the file, and that is not deisrable.

This commit was supported by the NSF-funded DataLad project.
2017-09-28 12:36:10 -04:00
Joey Hess
f4746da4ca
webdav: Improve error message for failed request to include the request method and path. 2017-09-28 12:01:58 -04:00
Joey Hess
76c9b580b3
releasing package git-annex version 6.20170925 2017-09-25 11:34:55 -04:00
Joey Hess
d71c65ca0a
add exporter thread to assistant
This is similar to the pusher thread, but a separate thread because git
pushes can be done in parallel with exports, and updating a big export
should not prevent other git pushes going out in the meantime.

The exportThread only runs at most every 30 seconds, since updating an
export is more expensive than pushing. This may need to be tuned.

Added a separate channel for export commits; the committer records a
commit in that channel.

Also, reconnectRemotes records a dummy commit, to make the exporter
thread wake up and make sure all exports are up-to-date. So,
connecting a drive with a directory special remote export will
immediately update it, and getting online will automatically
update S3 and WebDAV exports.

The transfer queue is not involved in exports. Instead, failed
exports are retried much like failed pushes.

This commit was sponsored by Ewen McNeill.
2017-09-20 15:29:13 -04:00
Joey Hess
cf51f40f0e
webdav: Changed path used on webdav server for temporary files.
Done to avoid a "tmp" directory appearing in webdav exports.

Also affects non-export webdav remotes, so interrupted uploads using the
old path will not overwrite it. However, PUT is quite likely to be
implemented atomically on web servers anyway, so I doubt this will cause
problems.
2017-09-15 15:52:31 -04:00
Joey Hess
e177bb1e25
webdav: Fix lack of url-escaping of filenames.
inDAVLocation does not url-escape, and so exporting a filename with spaces
to box.com at least resulted in a error 400.

It might also have affected storing keys on a webdav remote, if the key
contained a space or other problem character. Pretty unlikely.

I emailed Clint about the inDAVLocation gotcha, but seems best to fix it
here.

This commit was supported by the NSF-funded DataLad project.
2017-09-12 15:45:03 -04:00
Joey Hess
2ca1d3cc01
deal with box.com horrible infinite redirect behavior
webdav: Checking if a non-existent file is present on Box.com triggered a
bug in its webdav support that generates an infinite series of redirects.

It seems to redirect foo to foo/ to foo/index.php to
foo/index.php/index.php ... Why a webdav endpoint would behave this way
who knows.

Deal with such problems by assuming such behavior means the file is not
present.

Can't simply disable following redirects, because the webdav endpoint could
legitimately be redirected to a new endpoint. So, when this happens
10 redirects have to be followed, before it gives up and assumes this means
the file does not exist.

This commit was supported by the NSF-funded DataLad project.
2017-09-12 15:13:42 -04:00
Joey Hess
4d3a464e83
export to webdav
This basically works, but there's a bug when renaming a file that leaves
a .git-annex-temp-content-key file in the webdav store, that never gets
cleaned up.

Also, exporting files with spaces to box.com seems to fail; perhaps it
does not support it?

This commit was supported by the NSF-funded DataLad project.
2017-09-12 14:10:09 -04:00
Joey Hess
1332e6cec0
stop warning about removals from IA
In a test, I uploaded a pdf, and several files were derived from it.
After removing the pdf, the derived files went away after approximatly
half an hour. This window does not seem worth warning about every time.
Documented it in the tip.
2017-09-12 12:47:43 -04:00
Joey Hess
267f47c473
S3: Allow removing files from IA, but warn about derived versions potentially still existing there.
Removal works, only derives are a potential issue, so allow removing
with a warning. This way, unexporting a file works, and behavior is
consistent with IA remotes whether or not exporttree=yes.

Also tested exporting filenames containing unicode, spaces, underscores.
All worked, despite the IA's faq saying it doesn't.

This commit was sponsored by Trenton Cronholm on Patreon.
2017-09-12 12:35:58 -04:00
Joey Hess
44cd5ae313
S3 export (untested)
It opens a http connection per file exported, but then so does git
annex copy --to s3.

Decided not to munge exported filenames for IA. Too large a chance of
the munging having confusing results. Instead, export of files not
supported by IA, eg with spaces in their name, will fail.

This commit was supported by the NSF-funded DataLad project.
2017-09-08 15:46:24 -04:00
Joey Hess
a1b195d84c
External special remote protocol extended to support export.
Also updated example.sh to support export.

This commit was supported by the NSF-funded DataLad project.
2017-09-08 14:24:05 -04:00
Joey Hess
656797b4e8
update for export 2017-09-04 14:25:00 -04:00
Joey Hess
db2a06b66f
init: Display an additional message when it detects a filesystem that allows writing to files whose write bit is not set. 2017-08-28 13:21:18 -04:00
Joey Hess
ee2f096e3b
Support building with feed-1.0, while still supporting older versions.
This commit was sponsored by Jeff Goeke-Smith on Patreon.
2017-08-28 12:29:28 -04:00
Joey Hess
c76ba5a15e
CVE-2017-12976 2017-08-20 16:50:53 -04:00
Joey Hess
252994e095
releasing package git-annex version 6.20170818 2017-08-18 11:19:14 -04:00
Joey Hess
55495c5a98
git-annex.cabal: Deal with breaking changes in Cabal 2.0
https://github.com/haskell/cabal/issues/4655

This means that when a module is conditionally imported via ifdef
depending on the OS or build flags, the cabal file has to mirror the
same logic there to only list the module then.

Since there are lots of OS's and lots of combinations of build flags
here, it's rather difficult to know if the cabal file has been completelty
correctly updated to match the source code.

So I am very unhappy with needing to update things in two places. I've
only tested this on linux with most build flags enables; this will
probably need significant time and testing to catch every cabal file
tweak that this change to Cabal requires. And it will be a continual
source of compile failures going forward when the code is modified and
the cabal file not also updated.

DRY DRY DRY, I repeat myself, but: DRY! Sigh..

(Also, had to remove all Build.* that are standalone programs from the
Other-Modules list, because since cabal passes those modules to ghc when
building git-annex, it complains that they use module Main. Those
modules are only used when building with the Makefile anyway, so this
change shouldn't break anything.)

This commit was sponsored by Thomas Hochstein on Patreon.
2017-08-18 11:08:58 -04:00
Joey Hess
df11e54788
avoid the dashed ssh hostname class of security holes
Security fix: Disallow hostname starting with a dash, which would get
passed to ssh and be treated an option. This could be used by an attacker
who provides a crafted ssh url (for eg a git remote) to execute arbitrary
code via ssh -oProxyCommand.

No CVE has yet been assigned for this hole.
The same class of security hole recently affected git itself,
CVE-2017-1000117.

Method: Identified all places where ssh is run, by git grep '"ssh"'
Converted them all to use a SshHost, if they did not already, for
specifying the hostname.

SshHost was made a data type with a smart constructor, which rejects
hostnames starting with '-'.

Note that git-annex already contains extensive use of Utility.SafeCommand,
which fixes a similar class of problem where a filename starting with a
dash gets passed to a program which treats it as an option.

This commit was sponsored by Jochen Bartl on Patreon.
2017-08-17 22:11:31 -04:00
Joey Hess
fdbfe88168
fix external script for filenames with spaces from protocol
Fix the external special remotes git-annex-remote-ipfs,
git-annex-remote-torrent and the example.sh template to correctly support
filenames with spaces.

This commit was sponsored by John Peloquin on Patreon.
2017-08-17 16:20:09 -04:00
Joey Hess
dafafad115
external: nice error message for keys with spaces in their name
External special remotes will refuse to operate on keys with spaces in
their names. That has never worked correctly due to the design of the
external special remote protocol. Display an error message suggesting
migration.

Not super happy with this, but it's a pragmatic solution. Better than
complicating the external special remote interface and all external special
remotes.

Note that I only made it use SafeKey in Request, not Response. git-annex
does not construct a Response, so that would not add any safety. And
presumably, if git-annex avoids feeding any such keys to an external
special remote, it will never have a reason to make a Response using such a
key. If it did, it would result in a protocol error anyway.

There's still a Serializeable instance for Key; it's used by P2P.Protocol.
There, the Key is always in the final position, so it's ok if it contains
spaces.

Note that the protocol documentation has been fixed to say that the File
may contain spaces. One way that can happen, even though the Key can't,
is when using direct mode, and the work tree filename contains spaces.
When sending such a file to the external special remote the worktree
filename is used.

This commit was sponsored by Thom May on Patreon.
2017-08-17 16:18:34 -04:00
Joey Hess
96c055eda2
migrate: WORM keys containing spaces will be migrated to not contain spaces anymore
To work around the problem that the external special remote protocol does
not support keys containing spaces.

This commit was sponsored by Denis Dzyubenko on Patreon.
2017-08-17 15:09:38 -04:00
Joey Hess
51801cff6a
Prevent spaces from being embedded in the name of new WORM keys, as that handing spaces in keys would complicate things like the external special remote protocol. 2017-08-17 14:46:33 -04:00
Joey Hess
d39c120afa
add annex-ignore-command and annex-sync-command configs
Added remote configuration settings annex-ignore-command and
annex-sync-command, which are dynamic equivilants of the annex-ignore
and annex-sync configurations.

For this I needed a new DynamicConfig infrastructure. Its implementation
should be as fast as before when there is no dynamic config, and it caches
so shell commands are only run once.

Note that annex-ignore-command exits nonzero when the remote should be ignored.
While that may seem backwards, it allows using the same command for it as
for annex-sync-command when you want to disable both.

This commit was sponsored by Trenton Cronholm on Patreon.
2017-08-17 13:54:14 -04:00
Joey Hess
86428f6261
comment 2017-08-17 12:17:47 -04:00
Joey Hess
4173decf27
Windows: Win32 package has subsumed Win32-extras; update dependency. 2017-08-16 17:43:38 -04:00
Joey Hess
69dcb08d7a
Disable http-client's default 30 second response timeout when HEADing an url to check if it exists. Some web servers take quite a long time to answer a HEAD request. 2017-08-15 13:56:12 -04:00
Joey Hess
2eb6309d3e
move, copy: Support --batch. 2017-08-15 12:39:10 -04:00
Joey Hess
8526cd7c92
test: Avoid most situations involving failure to delete test directories
By forking a worker process and only deleting the test directory once it exits.

This way, if a test leaves files open, they'll get closed when the worker
exits, so avoiding failure to delete open files on Windows, and failure to
delete directories due to NFS lock files.

If a test leaves a git worker process running, the closed pipes should
cause the worker to exit too, also avoiding the problem there. The 10
second sleep ought to give plenty of time for such worker processes to
exit, although this is of course a race.

Finally, even if test directory fails to be deleted still,
it won't appear as if the last test in the test suite failed; the error
will be displayed at the very end.

This commit was supported by the NSF-funded DataLad project.
2017-08-14 16:29:47 -04:00