Commit graph

1493 commits

Author SHA1 Message Date
Joey Hess
6b13574827
Windows: include= and exclude= containing '/' will also match filenames that are written using '\'
And vice-versa, but it's better to use '/' for portability.

Notably, standardPreferredContent contains "archive/*" and that might not
match if the filename ends up coming in with the slashes the other way
around.
2020-12-15 12:39:34 -04:00
Joey Hess
94b323a8e8
use TotalSize more extensively 2020-12-11 12:10:43 -04:00
Joey Hess
447d798987
export encode_c' 2020-12-09 15:28:45 -04:00
Joey Hess
19777d1c6f
minor improvements
Adding new instance for Integer, and some parsers for more parameters.

The conversion of readish to readMaybe is done because a serialized
exit code cannot contain additional text after the number.
2020-12-09 15:28:11 -04:00
Joey Hess
41f2c308ff
stall detection is working
New config annex.stalldetection, remote.name.annex-stalldetection, which
can be used to deal with remotes that stall during transfers, or are
sometimes too slow to want to use.

This commit was sponsored by Luke Shumaker on Patreon.
2020-12-08 15:22:18 -04:00
Joey Hess
794fc72afb
avoid parseDuration succeeding on empty string 2020-12-08 12:51:56 -04:00
Joey Hess
72e5764a87
move TransferrerPool from assistant
This old code will now be useful for git-annex beyond the assistant.

git-annex won't use the CheckTransferrer part, and won't run transferkeys
as a batch process, and will want withTransferrer to not shut down
transferkeys processes. Still, the rest of this is a good fit for what I
need now.

Also removed some dead code, and simplified a little bit.

This commit was sponsored by Mark Reidenbach on Patreon.
2020-12-07 12:50:48 -04:00
Joey Hess
e5b170aa1c
switch back to POSIXTime
turned out not to need Read MeterState
2020-12-04 13:54:33 -04:00
Joey Hess
5a41e46bd4
start on serializing Messages
Json objects not yet handled, and some other special cases, but this is
the bulk of the messages.

For progress meters, POSIXTime does not have a Read instance (or a
suitable Show instance), so had to switch to using a Double for progress
meters.

This commit was sponsored by Ethan Aubin on Patreon.
2020-12-03 13:03:03 -04:00
Joey Hess
92136284b1
avoid hGetMetered 0 closing the handle
This is an edge case, which happened to be triggered by the P2P protocol
seeing DATA 0. When reading 0 bytes, getting an empty string does
not mean the handle has reached EOF.

I verified there was in fact a bug, where get of an empty file followed
by another file would get the empty file and then fail
with "handle is closed". This fixes it.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2020-12-01 15:39:22 -04:00
Joey Hess
a3b714ddd9
finish fixing removeLink on windows
9cb250f7be got the ones in RawFilePath,
but there were others that used the one from unix-compat, which fails at
runtime on windows. To avoid this,
import System.PosixCompat.Files hiding removeLink

This commit was sponsored by Ethan Aubin.
2020-11-24 13:20:44 -04:00
Joey Hess
dce0781391
squash remaining build warnings on windows 2020-11-24 12:35:09 -04:00
Joey Hess
804808d569
squash build warnings on windows 2020-11-23 14:00:17 -04:00
Joey Hess
b13c44cccc
convert processTranscript to use hGetLineUntilExitOrEOF
It does use it on both stdout and stderr. It seems unlikely the problem
could really affect stdout, but the unix implementation of it combines
both into a single handle in any case.
2020-11-19 16:36:37 -04:00
Joey Hess
ff0927bde9
converted reads from stderr to use hGetLineUntilExitOrEOF
These are all unlikely to suffer from the inherited stderr fd problem,
but who knows, it could happen.
2020-11-19 16:21:17 -04:00
Joey Hess
728afbc4b1
preserve other headers when adding resume header
It had lost the hAcceptEncoding header that is set as part of the
overriding of http-client's default decompression of compressed files.
Seems likely that would have caused resuming of compressed files to fail
in some cases.

This commit was sponsored by Brett Eisenberg on Patreon.
2020-11-19 14:45:22 -04:00
Joey Hess
b90b9b936d
don't rely on exception for http 416
Fix a bug that could make resuming a download from the web fail when the
entire content of the file is actually already present locally.

What a mess that Request can throw exceptions or not, depending on how
it's configured. Makes it very hard if you need to handle some specific
http status codes in a function like this! Implementing everything two
ways did not seem appealing, if possible at all, so I decided to
override the Request if it did come configured to throw exception on
non-2xx http status. Other exceptions, like from http-client-restricted,
or due to a redirect to a non-http url, still get thrown.

This commit was sponsored by Luke Shumaker on Patreon.
2020-11-19 14:44:42 -04:00
Joey Hess
4b739fc460
Fix build on Windows
Thanks to bug reporter for the patch.
2020-11-19 12:33:00 -04:00
Joey Hess
9cb250f7be
fix removeLink on windows
This removeLink was introduced in commit
e505c03bcc, which replaced code
that used removeFile on Windows. So, I know git-annex did not used to do
anything other than removeFile on Windows. If there were symlinks it
wanted to remove, this would not work on windows, but of course it does
not use symlinks on windows.
2020-11-19 12:20:18 -04:00
Joey Hess
682829c200
avoid throwing exception when the handle is closed
The handle could get closed eg, by cleanupProcess being called,
which forces the process to exit and closes all its handles.

At this point, the test case in https://git-annex.branchable.com/bugs/Buggy_external_special_remote_stalls_after_7245a9e/
is fixed.
2020-11-18 15:10:35 -04:00
Joey Hess
b021e2322f
avoid crash on EOF at end 2020-11-18 15:03:30 -04:00
Joey Hess
e6d741af79
finish conversion to hGetLineUntilExitOrEOF
started in aafae46bcb
2020-11-18 14:54:02 -04:00
Joey Hess
b483be8548
newline mode (mis)handling for windows
Unfortunately, there is no hGetNewLineMode. This seems like an oversight
that should be fixed in ghc, but for now, I paper over it with a windows
hack.
2020-11-18 14:48:50 -04:00
Joey Hess
787b39c7c1
working hGetLineUntilExitOrEOF
The problem with the old version seemed to be that hWaitForInput blocks
rather than timing out when being run concurrently with hGetLine on the
same handle.

This passes the bench test, and also works when run concurrently on
different handles.
2020-11-18 14:21:47 -04:00
Joey Hess
9af0000e0f
bench test for hGetLineUntilExitOrEOF
This seems to show that hWaitForInput does not seem to behave as
documented. It does not time out, so blocks forever in this situation.
This is with a 0 timeout and with larger timeouts. Unsure why, it looked
like it should work.
2020-11-18 12:23:15 -04:00
Joey Hess
aafae46bcb
WIP
for https://git-annex.branchable.com/bugs/Buggy_external_special_remote_stalls_after_7245a9e/
2020-11-17 17:31:08 -04:00
Joey Hess
bce8865824
fix build on windows 2020-11-17 11:58:45 -04:00
Joey Hess
49a14b16a1
fix build on windows 2020-11-16 09:31:45 -04:00
Joey Hess
ed7afabdb1
fix build on windows 2020-11-13 13:34:28 -04:00
Joey Hess
f240f0196c
fix build on windows 2020-11-12 11:39:29 -04:00
Joey Hess
6911d27d42
fix build on windows 2020-11-11 08:51:42 -04:00
Joey Hess
4c1eb28c40
fix build on windows 2020-11-10 11:21:03 -04:00
Joey Hess
885974be99
add newtypes for QuickCheck to avoid LANG=C issues
All properties changed to use them, except for
prop_encode_c_decode_c_roundtrip, which already filtered to ascii
for other reasons.

A few modules had to be split out, because Setup does not build-depend
on QuickCheck.
2020-11-09 20:21:18 -04:00
Joey Hess
dd52d8ebdc
update after RawFilePath transition 2020-11-09 12:12:25 -04:00
Joey Hess
907a0bcad6
avoid providing filename with NUL to quickcheck properties
instance Arbitrary [Char] allows that, and it's not a legal part of a
filename so can break processing them.

Noticed when prop_view_roundtrips failed.

The instance Arbitrary AssociatedFile avoids this problem.

This commit was sponsored by Mark Reidenbach on Patreon.
2020-11-06 15:15:33 -04:00
Joey Hess
1db49497e0
finished this stage of the RawFilePath conversion
This commit was sponsored by Denis Dzyubenko on Patreon.
2020-11-06 14:10:58 -04:00
Joey Hess
2c8cf06e75
more RawFilePath conversion
Converted file mode setting to it, and follow-on changes.

Compiles up through 369/646.

This commit was sponsored by Ethan Aubin.
2020-11-05 18:45:37 -04:00
Joey Hess
9b0dde834e
convert getFileSize to RawFilePath
Lots of nice wins from this in avoiding unncessary work, and I think
nothing got slower.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2020-11-05 11:32:57 -04:00
Joey Hess
5a1e73617d
finished this stage of the RawFilePath conversion
Finally compiles again, and test suite passes.

This commit was sponsored by Brock Spratlen on Patreon.
2020-11-04 14:20:37 -04:00
Joey Hess
eb42cd4d46
more RawFilePath conversion
535/645

This commit was sponsored by Brett Eisenberg on Patreon.
2020-11-03 10:11:04 -04:00
Joey Hess
b724236b35
remove unused imports 2020-11-02 15:36:11 -04:00
Joey Hess
f45ad178cb
more RawFilePath conversion
At 318/645 after 4k lines of changes

This commit was sponsored by Jake Vosloo on Patreon.
2020-10-29 12:03:50 -04:00
Joey Hess
e505c03bcc
more RawFilePath conversion
nukeFile replaced with removeWhenExistsWith removeLink, which allows
using RawFilePath. Utility.Directory cannot use RawFilePath since setup
does not depend on posix.

This commit was sponsored by Graham Spencer on Patreon.
2020-10-29 10:50:29 -04:00
Joey Hess
8d66f7ba0f
more RawFilePath conversion
Added a RawFilePath createDirectory and kept making stuff build.

Up to 296/645

This commit was sponsored by Mark Reidenbach on Patreon.
2020-10-28 17:25:59 -04:00
Joey Hess
6c29817748
RawFilePath version of getCurrentDirectory
This commit was sponsored by Jochen Bartl on Patreon
2020-10-28 16:03:45 -04:00
Joey Hess
08cbaee1f8
more RawFilePath conversion
Most of Git/ builds now.

Notable win is toTopFilePath no longer double converts

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2020-10-28 15:55:30 -04:00
Joey Hess
d6e94a6b2e
got configure working after Utility.Path ByteString conversion
Had to split out some modules because getWorkingDirectory needs unix,
which is not a build-dep of configure.

This commit was sponsored by Brock Spratlen on Patreon.
2020-10-28 15:01:19 -04:00
Joey Hess
e219aadbab
convert to RawByteString
This will break a lot of stuff that uses it, but once fixed should lead
to better performance.

Mostly mechanical.
Changes of note:

* upFrom now uses isPathSeparator, which is better on Windows where
  there is not just one
* splitShortExtensions used to take the length of a string,
  which would count wide unicode characters as a single character.
  Changing to B.length changes that. Note that, git-annex's
  annexMaxExtensionLength already changed to the length in bytes
  before this change. This function is only used in generating views,
  and the small behavior change should not be a problem.
* relHome still uses FilePath because it didn't seem worth changing(?)

This commit was sponsored by Jack Hill on Patreon.
2020-10-28 14:32:45 -04:00
Joey Hess
9a5cd96f0d
Fix a memory leak introduced in the last release
The problem was this line:

	cleanup = and <$> sequence (map snd v)

That caused all of v to be held onto until the end, when the cleanup action
was run.

I could not seem to find a bang pattern that avoided the leak, so I
resorted to a IORef, rather clunky, but not a performance problem because
it will only be written once per git ls-files, so typically just 1 time.

This commit was sponsored by Mark Reidenbach on Patreon.
2020-10-13 16:31:01 -04:00
Joey Hess
d54dd0ef9c
Fix build on Windows with network-3
inet_addr was removed, but all this needs is localhost, so hardcoding it
should work fine.

It may be that this windows ifdef is no longer needed. It was added in 2013
with a note that getAddrInfo didn't work on windows, but it seems likely
such a problem would have been fixed since.
2020-10-08 10:50:39 -04:00