Commit graph

448 commits

Author SHA1 Message Date
Joey Hess
9b98d3f630
better HTTP connection reuse
Enable HTTP connection reuse across multiple files, when git-annex
uses http-conduit. Before, a new Manager was created each time
Utility.Url used it. Now, a single Manager gets created the first time,
so connections are reused.

Doesn't help when external programs are used for url download,
but does speed up addurl --fast, fsck --from web, etc.

Testing fsck --fast --from web with 3 files, over high-latency
satellite internet, it sped up from 19.37s to 14.96s.

This commit was supported by the NSF-funded DataLad project.
2018-04-04 15:39:40 -04:00
Joey Hess
2ec07bc29f
Avoid running annex.http-headers-command more than once. 2018-04-04 15:15:08 -04:00
Joey Hess
ef389722ae
don't copy old date metadata when adding new version of a file
When adding a new version of a file, and annex.genmetadata is enabled,
don't copy the data metadata from the old version of the file, instead use
the mtime of the file. Rationalle being that the user has requested to
generate metadata and so would expect to get the new mtime into metadata.

Also, avoid warning about copying metadata when all the old metadata is
date metadata. Which was rather the harder part.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2018-04-04 13:58:16 -04:00
Joey Hess
961fa377d9
Also do forward retrying in cases where no exception is thrown, but the transfer failed.
I think this used to be the case, but it was accidentially lost way back in
commit 3887432c54. Normally, transfers do not
throw exceptions, so probably forward retrying was rarely done due to that
oversight.

This also affects the new annex.retry etc configuration. If a transfer
fails, without making any progress, eg because the file is not present on
the remote or the remote is not accessible, it will now retry when
configuration calls for it. In some cases such a retry is not desirable,
for example the remote could be accessible and not have a copy of the file
that the local repo thinks it has. I see no way to distinguish such cases
from cases where a retry should really be done. So, it'll be up to the user
to configure it to work for them.
2018-03-29 13:22:49 -04:00
Joey Hess
46d4316954
implement annex.retry et al
Added annex.retry, annex.retry-delay, and per-remote versions to configure
transfer retries.

This commit was supported by the NSF-funded DataLad project.
2018-03-29 13:04:07 -04:00
Joey Hess
2927618d35
Added adb special remote which allows exporting files to Android devices.
git annex testremote passes.

exportree not implemented yet, although the documentation talks about it,
since it will be the main way this remote will be used.

The adb push/pull progress is displayed for now; it would be better
to consume it and use it to update the git-annex progress bar.

This commit was sponsored by andrea rota.
2018-03-27 14:54:41 -04:00
Joey Hess
5574e25c54
OSX app: Work around libz/libPng/ImageIO.framework version skew
By not bundling libz, assuming OSX includes a suitable libz.1.dylib.

This commit was supported by the NSF-funded DataLad project.
2018-03-22 11:53:10 -04:00
Joey Hess
bebf541aa7
Fix calculation of estimated completion for progress meter.
Was estimating transfer of whole file, not remaining part of it.
2018-03-19 23:26:41 -04:00
Joey Hess
bc1d56bdfd
cabal sdist refuses to build with -O1
Sigh, disabled the full fix. What we have now will work once the ghc bug
is fixed, but not currently.
2018-03-16 12:56:28 -04:00
Joey Hess
14100f4f74
releasing package git-annex version 6.20180316 2018-03-16 12:40:27 -04:00
Joey Hess
ac6f58d642
fix ssh warmup hang
Fix race condition in ssh warmup that caused git-annex to get stuck and
never process some while when run with high levels of concurrency.

So far, I've isolated the problem to processTranscript, which hangs
reading output from ssh in this situation. I don't yet understand why
processTranscript behaves that way.

Since here we don't care about the ssh output, and only want to /dev/null
it, changed to not use processTranscript, avoiding its problem.

This commit was supported by the NSF-funded DataLad project.
2018-03-15 15:04:15 -04:00
Joey Hess
10d3b7fc62
Fix reversion introduced in 6.20171214 that caused concurrent transfers to incorrectly fail with "transfer already in progress".
Avoid creating transfer info file before transfer lock is created and
locked.

The wrong order for one thing caused transfer info to be overwritten
when a transfer was already in progress.

But worse, it caused checkTransfer to see the transfer info,
and so lock the transfer lock in order to verify the transfer was not in
progress. Which in a concurrent situation, prevented the transferrer
from locking the transfer lock, so it failed with "transfer already in
progress".

Note that the transferinfo command does not lock the transfer lock
before creating the transfer info. But, that's only run after
recvkey is running, and recvkey does lock the transfer lock, so that
seems more or less ok. (Other than being a super complicated legacy mess
that the P2P code has mostly obsoleted now.)

This commit was supported by the NSF-funded DataLad project.
2018-03-14 18:55:34 -04:00
Joey Hess
ba44ca80e6
Include amount of data transferred in progress display. 2018-03-14 13:39:14 -04:00
Joey Hess
050ada746f
Added backends for the BLAKE2 family of hashes.
There are a lot of different variants and sizes, I suppose we might as well
export all the common ones.

Bump dep to cryptonite to 0.16, earlier versions lacked BLAKE2 support.
Even android has 0.16 or newer.

On Debian, Blake2bp_512 is buggy, so I have omitted it for now.
http://bugs.debian.org/892855

This commit was sponsored by andrea rota.
2018-03-13 16:23:42 -04:00
Joey Hess
4015c5679a
force verification when resuming download
When resuming a download and not using a rolling checksummer like rsync,
the partial file we start with might contain garbage, in the case where a
file changed as it was being downloaded. So, disabling verification on
resumes risked a bad object being put into the annex.

Even downloads with rsync are currently affected. It didn't seem worth the
added complexity to special case those to prevent verification, especially
since git-annex is using rsync less often now.

This commit was sponsored by Brock Spratlen on Patreon.
2018-03-13 14:50:49 -04:00
Joey Hess
59e7f3cbb2
done for the day 2018-03-12 17:32:57 -04:00
Joey Hess
c3df5d1f10
avoid double-connect to unreachable ssh remote
When git-annex-shell p2pstdio fails with 255, it's because the ssh
server is not reachable. Avoid running the fallback action in this case,
since it would just try a second time to connect, and presumably fail.

Note that the closed P2PSshConnection will not be stored in the pool,
so the next request tries again to connect. This is just the right
behavior; when the remote becomes reachable again, the same git-annex
process will start using it.

This commit was sponsored by Ole-Morten Duesund on Patreon.
2018-03-12 16:50:21 -04:00
Joey Hess
989de2140b
wording 2018-03-09 13:56:21 -04:00
Joey Hess
08814327ff
use P2P protocol for checkpresent, retrieve, and store
Note that, due to not using rsync to transfer files to ssh remotes
any longer, permissions and other file metadata of annexed files
will no longer be preserved when copying them to ssh remotes.
Other remotes never supported preserving that information, so
this is not considered a regression. Added NEWS item about this.

Another significant side effect of this is that, even when rsync is run to
retrieve a file, its progress display will no longer be shown, and
instead the native git-annex progress display will appear. It would be
possible to use the rsync process display when rsync is used (old
git-annex-shell and also retrieval from a local repository), but it
would have complicated the code unncessarily, and been inconsistent
behavior.

(I'd been thinking for a while about eliminating the rsync progress
display, since it's got some annoying verbosities, including display of
the key and the "(xfr#1, to-chk=0/1)" bit and was already somewhat
inconsistent.)

retrieveKeyFileCheap still uses rsync, since that ensures that it gets
the actual file content from the remote. Using the P2P protocol would
use the local content, as long as the local and remote size are the
same.

This commit was sponsored by John Pellman on Patreon.
2018-03-09 13:25:16 -04:00
Joey Hess
5bc0ab3f31
going AGPL
Remote/Git.hs now contains AGPL licensed code, thus the license
of git-annex as a whole is AGPL. This was already the case when git-annex
was built with the webapp enabled.

The AGPL license will apply to all code added to Remote/Git.hs in the
future, which is going to include support for using
`git-annex-shell p2pstdio`.
2018-03-09 01:03:46 -04:00
Joey Hess
3dd43df9c2
Better ssh connection warmup when using -J for concurrency.
Avoids ugly messages when forced ssh command is not git-annex-shell.

This commit was sponsored by Ole-Morten Duesund on Patreon.
2018-03-07 17:30:14 -04:00
Joey Hess
6ddfa9807b
implemented git-annex-shell p2pstdio
Not yet used by git-annex, but this will allow faster transfers etc than
using individual ssh connections and rsync.

Not called git-annex-shell p2p, because git-annex p2p does something
else and I don't want two subcommands with the same name between the two
for sanity reasons.

This commit was sponsored by Øyvind Andersen Holm.
2018-03-07 15:38:01 -04:00
Joey Hess
f4103744c3
make sure that lockContentShared is always paired with an inAnnex check
lockContentShared had a screwy caveat that it didn't verify that the content
was present when locking it, but in the most common case, eg indirect mode,
it failed to lock when the content is not present.

That led to a few callers forgetting to check inAnnex when using it,
but the potential data loss was unlikely to be noticed because it only
affected direct mode I think.

Fix data loss bug when the local repository uses direct mode, and a
locally modified file is dropped from a remote repsitory. The bug
caused the modified file to be counted as a copy of the original file.
(This is not a severe bug because in such a situation, dropping
from the remote and then modifying the file is allowed and has the same
end result.)

And, in content locking over tor, when the remote repository is
in direct mode, it neglected to check that the content was actually
present when locking it. This could cause git annex drop to remove
the only copy of a file when it thought the tor remote had a copy.

So, make lockContentShared do its own inAnnex check. This could perhaps
be optimised for direct mode, to avoid the check then, since locking
the content necessarily verifies it exists there, but I have not bothered
with that.

This commit was sponsored by Jeff Goeke-Smith on Patreon.
2018-03-07 14:23:52 -04:00
Joey Hess
07e253b1fb
Improve SHA*E extension extraction code
Do not treat parts of the filename that contain punctuation or other
non-alphanumeric characters as extensions. Before, such characters were
filtered out.

Note that in 45308ec78b "foo.ba__________r"
was munged to ".bar" and so incorrectly treated as an extension. That was
fixed by changing the filter order, but not allowing punctuation seems a
better fix.

This assumes that extensions containing punctuation are rare. "_" seems the
most likely character; I used it in ikiwiki "._comment" files. But I can't
recall seeing it anywhere else. It certianly seems that no commonly used
extensions contain punctuation. If git-annex doesn't treat "._comment"
as an extension, it's not likely to break software that expects to see that
extension like some software expects to see .epub or .mp3.

This commit was sponsored by Jack Hill on Patreon.
2018-03-05 11:25:01 -04:00
Joey Hess
6063b3df3f
Dial back optimisation when building on arm
Prevent ghc and llc from running out of memory when optimising some
files.

Sean Whitton reported that doing this only in Test.hs was insufficient,
the build still OOMed by the time it got to Test.hs. He had earlier found
the build worked when these options are applied globally.

See https://ghc.haskell.org/trac/ghc/ticket/14821 for why it needs -O1;
once that's fixed it may suffice to use "GHC-Options: -O2 -optlo-O2",
although it may also be that the -O1 prevents ghc from using/leaking
as much memory.

os(arm) should match armel, armhf, armeb, and arm.
It probably also matches arm64, somewhat unfortunately since arm64
systems probably tend to have more memory. See list of arches in
https://hackage.haskell.org/package/Cabal-1.22.2.0/docs/src/Distribution-System.html

This commit was sponsored by Henrik Riomar on Patreon.
2018-03-04 19:48:07 -04:00
Joey Hess
bed6773346
Support exporttree=yes for rsync special remotes.
Renaming is not supported; it might be possible to use --fuzzy to get rsync
to notice the file is being renamed, but that is a bit ..fuzzy.

On the other hand, interrupted transfers of an exported file are resumed,
since rsync is great at that. Had to adjust the exporttree docs, which
said interrupted transfers would restart.

Note that remove no longer makes the empty directory dummy, instead
sending the top-level empty directory. This works just as well and I
noticed the dummy was unncessary when refactoring it into removeGeneric.
Verified that behavior of remove is not changed, and git annex
testremote does pass.

This commit was sponsored by Brock Spratlen on Patreon.
2018-02-28 13:36:20 -04:00
Joey Hess
8927ef7613
releasing package git-annex version 6.20180227 2018-02-27 12:05:35 -04:00
Joey Hess
8f71b316c9
remove chrpath hack
Makefile: Remove chrpath workaround for bug in cabal,  which is no longer
needed.

https://github.com/haskell/cabal/issues/2717 says it uses RUNPATH instead
of RPATH now, but I don't even see that for statically linked libraries;
the bug with that appears to be fixed.

cabal-install version 1.24.0.2
compiled using version 1.24.2.0 of the Cabal library

I left the rpath removal using otool on OSX because those straight up
broke the linker, and I don't know if the OSX autobuilder is updated to
a new enough cabal to not need it.

This commit was sponsored by Ewen McNeill on Patreon.
2018-02-27 12:02:38 -04:00
Joey Hess
f99af09322
wording 2018-02-26 14:36:40 -04:00
Joey Hess
db057dcff0
fix sync bug in direct mode
sync: Fix bug that prevented pulling changes into direct mode repositories
that were committed to remotes using git commit rather than git-annex sync.

This commit was supported by the NSF-funded DataLad project.
2018-02-26 14:10:03 -04:00
Joey Hess
42f154bd7e
Fix metadata hook script to support non-ascii characters
tips/automatically_adding_metadata/pre-commit-annex: Fix to not silently
skip filenames containing non-ascii characters.

git diff-index defaults to munging non-ascii characters. Using -z makes
it not do that, and then we just change the nulls to newlines.

This commit was sponsored by Jochen Bartl on Patreon.
2018-02-26 12:58:37 -04:00
Joey Hess
09e73a3ab6
annex.merge-annex-branches
Added annex.merge-annex-branches config setting which can be used to
disable automatic merge of git-annex branches.

I wonder if git-annex merge/sync/assistant should disable this
setting? Not sure yet, so have not done so. May be that users will not set
it in git config, but pass it via -c to commands that need it.

Checking the config setting adds a very small overhead, but it's
only checked once per command so should be insignificant.

This commit was supported by the NSF-funded DataLad project.
2018-02-22 14:25:32 -04:00
Joey Hess
cb3b73df6c
importfeed: Fix a failure when downloading with youtube-dl and the destination subdirectory does not exist yet.
Noticed while running this (which a user posted in a comment they deleted
for some reason):

git-annex importfeed https://vimeo.com/logiingimars/videos/rss

The filename that youtube-dl suggests included a subdirectory,
which didn't exist, so renaming to it failed.

This commit was sponsored by mo on Patreon.
2018-02-22 13:20:19 -04:00
Joey Hess
2e25185a9c
Remove temporary code added in 6.20160619 to prime the mergedrefs log.
Repositories that are upgraded from before that version to this
one will not break, but will just not see the benefit of the mergedrefs log
speeding things up, until one new ref gets merged in.
2018-02-22 12:31:27 -04:00
Joey Hess
fbfac04cef
wording 2018-02-19 15:41:56 -04:00
Joey Hess
6583448bab
add --json-error-messages (not yet implemented)
Added --json-error-messages option, which includes error messages in the
json output, rather than outputting them to stderr.

The actual rediretion of errors is not implemented yet, this is only
the docs and option plumbing.

This commit was supported by the NSF-funded DataLad project.
2018-02-19 14:32:15 -04:00
Joey Hess
fa65f1d240
fix --json-progress --json to be same as --json --json-progress
Fix behavior of --json-progress followed by --json, in which
the latter option disabled the former.

This commit was supported by the NSF-funded DataLad project.
2018-02-19 14:12:15 -04:00
Joey Hess
8ccfbd14d0
Split Test.hs and avoid optimising it much, to need less memory to compile.
The ghc options were found by Sean Whitton; the debian arm autobuilders
need those to build w/o OOM, and it seems to involve llvm using too much
memory to optimize Test.

This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
2018-02-18 11:48:48 -04:00
Joey Hess
bbdfdaa833
git-annex.cabal: Once more try to not build the assistant on the hurd, hopefully hackage finally recognises that OS.
Last try was in 2015 https://github.com/haskell/hackage-server/issues/269
2018-02-18 11:09:16 -04:00
Joey Hess
7e454ee341
--json: multi-line notes
--json: When there are multiple lines of notes about a file, make the note
field multiline, rather than the old behavior of only including the last
line.

Using newlines in the note is perhaps not ideal, but upgrading it to an
array in this case would be an annoying inconsistency to need to deal with.

This commit was sponsored by Ole-Morten Duesund on Patreon.
2018-02-16 13:27:17 -04:00
Joey Hess
aee9adbadc
Add gpg-agent to Build-Depends. Needed to run the test suite.
Merged from Debian.

I think what this actually deals with is the case where gpg is installed,
but gpg-agent is not, since Utility.Gpg.stdParams enables --use-agent
when GPG_BATCH is set, and the test suite enables GPG_BATCH. So, test suite
will work with gpg not installed, or with both gpg and gpg-agent installed,
but not with only gpg.

For this reason, I've also put in an explicit dep on gnupg, although
dpkg-dev recommends it and all debian package builds tend to have it
available implicitly.
2018-02-14 17:32:44 -04:00
Joey Hess
7f5c6a28a6
fsck: Warn when required content is not present in the repository that requires it.
This commit was sponsored by Jack Hill on Patreon.
2018-02-08 14:08:41 -04:00
Joey Hess
6f72bd0b47
datalad < 0.9.1 had a problem in its special remote protocol handling which is broken by EXTENSIONS. Make the debian git-annex package conflict with the problem version of datalad. 2018-02-07 16:16:33 -04:00
Joey Hess
d884e5b6fe
Added EXTENSIONS to external special remote protocol.
Allows using new special remote messages when git-annex supports them,
and avoiding using them when git-annex is too old. The new INFO is one
such message.

There's also the possibility, currently unused, for the special remote's
reply to include some kind of extensions of its own.

Merging this is blocked by https://github.com/datalad/datalad/issues/2124
since it seems it will break datalad. I checked all the other special
remotes and they will be ok.

This commit was supported by the NSF-funded DataLad project.
2018-02-07 15:02:12 -04:00
Joey Hess
7d9f0e0fbe
Added INFO to external special remote protocol.
It's left up to the special remote to detect when git-annex is new enough
to support the message; an old git-annex will blow up.

This commit was supported by the NSF-funded DataLad project.
2018-02-06 13:03:55 -04:00
Joey Hess
cfbfb3ab9a
inprogress: Avoid showing failures for files not in progress. 2018-01-24 20:43:19 -04:00
Joey Hess
84ea1eb762
releasing package git-annex version 6.20180112 2018-01-12 15:47:32 -04:00
Joey Hess
a28c541e23
add remote.<name>.annex-checkuuid
Added remote.<name>.annex-checkuuid config, which can be set to false to
disable the default checking of the uuid of remotes that point to
directories. This can be useful to avoid unncessary drive spin-ups and
automounting.

Note that the UUID check is still done before writing to the repository,
to avoid writing to the wrong repository if it got relocated. Check is
also done before checkPresent to avoid getting confused about what is in
which repo. This is effectively the same as the use of git-annex-shell
with a uuid to check that the remote repository is the expected one.
Did not bother with the check for retrieveKeyFile because it doesn't
matter if the wrong repo is used then.

This commit was sponsored by Trenton Cronholm on Patreon.
2018-01-10 14:21:18 -04:00
Joey Hess
2b66492d6e
Improve startup time for commands that do not operate on remotes
And for tab completion, by not unnessessarily statting paths to remotes,
which used to cause eg, spin-up of removable drives.

Got rid of the remotes member of Git.Repo. This was a bit painful.

Remote.Git modifies the list of remotes as it reads their configs,
so still need a persistent list of remotes. So, put it in as
Annex.gitremotes. It's only populated by getGitRemotes, so commands
like examinekey that don't care about remotes won't do so.

This commit was sponsored by Jake Vosloo on Patreon.
2018-01-09 16:22:07 -04:00
Joey Hess
edab547034
switch distribution name to "upstream"
to not imply that this changelog has anything to do with debian unstable
2018-01-05 13:35:35 -04:00
Joey Hess
24df95f0f6
Fix several places where files in .git/annex/ were written with modes that did not take the core.sharedRepository config into account.
git grep writeFile finds some more that might also be problems, but
for now I've concentrated on .git/annex/ log files. There are certianly
cases where writeFile is not a problem too.

This commit was sponsored by mo on Patreon.
2018-01-02 17:25:25 -04:00
Joey Hess
edd25f04d9
unused: Write .git/annex/unused etc files with appropriate permissions for the core.sharedRepository config.
This commit was sponsored by an anonymous bitcoin donor.
2018-01-02 16:25:27 -04:00
Joey Hess
25703e1413
finally really add back custom-setup stanza
Fourth or fifth try at this and finally found a way to make it work.

Absurd amount of busy-work forced on me by change in cabal's behavior.
Split up Utility modules that need posix stuff out of ones used by
Setup. Various other hacks around inability for Setup to use anything
that ifdefs a use of unix.

Probably lost a full day of my life to this.
This is how build systems make their users hate them. Just saying.
2017-12-31 16:36:39 -04:00
Joey Hess
2bfdd690e2
addurl: Fix encoding of filename queried from youtube-dl when in --fast mode.
And also now in non-fast mode, since it was just changed to query for the
filename separately.

And avoid processTranscript which mixed up stdout and stderr and could have
led to weirdness if there were warnings that didn't get suppressed.
2017-12-31 15:19:01 -04:00
Joey Hess
fcdd9ce788
repeated addurl behavior reversion fix
addurl: When the file youtube-dl will download is already an annexed file,
don't download it again and fail to overwrite it, instead just do nothing,
like it used to when quvi was used.

This commit was sponsored by Anthony DeRobertis on Patreon.
2017-12-31 14:55:51 -04:00
Joey Hess
1f5bf73af0
Revert "git-annex.cabal: Add back custom-setup stanza, so cabal new-build works."
This reverts commit 51228c2306.

No, still doesn't work when built with cabal. It did with stack; stack
must somehow make the unix package implicitly available.

With cabal, System.Posix.Process and System.Posix.Env are both missing.
2017-12-31 14:09:41 -04:00
Joey Hess
51228c2306
git-annex.cabal: Add back custom-setup stanza, so cabal new-build works.
Seems I had all the work in past commits to make this build, at least on
linux. I'm actually surprised it does, without a unix dep, Utility.Env
still builds ok somehow despite using System.Posix.Env.

This commit was sponsored by Fernando Jimenez on Patreon.
2017-12-31 13:54:41 -04:00
Joey Hess
beb67fbbad
Fix bug introduced in version 6.20171018 that caused some commands to print out "ok" twice after processing a file.
This commit was sponsored by Boyd Stephen Smith Jr. on Patreon
2017-12-28 17:31:16 -04:00
Joey Hess
67338fd7ac
Added inprogress command for accessing files as they are being downloaded.
Chose to make this only handle files actively being downloaded, not temp
files for downloads that were interrupted or files that have been fully
downloaded.

This commit was sponsored by Ole-Morten Duesund on Patreon.
2017-12-28 11:46:39 -04:00
Joey Hess
79857d7e9f
Removed the testsuite build flag
Test suite is always included.

Building with this flag disabled has actually been broken for some time,
since Command.TestRemote uses tasty. Fewer build flags are better, so good
time to drop it.

This commit was sponsored by Thomas Hochstein on Patreon.
2017-12-20 12:25:03 -04:00
Joey Hess
61b515d71d
releasing package git-annex version 6.20171214 2017-12-14 11:54:36 -04:00
Joey Hess
3cc94c1667
.noannex file
A top-level .noannex file will prevent git-annex init from being used in a
repository. This is useful for repositories that have a policy reason not
to use git-annex. The content of the file will be displayed to the user who
tries to run git-annex init.

This also affects git annex reinit and initialization via the webapp.
It does not affect automatic inits, when there's a sibling git-annex branch
already.

This commit was supported by the NSF-funded DataLad project.
2017-12-13 14:34:32 -04:00
Joey Hess
cfdfe4df6c
lookupkey absolute path support
lookupkey: Support being given an absolute filename to a file within the
current git repository.

This commit was supported by the NSF-funded DataLad project.
2017-12-08 15:35:02 -04:00
Joey Hess
ed701667aa
fix gpg subkey support typo
initremote, enableremote: Really support gpg subkeys suffixed with an
exclamation mark, which forces gpg to use a specific subkey. (Previous try
had a bug.)

This commit was sponsored by Jake Vosloo on Patreon.
2017-12-05 13:58:53 -04:00
Joey Hess
4e38c4f57f
Allow exporttree remotes to be marked as dead.
Union with max so that DeadTrusted wins over UnTrusted.

This commit was sponsored by Trenton Cronholm on Patreon.
2017-12-05 13:46:55 -04:00
Joey Hess
f00770d736
Removed no longer needed dependency on yesod-default.
This commit was sponsored by Nick Daly on Patreon.
2017-12-05 13:18:25 -04:00
Joey Hess
2528e3ddb0
rethought --relaxed change
Better to make it not be surprising and slow, than surprising and fast.
--raw can be used when it needs to be really fast.

Implemented adding a youtube-dl supported url to an existing file.

This commit was sponsored by andrea rota.
2017-11-30 14:13:20 -04:00
Joey Hess
24f27ec39d
convert importfeed to youtube-dl
Fully working, including --fast/--relaxed.

Note that, while git-annex addurl --relaxed is not going to check
youtube-dl, I kept git annex importfeed --relaxed checking it.
Thinking is that, let's not break people's importfeed cron jobs, and
importfeed does not typically have to check a large number of new items,
so it's ok if it's a little bit slower when used with youtube playlist
feeds.

importfeed's behavior is also improved (?) when a feed has links in it
to non-media files. Before, those were skipped. Now, the content of the
link is downloaded. This had to be done, because trying to use
youtube-dl is slow, and if those were skipped, it would have to check
every time importfeed was run. While this behavior change may not be
desirable for some feeds, that intersperse links to web pages with
enclosures, it will be desirable for other feeds, that have
non-enclosure directy links to media files.

Remove old quvi modules.

This commit was sponsored by Øyvind Andersen Holm.
2017-11-29 17:30:02 -04:00
Joey Hess
3febb79c8f
wip 2017-11-28 17:17:40 -04:00
Joey Hess
d6d8f72957
documentation update for youtube-dl
Code not updated yet.

This commit was sponsored by Thomas Hochstein on Patreon.
2017-11-28 14:05:58 -04:00
Joey Hess
28565919af
releasing package git-annex version 6.20171124 2017-11-24 10:49:57 -04:00
Joey Hess
ed9d5da2d5
Fix build with dns-3.0.
This commit was sponsored by Henrik Riomar on Patreon.
2017-11-24 10:49:31 -04:00
Joey Hess
e8ee6a73b4
remove changelog entry
glibc already didn't support old kernels, so nothing user-visible has changed
2017-11-22 15:30:43 -04:00
Joey Hess
10390a6378
Linux standalone arm build no longer supports Linux kernels older than 2.6.28
As it was getting too expensive to patch out use of the "new" syscalls

We could revisit this if someone has hardware with an older kernel
that's still being maintained, but I've verified that the Synology
NAS that had used a too old kernel version has been updated to 2.6.32.
2017-11-21 13:29:14 -04:00
Joey Hess
f5edb16729
Display progress meter when uploading a key without size information
Getting the size by statting the content file.

This commit was supported by the NSF-funded DataLad project.
2017-11-14 16:40:49 -04:00
Joey Hess
01d940dc10
releasing package git-annex version 6.20171109 2017-11-09 12:23:34 -04:00
Joey Hess
0e4bdd21a8
Fix directory special remote's cleanup of empty export directories.
Was trying to rmdir the file, which had already been deleted, and when that
failed, it skipped trying to delete the parent directories.

Noticed the bug through testremote, but it can't itself detect such
problems as there is no enumeration in the API.

This commit was sponsored by Brock Spratlen on Patreon.
2017-11-08 14:38:24 -04:00
Joey Hess
1d0bf44173
testremote: Test exporttree.
As long as the class of remotes supports exporting, it's tested whether
or not the remote is configured with exporttree=yes.

Also, made testremote of a remote configured with exporttree=yes
disable that configuration for testing non-export storage.

This commit was supported by the NSF-funded DataLad project.
2017-11-08 14:22:11 -04:00
Joey Hess
49663ab6a8
Makefile improvement for BUILDER=stack, use stack to run ghc. 2017-11-07 16:56:48 -04:00
Joey Hess
f88ff8d460
Makefile improvement for sudo make install. Thanks, Eric Siegerman 2017-11-07 16:29:58 -04:00
Joey Hess
9d129367db
Web.checkKey: Fix handling of multiple urls
When there are multiple urls for a file, still treat it as being present
in the web when some urls don't work, as long as at least one url does
work.

This is consistent with the other web methods handling of multiple urls.

This commit was sponsored by Ole-Morten Duesund on Patreon.
2017-11-07 16:15:44 -04:00
Joey Hess
75ec0227f8
unlock, lock: Support --json. 2017-10-30 14:44:11 -04:00
Joey Hess
68a9c867be
Fix exporting of non-annexed files to external special remotes.
Actual problem is the keyName was set to "Ref \"sha\"", which led to
this follow-on failure since it contained a space.

The bad data would also get into the export database when exporting to a
non-external special remote. Looking briefly at that, I don't think the bad
data will lead to anything more than a re-upload of the file content
now that the problem has been fixed.

This commit was sponsored by Peter Hogg on Patreon.
2017-10-30 12:47:13 -04:00
Joey Hess
24883e01cd
Fix export of subdir of a branch.
Seems I forgot to fully test that feature when documenting it.

git rev-parse needs a colon after a branch to de-reference the tree
it points to, rather than the commit. But that had it adding an extra
colon when the user specified "branch:subdir". So, check if there is a
colon before adding one.

This commit was sponsored by Francois Marier on Patreon.
2017-10-30 12:02:22 -04:00
Joey Hess
43d011a524
releasing package git-annex version 6.20171026 2017-10-26 14:02:40 -04:00
Joey Hess
0ae2ac282e
fix gitAnnexLink to not be absolute on Windows
Windows: Fix reversion that caused the path used to link to annexed
content include the drive letter and full path, rather than being
relative. (`git annex fix` will fix up after this problem).

I've not identified the commit that brought the reversion (probably it
happened this spring when I was removing MisingH and last touched
Utility.Path). Likely commit 18b9a4b8024115db67ae309fdaf54e1553037529?

The problem is that relPathDirToFile got called two paths that had the
slashes different ways around. Since takeDrive includes the first slash,
this made two paths on the same drive seem different and it bailed.

(ifdefs around this to avoid doing extra work on non-windows)

This commit was sponsored by Jack Hill on Patreon.
2017-10-25 19:36:29 -04:00
Joey Hess
e8eacf96d5
Add day to metadata when annex.genmetadata is enabled.
Thanks, Sean T Parsons
2017-10-25 15:11:38 -04:00
Joey Hess
64f1c09526
stack.yaml: Added nix packages section.
Thanks, Sean T Parsons
2017-10-25 15:09:18 -04:00
Joey Hess
833b3f06cd
build for windows with forked win32 package that has terminateProcessId
Get ugly reversion out of CHANGELOG.

Also, relocated the windows stack.yaml to top, and updated windows build
instructions.

This commit was sponsored by Henrik Riomar on Patreon.
2017-10-25 14:45:23 -04:00
Joey Hess
54e596894f
Windows: Remove wget from bundle; it needs libraries that are not included, and git for windows includes curl which git-annex will use instead.
wget was broken even in the previous old release of the windows bundle,
this is not new breakage. msys-idn-11.dll and probably more would be needed
to use it. git for windows includes msys-idn2-0.dll instead.
2017-10-25 13:51:31 -04:00
Joey Hess
3e839ab327
temporary hack to get windows build working
Code for terminating processes on Windows is not linking anymore;
made a warning be displayed instead. This breaks restarting the
assistant and git annex assistant --stop.

I hope to see the code added to the Win32 library, where it should fit
better and should avoid whatever problem is making the linker not like it
when included in git-annex. I opened an issue requesting its addition,
here: https://github.com/haskell/win32/issues/91

This commit was sponsored by Thomas Hochstein on Patreon.
2017-10-24 13:16:40 -04:00
Joey Hess
bb20b1ed35
releasing package git-annex version 6.20171018 2017-10-18 15:41:08 -04:00
Joey Hess
e0966bbb16
reorder 2017-10-18 13:36:43 -04:00
Joey Hess
0c1c55efea
changelog 2017-10-18 13:34:53 -04:00
Joey Hess
e1ac299ad0
better dup key with -J fix
This avoids all the complication about redundant work discussed in
the previous try at fixing this. At the expense of needing each command
that could have the problem to be patched to simply wrap the action in
onlyActionOn once the key is known. But there do not seem to be many
such commands.

onlyActionOn' should not be used with a CommandStart (or CommandPerform),
although the types do allow it. onlyActionOn handles running the whole
CommandStart chain. I couldn't immediately see a way to avoid mistken
use of onlyActionOn'.

This commit was supported by the NSF-funded DataLad project.
2017-10-17 18:48:53 -04:00
Joey Hess
68a49adcda
Improve behavior when -J transfers multiple files that point to the same key
After a false start, I found a fairly non-intrusive way to deal with it.
Although it only handles transfers -- there may be issues with eg
concurrent dropping of the same key, or other operations.

There is no added overhead when -J is not used, other than an added
inAnnex check. When -J is used, it has to maintain and check a small
Set, which should be negligible overhead.

It could output some message saying that the transfer is being done by
another thread. Or it could even display the same progress info for both
files that are being downloaded since they have the same content. But I
opted to keep it simple, since this is rather an edge case, so it just
doesn't say anything about the transfer of the file until the other
thread finishes.

Since the deferred transfer action still runs, actions that do more than
transfer content will still get a chance to do their other work. (An
example of something that needs to do such other work is P2P.Annex,
where the download always needs to receive the content from the peer.)
And, if the first thread fails to complete a transfer, the second thread
can resume it.

But, this unfortunately means that there's a risk of redundant work
being done to transfer a key that just got transferred.
That's not ideal, but should never cause breakage; the same
thing can occur when running two separate git-annex processes.

The get/move/copy/mirror --from commands had extra inAnnex checks added,
inside the download actions. Without those checks, the first thread
downloaded the content, and then the second thread woke up and
downloaded the same content redundantly.

move/copy/mirror --to is left doing redundant uploads for now. It
would need a second checkPresent of the remote inside the upload
to avoid them, which would be expensive. A better way to avoid
redundant work needs to be found..

This commit was supported by the NSF-funded DataLad project.
2017-10-17 17:10:50 -04:00
Joey Hess
193b5f4ca4
new lts 2017-10-17 13:00:22 -04:00
Joey Hess
0c7fee832a
close bug 2017-10-16 14:50:30 -04:00
Joey Hess
85ed38a574
Avoid repeated checking that files passed on the command line exist.
git annex add, git annex lock etc make multiple seek passes,
and each seek pass checked that files existed. That was unncessary
redundant work.

Fixed by adding a new WorkTreeItem type, make seek actions use it,
and check that the files exist when constructing it.

This commit was supported by the NSF-funded DataLad project.
2017-10-16 14:10:20 -04:00
Joey Hess
a461cf2ce6
add: Replace work tree file atomically.
Before, there was a window where interrupting an add could result in the
file being moved into the annex, with no symlink yet created.

This commit was supported by the NSF-funded DataLad project.
2017-10-16 12:57:28 -04:00