Commit graph

23 commits

Author SHA1 Message Date
Joey Hess
003fc2b7e1
add UrlOptions sum type 2014-02-24 22:00:25 -04:00
Joey Hess
86ffeb73d1 reorganize some files and imports 2014-01-26 16:25:55 -04:00
Joey Hess
34c8af74ba fix inversion of control in CommandSeek (no behavior changes)
I've been disliking how the command seek actions were written for some
time, with their inversion of control and ugly workarounds.

The last straw to fix it was sync --content, which didn't fit the
Annex [CommandStart] interface well at all. I have not yet made it take
advantage of the changed interface though.

The crucial change, and probably why I didn't do it this way from the
beginning, is to make each CommandStart action be run with exceptions
caught, and if it fails, increment a failure counter in annex state.
So I finally remove the very first code I wrote for git-annex, which
was before I had exception handling in the Annex monad, and so ran outside
that monad, passing state explicitly as it ran each CommandStart action.

This was a real slog from 1 to 5 am.

Test suite passes.

Memory usage is lower than before, sometimes by a couple of megabytes, and
remains constant, even when running in a large repo, and even when
repeatedly failing and incrementing the error counter. So no accidental
laziness space leaks.

Wall clock speed is identical, even in large repos.

This commit was sponsored by an anonymous bitcoiner.
2014-01-20 04:57:36 -04:00
Joey Hess
964f5d080e fix form of quvi url 2014-01-05 13:35:14 -04:00
Joey Hess
9e6b2144f9 check if quvi page url has already been downloaded before
That is the url that is typically stored in the git-annex branch.

Kept the check for the video url too, just in case.
2014-01-05 13:07:21 -04:00
Joey Hess
81f498559a importfeed: Support youtube playlists. 2013-12-29 15:52:20 -04:00
Joey Hess
94251de91e add readFileStrictAnyEncoding 2013-11-20 13:41:13 -04:00
Joey Hess
12f6b9693a Send a git-annex user-agent when downloading urls.
Overridable with --user-agent option.

Not yet done for S3 or WebDAV due to limitations of libraries used --
nether allows a user-agent header to be specified.

This commit sponsored by Michael Zehrer.
2013-09-28 14:35:21 -04:00
Joey Hess
b405295aee hlint
test suite still passes
2013-09-25 03:09:06 -04:00
Joey Hess
b51dffa46d fix error propigating when unable to download feed item 2013-09-03 14:39:07 -04:00
Joey Hess
d1bacccff4 importfeed: Also ignore transient problems with downloading content from feeds. 2013-09-03 14:32:26 -04:00
Joey Hess
46b6d75274 Youtube support! (And 53 other video hosts)
When quvi is installed, git-annex addurl automatically uses it to detect
when an page is a video, and downloads the video file.

web special remote: Also support using quvi, for getting files,
or checking if files exist in the web.

This commit was sponsored by Mark Hepburn. Thanks!
2013-08-22 18:50:43 -04:00
Joey Hess
b28023cb52 importfeed: Fix handling of dots in extensions. 2013-08-03 02:36:38 -04:00
Joey Hess
24c8a6042b importfeed: Ignores transient problems with feeds. Only exits nonzero when a feed has repeatedly had a problems for at least 1 day. 2013-08-03 01:40:21 -04:00
Joey Hess
03c76b5a30 improve importfeed --force; try to match existing files to avoid unncessary duplication 2013-08-01 11:57:05 -04:00
Joey Hess
42ca8aaa61 importfeed --force: re-download urls that have been seen before 2013-07-31 12:19:00 -04:00
Joey Hess
07a9910af7 improve comment 2013-07-28 20:15:20 -04:00
Joey Hess
ac08924ec3 fix bug in makeUnique
Returned the possibly non-unique file
2013-07-28 20:14:13 -04:00
Joey Hess
8c55970413 better extension handling
When there's no extension, don't use "none", but "".

When there is an extension, it starts with a dot, so don't put a redundant
dot in the default format.
2013-07-28 19:08:50 -04:00
Joey Hess
8c8488e01a if a feed cannot be downloaded or has no enclosures, fail 2013-07-28 18:16:24 -04:00
Joey Hess
18541bf3fa don't crash on encoding issues in feeds
filesystem encoding to the rescue once more!

IIRC this was the main bug in hpodder.
2013-07-28 17:24:30 -04:00
Joey Hess
66dfeaff44 show a side action when finding known urls 2013-07-28 17:19:21 -04:00
Joey Hess
7e66d260ea importfeed: git-annex becomes a podcatcher in 150 LOC 2013-07-28 16:55:42 -04:00