Commit graph

132 commits

Author SHA1 Message Date
Joey Hess
7b50b3c057 fix some mixed space+tab indentation
This fixes all instances of " \t" in the code base. Most common case
seems to be after a "where" line; probably vim copied the two space layout
of that line.

Done as a background task while listening to episode 2 of the Type Theory
podcast.
2014-10-09 15:09:11 -04:00
Joey Hess
d15bfc9061 fix build with old feed 2014-07-19 18:30:08 -04:00
Joey Hess
810bc10857 include feed url in repeated problems message 2014-07-14 13:07:13 -04:00
Joey Hess
fc80956092
really add non-date metadata too 2014-07-03 14:35:20 -04:00
Joey Hess
d0c1a22e7c import metadata from feeds
When annex.genmetadata is set, metadata from the feed is added to files
that are imported from it.

Reused the same feedtitle and itemtitle, feedauthor, itemauthor, etc names
that are used in --template.

Also added title and author, which are the item title/author if available,
falling back to the feed title/author. These are more likely to be common
metadata fields.

(There is a small bit of dupication here, but once git gets
around to packing the object, it will compress it away.)

The itempubdate field is not included in the metadata as a string; instead
it is used to generate year and month fields, same as is done when adding
files with annex.genmetadata set.

This commit was sponsored by Amitai Schlair, who cooincidentially
is responsible for ikiwiki generating nice feed metadata!
2014-07-03 14:15:00 -04:00
Joey Hess
291847bf88
allow building with old feed lib, w/o pubdate 2014-04-21 00:37:14 -04:00
Joey Hess
e880d0d22c replace (Key, Backend) with Key
Only fsck and reinject and the test suite used the Backend, and they can
look it up as needed from the Key. This simplifies the code and also speeds
it up.

There is a small behavior change here. Before, all commands would warn when
acting on an annexed file with an unknown backend. Now, only fsck and
reinject show that warning.
2014-04-17 18:03:39 -04:00
Joey Hess
836f2cc816 importfeed: Filename template can now contain an itempubdate variable. Needs feed 0.3.9.2. 2014-04-07 16:55:04 -04:00
Joey Hess
7ac37a7854 Probe for quvi version at run time.
Overhead: git annex addurl runs quvi --version once.
And more bloat to Annex state..
2014-02-28 14:54:02 -04:00
Joey Hess
003fc2b7e1
add UrlOptions sum type 2014-02-24 22:00:25 -04:00
Joey Hess
86ffeb73d1 reorganize some files and imports 2014-01-26 16:25:55 -04:00
Joey Hess
34c8af74ba fix inversion of control in CommandSeek (no behavior changes)
I've been disliking how the command seek actions were written for some
time, with their inversion of control and ugly workarounds.

The last straw to fix it was sync --content, which didn't fit the
Annex [CommandStart] interface well at all. I have not yet made it take
advantage of the changed interface though.

The crucial change, and probably why I didn't do it this way from the
beginning, is to make each CommandStart action be run with exceptions
caught, and if it fails, increment a failure counter in annex state.
So I finally remove the very first code I wrote for git-annex, which
was before I had exception handling in the Annex monad, and so ran outside
that monad, passing state explicitly as it ran each CommandStart action.

This was a real slog from 1 to 5 am.

Test suite passes.

Memory usage is lower than before, sometimes by a couple of megabytes, and
remains constant, even when running in a large repo, and even when
repeatedly failing and incrementing the error counter. So no accidental
laziness space leaks.

Wall clock speed is identical, even in large repos.

This commit was sponsored by an anonymous bitcoiner.
2014-01-20 04:57:36 -04:00
Joey Hess
964f5d080e fix form of quvi url 2014-01-05 13:35:14 -04:00
Joey Hess
9e6b2144f9 check if quvi page url has already been downloaded before
That is the url that is typically stored in the git-annex branch.

Kept the check for the video url too, just in case.
2014-01-05 13:07:21 -04:00
Joey Hess
81f498559a importfeed: Support youtube playlists. 2013-12-29 15:52:20 -04:00
Joey Hess
94251de91e add readFileStrictAnyEncoding 2013-11-20 13:41:13 -04:00
Joey Hess
12f6b9693a Send a git-annex user-agent when downloading urls.
Overridable with --user-agent option.

Not yet done for S3 or WebDAV due to limitations of libraries used --
nether allows a user-agent header to be specified.

This commit sponsored by Michael Zehrer.
2013-09-28 14:35:21 -04:00
Joey Hess
b405295aee hlint
test suite still passes
2013-09-25 03:09:06 -04:00
Joey Hess
b51dffa46d fix error propigating when unable to download feed item 2013-09-03 14:39:07 -04:00
Joey Hess
d1bacccff4 importfeed: Also ignore transient problems with downloading content from feeds. 2013-09-03 14:32:26 -04:00
Joey Hess
46b6d75274 Youtube support! (And 53 other video hosts)
When quvi is installed, git-annex addurl automatically uses it to detect
when an page is a video, and downloads the video file.

web special remote: Also support using quvi, for getting files,
or checking if files exist in the web.

This commit was sponsored by Mark Hepburn. Thanks!
2013-08-22 18:50:43 -04:00
Joey Hess
b28023cb52 importfeed: Fix handling of dots in extensions. 2013-08-03 02:36:38 -04:00
Joey Hess
24c8a6042b importfeed: Ignores transient problems with feeds. Only exits nonzero when a feed has repeatedly had a problems for at least 1 day. 2013-08-03 01:40:21 -04:00
Joey Hess
03c76b5a30 improve importfeed --force; try to match existing files to avoid unncessary duplication 2013-08-01 11:57:05 -04:00
Joey Hess
42ca8aaa61 importfeed --force: re-download urls that have been seen before 2013-07-31 12:19:00 -04:00
Joey Hess
07a9910af7 improve comment 2013-07-28 20:15:20 -04:00
Joey Hess
ac08924ec3 fix bug in makeUnique
Returned the possibly non-unique file
2013-07-28 20:14:13 -04:00
Joey Hess
8c55970413 better extension handling
When there's no extension, don't use "none", but "".

When there is an extension, it starts with a dot, so don't put a redundant
dot in the default format.
2013-07-28 19:08:50 -04:00
Joey Hess
8c8488e01a if a feed cannot be downloaded or has no enclosures, fail 2013-07-28 18:16:24 -04:00
Joey Hess
18541bf3fa don't crash on encoding issues in feeds
filesystem encoding to the rescue once more!

IIRC this was the main bug in hpodder.
2013-07-28 17:24:30 -04:00
Joey Hess
66dfeaff44 show a side action when finding known urls 2013-07-28 17:19:21 -04:00
Joey Hess
7e66d260ea importfeed: git-annex becomes a podcatcher in 150 LOC 2013-07-28 16:55:42 -04:00