Commit graph

48 commits

Author SHA1 Message Date
Joey Hess
77c43a388e fromkey, registerurl: Allow urls to be specified instead of keys, and generate URL keys.
This is especially useful because the caller doesn't need to generate valid
url keys, which involves some escaping of characters, and may involve
taking a md5sum of the url if it's too long.
2015-05-22 22:41:36 -04:00
Joey Hess
6c2d5b5e41 more time-1.5 fixes 2015-05-10 15:36:58 -04:00
Joey Hess
f395ca7a41 Improve quvi 0.4 output parsing to handle cases wher there is no known filename extension. This is currently the case when using quvi with youtube. In this case, the extension ".m" will be used. 2015-05-08 13:39:00 -04:00
Joey Hess
3695094454 importfeed: Error out when passed a non-url. 2015-04-09 13:09:06 -04:00
Joey Hess
aa31af5594 importfeed: Fix feed download when curl is used.
This bug used to only afflict file:/// urls, but became more exposed
by b8f0b7309f.
2015-04-09 13:02:57 -04:00
Joey Hess
9e25cbde20 importfeed: Avoid downloading a redundant item from a feed whose guid has been downloaded before, even when the url has changed.
To support this, always store itemid in metadata; before this was only done
when annex.genmetadata was set.
2015-03-31 13:30:13 -04:00
Joey Hess
504dda82a4 addurl: Added --raw option, which bypasses special handling of quvi, bittorrent etc urls. 2015-03-05 14:46:08 -04:00
Joey Hess
587f6a919b addurl: When a Content-Disposition header suggests a filename to use, addurl will consider using it, if it's reasonable and doesn't conflict with an existing file. (--file overrides this) 2015-01-22 14:52:52 -04:00
Joey Hess
afc5153157 update my email address and homepage url 2015-01-21 12:50:09 -04:00
Joey Hess
3bab5dfb1d revert parentDir change
Reverts 965e106f24

Unfortunately, this caused breakage on Windows, and possibly elsewhere,
because parentDir and takeDirectory do not behave the same when there is a
trailing directory separator.
2015-01-09 13:11:56 -04:00
Joey Hess
965e106f24 made parentDir return a Maybe FilePath; removed most uses of it
parentDir is less safe than takeDirectory, especially when working
with relative FilePaths. It's really only useful in loops that
want to terminate at /

This commit was sponsored by Audric SCHILTKNECHT.
2015-01-06 18:55:56 -04:00
Joey Hess
7e422269a6 move dummy uuids to Annex.UUID 2014-12-17 13:57:52 -04:00
Joey Hess
67c05daf5e sanitize filepaths provided by checkUrl 2014-12-11 20:08:49 -04:00
Joey Hess
f41ce65ef4 importfeed: Support for downloading urls claimed by remotes 2014-12-11 16:43:46 -04:00
Joey Hess
30685751ea add missing quvi ifdef 2014-12-11 13:12:52 -04:00
Joey Hess
59f88558d5 doh't use "def" for command definitions, it conflicts with Data.Default.def 2014-10-14 14:20:10 -04:00
Joey Hess
7b50b3c057 fix some mixed space+tab indentation
This fixes all instances of " \t" in the code base. Most common case
seems to be after a "where" line; probably vim copied the two space layout
of that line.

Done as a background task while listening to episode 2 of the Type Theory
podcast.
2014-10-09 15:09:11 -04:00
Joey Hess
d15bfc9061 fix build with old feed 2014-07-19 18:30:08 -04:00
Joey Hess
810bc10857 include feed url in repeated problems message 2014-07-14 13:07:13 -04:00
Joey Hess
fc80956092
really add non-date metadata too 2014-07-03 14:35:20 -04:00
Joey Hess
d0c1a22e7c import metadata from feeds
When annex.genmetadata is set, metadata from the feed is added to files
that are imported from it.

Reused the same feedtitle and itemtitle, feedauthor, itemauthor, etc names
that are used in --template.

Also added title and author, which are the item title/author if available,
falling back to the feed title/author. These are more likely to be common
metadata fields.

(There is a small bit of dupication here, but once git gets
around to packing the object, it will compress it away.)

The itempubdate field is not included in the metadata as a string; instead
it is used to generate year and month fields, same as is done when adding
files with annex.genmetadata set.

This commit was sponsored by Amitai Schlair, who cooincidentially
is responsible for ikiwiki generating nice feed metadata!
2014-07-03 14:15:00 -04:00
Joey Hess
291847bf88
allow building with old feed lib, w/o pubdate 2014-04-21 00:37:14 -04:00
Joey Hess
e880d0d22c replace (Key, Backend) with Key
Only fsck and reinject and the test suite used the Backend, and they can
look it up as needed from the Key. This simplifies the code and also speeds
it up.

There is a small behavior change here. Before, all commands would warn when
acting on an annexed file with an unknown backend. Now, only fsck and
reinject show that warning.
2014-04-17 18:03:39 -04:00
Joey Hess
836f2cc816 importfeed: Filename template can now contain an itempubdate variable. Needs feed 0.3.9.2. 2014-04-07 16:55:04 -04:00
Joey Hess
7ac37a7854 Probe for quvi version at run time.
Overhead: git annex addurl runs quvi --version once.
And more bloat to Annex state..
2014-02-28 14:54:02 -04:00
Joey Hess
003fc2b7e1
add UrlOptions sum type 2014-02-24 22:00:25 -04:00
Joey Hess
86ffeb73d1 reorganize some files and imports 2014-01-26 16:25:55 -04:00
Joey Hess
34c8af74ba fix inversion of control in CommandSeek (no behavior changes)
I've been disliking how the command seek actions were written for some
time, with their inversion of control and ugly workarounds.

The last straw to fix it was sync --content, which didn't fit the
Annex [CommandStart] interface well at all. I have not yet made it take
advantage of the changed interface though.

The crucial change, and probably why I didn't do it this way from the
beginning, is to make each CommandStart action be run with exceptions
caught, and if it fails, increment a failure counter in annex state.
So I finally remove the very first code I wrote for git-annex, which
was before I had exception handling in the Annex monad, and so ran outside
that monad, passing state explicitly as it ran each CommandStart action.

This was a real slog from 1 to 5 am.

Test suite passes.

Memory usage is lower than before, sometimes by a couple of megabytes, and
remains constant, even when running in a large repo, and even when
repeatedly failing and incrementing the error counter. So no accidental
laziness space leaks.

Wall clock speed is identical, even in large repos.

This commit was sponsored by an anonymous bitcoiner.
2014-01-20 04:57:36 -04:00
Joey Hess
964f5d080e fix form of quvi url 2014-01-05 13:35:14 -04:00
Joey Hess
9e6b2144f9 check if quvi page url has already been downloaded before
That is the url that is typically stored in the git-annex branch.

Kept the check for the video url too, just in case.
2014-01-05 13:07:21 -04:00
Joey Hess
81f498559a importfeed: Support youtube playlists. 2013-12-29 15:52:20 -04:00
Joey Hess
94251de91e add readFileStrictAnyEncoding 2013-11-20 13:41:13 -04:00
Joey Hess
12f6b9693a Send a git-annex user-agent when downloading urls.
Overridable with --user-agent option.

Not yet done for S3 or WebDAV due to limitations of libraries used --
nether allows a user-agent header to be specified.

This commit sponsored by Michael Zehrer.
2013-09-28 14:35:21 -04:00
Joey Hess
b405295aee hlint
test suite still passes
2013-09-25 03:09:06 -04:00
Joey Hess
b51dffa46d fix error propigating when unable to download feed item 2013-09-03 14:39:07 -04:00
Joey Hess
d1bacccff4 importfeed: Also ignore transient problems with downloading content from feeds. 2013-09-03 14:32:26 -04:00
Joey Hess
46b6d75274 Youtube support! (And 53 other video hosts)
When quvi is installed, git-annex addurl automatically uses it to detect
when an page is a video, and downloads the video file.

web special remote: Also support using quvi, for getting files,
or checking if files exist in the web.

This commit was sponsored by Mark Hepburn. Thanks!
2013-08-22 18:50:43 -04:00
Joey Hess
b28023cb52 importfeed: Fix handling of dots in extensions. 2013-08-03 02:36:38 -04:00
Joey Hess
24c8a6042b importfeed: Ignores transient problems with feeds. Only exits nonzero when a feed has repeatedly had a problems for at least 1 day. 2013-08-03 01:40:21 -04:00
Joey Hess
03c76b5a30 improve importfeed --force; try to match existing files to avoid unncessary duplication 2013-08-01 11:57:05 -04:00
Joey Hess
42ca8aaa61 importfeed --force: re-download urls that have been seen before 2013-07-31 12:19:00 -04:00
Joey Hess
07a9910af7 improve comment 2013-07-28 20:15:20 -04:00
Joey Hess
ac08924ec3 fix bug in makeUnique
Returned the possibly non-unique file
2013-07-28 20:14:13 -04:00
Joey Hess
8c55970413 better extension handling
When there's no extension, don't use "none", but "".

When there is an extension, it starts with a dot, so don't put a redundant
dot in the default format.
2013-07-28 19:08:50 -04:00
Joey Hess
8c8488e01a if a feed cannot be downloaded or has no enclosures, fail 2013-07-28 18:16:24 -04:00
Joey Hess
18541bf3fa don't crash on encoding issues in feeds
filesystem encoding to the rescue once more!

IIRC this was the main bug in hpodder.
2013-07-28 17:24:30 -04:00
Joey Hess
66dfeaff44 show a side action when finding known urls 2013-07-28 17:19:21 -04:00
Joey Hess
7e66d260ea importfeed: git-annex becomes a podcatcher in 150 LOC 2013-07-28 16:55:42 -04:00