git-annex/doc/git-annex-importfeed.mdwn

# NAME

git-annex importfeed - import files from podcast feeds

# SYNOPSIS

git annex importfeed `[url ...]`

# DESCRIPTION

Imports the contents of podcasts and other feeds. Only downloads files whose
content has not already been added to the repository before, so you can
delete, rename, etc the resulting files and repeated runs won't duplicate
them.

When `youtube-dl` is installed, it can be used to download links in the feed.
This allows importing e.g., YouTube playlists.
(However, this is disabled by default as it can be a security risk. 
See the documentation of annex.security.allowed-ip-addresses
in [[git-annex]](1) for details.)

To make the import process add metadata to the imported files from the feed,
`git config annex.genmetadata true`

# OPTIONS

* `--force`

  Force downloading items it's seen before.

* `--template`

  Controls where the files are stored.

  The default template is '${feedtitle}/${itemtitle}${extension}'
  
  The available variables in the template include these that
  are information about the feed: feedtitle, feedauthor

  And these that are information about individual items in the feed:
  itemtitle, itemauthor, itemsummary, itemdescription, itemrights,
  itemid.

  Also, title is itemtitle but falls back to feedtitle if the item has no
  title, and author is itemauthor but falls back to feedauthor.

  (All of the above are also added as metadata when annex.genmetadata is
  set.)

  The extension variable is the extension of the file in the feed,
  or sometimes ".m" if no extension can be determined.

  The template also has some variables for when an item was published.
  
  itempubyear (YYYY), itempubmonth (MM), itempubday (DD), itempubhour (HH),
  itempubminute (MM), itempubsecond (SS),
  itempubdate (YYYY-MM-DD or if the feed's date cannot be parsed, the raw
  value from the feed).
  
  (These use the UTC time zone, not the local time zone.)

* `--relaxed`, `--fast`, `--raw`

  These options behave the same as when using [[git-annex-addurl]](1).

* `--fast`

  Avoid immediately downloading urls. The url is still checked
  (via HEAD) to verify that it exists, and to get its size if possible.

* `--relaxed`

  Don't immediately download urls, and avoid storing the size of the
  url's content. This makes git-annex accept whatever content is there
  at a future point.

* `--raw`

  Prevent special handling of urls by youtube-dl, bittorrent, and other
  special remotes. This will for example, make importfeed
  download a .torrent file and not the contents it points to.

# SEE ALSO

[[git-annex]](1)

[[git-annex-addurl]](1)

# AUTHOR

Joey Hess <id@joeyh.name>

Warning: Automatically converted into a man page by mdwn2man. Edit with care.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			`# NAME`

			`git-annex importfeed - import files from podcast feeds`

			`# SYNOPSIS`

			git annex importfeed `[url ...]`

			`# DESCRIPTION`

convert importfeed to youtube-dl Fully working, including --fast/--relaxed. Note that, while git-annex addurl --relaxed is not going to check youtube-dl, I kept git annex importfeed --relaxed checking it. Thinking is that, let's not break people's importfeed cron jobs, and importfeed does not typically have to check a large number of new items, so it's ok if it's a little bit slower when used with youtube playlist feeds. importfeed's behavior is also improved (?) when a feed has links in it to non-media files. Before, those were skipped. Now, the content of the link is downloaded. This had to be done, because trying to use youtube-dl is slow, and if those were skipped, it would have to check every time importfeed was run. While this behavior change may not be desirable for some feeds, that intersperse links to web pages with enclosures, it will be desirable for other feeds, that have non-enclosure directy links to media files. Remove old quvi modules. This commit was sponsored by Øyvind Andersen Holm. 2017-11-29 21:05:27 +00:00			`Imports the contents of podcasts and other feeds. Only downloads files whose`
update importfeed man page, including a mention of annex.genmetadata 2015-03-31 17:48:13 +00:00			`content has not already been added to the repository before, so you can`
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			`delete, rename, etc the resulting files and repeated runs won't duplicate`
			`them.`

default to not using youtube-dl, for security Pity, but same reasoning as curl applies to it. This commit was sponsored by Peter on Patreon. 2018-06-17 18:46:22 +00:00			When `youtube-dl` is installed, it can be used to download links in the feed.
documentation update for youtube-dl Code not updated yet. This commit was sponsored by Thomas Hochstein on Patreon. 2017-11-28 18:05:58 +00:00			`This allows importing e.g., YouTube playlists.`
default to not using youtube-dl, for security Pity, but same reasoning as curl applies to it. This commit was sponsored by Peter on Patreon. 2018-06-17 18:46:22 +00:00			`(However, this is disabled by default as it can be a security risk.`
rename annex.security.allowed-http-addresses Renamed annex.security.allowed-http-addresses to annex.security.allowed-ip-addresses because it is not really specific to the http protocol, also limiting eg, git-annex's use of ftp and via youtube-dl, several other protocols. The old name for the config will still work. If both old and new name are set, the new name will win. 2019-05-30 16:43:40 +00:00			`See the documentation of annex.security.allowed-ip-addresses`
default to not using youtube-dl, for security Pity, but same reasoning as curl applies to it. This commit was sponsored by Peter on Patreon. 2018-06-17 18:46:22 +00:00			`in [[git-annex]](1) for details.)`
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00
update importfeed man page, including a mention of annex.genmetadata 2015-03-31 17:48:13 +00:00			`To make the import process add metadata to the imported files from the feed,`
			`git config annex.genmetadata true`

splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			`# OPTIONS`

			* `--force`

doc/*.mdwn: Various typo fixes 2015-05-30 14:54:14 +00:00			`Force downloading items it's seen before.`
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00
			* `--template`

			`Controls where the files are stored.`

			`The default template is '${feedtitle}/${itemtitle}${extension}'`

improve docs of template variables 2020-06-24 17:23:16 +00:00			`The available variables in the template include these that`
			`are information about the feed: feedtitle, feedauthor`

			`And these that are information about individual items in the feed:`
			`itemtitle, itemauthor, itemsummary, itemdescription, itemrights,`
			`itemid.`

			`Also, title is itemtitle but falls back to feedtitle if the item has no`
			`title, and author is itemauthor but falls back to feedauthor.`

			`(All of the above are also added as metadata when annex.genmetadata is`
			`set.)`

importfeed: Added some additional --template variables for date and time This commit was sponsored by Ethan Aubin. 2020-06-24 18:24:50 +00:00			`The extension variable is the extension of the file in the feed,`
			`or sometimes ".m" if no extension can be determined.`

improve docs of template variables 2020-06-24 17:23:16 +00:00			`The template also has some variables for when an item was published.`

importfeed: Added some additional --template variables for date and time This commit was sponsored by Ethan Aubin. 2020-06-24 18:24:50 +00:00			`itempubyear (YYYY), itempubmonth (MM), itempubday (DD), itempubhour (HH),`
			`itempubminute (MM), itempubsecond (SS),`
			`itempubdate (YYYY-MM-DD or if the feed's date cannot be parsed, the raw`
improve docs of template variables 2020-06-24 17:23:16 +00:00			`value from the feed).`

			`(These use the UTC time zone, not the local time zone.)`
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00
fix typo 2015-07-21 16:20:48 +00:00			* `--relaxed`, `--fast`, `--raw`
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00
			`These options behave the same as when using [[git-annex-addurl]](1).`

convert importfeed to youtube-dl Fully working, including --fast/--relaxed. Note that, while git-annex addurl --relaxed is not going to check youtube-dl, I kept git annex importfeed --relaxed checking it. Thinking is that, let's not break people's importfeed cron jobs, and importfeed does not typically have to check a large number of new items, so it's ok if it's a little bit slower when used with youtube playlist feeds. importfeed's behavior is also improved (?) when a feed has links in it to non-media files. Before, those were skipped. Now, the content of the link is downloaded. This had to be done, because trying to use youtube-dl is slow, and if those were skipped, it would have to check every time importfeed was run. While this behavior change may not be desirable for some feeds, that intersperse links to web pages with enclosures, it will be desirable for other feeds, that have non-enclosure directy links to media files. Remove old quvi modules. This commit was sponsored by Øyvind Andersen Holm. 2017-11-29 21:05:27 +00:00			* `--fast`

			`Avoid immediately downloading urls. The url is still checked`
			`(via HEAD) to verify that it exists, and to get its size if possible.`

			* `--relaxed`

			`Don't immediately download urls, and avoid storing the size of the`
			`url's content. This makes git-annex accept whatever content is there`
			`at a future point.`

			* `--raw`

			`Prevent special handling of urls by youtube-dl, bittorrent, and other`
			`special remotes. This will for example, make importfeed`
			`download a .torrent file and not the contents it points to.`

splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			`# SEE ALSO`

			`[[git-annex]](1)`

expand manpages cross-references significantly i found that most man pages only had references to the main git-annex manpage, which i stillfind pretty huge and hard to navigate through. i tried to sift through all the man pages and add cross-references between relevant pages. my general rule of thumb is that links should be both ways unless one of the pages is a more general page that would become ridiculously huge if all backlinks would be added (git-annex-preferred-content comes to mind). i have also make the links one per line as this is how it was done in the metadata pages so far. i did everything but the plumbing, utility and test commands, although some of those are linked from the other commands so cross-links were added there as well. 2015-05-29 16:12:55 +00:00			`[[git-annex-addurl]](1)`

splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			`# AUTHOR`

			`Joey Hess <id@joeyh.name>`

			`Warning: Automatically converted into a man page by mdwn2man. Edit with care.`