git-annex/doc/git-annex-importfeed.mdwn

# NAME

git-annex importfeed - import files from podcast feeds

# SYNOPSIS

git annex importfeed `[url ...]`

# DESCRIPTION

Imports the contents of podcasts and other feeds. Only downloads files whose
content has not already been added to the repository before, so you can
delete, rename, etc the resulting files and repeated runs won't duplicate
them.

When `youtube-dl` is installed, it can be used to download links in the feed.
This allows importing e.g., YouTube playlists.
(However, this is disabled by default as it can be a security risk. 
See the documentation of annex.security.allowed-ip-addresses
in [[git-annex]](1) for details.)

To make the import process add metadata to the imported files from the feed,
`git config annex.genmetadata true`

By default, the downloaded files are put in a directory with the title
of the feed, and files are named based on the title of the item in the
feed. This can be changed using the --template option.

Existing files are not overwritten by this command. If "some feed/foo.mp3"
already exists, it will instead write to "some feed/2\_foo.mp3"
(or 3, 4, etc). Sometimes a feed will change an item's url,
resulting in the new url being downloaded to such a filename.

# OPTIONS

* `--force`

  Force downloading items it's seen before.

* `--relaxed`, `--fast`, `--raw`

  These options behave the same as when using [[git-annex-addurl]](1).

* `--fast`

  Avoid immediately downloading urls. The url is still checked
  (via HEAD) to verify that it exists, and to get its size if possible.

* `--relaxed`

  Don't immediately download urls, and avoid storing the size of the
  url's content. This makes git-annex accept whatever content is there
  at a future point.

* `--raw`

  Prevent special handling of urls by youtube-dl, bittorrent, and other
  special remotes. This will for example, make importfeed
  download a .torrent file and not the contents it points to.

* `--template`

  Controls where the files are stored.

  The default template is '${feedtitle}/${itemtitle}${extension}'
  
  The available variables in the template include these that
  are information about the feed: feedtitle, feedauthor

  And these that are information about individual items in the feed:
  itemtitle, itemauthor, itemsummary, itemdescription, itemrights,
  itemid.

  Also, title is itemtitle but falls back to feedtitle if the item has no
  title, and author is itemauthor but falls back to feedauthor.

  (All of the above are also added as metadata when annex.genmetadata is
  set.)

  The extension variable is the extension of the file in the feed,
  or sometimes ".m" if no extension can be determined.

  The template also has some variables for when an item was published.
  
  itempubyear (YYYY), itempubmonth (MM), itempubday (DD), itempubhour (HH),
  itempubminute (MM), itempubsecond (SS),
  itempubdate (YYYY-MM-DD or if the feed's date cannot be parsed, the raw
  value from the feed).
  
  (These use the UTC time zone, not the local time zone.)

* `--no-check-gitignore`

  By default, gitignores are honored and it will refuse to download an
  url to a file that would be ignored. This makes such files be added
  despite any ignores.

# SEE ALSO

[[git-annex]](1)

[[git-annex-addurl]](1)

# AUTHOR

Joey Hess <id@joeyh.name>

Warning: Automatically converted into a man page by mdwn2man. Edit with care.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			`# NAME`

			`git-annex importfeed - import files from podcast feeds`

			`# SYNOPSIS`

			git annex importfeed `[url ...]`

			`# DESCRIPTION`

convert importfeed to youtube-dl Fully working, including --fast/--relaxed. Note that, while git-annex addurl --relaxed is not going to check youtube-dl, I kept git annex importfeed --relaxed checking it. Thinking is that, let's not break people's importfeed cron jobs, and importfeed does not typically have to check a large number of new items, so it's ok if it's a little bit slower when used with youtube playlist feeds. importfeed's behavior is also improved (?) when a feed has links in it to non-media files. Before, those were skipped. Now, the content of the link is downloaded. This had to be done, because trying to use youtube-dl is slow, and if those were skipped, it would have to check every time importfeed was run. While this behavior change may not be desirable for some feeds, that intersperse links to web pages with enclosures, it will be desirable for other feeds, that have non-enclosure directy links to media files. Remove old quvi modules. This commit was sponsored by Øyvind Andersen Holm. 2017-11-29 21:05:27 +00:00			`Imports the contents of podcasts and other feeds. Only downloads files whose`
update importfeed man page, including a mention of annex.genmetadata 2015-03-31 17:48:13 +00:00			`content has not already been added to the repository before, so you can`
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			`delete, rename, etc the resulting files and repeated runs won't duplicate`
			`them.`

default to not using youtube-dl, for security Pity, but same reasoning as curl applies to it. This commit was sponsored by Peter on Patreon. 2018-06-17 18:46:22 +00:00			When `youtube-dl` is installed, it can be used to download links in the feed.
documentation update for youtube-dl Code not updated yet. This commit was sponsored by Thomas Hochstein on Patreon. 2017-11-28 18:05:58 +00:00			`This allows importing e.g., YouTube playlists.`
default to not using youtube-dl, for security Pity, but same reasoning as curl applies to it. This commit was sponsored by Peter on Patreon. 2018-06-17 18:46:22 +00:00			`(However, this is disabled by default as it can be a security risk.`
rename annex.security.allowed-http-addresses Renamed annex.security.allowed-http-addresses to annex.security.allowed-ip-addresses because it is not really specific to the http protocol, also limiting eg, git-annex's use of ftp and via youtube-dl, several other protocols. The old name for the config will still work. If both old and new name are set, the new name will win. 2019-05-30 16:43:40 +00:00			`See the documentation of annex.security.allowed-ip-addresses`
default to not using youtube-dl, for security Pity, but same reasoning as curl applies to it. This commit was sponsored by Peter on Patreon. 2018-06-17 18:46:22 +00:00			`in [[git-annex]](1) for details.)`
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00
update importfeed man page, including a mention of annex.genmetadata 2015-03-31 17:48:13 +00:00			`To make the import process add metadata to the imported files from the feed,`
			`git config annex.genmetadata true`

document the 2_ files 2020-06-24 18:31:46 +00:00			`By default, the downloaded files are put in a directory with the title`
			`of the feed, and files are named based on the title of the item in the`
			`feed. This can be changed using the --template option.`

			`Existing files are not overwritten by this command. If "some feed/foo.mp3"`
			`already exists, it will instead write to "some feed/2\_foo.mp3"`
			`(or 3, 4, etc). Sometimes a feed will change an item's url,`
			`resulting in the new url being downloaded to such a filename.`

splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			`# OPTIONS`

			* `--force`

doc/*.mdwn: Various typo fixes 2015-05-30 14:54:14 +00:00			`Force downloading items it's seen before.`
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00
document the 2_ files 2020-06-24 18:31:46 +00:00			* `--relaxed`, `--fast`, `--raw`

			`These options behave the same as when using [[git-annex-addurl]](1).`

			* `--fast`

			`Avoid immediately downloading urls. The url is still checked`
			`(via HEAD) to verify that it exists, and to get its size if possible.`

			* `--relaxed`

			`Don't immediately download urls, and avoid storing the size of the`
			`url's content. This makes git-annex accept whatever content is there`
			`at a future point.`

			* `--raw`

			`Prevent special handling of urls by youtube-dl, bittorrent, and other`
			`special remotes. This will for example, make importfeed`
			`download a .torrent file and not the contents it points to.`

splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			* `--template`

			`Controls where the files are stored.`

			`The default template is '${feedtitle}/${itemtitle}${extension}'`

improve docs of template variables 2020-06-24 17:23:16 +00:00			`The available variables in the template include these that`
			`are information about the feed: feedtitle, feedauthor`

			`And these that are information about individual items in the feed:`
			`itemtitle, itemauthor, itemsummary, itemdescription, itemrights,`
			`itemid.`

			`Also, title is itemtitle but falls back to feedtitle if the item has no`
			`title, and author is itemauthor but falls back to feedauthor.`

			`(All of the above are also added as metadata when annex.genmetadata is`
			`set.)`

importfeed: Added some additional --template variables for date and time This commit was sponsored by Ethan Aubin. 2020-06-24 18:24:50 +00:00			`The extension variable is the extension of the file in the feed,`
			`or sometimes ".m" if no extension can be determined.`

improve docs of template variables 2020-06-24 17:23:16 +00:00			`The template also has some variables for when an item was published.`

importfeed: Added some additional --template variables for date and time This commit was sponsored by Ethan Aubin. 2020-06-24 18:24:50 +00:00			`itempubyear (YYYY), itempubmonth (MM), itempubday (DD), itempubhour (HH),`
			`itempubminute (MM), itempubsecond (SS),`
			`itempubdate (YYYY-MM-DD or if the feed's date cannot be parsed, the raw`
improve docs of template variables 2020-06-24 17:23:16 +00:00			`value from the feed).`

			`(These use the UTC time zone, not the local time zone.)`
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00
Added --no-check-gitignore option for finer grained control than using --force. add, addurl, importfeed, import: Added --no-check-gitignore option for finer grained control than using --force. (--force is used for too many different things, and at least one of these also uses it for something else. I would like to reduce --force's footprint until it only forces drops or a few other data losses. For now, --force still disables checking ignores too.) addunused: Don't check .gitignores when adding files. This is a behavior change, but I justify it by analogy with git add of a gitignored file adding it, asking to add all unused files back should add them all back, not skip some. The old behavior was surprising. In Command.Lock and Command.ReKey, CheckGitIgnore False does not change behavior, it only makes explicit what is done. Since these commands are run on annexed files, the file is already checked into git, so git add won't check ignores. 2020-09-18 17:12:04 +00:00			* `--no-check-gitignore`

			`By default, gitignores are honored and it will refuse to download an`
			`url to a file that would be ignored. This makes such files be added`
			`despite any ignores.`

splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			`# SEE ALSO`

			`[[git-annex]](1)`

expand manpages cross-references significantly i found that most man pages only had references to the main git-annex manpage, which i stillfind pretty huge and hard to navigate through. i tried to sift through all the man pages and add cross-references between relevant pages. my general rule of thumb is that links should be both ways unless one of the pages is a more general page that would become ridiculously huge if all backlinks would be added (git-annex-preferred-content comes to mind). i have also make the links one per line as this is how it was done in the metadata pages so far. i did everything but the plumbing, utility and test commands, although some of those are linked from the other commands so cross-links were added there as well. 2015-05-29 16:12:55 +00:00			`[[git-annex-addurl]](1)`

splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands.. 2015-03-23 19:36:10 +00:00			`# AUTHOR`

			`Joey Hess <id@joeyh.name>`

			`Warning: Automatically converted into a man page by mdwn2man. Edit with care.`