mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc/git-annex-importfeed.mdwn

142 lines
4.1 KiB
Text
Raw Normal View History

splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
# NAME
git-annex importfeed - import files from podcast feeds
# SYNOPSIS
git annex importfeed `[url ...]`
# DESCRIPTION
convert importfeed to youtube-dl Fully working, including --fast/--relaxed. Note that, while git-annex addurl --relaxed is not going to check youtube-dl, I kept git annex importfeed --relaxed checking it. Thinking is that, let's not break people's importfeed cron jobs, and importfeed does not typically have to check a large number of new items, so it's ok if it's a little bit slower when used with youtube playlist feeds. importfeed's behavior is also improved (?) when a feed has links in it to non-media files. Before, those were skipped. Now, the content of the link is downloaded. This had to be done, because trying to use youtube-dl is slow, and if those were skipped, it would have to check every time importfeed was run. While this behavior change may not be desirable for some feeds, that intersperse links to web pages with enclosures, it will be desirable for other feeds, that have non-enclosure directy links to media files. Remove old quvi modules. This commit was sponsored by Øyvind Andersen Holm.
2017-11-29 21:05:27 +00:00
Imports the contents of podcasts and other feeds. Only downloads files whose
update importfeed man page, including a mention of annex.genmetadata
2015-03-31 17:48:13 +00:00
content has not already been added to the repository before, so you can
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
delete, rename, etc the resulting files and repeated runs won't duplicate
them.
default to yt-dlp and fix progress parsing bugs I noticed git-annex was using a lot of CPU when downloading from youtube, and was not displaying progress. Turns out that yt-dlp (and I think also youtube-dl) sometimes only knows an estimated size, not the actual size, and displays the progress output slightly differently for that. That broke the parser. And, the parser was feeding chunks that failed to parse back as a remainder, which caused it to try to re-parse the entire output each time, so it got slower and slower. Using --progress-template like this should avoid parsing problems as well as future proof against output changes. But it will work with only yt-dlp. So, this seemed like the right time to deprecate youtube-dl, and default to yt-dlp when available. git-annex will still use youtube-dl if that's all that's available. However, since the progress parser for youtube-dl was buggy, and I don't want to maintain two different progress parsers (especially since youtube-dl is no longer in debian unstable having been replaced by yt-dlp), made git-annex no longer try to parse youtube-dl's progress. Also, updated docs for yt-dlp being default. It did not seem worth renaming annex.youtube-dl-options and annex.youtube-dl-command. Note that yt-dlp does not seem to document the fields available in the progress template. I found them by reading the source and looking at the templates it uses internally. Also note that the use of "i" (rather than "s") in progressTemplate makes it display floats rounded to integers; particularly the estimated total size can be a float. That also does not seem to be documented but I assume is a python thing? Sponsored-by: Joshua Antonishen on Patreon
2023-05-27 16:45:16 +00:00
When `yt-dlp` is installed, it can be used to download links in the feed.
documentation update for youtube-dl Code not updated yet. This commit was sponsored by Thomas Hochstein on Patreon.
2017-11-28 18:05:58 +00:00
This allows importing e.g., YouTube playlists.
default to not using youtube-dl, for security Pity, but same reasoning as curl applies to it. This commit was sponsored by Peter on Patreon.
2018-06-17 18:46:22 +00:00
(However, this is disabled by default as it can be a security risk.
rename annex.security.allowed-http-addresses Renamed annex.security.allowed-http-addresses to annex.security.allowed-ip-addresses because it is not really specific to the http protocol, also limiting eg, git-annex's use of ftp and via youtube-dl, several other protocols. The old name for the config will still work. If both old and new name are set, the new name will win.
2019-05-30 16:43:40 +00:00
See the documentation of annex.security.allowed-ip-addresses
default to not using youtube-dl, for security Pity, but same reasoning as curl applies to it. This commit was sponsored by Peter on Patreon.
2018-06-17 18:46:22 +00:00
in [[git-annex]](1) for details.)
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
update importfeed man page, including a mention of annex.genmetadata
2015-03-31 17:48:13 +00:00
To make the import process add metadata to the imported files from the feed,
`git config annex.genmetadata true`
document the 2_ files
2020-06-24 18:31:46 +00:00
By default, the downloaded files are put in a directory with the title
of the feed, and files are named based on the title of the item in the
feed. This can be changed using the --template option.
Existing files are not overwritten by this command. If "some feed/foo.mp3"
already exists, it will instead write to "some feed/2\_foo.mp3"
(or 3, 4, etc). Sometimes a feed will change an item's url,
resulting in the new url being downloaded to such a filename.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
# OPTIONS
* `--force`
doc/*.mdwn: Various typo fixes
2015-05-30 14:54:14 +00:00
Force downloading items it's seen before.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
document the 2_ files
2020-06-24 18:31:46 +00:00
* `--relaxed`, `--fast`, `--raw`
These options behave the same as when using [[git-annex-addurl]](1).
* `--fast`
Avoid immediately downloading urls. The url is still checked
(via HEAD) to verify that it exists, and to get its size if possible.
* `--relaxed`
Don't immediately download urls, and avoid storing the size of the
url's content. This makes git-annex accept whatever content is there
at a future point.
* `--raw`
default to yt-dlp and fix progress parsing bugs I noticed git-annex was using a lot of CPU when downloading from youtube, and was not displaying progress. Turns out that yt-dlp (and I think also youtube-dl) sometimes only knows an estimated size, not the actual size, and displays the progress output slightly differently for that. That broke the parser. And, the parser was feeding chunks that failed to parse back as a remainder, which caused it to try to re-parse the entire output each time, so it got slower and slower. Using --progress-template like this should avoid parsing problems as well as future proof against output changes. But it will work with only yt-dlp. So, this seemed like the right time to deprecate youtube-dl, and default to yt-dlp when available. git-annex will still use youtube-dl if that's all that's available. However, since the progress parser for youtube-dl was buggy, and I don't want to maintain two different progress parsers (especially since youtube-dl is no longer in debian unstable having been replaced by yt-dlp), made git-annex no longer try to parse youtube-dl's progress. Also, updated docs for yt-dlp being default. It did not seem worth renaming annex.youtube-dl-options and annex.youtube-dl-command. Note that yt-dlp does not seem to document the fields available in the progress template. I found them by reading the source and looking at the templates it uses internally. Also note that the use of "i" (rather than "s") in progressTemplate makes it display floats rounded to integers; particularly the estimated total size can be a float. That also does not seem to be documented but I assume is a python thing? Sponsored-by: Joshua Antonishen on Patreon
2023-05-27 16:45:16 +00:00
Prevent special handling of urls by yt-dlp, bittorrent, and other
document the 2_ files
2020-06-24 18:31:46 +00:00
special remotes. This will for example, make importfeed
download a .torrent file and not the contents it points to.
addurl, importfeed: Added --no-raw option Forces eg, download with youtube-dl without falling back to raw download. Since youtube-dl failing due to an url not being supported is difficult to distinguish from it failing due to being blocked in some way, this can be useful to avoid the fallback of git-annex downloading the raw web page and adding that. Since --raw also prevents using special remotes, --no-raw also allows special remote downloads. Although it's always possible that some special remote may claim an url and fall back to raw download of the content, which --no-raw cannot prevent. Sponsored-by: Boyd Stephen Smith Jr. on Patreon
2021-06-27 15:13:38 +00:00
* `--no-raw`
default to yt-dlp and fix progress parsing bugs I noticed git-annex was using a lot of CPU when downloading from youtube, and was not displaying progress. Turns out that yt-dlp (and I think also youtube-dl) sometimes only knows an estimated size, not the actual size, and displays the progress output slightly differently for that. That broke the parser. And, the parser was feeding chunks that failed to parse back as a remainder, which caused it to try to re-parse the entire output each time, so it got slower and slower. Using --progress-template like this should avoid parsing problems as well as future proof against output changes. But it will work with only yt-dlp. So, this seemed like the right time to deprecate youtube-dl, and default to yt-dlp when available. git-annex will still use youtube-dl if that's all that's available. However, since the progress parser for youtube-dl was buggy, and I don't want to maintain two different progress parsers (especially since youtube-dl is no longer in debian unstable having been replaced by yt-dlp), made git-annex no longer try to parse youtube-dl's progress. Also, updated docs for yt-dlp being default. It did not seem worth renaming annex.youtube-dl-options and annex.youtube-dl-command. Note that yt-dlp does not seem to document the fields available in the progress template. I found them by reading the source and looking at the templates it uses internally. Also note that the use of "i" (rather than "s") in progressTemplate makes it display floats rounded to integers; particularly the estimated total size can be a float. That also does not seem to be documented but I assume is a python thing? Sponsored-by: Joshua Antonishen on Patreon
2023-05-27 16:45:16 +00:00
Require content pointed to by the url to be downloaded using yt-dlp
addurl, importfeed: Added --no-raw option Forces eg, download with youtube-dl without falling back to raw download. Since youtube-dl failing due to an url not being supported is difficult to distinguish from it failing due to being blocked in some way, this can be useful to avoid the fallback of git-annex downloading the raw web page and adding that. Since --raw also prevents using special remotes, --no-raw also allows special remote downloads. Although it's always possible that some special remote may claim an url and fall back to raw download of the content, which --no-raw cannot prevent. Sponsored-by: Boyd Stephen Smith Jr. on Patreon
2021-06-27 15:13:38 +00:00
or a special remote, rather than the raw content of the url. if that
cannot be done, the import will fail, and the next import of the feed
will retry.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
* `--template`
Controls where the files are stored.
The default template is '${feedtitle}/${itemtitle}${extension}'
improve docs of template variables
2020-06-24 17:23:16 +00:00
The available variables in the template include these that
importfeed: Add feedurl to the metadata (And allow it to be used in the --template although that seems unlikely to be very useful.) My use case for this is that one of the podcast feeds I subscribe to is sometimes leaking episodes of some other podcast. The other podcast is also very close to spam, so this may be a form of intentional spamming. I have not been able to catch the podcast feed containing those episodes, so I don't know which one is at fault. So putting this in the metadata will let me eventually catch it.
2023-07-06 04:10:19 +00:00
are information about the feed: feedtitle, feedauthor, feedurl
improve docs of template variables
2020-06-24 17:23:16 +00:00
And these that are information about individual items in the feed:
itemtitle, itemauthor, itemsummary, itemdescription, itemrights,
itemid.
Also, title is itemtitle but falls back to feedtitle if the item has no
title, and author is itemauthor but falls back to feedauthor.
(All of the above are also added as metadata when annex.genmetadata is
set.)
importfeed: Added some additional --template variables for date and time This commit was sponsored by Ethan Aubin.
2020-06-24 18:24:50 +00:00
The extension variable is the extension of the file in the feed,
or sometimes ".m" if no extension can be determined.
improve docs of template variables
2020-06-24 17:23:16 +00:00
The template also has some variables for when an item was published.
importfeed: Added some additional --template variables for date and time This commit was sponsored by Ethan Aubin.
2020-06-24 18:24:50 +00:00
itempubyear (YYYY), itempubmonth (MM), itempubday (DD), itempubhour (HH),
itempubminute (MM), itempubsecond (SS),
itempubdate (YYYY-MM-DD or if the feed's date cannot be parsed, the raw
improve docs of template variables
2020-06-24 17:23:16 +00:00
value from the feed).
(These use the UTC time zone, not the local time zone.)
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
Added --no-check-gitignore option for finer grained control than using --force. add, addurl, importfeed, import: Added --no-check-gitignore option for finer grained control than using --force. (--force is used for too many different things, and at least one of these also uses it for something else. I would like to reduce --force's footprint until it only forces drops or a few other data losses. For now, --force still disables checking ignores too.) addunused: Don't check .gitignores when adding files. This is a behavior change, but I justify it by analogy with git add of a gitignored file adding it, asking to add all unused files back should add them all back, not skip some. The old behavior was surprising. In Command.Lock and Command.ReKey, CheckGitIgnore False does not change behavior, it only makes explicit what is done. Since these commands are run on annexed files, the file is already checked into git, so git add won't check ignores.
2020-09-18 17:12:04 +00:00
* `--no-check-gitignore`
By default, gitignores are honored and it will refuse to download an
url to a file that would be ignored. This makes such files be added
despite any ignores.
importfeed: Support -J (and work toward supporting --json) Both -J and --json needed importfeed to be refactored to use commandAction. That was difficult, because of the interrelated nature of downloading feeds and then downloading files from feeds, both of which needed to use commandAction. And then checking for problems in feeds has to come after these actions, which may be run as background jobs. As for --json support, it's most of the way there, but still has some warts, so I didn't enable jsonOptions yet. The warts include: - An initial empty json record is displayed by getCache. - Input is not populated, should be feed url - feedProblem at end will not be captured by --json-error-messages (see FIXME) Sponsored-By: the NIH-funded NICEMAN (ReproNim TR&D3) project
2023-05-09 19:49:05 +00:00
* `--jobs=N` `-JN`
Runs multiple downloads parallel. For example: `-J4`
Setting this to "cpus" will run one job per CPU core.
support --backend again in addurl and importfeed Missed these two when converting from a global option. Sponsored-by: Dartmouth College's Datalad project
2022-07-05 19:34:49 +00:00
* `--backend`
Specifies which key-value backend to use.
importfeed: Support --json and --json-error-messages and --json-progress Sponsored-By: the NIH-funded NICEMAN (ReproNim TR&D3) project
2023-05-09 20:43:16 +00:00
* `--json`
Enable JSON output. This is intended to be parsed by programs that use
git-annex. Each line of output is a JSON object.
* `--json-progress`
Include progress objects in JSON output.
* `--json-error-messages`
Messages that would normally be output to standard error are included in
the JSON instead.
split out common options to its own page and mention it on each subcommand page Sometimes users would get confused because an option they were looking for was not mentioned on a subcommand's man page, and they had not noticed that the main git-annex man page had a list of common options. This change lets each subcommand mention the common options, similarly to how the matching options are handled. This commit was sponsored by Svenne Krap on Patreon.
2021-05-10 19:00:13 +00:00
* Also the [[git-annex-common-options]](1) can be used.
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
# SEE ALSO
[[git-annex]](1)
expand manpages cross-references significantly i found that most man pages only had references to the main git-annex manpage, which i stillfind pretty huge and hard to navigate through. i tried to sift through all the man pages and add cross-references between relevant pages. my general rule of thumb is that links should be both ways unless one of the pages is a more general page that would become ridiculously huge if all backlinks would be added (git-annex-preferred-content comes to mind). i have also make the links one per line as this is how it was done in the metadata pages so far. i did everything but the plumbing, utility and test commands, although some of those are linked from the other commands so cross-links were added there as well.
2015-05-29 16:12:55 +00:00
[[git-annex-addurl]](1)
splitting up the man page Common command man pages all split out and often expanded. A few sections split out into their own pages. Still need to do all the other commands..
2015-03-23 19:36:10 +00:00
# AUTHOR
Joey Hess <id@joeyh.name>
Warning: Automatically converted into a man page by mdwn2man. Edit with care.
Reference in a new issue Copy permalink