mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

default to yt-dlp and fix progress parsing bugs

Browse source 
I noticed git-annex was using a lot of CPU when downloading from youtube,
and was not displaying progress. Turns out that yt-dlp (and I think also
youtube-dl) sometimes only knows an estimated size, not the actual size,
and displays the progress output slightly differently for that. That broke
the parser. And, the parser was feeding chunks that failed to parse back
as a remainder, which caused it to try to re-parse the entire output each
time, so it got slower and slower.

Using --progress-template like this should avoid parsing problems as well
as future proof against output changes. But it will work with only yt-dlp.

So, this seemed like the right time to deprecate youtube-dl, and default
to yt-dlp when available.

git-annex will still use youtube-dl if that's all that's available.
However, since the progress parser for youtube-dl was buggy, and I don't
want to maintain two different progress parsers (especially since
youtube-dl is no longer in debian unstable having been replaced by
yt-dlp), made git-annex no longer try to parse youtube-dl's progress.

Also, updated docs for yt-dlp being default. It did not seem worth
renaming annex.youtube-dl-options and annex.youtube-dl-command.

Note that yt-dlp does not seem to document the fields available in the
progress template. I found them by reading the source and looking at
the templates it uses internally. Also note that the use of "i" (rather
than "s") in progressTemplate makes it display floats rounded to integers;
particularly the estimated total size can be a float. That also does not
seem to be documented but I assume is a python thing?

Sponsored-by: Joshua Antonishen on Patreon
This commit is contained in:
Joey Hess 2023-05-27 12:45:16 -04:00
parent f1cdb79ca4
commit f2db6da938
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
7 changed files with 79 additions and 73 deletions
Download patch file Download diff file Expand all files Collapse all files

6
doc/git-annex-addurl.mdwn
View file

@ -10,7 +10,7 @@ git annex addurl `[url ...]`
Downloads each url to its own file, which is added to the annex.
When `youtube-dl` is installed, it can be used to check for a video
When `yt-dlp` is installed, it can be used to check for a video
embedded in a web page at the url, and that is added to the annex instead.
(However, this is disabled by default as it can be a security risk.
See the documentation of annex.security.allowed-ip-addresses
@ -45,13 +45,13 @@ be used to get better filenames.
* `--raw`
Prevent special handling of urls by youtube-dl, bittorrent, and other
Prevent special handling of urls by yt-dlp, bittorrent, and other
special remotes. This will for example, make addurl
download the .torrent file and not the contents it points to.
* `--no-raw`
Require content pointed to by the url to be downloaded using youtube-dl
Require content pointed to by the url to be downloaded using yt-dlp
or a special remote, rather than the raw content of the url. if that
cannot be done, the add will fail.

6
doc/git-annex-importfeed.mdwn
View file

@ -13,7 +13,7 @@ content has not already been added to the repository before, so you can
delete, rename, etc the resulting files and repeated runs won't duplicate
them.
When `youtube-dl` is installed, it can be used to download links in the feed.
When `yt-dlp` is installed, it can be used to download links in the feed.
This allows importing e.g., YouTube playlists.
(However, this is disabled by default as it can be a security risk.
See the documentation of annex.security.allowed-ip-addresses
@ -54,13 +54,13 @@ resulting in the new url being downloaded to such a filename.
* `--raw`
Prevent special handling of urls by youtube-dl, bittorrent, and other
Prevent special handling of urls by yt-dlp, bittorrent, and other
special remotes. This will for example, make importfeed
download a .torrent file and not the contents it points to.
* `--no-raw`
Require content pointed to by the url to be downloaded using youtube-dl
Require content pointed to by the url to be downloaded using yt-dlp
or a special remote, rather than the raw content of the url. if that
cannot be done, the import will fail, and the next import of the feed
will retry.

18
doc/git-annex.mdwn
View file

@ -1774,19 +1774,19 @@ Remotes are configured using these settings in `.git/config`.
* `annex.youtube-dl-options`
Options to pass to youtube-dl (or yt-dlp) when using it to find the url
to download for a video.
Options to pass to yt-dlp (or deprecated youtube-dl) when using it to
find the url to download for a video.
Some options may break git-annex's integration with youtube-dl. For
Some options may break git-annex's integration with yt-dlp. For
example, the --output option could cause it to store files somewhere
git-annex won't find them. Avoid setting here or in the youtube-dl config
file any options that cause youtube-dl to download more than one file,
git-annex won't find them. Avoid setting here or in the yt-dlp config
file any options that cause it to download more than one file,
or to store the file anywhere other than the current working directory.
* `annex.youtube-dl-command`
Command to run for youtube-dl. Default is to use "youtube-dl" or
if that is not available in the PATH, to use "yt-dlp".
Default is to use "yt-dlp" or if that is not available in the PATH,
to use "youtube-dl".
* `annex.aria-torrent-options`
@ -1837,8 +1837,8 @@ Remotes are configured using these settings in `.git/config`.
causing it to be downloaded into your repository and transferred to
other remotes, exposing its content.
Note that, since the interfaces of curl and youtube-dl do not allow
these IP address restrictions to be enforced, curl and youtube-dl will
Note that, since the interfaces of curl and yt-dlp do not allow
these IP address restrictions to be enforced, curl and yt-dlp will
never be used unless annex.security.allowed-ip-addresses=all.
To allow accessing local or private IP addresses on only specific ports,

4
doc/tips/downloading_podcasts.mdwn
View file

@ -74,7 +74,7 @@ and transferring to your laptop on demand.
## youtube channels
You can also use `git annex importfeed` on youtube channels.
It will use youtube-dl to automatically
It will use yt-dlp to automatically
download the videos.
To download a youtube channel, you need to find the feed associated with that
@ -84,7 +84,7 @@ manually. For a channel url like
"https://www.youtube.com/channel/$foo", the
feed is "https://www.youtube.com/feeds/videos.xml?channel_id=$foo"
Use of youtube-dl is disabled by default as it can be a security risk.
Use of yt-dlp is disabled by default as it can be a security risk.
See the documentation of annex.security.allowed-ip-addresses
in [[git-annex]] for details.)

12
doc/tips/using_the_web_as_a_special_remote.mdwn
View file

@ -75,9 +75,9 @@ number takes that many paths from the end.
<a name=videos></a>
There's support for downloading videos from sites like YouTube, Vimeo,
and many more. This relies on youtube-dl to download the videos.
and many more. This relies on yt-dlp to download the videos.
When you have youtube-dl installed, you can just
When you have yt-dlp installed, you can just
`git annex addurl http://youtube.com/foo` and it will detect that
it is a video and download the video content for offline viewing.
@ -86,16 +86,14 @@ See the documentation of annex.security.allowed-ip-addresses
in [[git-annex]] for details.)
Later, in another clone of the repository, you can run `git annex get` on
the file and it will also be downloaded with youtube-dl. This works
the file and it will also be downloaded with yt-dlp. This works
even if the video host has transcoded or otherwise changed the video
in the meantime; the assumption is that these video files are equivalent.
There is an `annex.youtube-dl-options` configuration setting that can be used
to pass parameters to quvi. For example, you could set `git config
to pass parameters to yt-dlp. For example, you could set `git config
annex.youtube-dl-options "--format worst"` to configure it to download low
quality videos from YouTube. Note that the youtube-dl configuration files
are not read when git-annex runs youtube-dl, to avoid config settings that
break its integration.
quality videos from YouTube.
To download a youtube channel, you need to find the RSS feed associated with
that channel, and pass it to `git annex importfeed`. There does not seem to