mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc
History
Joey Hess f2db6da938
default to yt-dlp and fix progress parsing bugs 
I noticed git-annex was using a lot of CPU when downloading from youtube,
and was not displaying progress. Turns out that yt-dlp (and I think also
youtube-dl) sometimes only knows an estimated size, not the actual size,
and displays the progress output slightly differently for that. That broke
the parser. And, the parser was feeding chunks that failed to parse back
as a remainder, which caused it to try to re-parse the entire output each
time, so it got slower and slower.

Using --progress-template like this should avoid parsing problems as well
as future proof against output changes. But it will work with only yt-dlp.

So, this seemed like the right time to deprecate youtube-dl, and default
to yt-dlp when available.

git-annex will still use youtube-dl if that's all that's available.
However, since the progress parser for youtube-dl was buggy, and I don't
want to maintain two different progress parsers (especially since
youtube-dl is no longer in debian unstable having been replaced by
yt-dlp), made git-annex no longer try to parse youtube-dl's progress.

Also, updated docs for yt-dlp being default. It did not seem worth
renaming annex.youtube-dl-options and annex.youtube-dl-command.

Note that yt-dlp does not seem to document the fields available in the
progress template. I found them by reading the source and looking at
the templates it uses internally. Also note that the use of "i" (rather
than "s") in progressTemplate makes it display floats rounded to integers;
particularly the estimated total size can be a float. That also does not
seem to be documented but I assume is a python thing?

Sponsored-by: Joshua Antonishen on Patreon
2023-05-27 13:04:53 -04:00
..
Android Added a comment 2022-07-18 13:19:44 +00:00
android
assistant
automatic_conflict_resolution
backends
bare_repositories
bugs assist: honor gitignore 2023-05-24 14:04:09 -04:00
chunking comment 2023-02-27 12:03:09 -04:00
coding_style
contribute update comment about updating comments :) 2022-07-27 19:29:43 -04:00
copies
design response 2023-04-07 13:00:56 -04:00
devblog devblog 2023-04-12 15:03:01 -04:00
direct_mode
download
encryption
footer
forum Added a comment 2023-05-19 19:06:01 +00:00
future_proofing
git-annex-add
git-annex-addurl promote comment to bug 2023-03-27 14:10:32 -04:00
git-annex-adjust
git-annex-benchmark comment 2022-12-20 14:30:18 -04:00
git-annex-checkpresentkey
git-annex-config
git-annex-copy
git-annex-direct
git-annex-drop
git-annex-enableremote
git-annex-find findkeys: New command, very similar to git-annex find but operating on keys 2023-01-17 14:51:57 -04:00
git-annex-import
git-annex-importfeed
git-annex-lookupkey
git-annex-metadata
git-annex-move
git-annex-p2p
git-annex-preferred-content Support "inbackend" in preferred content expressions 2022-09-26 16:06:49 -04:00
git-annex-reinject
git-annex-repair Added a comment: When a weird 'no space left on device' error appears 2022-12-21 16:46:44 +00:00
git-annex-rmurl
git-annex-setpresentkey
git-annex-sync comment and improve docs 2023-04-19 12:20:26 -04:00
git-annex-test
git-annex-unannex Added a comment: Re: Weird behavior of git archive in combination with largefiles configuration 2023-02-26 18:45:31 +00:00
git-annex-undo
git-annex-uninit
git-annex-unlock
git-annex-unused
git-annex-whereis
git-annex-whereused comment 2023-05-05 13:36:57 -04:00
how_it_works
install Apply codespell -w throughout 2023-03-17 15:14:58 -04:00
internals convert renameFile to moveFile to support cross-device moves 2022-12-20 15:17:50 -04:00
license
links
metadata
news add news item for git-annex 10.20230407 2023-04-07 13:37:18 -04:00
not
polls
preferred_content
profiling
projects rename an old closed bug to avoid filename too long on windows checkout 2023-03-18 12:24:58 -04:00
required_content
scalability
security
special_remotes update docs to account for exporttree 2023-04-25 12:15:14 -04:00
stickers
submodules analysis and followup 2023-04-20 14:11:04 -04:00
sync this comment section is not a BTS, part N+1 2023-03-21 13:48:34 -04:00
templates
testimonials
thanks Revert "update" 2023-05-07 12:54:29 -04:00
tips default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
todo Added a comment 2023-05-23 16:12:42 +00:00
trust
tuning
upgrades
use_case
users 2023-03-29 22:32:06 +00:00
videos
walkthrough git-annex pull and push 2023-05-16 16:51:07 -04:00
workflow
Android.mdwn
assistant.mdwn
automatic_conflict_resolution.mdwn
backends.mdwn fix typo 2022-12-26 12:36:56 -04:00
bare_repositories.mdwn
bugs.mdwn
builds.mdwn remove appveyor badge, which does not work 2023-03-22 10:55:38 -04:00
chunking.mdwn
coding_style.mdwn
comments.mdwn
contact.mdwn
contribute.mdwn
copies.mdwn
design.mdwn
devblog.mdwn remove calendar 2023-01-05 14:50:26 -04:00
direct_mode.mdwn
distributed_version_control.mdwn
download.mdwn
encryption.mdwn correct documentation, keyid= only works once 2022-08-15 14:25:13 -04:00
favicon.ico
forum.mdwn
future_proofing.mdwn
git-annex-add.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-addunused.mdwn addunused, dropunused: Support --json and --json-error-messages 2023-05-05 14:01:40 -04:00
git-annex-addurl.mdwn default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
git-annex-adjust.mdwn support --unlock-present adjustment of view branches 2023-02-27 15:37:57 -04:00
git-annex-assist.mdwn document -m 2023-05-23 11:46:54 -04:00
git-annex-assistant.mdwn git-annex assist 2023-05-18 14:37:43 -04:00
git-annex-backends.mdwn improve documentation about backends 2022-09-26 15:59:10 -04:00
git-annex-benchmark.mdwn Typo fix unncessary -> unnecessary. 2022-08-20 09:40:19 -04:00
git-annex-calckey.mdwn typo fix 2022-08-03 11:11:10 -04:00
git-annex-checkpresentkey.mdwn
git-annex-common-options.mdwn Improved handling of --time-limit when combined with -J 2022-09-22 12:54:52 -04:00
git-annex-config.mdwn git-annex assist 2023-05-18 14:37:43 -04:00
git-annex-configremote.mdwn configremote 2023-04-18 15:30:49 -04:00
git-annex-contentlocation.mdwn
git-annex-copy.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-dead.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-describe.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-diffdriver.mdwn
git-annex-direct.mdwn
git-annex-drop.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-dropkey.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-dropunused.mdwn addunused, dropunused: Support --json and --json-error-messages 2023-05-05 14:01:40 -04:00
git-annex-edit.mdwn
git-annex-enable-tor.mdwn
git-annex-enableremote.mdwn enableremote: Support --json and --json-error-messages 2023-05-10 14:09:27 -04:00
git-annex-examinekey.mdwn avoid quoting spaces in git-annex find output to terminal 2023-04-26 00:18:30 -04:00
git-annex-expire.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-export.mdwn git-annex pull and push 2023-05-16 16:51:07 -04:00
git-annex-filter-branch.mdwn fix link to git-annex-initremote 2022-08-02 13:51:47 +00:00
git-annex-filter-process.mdwn Typo fix unncessary -> unnecessary. 2022-08-20 09:40:19 -04:00
git-annex-find.mdwn avoid quoting spaces in git-annex find output to terminal 2023-04-26 00:18:30 -04:00
git-annex-findkeys.mdwn avoid quoting spaces in git-annex find output to terminal 2023-04-26 00:18:30 -04:00
git-annex-findref.mdwn
git-annex-fix.mdwn fix: Support --json and --json-error-messages 2023-05-04 14:28:21 -04:00
git-annex-forget.mdwn
git-annex-fromkey.mdwn
git-annex-fsck.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-fuzztest.mdwn
git-annex-get.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-group.mdwn
git-annex-groupwanted.mdwn
git-annex-import.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-importfeed.mdwn default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
git-annex-indirect.mdwn
git-annex-info.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-init.mdwn init: Support --json and --json-error-messages 2023-05-08 14:58:08 -04:00
git-annex-initremote.mdwn initremote: Support --json and --json-error-messages 2023-05-10 14:03:40 -04:00
git-annex-inprogress.mdwn
git-annex-list.mdwn
git-annex-lock.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-log.mdwn log: Support --json and --json-error-messages 2023-05-04 12:36:31 -04:00
git-annex-lookupkey.mdwn
git-annex-map.mdwn
git-annex-matchexpression.mdwn
git-annex-matching-expression.mdwn
git-annex-matching-options.mdwn --anything and --nothing 2022-12-20 15:44:09 -04:00
git-annex-merge.mdwn git-annex pull and push 2023-05-16 16:51:07 -04:00
git-annex-metadata.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-migrate.mdwn migrate, undo: Support --json and --json-error-messages 2023-05-04 16:34:35 -04:00
git-annex-mincopies.mdwn
git-annex-mirror.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-move.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-multicast.mdwn
git-annex-numcopies.mdwn
git-annex-p2p.mdwn
git-annex-post-receive.mdwn
git-annex-pre-commit.mdwn
git-annex-preferred-content.mdwn document how to include= a path with a space in it 2023-05-15 16:02:06 -04:00
git-annex-proxy.mdwn
git-annex-pull.mdwn sync: Added -g as a short option for --no-content 2023-05-17 12:34:26 -04:00
git-annex-push.mdwn push: Support --cleanup 2023-05-17 12:51:49 -04:00
git-annex-readpresentkey.mdwn
git-annex-registerurl.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-reinit.mdwn reinit: Support --json and --json-error-messages 2023-05-08 15:07:40 -04:00
git-annex-reinject.mdwn reinject: Support --json and --json-error-messages 2023-05-08 15:43:37 -04:00
git-annex-rekey.mdwn rekey, setpresentkey: Support --json and --json-error-messages 2023-05-04 16:03:54 -04:00
git-annex-remotedaemon.mdwn
git-annex-renameremote.mdwn configremote 2023-04-18 15:30:49 -04:00
git-annex-repair.mdwn
git-annex-required.mdwn
git-annex-resolvemerge.mdwn
git-annex-restage.mdwn restage: New git-annex command, handles restaging unlocked files 2022-09-23 16:29:59 -04:00
git-annex-rmurl.mdwn rmurl: Support --json and --json-error-messages 2023-05-04 11:28:27 -04:00
git-annex-schedule.mdwn
git-annex-semitrust.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-setkey.mdwn
git-annex-setpresentkey.mdwn rekey, setpresentkey: Support --json and --json-error-messages 2023-05-04 16:03:54 -04:00
git-annex-shell.mdwn
git-annex-smudge.mdwn
git-annex-status.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-sync.mdwn document -m 2023-05-23 11:46:54 -04:00
git-annex-test.mdwn test: Add --test-debug option 2022-11-28 15:12:53 -04:00
git-annex-testremote.mdwn
git-annex-transferkey.mdwn
git-annex-transferkeys.mdwn
git-annex-transferrer.mdwn
git-annex-trust.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-unannex.mdwn unannex: Support --json and --json-error-messages 2023-05-03 15:56:20 -04:00
git-annex-undo.mdwn migrate, undo: Support --json and --json-error-messages 2023-05-04 16:34:35 -04:00
git-annex-ungroup.mdwn
git-annex-uninit.mdwn uninit: Support --json and --json-error-messages 2023-05-11 13:43:02 -04:00
git-annex-unlock.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-unregisterurl.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-untrust.mdwn expire, trust et al, dead, describe: Support --json and --json-error-messages 2023-05-05 15:33:30 -04:00
git-annex-unused.mdwn unused: Support --json and --json-error-messages 2023-05-08 14:39:57 -04:00
git-annex-upgrade.mdwn upgrade: Support --json and --json-error-messages and --json-progress 2023-05-10 12:54:48 -04:00
git-annex-vadd.mdwn add directory to views for files that lack specified metadata 2023-02-07 16:28:46 -04:00
git-annex-vcycle.mdwn
git-annex-version.mdwn
git-annex-vfilter.mdwn add directory to views for files that lack specified metadata 2023-02-07 16:28:46 -04:00
git-annex-vicfg.mdwn
git-annex-view.mdwn annex.maxextensionlength for view 2023-03-24 14:01:38 -04:00
git-annex-vpop.mdwn
git-annex-wanted.mdwn
git-annex-watch.mdwn git-annex assist 2023-05-18 14:37:43 -04:00
git-annex-webapp.mdwn
git-annex-whereis.mdwn Revert "--json-exceptions" 2023-04-25 17:37:34 -04:00
git-annex-whereused.mdwn migrate, undo: Support --json and --json-error-messages 2023-05-04 16:34:35 -04:00
git-annex.mdwn default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
git-remote-tor-annex.mdwn
git-union-merge.mdwn
how_it_works.mdwn
index.mdwn
install.mdwn move a paragraph into the pages that the links it talks about were long ago moved to 2023-02-11 16:07:05 -04:00
internals.mdwn convert renameFile to moveFile to support cross-device moves 2022-12-20 15:17:50 -04:00
license.mdwn
location_tracking.mdwn
logo-old-bw.svg
logo-old.png
logo-old.svg
logo-old_small.png
logo.mdwn
logo.svg
logo_16x16.png
logo_32x32.png
logo_small.png
meta.mdwn
metadata.mdwn
news.mdwn
not.mdwn
polls.mdwn
preferred_content.mdwn
privacy.mdwn
profiling.mdwn
projects.mdwn
publicrepos.mdwn
related_software.mdwn
repomap.png
required_content.mdwn
scalability.mdwn
security.mdwn
shortcuts.mdwn
sidebar.mdwn
sitemap.mdwn
special_remotes.mdwn add NNCP special remote! 2022-09-02 16:55:36 -04:00
stickers.mdwn
submodules.mdwn
summary.mdwn
sync.mdwn comment and improve docs 2023-04-19 12:20:26 -04:00
testimonials.mdwn
thanks.mdwn update 2023-01-07 13:56:06 -04:00
tips.mdwn
todo.mdwn
transferring_data.mdwn
trust.mdwn
tuning.mdwn
upgrades.mdwn v8 repositories automatically upgrade to v9 2022-07-25 16:20:04 -04:00
users.mdwn
videos.mdwn
walkthrough.mdwn
workflow.mdwn