mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc/tips
History
Joey Hess f2db6da938
default to yt-dlp and fix progress parsing bugs 
I noticed git-annex was using a lot of CPU when downloading from youtube,
and was not displaying progress. Turns out that yt-dlp (and I think also
youtube-dl) sometimes only knows an estimated size, not the actual size,
and displays the progress output slightly differently for that. That broke
the parser. And, the parser was feeding chunks that failed to parse back
as a remainder, which caused it to try to re-parse the entire output each
time, so it got slower and slower.

Using --progress-template like this should avoid parsing problems as well
as future proof against output changes. But it will work with only yt-dlp.

So, this seemed like the right time to deprecate youtube-dl, and default
to yt-dlp when available.

git-annex will still use youtube-dl if that's all that's available.
However, since the progress parser for youtube-dl was buggy, and I don't
want to maintain two different progress parsers (especially since
youtube-dl is no longer in debian unstable having been replaced by
yt-dlp), made git-annex no longer try to parse youtube-dl's progress.

Also, updated docs for yt-dlp being default. It did not seem worth
renaming annex.youtube-dl-options and annex.youtube-dl-command.

Note that yt-dlp does not seem to document the fields available in the
progress template. I found them by reading the source and looking at
the templates it uses internally. Also note that the use of "i" (rather
than "s") in progressTemplate makes it display floats rounded to integers;
particularly the estimated total size can be a float. That also does not
seem to be documented but I assume is a python thing?

Sponsored-by: Joshua Antonishen on Patreon
2023-05-27 13:04:53 -04:00
..
a_gui_for_metadata_operations git-annex-metadata-gui yay! 2016-12-07 16:00:12 -04:00
Accessing_files_in_bare_remotes_without_git-annex Added a comment: Updated 2016-01-12 22:47:49 +00:00
android_sync_with_adb response to misplaced bug report 2022-07-13 14:58:04 -04:00
assume-unstaged
automatically_adding_metadata comment 2021-09-03 11:41:38 -04:00
beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes
Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__
Bup_repositories_in_git-annex Added a comment: bup.bloom must be unlocked 2015-07-17 17:45:24 +00:00
centralised_repository__58___starting_from_nothing rename files containing : 2016-03-12 12:57:18 -04:00
centralized_git_repository_tutorial update tutorial to work with 6 year old version of git-annex-shell 2021-02-02 12:26:37 -04:00
Decentralized_repository_behind_a_Firewall
Decrypting_files_in_special_remotes_without_git-annex Added a comment 2017-06-04 19:26:47 +00:00
Delay_Assistant_Startup_on_Login
deleting_unwanted_files Added a comment 2020-10-15 14:46:33 +00:00
disabling_a_special_remote comment 2021-06-16 12:07:08 -04:00
downloading_podcasts Added a comment: howto importfeed youtube playlists (not entire channels) 2022-09-01 19:26:20 +00:00
dumb_metadata_extraction_from_xbmc
Faster_bash_autocompletion_with_big_annex_repos patchneeds to be sent to git 2017-05-11 13:30:15 -04:00
file_manager_integration
finding_duplicate_files removed 2021-01-29 17:16:46 +00:00
finding_which_file_matches_a_key Added a comment: right! forgot about log -S 2019-10-23 20:16:26 +00:00
flickrannex Added a comment: Dead project, or just working smoothly? Also, which hat color? 2016-10-23 18:23:34 +00:00
fully_encrypted_git_repositories_with_gcrypt Added a comment: Cloning and syncing leads to confusing "gcrypt: Failed to decrypt manifest!" errors. 2018-04-08 00:50:15 +00:00
git-annex_extensions_for_ranger__44___the_file_manager Added a comment: related work 2016-10-11 13:35:08 +00:00
git-annex_on_NFS comment 2016-09-21 15:39:32 -04:00
Git_annex_and_Calibre Added a comment 2016-06-02 18:10:33 +00:00
hiding_missing_files fix wrong option, followup 2019-03-22 09:51:42 -04:00
How_to_retroactively_annex_a_file_already_in_a_git_repo Added a comment: Walkthrough of a prudent retroactive annex. 2016-11-24 11:27:59 +00:00
install_on_Android_in_Termux move comment 2018-10-11 13:35:13 -04:00
Internet_Archive_via_S3
largefiles Added a comment 2022-04-21 01:18:21 +00:00
local_caching_of_annexed_files Added a comment: Hardlinks 2021-04-16 08:17:36 +00:00
megaannex
metadata_driven_views
migrating_two_seperate_disconnected_directories_to_git_annex comment, update tip 2022-10-31 12:16:36 -04:00
offline_archive_drives comment 2023-05-03 15:25:17 -04:00
owncloudannex Added a comment: asked upstream 2018-10-07 18:07:05 +00:00
peer_to_peer_network_with_tor Added a comment: Issue on openSUSE with Tor's requirement for Python 2.7 2019-09-29 16:41:15 +00:00
playlist_fetch Added a comment: an alternative 2021-06-23 01:29:23 +00:00
public_Amazon_S3_remote S3: fix documentation of publicurl 2018-07-02 12:30:39 -04:00
publishing_your_files_to_the_public Fix typos "=yet" -> "=yes" 2023-03-10 18:07:20 +01:00
recover_data_from_lost+found comment 2017-03-14 19:45:28 -04:00
recovering_from_a_corrupt_git_repository
replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant
Repositories_with_large_number_of_files Added a comment 2022-09-05 23:57:01 +00:00
semi-synchronized_remotes Added remote.<name>.annex-push and remote.<name>.annex-pull 2017-04-05 13:22:35 -04:00
setup_a_public_repository_on_a_web_site Added a comment 2022-09-07 23:22:40 +00:00
shared_git_annex_directory_between_multiple_users Added a comment: Also problem with USB drives 2020-03-31 06:36:14 +00:00
skydriveannex Added a comment 2015-10-03 19:18:24 +00:00
splitting_a_repository Added a comment 2023-05-10 12:57:54 +00:00
Synology_NAS_and_git_annex removed 2019-07-23 18:01:46 +00:00
Systemd_unit comment 2017-04-05 11:44:29 -04:00
The_perfect_preferred_content_settings_for_my_android_phone
transmission_integration
unlocked_files Added a comment: Windows 10 NTFS hardlinks not working 2020-12-03 11:43:08 +00:00
using_Amazon_Glacier response 2015-12-08 11:13:55 -04:00
using_Amazon_S3 comment 2017-09-29 13:46:33 -04:00
using_box.com_as_a_special_remote
Using_git-annex_on_NTFS_with_WSL1 comment 2022-09-13 15:07:58 -04:00
Using_git-worktree_with_annex Added a comment: worktrees vs. bare repositories 2022-12-15 13:33:23 +00:00
using_git_annex_with_no_fixed_hostname_and_optimising_ssh
using_gitolite_with_git-annex Added a comment: Where does gitolite stores the annex objects contents? 2018-04-27 21:06:42 +00:00
using_Google_Cloud_Storage Added a comment: Coldline 2016-11-21 00:49:23 +00:00
using_nested_git_repositories commeent 2022-01-19 11:56:00 -04:00
using_signed_git_commits comment 2022-01-11 13:07:51 -04:00
using_the_web_as_a_special_remote Added a comment: Download either video and audio 2019-09-23 09:14:29 +00:00
visualizing_repositories_with_gource
what_to_do_when_a_repository_is_corrupted Added a comment: Thanks! 2021-10-05 21:09:44 +00:00
what_to_do_when_you_lose_a_repository response 2016-05-03 15:17:39 -04:00
yet_another_simple_disk_usage_like_utility
ZSH_completion Added a comment 2017-11-09 17:35:14 +00:00
a_gui_for_metadata_operations.mdwn 2016-12-05 19:34:55 +00:00
Accessing_files_in_bare_remotes_without_git-annex.mdwn change a few last v6 references to v7 2019-07-17 11:44:57 -04:00
android_sync_with_adb.mdwn add missing command name 2020-08-10 15:00:58 -04:00
Announcing_recastex_-___40__re__41__podcast__from_your_annex.mdwn 2018-04-06 06:51:36 +00:00
antipatterns.mdwn Support working trees set up by git-worktree. 2018-07-18 14:27:26 -04:00
assume-unstaged.mdwn
automatically_adding_metadata.mdwn The pre-commit-annex hook script that automatically extracts metadata has been updated to also use exiftool. Thanks, Klaus Ethgen. 2016-02-28 11:10:55 -04:00
automatically_getting_files_on_checkout.mdwn
beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes.mdwn
Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__.mdwn
Bup_repositories_in_git-annex.mdwn
centralised_repository__58___starting_from_nothing.mdwn rename files containing : 2016-03-12 12:57:18 -04:00
centralized_git_repository_tutorial.mdwn caps 2015-07-22 18:07:31 -04:00
cloning_a_repository_privately.mdwn remove misleading para 2021-04-23 16:01:12 -04:00
Crude_Windows_Sync.mdwn
Decentralized_repository_behind_a_Firewall.mdwn
Decrypting_files_in_special_remotes_without_git-annex.mdwn 2016-01-08 07:38:01 +00:00
Delay_Assistant_Startup_on_Login.mdwn
deleting_unwanted_files.mdwn Automatically convert direct mode repositories to v7 with adjusted unlocked branches 2019-08-26 15:05:25 -04:00
disabling_a_special_remote.mdwn fix a transcription error - to disable a remote, we set ignore to true, not false! 2020-01-08 15:30:51 +00:00
downloading_podcasts.mdwn default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
dropboxannex.mdwn 2021-07-06 19:50:36 +00:00
dumb_metadata_extraction_from_xbmc.mdwn
emacs_integration.mdwn Fixes markup error 2016-08-31 18:43:43 +00:00
enable_tor_on_nixos.mdwn Add a tip explaining how to enable-tor on NixOS 2021-08-24 04:34:55 +00:00
Faster_bash_autocompletion_with_big_annex_repos.mdwn 2017-04-14 22:11:55 +00:00
file_manager_integration.mdwn update for https://pypi.org/project/thunar-plugins/ 2023-01-01 18:10:35 -04:00
finding_duplicate_files.mdwn
finding_which_file_matches_a_key.mdwn update from comments 2019-10-23 20:15:40 +00:00
flickrannex.mdwn
fully_encrypted_git_repositories_with_gcrypt.mdwn change back to git-remote-gcrypt 1.4 url forms, with a note that that version is needed 2021-04-13 12:55:50 -04:00
get_git-annex-shell_into_PATH.mdwn tip for git-annex-shell PATH issues 2015-08-09 18:09:05 -04:00
git-annex_extensions_for_ranger__44___the_file_manager.mdwn added :ga_get and :ga_set 2016-08-11 03:12:04 +00:00
git-annex_on_NFS.mdwn my NFS tricks for git-annex 2015-07-17 22:14:25 +00:00
Git_annex_and_Calibre.mdwn Automatically convert direct mode repositories to v7 with adjusted unlocked branches 2019-08-26 15:05:25 -04:00
hashdeep_integration.mdwn 2020-12-10 15:53:37 +00:00
hiding_missing_files.mdwn fix wrong option, followup 2019-03-22 09:51:42 -04:00
How_to_retroactively_annex_a_file_already_in_a_git_repo.mdwn
imapannex.mdwn
install_on_Android_in_Termux.mdwn update android docs for termux 2018-10-13 12:11:47 -04:00
Internet_Archive_via_S3.mdwn upload to the test collection, giving users an idea of how to change collections as well as forcing people to think before they upload 2018-11-05 16:02:23 +00:00
largefiles.mdwn document version for --force-large/--force-small 2020-10-26 11:34:30 -04:00
local_caching_of_annexed_files.mdwn prevent fetch/merge/push to cache 2018-08-03 14:10:05 -04:00
making_a_remote_repo_update_when_changes_are_pushed_to_it.mdwn linkify 2017-02-17 15:58:32 -04:00
mc_menu_integration.mdwn 2017-01-31 16:53:58 +00:00
megaannex.mdwn improve special remotes list 2020-03-04 22:27:56 -04:00
metadata_driven_views.mdwn
migrating_data_to_a_new_backend.mdwn
migrating_two_seperate_disconnected_directories_to_git_annex.mdwn comment, update tip 2022-10-31 12:16:36 -04:00
multiple_remotes_accessing_the_same_data_store.mdwn httpalso: support exporttree=yes 2020-09-02 11:26:00 -04:00
offline_archive_drives.mdwn
owncloudannex.mdwn two more services with multiple special remotes 2020-03-04 22:42:33 -04:00
peer_to_peer_network_with_tor.mdwn Fix typo and incorrect references to "git annex peer" 2019-09-29 23:34:25 +00:00
playlist_fetch.mdwn explain a bit more 2016-03-31 10:55:20 -04:00
powerful_file_matching.mdwn
public_Amazon_S3_remote.mdwn S3: fix documentation of publicurl 2018-07-02 12:30:39 -04:00
publishing_your_files_to_the_public.mdwn fix typo in git-annex command to create public S3 remote 2020-11-25 20:31:23 +00:00
recover_data_from_lost+found.mdwn update 2015-11-13 13:46:11 -04:00
recovering_from_a_corrupt_git_repository.mdwn
remote_webapp_setup.mdwn
replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant.mdwn rework largefiles documentation 2016-02-02 16:50:58 -04:00
Repositories_with_large_number_of_files.mdwn add signatures to the various sections 2017-04-24 13:40:59 +00:00
semi-synchronized_remotes.mdwn add annex-sync setting sample 2017-03-27 19:50:43 +00:00
setup_a_public_repository_on_a_web_site.mdwn update to work with modern git 2022-09-13 15:09:02 -04:00
Shamir_secret_sharing_and_git-annex.mdwn
shared_git_annex_directory_between_multiple_users.mdwn actually, i screwed up, it's actually the litteral "group" 2015-05-27 20:56:05 +00:00
skydriveannex.mdwn
splitting_a_repository.mdwn update for filter-branch 2021-05-17 15:03:47 -04:00
storing_data_in_git-lfs.mdwn git-lfs: remember urls, and autoenable remotes using known urls 2019-11-18 16:09:09 -04:00
Synology_NAS_and_git_annex.mdwn
Systemd_unit.mdwn prefer per-user, fix ExecStart syntax 2016-11-10 06:34:44 +00:00
The_perfect_preferred_content_settings_for_my_android_phone.mdwn
transmission_integration.mdwn
unlocked_files.mdwn close as git-annex already makes reflinks when supported 2022-11-14 13:29:27 -04:00
untrusted_repositories.mdwn
using_Amazon_Glacier.mdwn Behavior change: --trust-glacier option no longer overrides trust 2021-01-07 10:37:43 -04:00
using_Amazon_S3.mdwn reorder 2019-04-23 14:45:34 -04:00
using_Backblaze_B2.mdwn fix paste 2020-05-12 09:55:54 -04:00
using_borg_for_efficient_storage_of_old_annexed_files.mdwn improve docs based on forum feedback 2021-06-14 13:04:58 -04:00
using_box.com_as_a_special_remote.mdwn 2019-02-04 14:40:03 +00:00
Using_Git-annex_as_a_web_browsing_assistant.mdwn limit url downloads to whitelisted schemes 2018-06-16 11:57:50 -04:00
using_git-annex_from_your_program.mdwn tip documenting mostly --batch --json including with -J 2020-09-16 13:00:47 -04:00
Using_git-annex_on_NTFS_with_WSL1.mdwn Update WSL1 tips 2022-10-13 15:29:12 +00:00
Using_git-worktree_with_annex.mdwn 2018-11-28 09:02:12 +00:00
using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn
using_gitolite_with_git-annex.mdwn
using_Google_Cloud_Storage.mdwn improve special remotes list 2020-03-04 22:27:56 -04:00
using_Google_Drive.mdwn improve special remotes list 2020-03-04 22:27:56 -04:00
using_Hubic.mdwn improve special remotes list 2020-03-04 22:27:56 -04:00
using_nested_git_repositories.mdwn 2022-01-13 18:08:25 +00:00
using_signed_git_commits.mdwn better headings 2017-02-27 16:18:20 -04:00
using_the_web_as_a_special_remote.mdwn default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
visualizing_repositories_with_gource.mdwn
what_to_do_when_a_repository_is_corrupted.mdwn
what_to_do_when_you_lose_a_repository.mdwn
yet_another_simple_disk_usage_like_utility.mdwn
ZSH_completion.mdwn