mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions 1
git-annex/doc/tips
History
Joey Hess f2db6da938
default to yt-dlp and fix progress parsing bugs 
I noticed git-annex was using a lot of CPU when downloading from youtube,
and was not displaying progress. Turns out that yt-dlp (and I think also
youtube-dl) sometimes only knows an estimated size, not the actual size,
and displays the progress output slightly differently for that. That broke
the parser. And, the parser was feeding chunks that failed to parse back
as a remainder, which caused it to try to re-parse the entire output each
time, so it got slower and slower.

Using --progress-template like this should avoid parsing problems as well
as future proof against output changes. But it will work with only yt-dlp.

So, this seemed like the right time to deprecate youtube-dl, and default
to yt-dlp when available.

git-annex will still use youtube-dl if that's all that's available.
However, since the progress parser for youtube-dl was buggy, and I don't
want to maintain two different progress parsers (especially since
youtube-dl is no longer in debian unstable having been replaced by
yt-dlp), made git-annex no longer try to parse youtube-dl's progress.

Also, updated docs for yt-dlp being default. It did not seem worth
renaming annex.youtube-dl-options and annex.youtube-dl-command.

Note that yt-dlp does not seem to document the fields available in the
progress template. I found them by reading the source and looking at
the templates it uses internally. Also note that the use of "i" (rather
than "s") in progressTemplate makes it display floats rounded to integers;
particularly the estimated total size can be a float. That also does not
seem to be documented but I assume is a python thing?

Sponsored-by: Joshua Antonishen on Patreon
2023-05-27 13:04:53 -04:00
..
a_gui_for_metadata_operations
Accessing_files_in_bare_remotes_without_git-annex
android_sync_with_adb response to misplaced bug report 2022-07-13 14:58:04 -04:00
assume-unstaged
automatically_adding_metadata comment 2021-09-03 11:41:38 -04:00
beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes
Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__
Bup_repositories_in_git-annex
centralised_repository__58___starting_from_nothing
centralized_git_repository_tutorial update tutorial to work with 6 year old version of git-annex-shell 2021-02-02 12:26:37 -04:00
Decentralized_repository_behind_a_Firewall
Decrypting_files_in_special_remotes_without_git-annex
Delay_Assistant_Startup_on_Login
deleting_unwanted_files Added a comment 2020-10-15 14:46:33 +00:00
disabling_a_special_remote comment 2021-06-16 12:07:08 -04:00
downloading_podcasts Added a comment: howto importfeed youtube playlists (not entire channels) 2022-09-01 19:26:20 +00:00
dumb_metadata_extraction_from_xbmc
Faster_bash_autocompletion_with_big_annex_repos
file_manager_integration
finding_duplicate_files removed 2021-01-29 17:16:46 +00:00
finding_which_file_matches_a_key Added a comment: right! forgot about log -S 2019-10-23 20:16:26 +00:00
flickrannex
fully_encrypted_git_repositories_with_gcrypt
git-annex_extensions_for_ranger__44___the_file_manager
git-annex_on_NFS
Git_annex_and_Calibre
hiding_missing_files fix wrong option, followup 2019-03-22 09:51:42 -04:00
How_to_retroactively_annex_a_file_already_in_a_git_repo
install_on_Android_in_Termux
Internet_Archive_via_S3
largefiles Added a comment 2022-04-21 01:18:21 +00:00
local_caching_of_annexed_files Added a comment: Hardlinks 2021-04-16 08:17:36 +00:00
megaannex
metadata_driven_views
migrating_two_seperate_disconnected_directories_to_git_annex comment, update tip 2022-10-31 12:16:36 -04:00
offline_archive_drives comment 2023-05-03 15:25:17 -04:00
owncloudannex
peer_to_peer_network_with_tor Added a comment: Issue on openSUSE with Tor's requirement for Python 2.7 2019-09-29 16:41:15 +00:00
playlist_fetch Added a comment: an alternative 2021-06-23 01:29:23 +00:00
public_Amazon_S3_remote
publishing_your_files_to_the_public Fix typos "=yet" -> "=yes" 2023-03-10 18:07:20 +01:00
recover_data_from_lost+found
recovering_from_a_corrupt_git_repository
replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant
Repositories_with_large_number_of_files Added a comment 2022-09-05 23:57:01 +00:00
semi-synchronized_remotes
setup_a_public_repository_on_a_web_site Added a comment 2022-09-07 23:22:40 +00:00
shared_git_annex_directory_between_multiple_users Added a comment: Also problem with USB drives 2020-03-31 06:36:14 +00:00
skydriveannex
splitting_a_repository Added a comment 2023-05-10 12:57:54 +00:00
Synology_NAS_and_git_annex removed 2019-07-23 18:01:46 +00:00
Systemd_unit
The_perfect_preferred_content_settings_for_my_android_phone
transmission_integration
unlocked_files Added a comment: Windows 10 NTFS hardlinks not working 2020-12-03 11:43:08 +00:00
using_Amazon_Glacier
using_Amazon_S3
using_box.com_as_a_special_remote
Using_git-annex_on_NTFS_with_WSL1 comment 2022-09-13 15:07:58 -04:00
Using_git-worktree_with_annex Added a comment: worktrees vs. bare repositories 2022-12-15 13:33:23 +00:00
using_git_annex_with_no_fixed_hostname_and_optimising_ssh
using_gitolite_with_git-annex
using_Google_Cloud_Storage
using_nested_git_repositories commeent 2022-01-19 11:56:00 -04:00
using_signed_git_commits comment 2022-01-11 13:07:51 -04:00
using_the_web_as_a_special_remote Added a comment: Download either video and audio 2019-09-23 09:14:29 +00:00
visualizing_repositories_with_gource
what_to_do_when_a_repository_is_corrupted Added a comment: Thanks! 2021-10-05 21:09:44 +00:00
what_to_do_when_you_lose_a_repository
yet_another_simple_disk_usage_like_utility
ZSH_completion
a_gui_for_metadata_operations.mdwn
Accessing_files_in_bare_remotes_without_git-annex.mdwn change a few last v6 references to v7 2019-07-17 11:44:57 -04:00
android_sync_with_adb.mdwn add missing command name 2020-08-10 15:00:58 -04:00
Announcing_recastex_-___40__re__41__podcast__from_your_annex.mdwn
antipatterns.mdwn
assume-unstaged.mdwn
automatically_adding_metadata.mdwn
automatically_getting_files_on_checkout.mdwn
beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes.mdwn
Building_git-annex_on_Debian_OR___37____164____35____34____164____37____38____34____35___Haskell__33__.mdwn
Bup_repositories_in_git-annex.mdwn
centralised_repository__58___starting_from_nothing.mdwn
centralized_git_repository_tutorial.mdwn
cloning_a_repository_privately.mdwn remove misleading para 2021-04-23 16:01:12 -04:00
Crude_Windows_Sync.mdwn
Decentralized_repository_behind_a_Firewall.mdwn
Decrypting_files_in_special_remotes_without_git-annex.mdwn
Delay_Assistant_Startup_on_Login.mdwn
deleting_unwanted_files.mdwn Automatically convert direct mode repositories to v7 with adjusted unlocked branches 2019-08-26 15:05:25 -04:00
disabling_a_special_remote.mdwn fix a transcription error - to disable a remote, we set ignore to true, not false! 2020-01-08 15:30:51 +00:00
downloading_podcasts.mdwn default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
dropboxannex.mdwn 2021-07-06 19:50:36 +00:00
dumb_metadata_extraction_from_xbmc.mdwn
emacs_integration.mdwn
enable_tor_on_nixos.mdwn Add a tip explaining how to enable-tor on NixOS 2021-08-24 04:34:55 +00:00
Faster_bash_autocompletion_with_big_annex_repos.mdwn
file_manager_integration.mdwn update for https://pypi.org/project/thunar-plugins/ 2023-01-01 18:10:35 -04:00
finding_duplicate_files.mdwn
finding_which_file_matches_a_key.mdwn update from comments 2019-10-23 20:15:40 +00:00
flickrannex.mdwn
fully_encrypted_git_repositories_with_gcrypt.mdwn change back to git-remote-gcrypt 1.4 url forms, with a note that that version is needed 2021-04-13 12:55:50 -04:00
get_git-annex-shell_into_PATH.mdwn
git-annex_extensions_for_ranger__44___the_file_manager.mdwn
git-annex_on_NFS.mdwn
Git_annex_and_Calibre.mdwn Automatically convert direct mode repositories to v7 with adjusted unlocked branches 2019-08-26 15:05:25 -04:00
hashdeep_integration.mdwn 2020-12-10 15:53:37 +00:00
hiding_missing_files.mdwn fix wrong option, followup 2019-03-22 09:51:42 -04:00
How_to_retroactively_annex_a_file_already_in_a_git_repo.mdwn
imapannex.mdwn
install_on_Android_in_Termux.mdwn
Internet_Archive_via_S3.mdwn
largefiles.mdwn document version for --force-large/--force-small 2020-10-26 11:34:30 -04:00
local_caching_of_annexed_files.mdwn
making_a_remote_repo_update_when_changes_are_pushed_to_it.mdwn
mc_menu_integration.mdwn
megaannex.mdwn improve special remotes list 2020-03-04 22:27:56 -04:00
metadata_driven_views.mdwn
migrating_data_to_a_new_backend.mdwn
migrating_two_seperate_disconnected_directories_to_git_annex.mdwn comment, update tip 2022-10-31 12:16:36 -04:00
multiple_remotes_accessing_the_same_data_store.mdwn httpalso: support exporttree=yes 2020-09-02 11:26:00 -04:00
offline_archive_drives.mdwn
owncloudannex.mdwn two more services with multiple special remotes 2020-03-04 22:42:33 -04:00
peer_to_peer_network_with_tor.mdwn Fix typo and incorrect references to "git annex peer" 2019-09-29 23:34:25 +00:00
playlist_fetch.mdwn
powerful_file_matching.mdwn
public_Amazon_S3_remote.mdwn
publishing_your_files_to_the_public.mdwn fix typo in git-annex command to create public S3 remote 2020-11-25 20:31:23 +00:00
recover_data_from_lost+found.mdwn
recovering_from_a_corrupt_git_repository.mdwn
remote_webapp_setup.mdwn
replacing_Sparkleshare_or_dvcs-autosync_with_the_assistant.mdwn
Repositories_with_large_number_of_files.mdwn
semi-synchronized_remotes.mdwn
setup_a_public_repository_on_a_web_site.mdwn update to work with modern git 2022-09-13 15:09:02 -04:00
Shamir_secret_sharing_and_git-annex.mdwn
shared_git_annex_directory_between_multiple_users.mdwn
skydriveannex.mdwn
splitting_a_repository.mdwn update for filter-branch 2021-05-17 15:03:47 -04:00
storing_data_in_git-lfs.mdwn git-lfs: remember urls, and autoenable remotes using known urls 2019-11-18 16:09:09 -04:00
Synology_NAS_and_git_annex.mdwn
Systemd_unit.mdwn
The_perfect_preferred_content_settings_for_my_android_phone.mdwn
transmission_integration.mdwn
unlocked_files.mdwn close as git-annex already makes reflinks when supported 2022-11-14 13:29:27 -04:00
untrusted_repositories.mdwn
using_Amazon_Glacier.mdwn Behavior change: --trust-glacier option no longer overrides trust 2021-01-07 10:37:43 -04:00
using_Amazon_S3.mdwn reorder 2019-04-23 14:45:34 -04:00
using_Backblaze_B2.mdwn fix paste 2020-05-12 09:55:54 -04:00
using_borg_for_efficient_storage_of_old_annexed_files.mdwn improve docs based on forum feedback 2021-06-14 13:04:58 -04:00
using_box.com_as_a_special_remote.mdwn 2019-02-04 14:40:03 +00:00
Using_Git-annex_as_a_web_browsing_assistant.mdwn
using_git-annex_from_your_program.mdwn tip documenting mostly --batch --json including with -J 2020-09-16 13:00:47 -04:00
Using_git-annex_on_NTFS_with_WSL1.mdwn Update WSL1 tips 2022-10-13 15:29:12 +00:00
Using_git-worktree_with_annex.mdwn 2018-11-28 09:02:12 +00:00
using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn
using_gitolite_with_git-annex.mdwn
using_Google_Cloud_Storage.mdwn improve special remotes list 2020-03-04 22:27:56 -04:00
using_Google_Drive.mdwn improve special remotes list 2020-03-04 22:27:56 -04:00
using_Hubic.mdwn improve special remotes list 2020-03-04 22:27:56 -04:00
using_nested_git_repositories.mdwn 2022-01-13 18:08:25 +00:00
using_signed_git_commits.mdwn
using_the_web_as_a_special_remote.mdwn default to yt-dlp and fix progress parsing bugs 2023-05-27 13:04:53 -04:00
visualizing_repositories_with_gource.mdwn
what_to_do_when_a_repository_is_corrupted.mdwn
what_to_do_when_you_lose_a_repository.mdwn
yet_another_simple_disk_usage_like_utility.mdwn
ZSH_completion.mdwn