git-annex/doc/todo/make_annex_info_more_efficient.mdwn

ATM it takes about a minute for 'git annex info' on a sizeable but not huge repository with only ~450 files under annex but a few thousand of files  (~7000) in the tree.  I am not quite sure why it takes that long since it seems to care only about annexed files.  Also it might be of benefit to parallelize some traversal operations to take advantage of multiple cpu/cores

[[!meta author=yoh]]
[[!tag projects/datalad]]

> I sense this one has reached its end, it's fast enough, so [[done]] --[[Joey]]
2016-02-15 15:26:36 +00:00			`ATM it takes about a minute for 'git annex info' on a sizeable but not huge repository with only ~450 files under annex but a few thousand of files (~7000) in the tree. I am not quite sure why it takes that long since it seems to care only about annexed files. Also it might be of benefit to parallelize some traversal operations to take advantage of multiple cpu/cores`
added [[!meta author=yoh]] 2016-02-15 15:29:22 +00:00
			`[[!meta author=yoh]]`
Embed projects/{datalad,repronim} tags for the past todo/bugs Assignment between datalad and repronim might not be 100% right but approx there depending on the date Here is the command and its output: $> git grep -l '\[\[!meta author=\(yoh\\|ben\\|yarikoptic\\|kyle\)\]\]' \| grep -v comment \| grep -v assure_correct_names___40 \| while read f; do d=$(git log --date=iso $f\| awk '/^Date/{print $2;}' \| tail -n 1 ); p=datalad; if [[ "$d" =~ 2019-.* ]] \|\| [[ "$d" =~ 2018-1.* ]]; then p=repronim; fi; echo $d $p: $f; sed -i -e "s,\(\[\[\!meta author.*\),\1\n\[\[\!tag projects/$p\]\],g" $f; done 2017-10-10 datalad: doc/bugs/--shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__.mdwn 2017-08-12 datalad: doc/bugs/2_ssh_connection_prompts_for_password.mdwn 2018-10-22 repronim: doc/bugs/3_tests_fail_while_trying_to_build_fresh_git_annex_6.20181011+git109-gff9ba1f4d.mdwn 2018-12-20 repronim: doc/bugs/7.20181211+git29-gab4a1bed9_fails_tests_during_neurodebian_build.mdwn 2016-03-12 datalad: doc/bugs/OSX__58___addurl_--batch_--json_spits_out_shortened_output_string__dies_off_with_4.mdwn 2018-11-16 repronim: doc/bugs/Please_include_version_of_used_to_build_ghc_within___34__git_annex_version__34__.mdwn 2016-03-10 datalad: doc/bugs/R__47__O_permissions_are_stalking_me_somehow.mdwn 2019-07-03 repronim: doc/bugs/Regression_in___96__find_--json__96___output.mdwn 2017-01-27 datalad: doc/bugs/Too_difficult_if_not_impossible_to_explicitly_add__47__keep_file_under_git___40__not_annex__41___in_v6_without_employing_.gitattributes.mdwn 2018-12-11 repronim: doc/bugs/Unable_to_addurl_file__58____47____47____47___on_Windows.mdwn 2019-04-10 repronim: doc/bugs/Unable_to_get__47__addurl_to_http_link__58___download_failed__58___InvalidHeader___34__preload__34__.mdwn 2019-09-19 repronim: doc/bugs/add_--json_should_provide_a___34__note__34___with_an_error_message_in_case_of_an_error.mdwn 2016-11-20 datalad: doc/bugs/adds_file_destined_for_annex_into_git_in___39__addurl__39__.mdwn 2016-01-12 datalad: doc/bugs/addurl_--batch__--with-files_doesn__39__t_add_file_into_git_until_pipe_is_closed.mdwn 2016-01-15 datalad: doc/bugs/addurl_--batch___40__--json_or_not__41___doesn__39__t_report_failure_correctly_if_non-annexed_file_exists.mdwn 2016-05-23 datalad: doc/bugs/addurl_--batch_decides_to_talk_to_ssh_remotes_for_some_reason.mdwn 2016-01-11 datalad: doc/bugs/addurl_--file__causes_file_redownload_even_if_it_already_present.mdwn 2019-03-18 repronim: doc/bugs/addurl_fails_with_non-youtube_url_with___34__This_url_is_supported_by_youtube-dl__44___but_youtube-dl_could_potentially_access_any_address__34__.mdwn 2016-12-04 datalad: doc/bugs/annex_add_ignores_.-prefixed_directories.mdwn 2017-03-22 datalad: doc/bugs/annex_doesn__39__t_fixup_symlinks_when___34__git_commit_path__95__to__95__repo__34___is_used.mdwn 2016-07-12 datalad: doc/bugs/annex_drop_fails_to_determine_availability_on_a_http_url_redirecting_to_ftp.mdwn 2016-04-26 datalad: doc/bugs/annex_drop_is_not___34__in_effect__34___for_load_which_was___34__addurl_--batch__34__ed_but_not_yet_committed.mdwn 2018-09-25 datalad: doc/bugs/annex_get_-J_16_via_ssh_stalls_.mdwn 2019-06-05 repronim: doc/bugs/annex_init_no_longer_generates_default_description.mdwn 2018-03-27 datalad: doc/bugs/annex_metadata___40__not_--batch__39__ed__41___is_not_aware_of_files_added_via_addurls_--batch.mdwn 2018-11-12 repronim: doc/bugs/annex_sets_remote___40__regular_http__41___to_be_ignored_while_running_in_docker_container.mdwn 2018-02-23 datalad: doc/bugs/annex_sync___40__in_direct_mode_only__41___does_not_actually_update_local_branch__47__tree.mdwn 2018-05-04 datalad: doc/bugs/annex_view_barfs__fatal__58___Unable_to_add___40__null__41___to_database.mdwn 2016-09-09 datalad: doc/bugs/autoenable__61__true_seems_to_not_work_any_longer.mdwn 2015-11-20 datalad: doc/bugs/autostart__47__git-annex.desktop_gets_resurrected.mdwn 2016-03-29 datalad: doc/bugs/awkward_error_from_annex_whenever_operating_on_a_clone_with_submodules.mdwn 2016-11-08 datalad: doc/bugs/cannot___34__install__34___standalone_git_annex_within_afs_mount.mdwn 2017-03-17 datalad: doc/bugs/cannot___40__or_how__63____41___to_pass_socket_path_with_a_space_in_its_path_via_annex-ssh-options.mdwn 2018-08-08 datalad: doc/bugs/cannot_commit___34__annex_add__34__ed_modified_file_which_switched_its_largefile_status_to_be_committed_to_git_now.mdwn 2017-01-20 datalad: doc/bugs/clash_of_-j__in_copy_for_--json_--json-progress.mdwn 2017-03-01 datalad: doc/bugs/copy_does_not_reflect_some_failed_copies_in_--json_output.mdwn 2017-02-10 datalad: doc/bugs/could_standalone_copy_of_git_ship__47__deploy_hooks_samples_as_well_as_stock_git__63__.mdwn 2015-11-12 datalad: doc/bugs/does_not_complain__47__fail_if_by_mistake_option_value_passed_within_arg_for__-c.mdwn 2016-07-14 datalad: doc/bugs/drop_blows_on_lustre__58___SQLite3_returned_ErrorIO.mdwn 2016-04-29 datalad: doc/bugs/dropkey_--batch_--json_--force_is_always_succesfull.mdwn 2016-02-24 datalad: doc/bugs/duplicate_progress_reports_in_parallel___39__get__39__.mdwn 2016-04-27 datalad: doc/bugs/encfs_support_--_shouldn__39__t_it_be_treated_as_crippled_already__63__.mdwn 2019-08-03 repronim: doc/bugs/fails_to___96__get__96___in_parallel_for_a_freshly_clone_from_public_s3_bucket_where_versioning_info_was_forgotten.mdwn 2019-09-02 repronim: doc/bugs/fails_to_init_under_a_directory_with_a___34__tricky__34___name.mdwn 2017-07-26 datalad: doc/bugs/fails_to_verify_presence_via_http_while_wget_fetches_it_just_fine.mdwn 2019-08-13 repronim: doc/bugs/fresh_build_for_neurodebian__58___test_failure.mdwn 2017-09-13 datalad: doc/bugs/fsck_does_not_detect__47__fix_some_key_directories_correctly.mdwn 2017-07-30 datalad: doc/bugs/get_--json_fails_whenever_plain_get_works___40__with_https_urls__41__.mdwn 2018-03-09 datalad: doc/bugs/get_-JX__58____transfer_already_in_progress_..._for_some_files.mdwn 2017-08-30 datalad: doc/bugs/get_-J___34__fails__34___to_get_files_with_the_same_key.mdwn 2017-03-15 datalad: doc/bugs/get_-J_cannot_be_used_with_password-based_authentication.mdwn 2018-02-10 datalad: doc/bugs/get_-J_from_ssh_remote_tries_to_lock_in_home_directory__63__.mdwn 2019-07-11 repronim: doc/bugs/get_fails_to_place_v7_unlocked_file_content_into_the_file_tree_in_v7_in_repo_with_detached_HEAD.mdwn 2018-05-06 datalad: doc/bugs/get_from_the_web_remote_fails___40__redirect__63____41___-_lack_of_further_debug_info.mdwn 2018-09-22 datalad: doc/bugs/get_over_ssh_fails_with___fd__58__19__58___hClose__58___resource_vanished.mdwn 2016-03-12 datalad: doc/bugs/git-annex__58___content_is_locked__while_trying_to_move_under_NFS_and_pidlock.mdwn 2019-03-18 repronim: doc/bugs/git-annex_precommit_.__gets_stuck_after_likely_interrupted___96__annex_add__96___call.mdwn 2016-06-02 datalad: doc/bugs/git_annex_adjust_--unlock_seems_to_cause_migration_of_a_file_to_another_backend.mdwn 2017-02-06 datalad: doc/bugs/git_annex_info_fails_on_NFS__58___waitToSetLock__58___resource_exhausted___40__No_locks_available__41__.mdwn 2017-10-25 datalad: doc/bugs/graft__47__graft_cleanup_commits_--_really_needed__63__.mdwn 2018-03-05 datalad: doc/bugs/howto_guarantee_a_single_instance_of_a_special_remote__63__.mdwn 2018-09-25 datalad: doc/bugs/impossible_to_login_to_the_website_at_times.mdwn 2018-02-20 datalad: doc/bugs/impossible_to_perform___34__read-only__34___git_annex_info_without_write_permissions.mdwn 2016-01-11 datalad: doc/bugs/inconsistent_output_upon_addurl_--batch_complicates_if_not_forbids_reliable_parsing_of_output.mdwn 2015-12-21 datalad: doc/bugs/info_--json_lists_backend_usage_stats_as_a_list_of_lists.mdwn 2018-10-11 repronim: doc/bugs/initial_get_-J_on_exported_S3_bucket_fails.mdwn 2016-01-28 datalad: doc/bugs/lookupkey_started_to_spit_out___34__debug__34___messages_to_stdout.mdwn 2016-02-09 datalad: doc/bugs/man_page_for_command_misses_actual_command_in_the_synopsis_for_git-annex-checkpresentkey.mdwn 2018-11-08 repronim: doc/bugs/multiple_ssh_prompts__44___and_thread_blocked_indefinitely_in_an___63____63____63___transaction.mdwn 2017-02-17 datalad: doc/bugs/mysterious_dependency_of_git_annex_status_output_of_the_added_file.mdwn 2016-01-15 datalad: doc/bugs/new_whereis_--json_lost_information_about_web_urls_if_other_special_remotes_provide_them.mdwn 2017-05-25 datalad: doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_.mdwn 2016-05-01 datalad: doc/bugs/prematurely___40__can__39__t_check_offline__41___marks_remote_as_annex-ignore.mdwn 2018-05-08 datalad: doc/bugs/regression_-_fails_to_drop_._Exit_code_11_wo_--debug__44___and_1_with_--debug.mdwn 2017-12-06 datalad: doc/bugs/regression_-_yt__58___prefix_for___34__regular__34___urls.mdwn 2019-08-23 repronim: doc/bugs/regression__58___fails_to_detect_need_for_pidlock_on_an_NSF_mount.mdwn 2019-05-09 repronim: doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported.mdwn 2016-10-17 datalad: doc/bugs/remote_repository_must_be_version_6_as_well_to_have_copied_files_appear_correctly.mdwn 2016-08-05 datalad: doc/bugs/reports_success_when_addurl_--batch__a_file_which_is_.gitignore__39__d.mdwn 2017-09-26 datalad: doc/bugs/set_metadata_leaks_from_one___40__staged__41___key_to_another_during_rename_of_file.mdwn 2016-02-15 datalad: doc/bugs/shouldn__39__t_keep_permissions_of_the_ssh_remote__63__.mdwn 2016-03-09 datalad: doc/bugs/some_tests_fail_while_running_under_NFS.mdwn 2018-09-18 datalad: doc/bugs/some_transfers_for_get_-J4_fail_over_ssh__47__rsync_and_require_retry.mdwn 2018-09-25 datalad: doc/bugs/standalone_build_is_slightly_out_of_date.mdwn 2015-10-28 datalad: doc/bugs/standalone_builds_shouldn__39__t_pollute___126____47__.ssh_with_helpers_merely_upon_annex_init.mdwn 2016-05-25 datalad: doc/bugs/strips___95___from_extensions_in_E_backends__63__.mdwn 2016-03-01 datalad: doc/bugs/thread_blocked_indefinitely_in_an_STM_transaction__while_moving_within__a_local_clone.mdwn 2016-01-15 datalad: doc/bugs/treatment_of_largefiles_is_not_working_for_addurl_--fast___40__or_--relaxed__41__.mdwn 2018-09-11 datalad: doc/bugs/unable_to_get_from_public_S3_remote_without_clear_reasoning_why.mdwn 2018-09-25 datalad: doc/bugs/unclear_why_annex_fails_to_get_a_file_from_a_public_S3_bucket_where_it_exported_before.mdwn 2016-02-08 datalad: doc/bugs/use_of_annex_in_submodule_replaces_.git_with_incorrect_symlink.mdwn 2016-02-26 datalad: doc/bugs/using_regular_magic_file__warning_pollutes_stderr.mdwn 2018-09-06 datalad: doc/bugs/v6_-_under_subdir__58___git_add___34__whines__34____44___git_commit___34__blows__34__.mdwn 2018-09-12 datalad: doc/bugs/whereis_shouldn__39__t_just_crash_if_one_of_the_special_remotes___34__drivers__34___is_not_available.mdwn 2019-01-21 repronim: doc/bugs/windows__58___autostart_for_annex_64bit_looks_under_x86.mdwn 2016-01-18 datalad: doc/todo/--batch_for_add.mdwn 2016-01-18 datalad: doc/todo/--batch_for_find.mdwn 2016-01-05 datalad: doc/todo/--batch_for_info.mdwn 2016-10-24 datalad: doc/todo/Allow_for_TRANSFER-SUCCESS_to_report_also_a_URL_where_key_could_now_be_obtained_from.mdwn 2018-02-05 datalad: doc/todo/INFO_message_for_custom_special_remotes.mdwn 2018-09-19 datalad: doc/todo/Natively_support_s3__58____47____47___urls___40__for_addurl__44___get__44___etc__41__.mdwn 2016-07-28 datalad: doc/todo/__39__info_filename__39___to_provide_information_either_content_is_locally_present.mdwn 2019-04-25 repronim: doc/todo/add_--json-progress_to___96__git_annex_add__96__.mdwn 2016-04-22 datalad: doc/todo/add_option_to_whereis_to_avoid_network_interactions.mdwn 2017-03-23 datalad: doc/todo/annex_add___40__-u__124__--update__41___mode.mdwn 2016-08-05 datalad: doc/todo/annex_merge_--remotes.mdwn 2017-02-06 datalad: doc/todo/be_able_to_specify_custom_commit_message_for_git-annex_branch_commit.mdwn 2016-02-10 datalad: doc/todo/checkpresentkey_without_explicit_remote.mdwn 2018-09-28 datalad: doc/todo/config_setting_to_force_--progress_reporting_for_some_git_calls.mdwn 2018-09-27 datalad: doc/todo/configuration_option_for_default___34__mode__34___on_crippled_file_systems.mdwn 2016-06-09 datalad: doc/todo/could_standalone___39__fixed__39___git-annex_binaries_be_prelinked__63__.mdwn 2016-04-25 datalad: doc/todo/drop_--batch.mdwn 2016-06-16 datalad: doc/todo/get_--batch.mdwn 2017-07-21 datalad: doc/todo/git_annex_info___60__remote__62___does_not_list_all_the_parameters_for_the_remote.mdwn 2016-06-09 datalad: doc/todo/git_annex_info_to_include_information_about_repo_version__63__.mdwn 2019-04-05 repronim: doc/todo/globus_special_remote_as_a___34__transport__34___layer.mdwn 2018-02-06 datalad: doc/todo/include_msg_with_possible_reason_why_command___40__e.g._add__41___failed_into_--json_output.mdwn 2016-07-05 datalad: doc/todo/interface_to_the___34__progress__34___of_annex_operations.mdwn 2018-05-22 datalad: doc/todo/machine_readable_information_about_reason_of_failure_in_enableremote.mdwn 2015-11-02 datalad: doc/todo/make_addurl_respect_annex.largefiles_option.mdwn 2016-02-15 datalad: doc/todo/make_annex_info_more_efficient.mdwn 2016-05-19 datalad: doc/todo/make_copy_--fast__faster.mdwn 2016-02-12 datalad: doc/todo/metadata_--batch.mdwn 2017-02-16 datalad: doc/todo/output_of_wanted___40__and_possibly_group_etc__41___should_not_be_polluted_with___34__informational__34___messages.mdwn 2014-12-18 datalad: doc/todo/parallel_get.mdwn 2018-09-28 datalad: doc/todo/prevent_directly_printing_to_stderr_error_from_an_attempt_to_download_remote___47__config_file.mdwn 2018-08-04 datalad: doc/todo/provide___39__file__39___in_--json-progress_record_for_addurl.mdwn 2019-01-25 repronim: doc/todo/publicurl_config_for_all_special_remotes.mdwn 2016-01-19 datalad: doc/todo/return___34__key__34___entry_in_--json_output_for_addurl___40__and_future_add__41___--batch.mdwn 2019-01-25 repronim: doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn 2018-04-28 datalad: doc/todo/support___126____47__.netrc_for_http_access_authentication_specs_as_was_done_when_wget_was_used.mdwn 2018-01-12 datalad: doc/todo/support_ssh__58____47____47___or_sftp__58____47____47___urls_via___34__built-in__34___ssh_support.mdwn 2016-08-05 datalad: doc/todo/sync_--branches__to_sync_only_specified_branches___40__e.g._git-annex__41__.mdwn 2016-10-28 datalad: doc/todo/unlock_--json.mdwn 2016-10-17 datalad: doc/todo/unlock_--read-only.mdwn 2019-09-22 01:07:09 +00:00			`[[!tag projects/datalad]]`
tagging todo items 2020-01-29 15:28:32 +00:00
			`> I sense this one has reached its end, it's fast enough, so [[done]] --[[Joey]]`