mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions
git-annex/doc
History
Joey Hess 34530e59d9
Avoid using a lot of memory when large objects are present in the git repository 
.. and have to be checked to see if they are a pointed to an annexed file.

Cases where such memory use could occur included, but were not limited to:
  - git commit -a of a large unlocked file (in v5 mode)
  - git-annex adjust when a large file was checked into git directly
Generally, any use of catKey was a potential problem.

Fix by using git cat-file --batch-check to check size before catting.
This adds another git batch process, which is included in the CatFileHandle
for simplicity.

There could be performance impact, anywhere catKey is used. Particularly
likely to affect adjusted branch generation speed, and operations on
unlocked files in v6 mode. Hopefully since the --batch-check and
--batch read the same data, disk buffering will avoid most overhead.
Leaving only the overhead of talking to the process over the pipe and
whatever computation --batch-check needs to do.

This commit was sponsored by Bruno BEAUFILS on Patreon.
2016-10-05 15:24:13 -04:00
..
android
Android
assistant Revert "removed" 2016-06-02 22:29:55 -04:00
automatic_conflict_resolution
backends comment 2016-09-05 15:49:19 -04:00
bare_repositories
bugs Avoid using a lot of memory when large objects are present in the git repository 2016-10-05 15:24:13 -04:00
coding_style
contact
contribute comment 2016-05-27 11:46:24 -04:00
copies
design poll vote (OpenStack SWIFT) 2016-10-04 11:57:43 +00:00
devblog comments 2016-10-05 11:33:08 -04:00
direct_mode
download remove spam 2016-06-08 13:29:46 -04:00
ekg
encryption comment 2016-04-04 14:52:43 -04:00
footer
forum Linux standalone: Add back the LOCPATH=/dev/null hack to avoid the system locale-archive being read. 2016-10-04 12:53:09 -04:00
future_proofing
git-annex-copy response 2016-04-04 16:32:38 -04:00
git-annex-direct
git-annex-drop response 2016-05-04 13:29:37 -04:00
git-annex-importfeed
git-annex-preferred-content comment and improve docs 2016-04-04 16:10:05 -04:00
git-annex-unused
how_it_works
install response 2016-09-21 13:22:47 -04:00
internals
license remove LGPL, after removing the LGPLed file 2016-03-05 11:24:05 -04:00
links
metadata
news cleanup 2016-09-23 09:47:48 -04:00
not
polls
preferred_content comment 2016-01-29 13:37:17 -04:00
required_content
scalability
special_remotes New url for git-remote-gcrypt, now maintained by spwhitton. 2016-07-05 11:30:58 -04:00
stickers
sync bug 2016-09-21 14:57:20 -04:00
templates
testimonials
tips Added a comment: Using gitolite 3.6.6 mirror not working with annex 2016-09-28 18:12:56 +00:00
todo Added a comment: Bash on Windows 2016-09-30 19:14:34 +00:00
trust
tuning
upgrades
use_case
users Added page for mih user 2016-03-17 21:17:15 +00:00
videos
walkthrough Added a comment: Nevermind - broken symlinks will be fixed on commit 2016-08-23 21:53:05 +00:00
Android.mdwn
assistant.mdwn
automatic_conflict_resolution.mdwn
backends.mdwn
bare_repositories.mdwn mention --branch 2016-07-20 13:39:13 -04:00
bugs.mdwn
builds.mdwn show OSX test result 2016-01-15 16:55:13 -04:00
chunking.mdwn
coding_style.mdwn
comments.mdwn
contact.mdwn
contribute.mdwn
copies.mdwn
design.mdwn
devblog.mdwn
direct_mode.mdwn
distributed_version_control.mdwn
download.mdwn Updated cabal file explictly lists source files. 2016-05-24 01:28:07 -04:00
ekg.mdwn
encryption.mdwn Change git annex info remote encryption description to use wording closer to what's used in initremote. 2016-05-11 16:09:39 -04:00
favicon.ico
forum.mdwn This reverts commit f7ced591d3 2016-04-23 09:41:16 +00:00
future_proofing.mdwn
git-annex-add.mdwn annex.addunlocked 2016-02-16 14:43:43 -04:00
git-annex-addunused.mdwn
git-annex-addurl.mdwn addurl, get: Added --json-progress option, which adds progress objects to the json output. 2016-09-09 15:06:54 -04:00
git-annex-adjust.mdwn adjust: Add --fix adjustment, which is useful when the git directory is in a nonstandard place. 2016-05-16 17:18:33 -04:00
git-annex-assistant.mdwn
git-annex-calckey.mdwn calckey can use --backend 2016-04-21 11:23:17 -04:00
git-annex-checkpresentkey.mdwn checkpresentkey: Allow to be run without an explicit remote and add --batch 2016-02-12 16:43:51 -04:00
git-annex-contentlocation.mdwn
git-annex-copy.mdwn copy, move, mirror: Support --json and --json-progress. 2016-09-09 16:24:26 -04:00
git-annex-dead.mdwn
git-annex-describe.mdwn
git-annex-diffdriver.mdwn
git-annex-direct.mdwn docs for git-annex adjust 2016-03-29 11:39:43 -04:00
git-annex-drop.mdwn --branch, stage 1 2016-07-20 12:05:26 -04:00
git-annex-dropkey.mdwn dropkey --json 2016-03-06 12:58:36 -04:00
git-annex-dropunused.mdwn
git-annex-edit.mdwn
git-annex-enableremote.mdwn enableremote: Remove annex-ignore configuration from a remote. 2016-05-24 15:58:27 -04:00
git-annex-examinekey.mdwn
git-annex-expire.mdwn
git-annex-find.mdwn find --batch 2016-01-20 13:04:07 -04:00
git-annex-findref.mdwn
git-annex-fix.mdwn
git-annex-forget.mdwn
git-annex-fromkey.mdwn
git-annex-fsck.mdwn --branch, stage 1 2016-07-20 12:05:26 -04:00
git-annex-fuzztest.mdwn finished splitting out man pages for all commands 2015-03-25 12:09:49 -04:00
git-annex-get.mdwn addurl, get: Added --json-progress option, which adds progress objects to the json output. 2016-09-09 15:06:54 -04:00
git-annex-group.mdwn
git-annex-groupwanted.mdwn
git-annex-import.mdwn document annex.largefiles handling 2016-01-25 13:41:21 -04:00
git-annex-importfeed.mdwn doc/*.mdwn: Minor fixes (typos, letter case) 2015-07-26 04:21:06 +02:00
git-annex-indirect.mdwn
git-annex-info.mdwn info: Support being passed a treeish, and show info about the annexed files in it similar to how a directory is handled. 2016-09-15 12:51:00 -04:00
git-annex-init.mdwn
git-annex-initremote.mdwn replace redudundant and incomplete encryption para with link 2016-05-24 14:38:59 -04:00
git-annex-list.mdwn
git-annex-lock.mdwn
git-annex-log.mdwn log: Added --all option. 2016-07-17 15:15:08 -04:00
git-annex-lookupkey.mdwn
git-annex-map.mdwn legend 2016-05-04 16:43:46 -04:00
git-annex-matchexpression.mdwn matchexpression: Added --largefiles option to parse an annex.largefiles expression. 2016-02-03 16:58:36 -04:00
git-annex-matching-options.mdwn better example 2016-04-01 10:58:21 -04:00
git-annex-merge.mdwn
git-annex-metadata.mdwn Added metadata --batch option, which allows getting, setting, deleting, and modifying metadata for multiple files/keys. 2016-07-27 10:46:25 -04:00
git-annex-migrate.mdwn
git-annex-mirror.mdwn copy, move, mirror: Support --json and --json-progress. 2016-09-09 16:24:26 -04:00
git-annex-move.mdwn copy, move, mirror: Support --json and --json-progress. 2016-09-09 16:24:26 -04:00
git-annex-numcopies.mdwn
git-annex-pre-commit.mdwn
git-annex-preferred-content.mdwn Fixed broken link to standard groups page 2016-04-24 16:19:20 +00:00
git-annex-proxy.mdwn
git-annex-readpresentkey.mdwn fix link 2016-04-18 16:36:16 -04:00
git-annex-registerurl.mdwn
git-annex-reinit.mdwn
git-annex-reinject.mdwn reinject: Added new mode which can reinject known files into the annex. 2016-04-22 13:49:32 -04:00
git-annex-rekey.mdwn
git-annex-remotedaemon.mdwn
git-annex-repair.mdwn
git-annex-required.mdwn
git-annex-resolvemerge.mdwn
git-annex-rmurl.mdwn
git-annex-schedule.mdwn
git-annex-semitrust.mdwn
git-annex-setkey.mdwn
git-annex-setpresentkey.mdwn
git-annex-shell.mdwn wording 2016-04-04 16:57:13 -04:00
git-annex-smudge.mdwn Fix path in git-annex-smudge(1) 2016-08-16 11:50:19 -04:00
git-annex-status.mdwn
git-annex-sync.mdwn
git-annex-test.mdwn
git-annex-testremote.mdwn
git-annex-transferkey.mdwn
git-annex-transferkeys.mdwn
git-annex-trust.mdwn
git-annex-unannex.mdwn
git-annex-undo.mdwn
git-annex-ungroup.mdwn
git-annex-uninit.mdwn
git-annex-unlock.mdwn Typo 2016-04-01 21:38:42 +00:00
git-annex-untrust.mdwn
git-annex-unused.mdwn
git-annex-upgrade.mdwn
git-annex-vadd.mdwn
git-annex-vcycle.mdwn
git-annex-version.mdwn
git-annex-vfilter.mdwn
git-annex-vicfg.mdwn
git-annex-view.mdwn document ! syntax 2016-05-03 14:02:22 -04:00
git-annex-vpop.mdwn
git-annex-wanted.mdwn
git-annex-watch.mdwn
git-annex-webapp.mdwn
git-annex-whereis.mdwn --branch, stage 1 2016-07-20 12:05:26 -04:00
git-annex-xmppgit.mdwn
git-annex.mdwn Added annex.gnupg-decrypt-options and remote.<name>.annex-gnupg-decrypt-options, which are passed to gpg when it's decrypting data. 2016-05-10 13:03:56 -04:00
git-union-merge.mdwn
how_it_works.mdwn
index.mdwn
install.mdwn move tumbleweed stuff to the openSUSE main page 2016-07-05 11:50:01 -04:00
internals.mdwn
license.mdwn remove LGPL, after removing the LGPLed file 2016-03-05 11:24:05 -04:00
location_tracking.mdwn
logo-old-bw.svg
logo-old.png
logo-old.svg
logo-old_small.png
logo.mdwn
logo.svg
logo_16x16.png
logo_32x32.png
logo_small.png
meta.mdwn
metadata.mdwn
news.mdwn
not.mdwn update 2016-01-19 12:07:46 -04:00
polls.mdwn
preferred_content.mdwn Support --metadata field<number, --metadata field>number etc to match ranges of numeric values. 2016-02-27 10:55:02 -04:00
privacy.mdwn
publicrepos.mdwn
related_software.mdwn add densho 2016-02-12 12:21:04 -04:00
repomap.png
required_content.mdwn
scalability.mdwn
shortcuts.mdwn
sidebar.mdwn remove flattr link, add patreon link 2016-08-03 17:40:26 -04:00
sitemap.mdwn
special_remotes.mdwn added rclone supported providers 2016-04-23 19:22:16 +00:00
stickers.mdwn improve 2015-11-19 18:56:24 -04:00
submodules.mdwn
summary.mdwn
sync.mdwn
testimonials.mdwn
thanks.mdwn update 2016-09-21 11:32:40 -04:00
tips.mdwn
todo.mdwn
transferring_data.mdwn
trust.mdwn
tuning.mdwn response; improve docs 2016-04-22 14:49:15 -04:00
upgrades.mdwn Upgrading a direct mode repository to v6 has changed to enter an adjusted unlocked branch. 2016-04-04 13:17:24 -04:00
users.mdwn
videos.mdwn
Void.mdwn
walkthrough.mdwn fix links 2016-03-12 13:47:31 -04:00