mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/doc
History
Joey Hess 69f8e6c7c0
ImportableContentsChunkable 
This improves the borg special remote memory usage, by
letting it only load one archive's worth of filenames into memory at a
time, and building up a larger tree out of the chunks.

When a borg repository has many archives, git-annex could easily OOM
before. Now, it will use only memory proportional to the number of
annexed keys in an archive.

Minor implementation wart: Each new chunk re-opens the content
identifier database, and also a new vector clock is used for each chunk.
This is a minor innefficiency only; the use of continuations makes
it hard to avoid, although putting the database handle into a Reader
monad would be one way to fix it.

It may later be possible to extend the ImportableContentsChunkable
interface to remotes that are not third-party populated. However, that
would perhaps need an interface that does not use continuations.

The ImportableContentsChunkable interface currently does not allow
populating the top of the tree with anything other than subtrees. It
would be easy to extend it to allow putting files in that tree, but borg
doesn't need that so I left it out for now.

Sponsored-by: Noam Kremen on Patreon
2021-10-08 13:15:22 -04:00
..
android
Android remove old android comments/bug reports 2021-06-21 13:21:27 -04:00
assistant
automatic_conflict_resolution
backends
bare_repositories
bugs ImportableContentsChunkable 2021-10-08 13:15:22 -04:00
chunking
coding_style
contribute
copies
design response 2021-10-05 12:16:18 -04:00
devblog devblog 2021-08-03 17:14:06 -04:00
direct_mode
download
encryption Added a comment 2021-04-18 15:40:55 +00:00
footer
forum Added a comment 2021-10-06 03:36:02 +00:00
future_proofing
git-annex-add
git-annex-addurl
git-annex-adjust
git-annex-checkpresentkey
git-annex-config
git-annex-copy
git-annex-direct
git-annex-drop
git-annex-enableremote
git-annex-find
git-annex-import
git-annex-importfeed
git-annex-lookupkey
git-annex-metadata
git-annex-move
git-annex-p2p
git-annex-preferred-content
git-annex-reinject
git-annex-rmurl
git-annex-setpresentkey
git-annex-sync
git-annex-test
git-annex-unannex comment 2021-05-27 12:23:57 -04:00
git-annex-undo
git-annex-uninit
git-annex-unlock Added a comment: re: clarifying unlocked files 2021-04-21 16:34:50 +00:00
git-annex-unused
git-annex-whereis
how_it_works Added a comment 2021-08-18 21:07:12 +00:00
install remove information about old version of debian 2021-10-02 12:32:00 -04:00
internals Added annex.freezecontent-command and annex.thawcontent-command configs 2021-06-21 14:40:52 -04:00
license
links
metadata
news add news item for git-annex 8.20210903 2021-09-03 12:01:35 -04:00
not
polls
preferred_content
profiling
projects
required_content
scalability
security
special_remotes promote comment to bug 2021-10-05 11:55:33 -04:00
stickers
submodules
sync
templates
testimonials
thanks update 2021-10-05 10:43:27 -04:00
tips Added a comment: Thanks! 2021-10-05 21:09:44 +00:00
todo Added a comment 2021-10-02 17:04:02 +00:00
trust
tuning
upgrades
use_case
users removed 2021-05-18 17:16:12 +00:00
videos
walkthrough comment 2021-09-21 13:23:57 -04:00
workflow
Android.mdwn
assistant.mdwn
automatic_conflict_resolution.mdwn
backends.mdwn
bare_repositories.mdwn
bugs.mdwn fix name 2021-07-15 12:56:44 -04:00
builds.mdwn update for autobuilder changes 2021-07-02 13:03:19 -04:00
chunking.mdwn
coding_style.mdwn
comments.mdwn
contact.mdwn
contribute.mdwn
copies.mdwn
design.mdwn
devblog.mdwn
direct_mode.mdwn
distributed_version_control.mdwn
download.mdwn
encryption.mdwn
favicon.ico
forum.mdwn rename tag page 2021-07-15 12:55:05 -04:00
future_proofing.mdwn
git-annex-add.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-addunused.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-addurl.mdwn addurl, importfeed: Added --no-raw option 2021-06-27 11:14:51 -04:00
git-annex-adjust.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-assistant.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-benchmark.mdwn
git-annex-calckey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-checkpresentkey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-common-options.mdwn --size-limit exit 101 2021-06-04 16:43:47 -04:00
git-annex-config.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-contentlocation.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-copy.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-dead.mdwn fix innacuracy 2021-06-25 14:48:36 -04:00
git-annex-describe.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-diffdriver.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-direct.mdwn
git-annex-drop.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-dropkey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-dropunused.mdwn dropping unused marks as dead 2021-06-25 15:22:26 -04:00
git-annex-edit.mdwn
git-annex-enable-tor.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-enableremote.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-examinekey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-expire.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-export.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-filter-branch.mdwn decided not to include export/import trees 2021-05-17 14:12:15 -04:00
git-annex-find.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-findref.mdwn
git-annex-fix.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-forget.mdwn git-annex-filter-branch man page 2021-05-13 16:17:45 -04:00
git-annex-fromkey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-fsck.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-fuzztest.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-get.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-group.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-groupwanted.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-import.mdwn sync, merge: Added --allow-unrelated-histories option 2021-07-19 12:14:26 -04:00
git-annex-importfeed.mdwn addurl, importfeed: Added --no-raw option 2021-06-27 11:14:51 -04:00
git-annex-indirect.mdwn
git-annex-info.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-init.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-initremote.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-inprogress.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-list.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-lock.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-log.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-lookupkey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-map.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-matchexpression.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-matching-expression.mdwn
git-annex-matching-options.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-merge.mdwn sync, merge: Added --allow-unrelated-histories option 2021-07-19 12:14:26 -04:00
git-annex-metadata.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-migrate.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-mincopies.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-mirror.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-move.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-multicast.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-numcopies.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-p2p.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-post-receive.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-pre-commit.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-preferred-content.mdwn
git-annex-proxy.mdwn
git-annex-readpresentkey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-registerurl.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-reinit.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-reinject.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-rekey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-remotedaemon.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-renameremote.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-repair.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-required.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-resolvemerge.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-rmurl.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-schedule.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-semitrust.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-setkey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-setpresentkey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-shell.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-smudge.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-status.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-sync.mdwn sync, merge: Added --allow-unrelated-histories option 2021-07-19 12:14:26 -04:00
git-annex-test.mdwn
git-annex-testremote.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-transferkey.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-transferkeys.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-transferrer.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-trust.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-unannex.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-undo.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-ungroup.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-uninit.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-unlock.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-unregisterurl.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-untrust.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-unused.mdwn improve docs for refspec format 2021-09-21 13:19:57 -04:00
git-annex-upgrade.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-vadd.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-vcycle.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-version.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-vfilter.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-vicfg.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-view.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-vpop.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-wanted.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-watch.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-webapp.mdwn split out common options to its own page and mention it on each subcommand page 2021-05-10 15:00:13 -04:00
git-annex-whereis.mdwn --batch-keys 2021-08-25 14:21:12 -04:00
git-annex-whereused.mdwn wording 2021-07-14 17:08:38 -04:00
git-annex.mdwn improved bwrate limiting implementation 2021-09-22 15:27:16 -04:00
git-remote-tor-annex.mdwn
git-union-merge.mdwn
how_it_works.mdwn
index.mdwn
install.mdwn
internals.mdwn
license.mdwn
location_tracking.mdwn
logo-old-bw.svg
logo-old.png
logo-old.svg
logo-old_small.png
logo.mdwn
logo.svg
logo_16x16.png
logo_32x32.png
logo_small.png
meta.mdwn
metadata.mdwn
news.mdwn
not.mdwn
polls.mdwn
preferred_content.mdwn
privacy.mdwn
profiling.mdwn
projects.mdwn
publicrepos.mdwn
related_software.mdwn
repomap.png
required_content.mdwn
scalability.mdwn
security.mdwn
shortcuts.mdwn
sidebar.mdwn
sitemap.mdwn
special_remotes.mdwn
stickers.mdwn
submodules.mdwn
summary.mdwn
sync.mdwn
testimonials.mdwn
thanks.mdwn
tips.mdwn
todo.mdwn
transferring_data.mdwn
trust.mdwn
tuning.mdwn
upgrades.mdwn
users.mdwn
videos.mdwn
walkthrough.mdwn
workflow.mdwn