69f8e6c7c0
This improves the borg special remote memory usage, by letting it only load one archive's worth of filenames into memory at a time, and building up a larger tree out of the chunks. When a borg repository has many archives, git-annex could easily OOM before. Now, it will use only memory proportional to the number of annexed keys in an archive. Minor implementation wart: Each new chunk re-opens the content identifier database, and also a new vector clock is used for each chunk. This is a minor innefficiency only; the use of continuations makes it hard to avoid, although putting the database handle into a Reader monad would be one way to fix it. It may later be possible to extend the ImportableContentsChunkable interface to remotes that are not third-party populated. However, that would perhaps need an interface that does not use continuations. The ImportableContentsChunkable interface currently does not allow populating the top of the tree with anything other than subtrees. It would be easy to extend it to allow putting files in that tree, but borg doesn't need that so I left it out for now. Sponsored-by: Noam Kremen on Patreon |
||
|---|---|---|
| .. | ||
| ActionItem.hs | ||
| AdjustedBranch.hs | ||
| Availability.hs | ||
| Backend.hs | ||
| Benchmark.hs | ||
| BranchState.hs | ||
| CatFileHandles.hs | ||
| CleanupActions.hs | ||
| Command.hs | ||
| Concurrency.hs | ||
| Creds.hs | ||
| Crypto.hs | ||
| DeferredParse.hs | ||
| DesktopNotify.hs | ||
| Difference.hs | ||
| Distribution.hs | ||
| Export.hs | ||
| FileMatcher.hs | ||
| GitConfig.hs | ||
| Group.hs | ||
| Import.hs | ||
| IndexFiles.hs | ||
| Key.hs | ||
| KeySource.hs | ||
| Link.hs | ||
| LockCache.hs | ||
| Messages.hs | ||
| MetaData.hs | ||
| Mime.hs | ||
| NumCopies.hs | ||
| ProposedAccepted.hs | ||
| RefSpec.hs | ||
| Remote.hs | ||
| RemoteConfig.hs | ||
| RemoteState.hs | ||
| RepoVersion.hs | ||
| ScheduledActivity.hs | ||
| StallDetection.hs | ||
| StandardGroups.hs | ||
| StoreRetrieve.hs | ||
| Test.hs | ||
| Transfer.hs | ||
| Transferrer.hs | ||
| TransferrerPool.hs | ||
| TrustLevel.hs | ||
| UrlContents.hs | ||
| UUID.hs | ||
| VectorClock.hs | ||
| View.hs | ||
| WorkerPool.hs | ||