mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Annex
History
Joey Hess 69f8e6c7c0
ImportableContentsChunkable 
This improves the borg special remote memory usage, by
letting it only load one archive's worth of filenames into memory at a
time, and building up a larger tree out of the chunks.

When a borg repository has many archives, git-annex could easily OOM
before. Now, it will use only memory proportional to the number of
annexed keys in an archive.

Minor implementation wart: Each new chunk re-opens the content
identifier database, and also a new vector clock is used for each chunk.
This is a minor innefficiency only; the use of continuations makes
it hard to avoid, although putting the database handle into a Reader
monad would be one way to fix it.

It may later be possible to extend the ImportableContentsChunkable
interface to remotes that are not third-party populated. However, that
would perhaps need an interface that does not use continuations.

The ImportableContentsChunkable interface currently does not allow
populating the top of the tree with anything other than subtrees. It
would be easy to extend it to allow putting files in that tree, but borg
doesn't need that so I left it out for now.

Sponsored-by: Noam Kremen on Patreon
2021-10-08 13:15:22 -04:00
..
AdjustedBranch annex.adjustedbranchrefresh 2020-11-16 14:27:28 -04:00
Branch implemented filter-branch for key info 2021-05-17 11:11:39 -04:00
Concurrent differentiate between concurrency enabled at command line and by git config 2020-09-16 11:47:12 -04:00
Content split verifyKeyContent 2021-07-29 13:58:40 -04:00
Debug implement fastDebug 2021-04-06 15:24:28 -04:00
LockPool Display a message when git-annex has to wait for a pid lock file held by another process 2020-08-26 13:05:34 -04:00
MetaData update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
SpecialRemote remove redundant imports 2020-06-22 11:05:34 -04:00
VectorClock deal better with clock skew situations, using vector clocks 2021-08-04 12:33:46 -04:00
View Fix test suite failure on Windows 2021-08-24 14:03:29 -04:00
Action.hs start splitting out readonly values from AnnexState 2021-04-02 15:51:44 -04:00
AdjustedBranch.hs sync --quiet 2021-07-19 11:28:47 -04:00
AutoMerge.hs sync --quiet 2021-07-19 11:28:47 -04:00
BloomFilter.hs Revert "data type that starts off using a set but converts to a bloom filter when large" 2020-07-01 20:12:19 -04:00
Branch.hs decorate openTempFile errors with the template name 2021-08-30 13:05:02 -04:00
BranchState.hs Avoid excess commits to the git-annex branch when stall detection is enabled 2021-04-02 11:57:18 -04:00
CatFile.hs reinject: Fix crash when reinjecting a file from outside the repository 2021-10-01 14:06:34 -04:00
ChangedRefs.hs more RawFilePath conversion 2020-10-29 14:20:57 -04:00
CheckAttr.hs mincopies 2021-01-06 14:15:19 -04:00
CheckIgnore.hs more RawFilePath conversion 2020-11-03 10:11:04 -04:00
Common.hs use fastDebug everywhere it can be used 2021-04-06 15:41:24 -04:00
Concurrent.hs remove unused import 2021-06-15 11:31:59 -04:00
Content.hs improve url download failure display 2021-09-01 15:33:38 -04:00
CopyFile.hs resume properly when copying a file to/from a local git remote is interrupted (take 2) 2021-09-27 16:03:01 -04:00
CurrentBranch.hs refactor getCurrentBranch 2018-10-19 17:29:18 -04:00
Debug.hs fix fastDebug to check if debugging is actually enabled 2021-04-06 16:28:37 -04:00
Difference.hs include git-annex-shell back in 2019-12-02 11:51:52 -04:00
DirHashes.hs Added http special remote, which is useful for accessing other remotes that publish content stored in them via http/https. 2020-09-01 15:16:35 -04:00
Drop.hs dropping unused marks as dead 2021-06-25 15:22:26 -04:00
Environment.hs include git-annex-shell back in 2019-12-02 11:51:52 -04:00
Export.hs convert Key to ShortByteString 2021-10-05 20:20:08 -04:00
ExternalAddonProcess.hs use fastDebug everywhere it can be used 2021-04-06 15:41:24 -04:00
FileMatcher.hs prep for fixing find --branch --unlocked 2021-03-02 13:39:31 -04:00
Fixup.hs fix a bug that prevented git-annex init from working in a submodule 2021-01-21 15:33:15 -04:00
GitOverlay.hs add: Significantly speed up adding lots of non-large files to git 2021-01-04 13:12:28 -04:00
HashObject.hs more RawFilePath conversion 2020-10-28 17:25:59 -04:00
Hook.hs don't try to remove pre-commit-annex and post-update-annex-hooks 2020-10-19 13:13:49 -04:00
Import.hs ImportableContentsChunkable 2021-10-08 13:15:22 -04:00
Ingest.hs defer write permissions checking in import until after copy to repo 2021-09-02 13:45:21 -04:00
Init.hs fix windows build 2021-09-02 12:21:25 -04:00
InodeSentinal.hs add debugging in sameInodeCache 2021-07-26 10:58:07 -04:00
Journal.hs implement annex.privateremote and remote.name.private configs 2021-04-23 14:21:57 -04:00
Link.hs check symlink before reading file 2021-06-14 11:53:51 -04:00
Locations.hs split annexLocations 2021-07-16 14:17:56 -04:00
LockFile.hs more RawFilePath conversion 2020-10-29 10:50:29 -04:00
LockPool.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Magic.hs Serialize use of C magic library, which is not thread safe. 2020-09-17 17:27:42 -04:00
MetaData.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Multicast.hs use programPath consistently, not readProgramFile 2020-03-30 16:06:27 -04:00
Notification.hs wip RawFilePath 2x git-annex find speedup 2019-11-26 16:01:58 -04:00
NumCopies.hs drop, move, mirror: when two files have the same content, honor the max numcopies and requiredcopies 2021-06-15 11:38:44 -04:00
Path.hs assistant: Fix a crash on startup by avoiding using forkProcess 2021-05-12 15:08:03 -04:00
Perms.hs init: check for filesystem where write bit cannot be removed 2021-09-01 10:27:28 -04:00
PidLock.hs propagate git-annex -c on to transferrer child process 2020-12-15 11:36:25 -04:00
Queue.hs Git.Queue: allow providing git common options like -c 2021-01-04 12:51:55 -04:00
RemoteTrackingBranch.hs refactor 2019-11-11 19:10:52 -04:00
ReplaceFile.hs fix test suite 2021-08-02 13:59:23 -04:00
SpecialRemote.hs avoid getting creds from environment during autoenable 2021-03-17 09:41:12 -04:00
Ssh.hs Added annex.adviceNoSshCaching config. 2021-05-27 12:37:49 -04:00
StallDetection.hs bwlimit 2021-09-21 16:58:10 -04:00
TaggedPush.hs Ref ByteString conversion done 2020-04-07 17:41:09 -04:00
Tmp.hs propagate signals to the transferrer process group 2020-12-11 15:32:00 -04:00
Transfer.hs simplify annex.bwlimit handling 2021-09-22 10:52:01 -04:00
TransferrerPool.hs avoid using temp file size when deciding whether to retry failed transfer 2021-06-25 12:04:23 -04:00
UntrustedFilePath.hs importfeed: Fix reversion that caused some '.' in filenames to be replaced with '_' 2020-08-05 11:35:00 -04:00
UpdateInstead.hs v7 for all repositories 2019-08-30 14:09:14 -04:00
Url.hs incremental verification for web special remote 2021-08-18 15:02:22 -04:00
UUID.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
VariantFile.hs more RawFilePath 2019-12-18 17:10:28 -04:00
VectorClock.hs deal better with clock skew situations, using vector clocks 2021-08-04 12:33:46 -04:00
Verify.hs fix OSX build 2021-08-18 16:35:26 -04:00
Version.hs jump right to v8, don't stop part way 2020-02-24 13:21:00 -04:00
View.hs Fix a bug in view filename generation when a metadata value ended with "/" 2021-01-22 14:05:14 -04:00
Wanted.hs prevent dropping required content of other file using same content 2021-05-25 11:34:06 -04:00
WorkerPool.hs start splitting out readonly values from AnnexState 2021-04-02 15:51:44 -04:00
WorkTree.hs work around strange auto-init bug 2021-07-30 18:36:03 -04:00
YoutubeDl.hs bwlimit 2021-09-21 16:58:10 -04:00