mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions
git-annex/Command
History
Joey Hess 69f8e6c7c0
ImportableContentsChunkable 
This improves the borg special remote memory usage, by
letting it only load one archive's worth of filenames into memory at a
time, and building up a larger tree out of the chunks.

When a borg repository has many archives, git-annex could easily OOM
before. Now, it will use only memory proportional to the number of
annexed keys in an archive.

Minor implementation wart: Each new chunk re-opens the content
identifier database, and also a new vector clock is used for each chunk.
This is a minor innefficiency only; the use of continuations makes
it hard to avoid, although putting the database handle into a Reader
monad would be one way to fix it.

It may later be possible to extend the ImportableContentsChunkable
interface to remotes that are not third-party populated. However, that
would perhaps need an interface that does not use continuations.

The ImportableContentsChunkable interface currently does not allow
populating the top of the tree with anything other than subtrees. It
would be easy to extend it to allow putting files in that tree, but borg
doesn't need that so I left it out for now.

Sponsored-by: Noam Kremen on Patreon
2021-10-08 13:15:22 -04:00
..
Add.hs bwlimit 2021-09-21 16:58:10 -04:00
AddUnused.hs more RawFilePath conversion 2020-11-03 10:11:04 -04:00
AddUrl.hs improve url download failure display 2021-09-01 15:33:38 -04:00
Adjust.hs rename to --unlock-present and better reverse adjusting 2020-11-13 14:56:43 -04:00
Assistant.hs add searchPathContents 2021-02-02 19:06:15 -04:00
Benchmark.hs improve benchmark --databases 2019-11-21 17:25:20 -04:00
CalcKey.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
CheckPresentKey.hs --batch-keys 2021-08-25 14:21:12 -04:00
Commit.hs use Branch.name instead of hard coding the branch name 2020-09-15 15:47:22 -04:00
Config.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
ConfigList.hs work around strange auto-init bug 2021-07-30 18:36:03 -04:00
ContentLocation.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Copy.hs --batch-keys 2021-08-25 14:21:12 -04:00
Dead.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Describe.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
DiffDriver.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Direct.hs Automatically convert direct mode repositories to v7 with adjusted unlocked branches 2019-08-26 15:05:25 -04:00
Drop.hs --batch-keys 2021-08-25 14:21:12 -04:00
DropKey.hs --batch-keys 2021-08-25 14:21:12 -04:00
DropUnused.hs dropping unused marks as dead 2021-06-25 15:22:26 -04:00
EnableRemote.hs cache remote.log 2020-09-22 13:52:26 -04:00
EnableTor.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
ExamineKey.hs examinekey: Added a "file" format variable 2020-11-16 09:59:11 -04:00
Expire.hs Future proof activity log parsing 2021-06-14 14:18:19 -04:00
Export.hs add git ls-tree --long parser 2021-03-23 12:47:00 -04:00
FilterBranch.hs decided not to include export/import trees 2021-05-17 14:12:15 -04:00
Find.hs convert Key to ShortByteString 2021-10-05 20:20:08 -04:00
FindRef.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Fix.hs fix potential race in updating inode cache 2021-07-27 13:08:08 -04:00
Forget.hs start implementing hidden git-annex repositories 2021-04-20 15:04:53 -04:00
FromKey.hs --batch-keys 2021-08-25 14:21:12 -04:00
Fsck.hs add: Detect when xattrs or perhaps ACLs prevent locking down a file's content 2021-08-27 14:33:01 -04:00
FuzzTest.hs finish fixing removeLink on windows 2020-11-24 13:20:44 -04:00
GCryptSetup.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Get.hs --batch-keys 2021-08-25 14:21:12 -04:00
Group.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
GroupWanted.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Help.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Import.hs ImportableContentsChunkable 2021-10-08 13:15:22 -04:00
ImportFeed.hs incremental verification for web special remote 2021-08-18 15:02:22 -04:00
InAnnex.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Indirect.hs Automatically convert direct mode repositories to v7 with adjusted unlocked branches 2019-08-26 15:05:25 -04:00
Info.hs --batch-keys 2021-08-25 14:21:12 -04:00
Init.hs work around strange auto-init bug 2021-07-30 18:36:03 -04:00
InitRemote.hs initremote --private 2021-04-23 14:47:46 -04:00
Inprogress.hs more RawFilePath conversion 2020-11-02 16:31:28 -04:00
List.hs add "input" field to json output 2020-09-15 16:22:44 -04:00
Lock.hs fix a place where the inode cache could potentially have gotten stale 2021-07-26 14:12:58 -04:00
LockContent.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Log.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
LookupKey.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Map.hs avoid making absolute git remote path relative 2021-02-08 13:18:01 -04:00
MatchExpression.hs split ProvidedInfo and UserProvidedInfo 2020-09-28 12:12:38 -04:00
Merge.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
MetaData.hs --batch-keys 2021-08-25 14:21:12 -04:00
Migrate.hs include locked files in the keys database associated files 2021-05-21 16:24:37 -04:00
MinCopies.hs mincopies 2021-01-06 14:15:19 -04:00
Mirror.hs dropping unused marks as dead 2021-06-25 15:22:26 -04:00
Move.hs --batch-keys 2021-08-25 14:21:12 -04:00
Multicast.hs Command/Multicast: use proper hyphen 2021-05-04 05:44:31 +02:00
NotifyChanges.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
NumCopies.hs mincopies 2021-01-06 14:15:19 -04:00
P2P.hs finish fixing removeLink on windows 2020-11-24 13:20:44 -04:00
P2PStdIO.hs finish CommandStart transition 2019-06-12 13:24:01 -04:00
PostReceive.hs remove direct mode remnant of merging unrelated histories 2021-07-19 11:41:26 -04:00
PreCommit.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Proxy.hs Automatically convert direct mode repositories to v7 with adjusted unlocked branches 2019-08-26 15:05:25 -04:00
ReadPresentKey.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
RecvKey.hs make getViaTmpFrom no longer update location log 2020-12-11 11:50:13 -04:00
RegisterUrl.hs --batch-keys 2021-08-25 14:21:12 -04:00
Reinit.hs work around strange auto-init bug 2021-07-30 18:36:03 -04:00
Reinject.hs split verifyKeyContent 2021-07-29 13:58:40 -04:00
ReKey.hs --batch-keys 2021-08-25 14:21:12 -04:00
RemoteDaemon.hs assistant: Fix a crash on startup by avoiding using forkProcess 2021-05-12 15:08:03 -04:00
RenameRemote.hs cache remote.log 2020-09-22 13:52:26 -04:00
Repair.hs finished this stage of the RawFilePath conversion 2020-11-06 14:10:58 -04:00
Required.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
ResolveMerge.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
RmUrl.hs --batch-keys 2021-08-25 14:21:12 -04:00
Schedule.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Semitrust.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
SendKey.hs use fastDebug everywhere it can be used 2021-04-06 15:41:24 -04:00
SetKey.hs annex.adjustedbranchrefresh 2020-11-16 14:27:28 -04:00
SetPresentKey.hs --batch-keys 2021-08-25 14:21:12 -04:00
Smudge.hs reinject: Fix crash when reinjecting a file from outside the repository 2021-10-01 14:06:34 -04:00
Status.hs more RawFilePath conversion 2020-11-02 16:31:28 -04:00
Sync.hs avoid uncessary call to inAnnex 2021-10-01 12:02:35 -04:00
Test.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
TestRemote.hs plumb VerifyConfig into retrieveKeyFile 2021-08-17 12:43:13 -04:00
TransferInfo.hs more RawFilePath conversion 2020-11-02 16:31:28 -04:00
TransferKey.hs plumb VerifyConfig into retrieveKeyFile 2021-08-17 12:43:13 -04:00
TransferKeys.hs plumb VerifyConfig into retrieveKeyFile 2021-08-17 12:43:13 -04:00
Transferrer.hs plumb VerifyConfig into retrieveKeyFile 2021-08-17 12:43:13 -04:00
Trust.hs fix format of message 2021-01-11 13:14:09 -04:00
Unannex.hs remove inode cache in unannex 2021-05-12 11:09:38 -04:00
Undo.hs finished this stage of the RawFilePath conversion 2020-11-04 14:20:37 -04:00
Ungroup.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Uninit.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
Unlock.hs fix potential race in updating inode cache 2021-07-27 13:08:08 -04:00
UnregisterUrl.hs --batch-keys 2021-08-25 14:21:12 -04:00
Untrust.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Unused.hs unused: Skip the refs/annex/last-index ref that git-annex recently started creating 2021-08-24 12:58:14 -04:00
Upgrade.hs work around strange auto-init bug 2021-07-30 18:36:03 -04:00
VAdd.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
VCycle.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Version.hs external backends wip 2020-07-29 15:23:18 -04:00
VFilter.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Vicfg.hs simplify and speed up Utility.FileSystemEncoding 2021-08-11 12:13:31 -04:00
View.hs more RawFilePath conversion 2020-10-30 15:55:59 -04:00
VPop.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Wanted.hs add SeekInput (not yet used) 2020-09-15 15:41:13 -04:00
Watch.hs remotedaemon: Don't list --stop in help since it's not supported. 2019-09-30 14:40:46 -04:00
WebApp.hs finished this stage of the RawFilePath conversion 2020-11-04 14:20:37 -04:00
Whereis.hs --batch-keys 2021-08-25 14:21:12 -04:00
WhereUsed.hs better key matching with a regexp 2021-07-14 16:31:17 -04:00