git-annex/Annex
Joey Hess 9a2c8757f3
add thirdPartyPopulated interface
This is to support, eg a borg repo as a special remote, which is
populated not by running git-annex commands, but by using borg. Then
git-annex sync lists the content of the remote, learns which files are
annex objects, and treats those as present in the remote.

So, most of the import machinery is reused, to a new purpose. While
normally importtree maintains a remote tracking branch, this does not,
because the files stored in the remote are annex object files, not
user-visible filenames. But, internally, a git tree is still generated,
of the files on the remote that are annex objects. This tree is used
by retrieveExportWithContentIdentifier, etc. As with other import/export
remotes, that  the tree is recorded in the export log, and gets grafted
into the git-annex branch.

importKey changed to be able to return Nothing, to indicate when an
ImportLocation is not an annex object and so should be skipped from
being included in the tree.

It did not seem to make sense to have git-annex import do this, since
from the user's perspective, it's not like other imports. So only
git-annex sync does it.

Note that, git-annex sync does not yet download objects from such
remotes that are preferred content. importKeys is run with
content downloading disabled, to avoid getting the content of all
objects. Perhaps what's needed is for seekSyncContent to be run with these
remotes, but I don't know if it will just work (in particular, it needs
to avoid trying to transfer objects to them), so I skipped that for now.

(Untested and unused as of yet.)

This commit was sponsored by Jochen Bartl on Patreon.
2020-12-18 15:23:58 -04:00
..
AdjustedBranch annex.adjustedbranchrefresh 2020-11-16 14:27:28 -04:00
Branch fix warning 2020-02-17 15:06:28 -04:00
Concurrent differentiate between concurrency enabled at command line and by git config 2020-09-16 11:47:12 -04:00
Content fix build on windows 2020-11-25 06:24:49 -04:00
LockPool Display a message when git-annex has to wait for a pid lock file held by another process 2020-08-26 13:05:34 -04:00
MetaData update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
SpecialRemote remove redundant imports 2020-06-22 11:05:34 -04:00
View fix test failure on windows 2020-11-26 11:48:52 -04:00
Action.hs propagate signals to the transferrer process group 2020-12-11 15:32:00 -04:00
AdjustedBranch.hs propagate signals to the transferrer process group 2020-12-11 15:32:00 -04:00
AutoMerge.hs finish fixing removeLink on windows 2020-11-24 13:20:44 -04:00
BloomFilter.hs Revert "data type that starts off using a set but converts to a bloom filter when large" 2020-07-01 20:12:19 -04:00
Branch.hs finish fixing removeLink on windows 2020-11-24 13:20:44 -04:00
BranchState.hs cache one more log file for metadata 2020-07-07 14:18:55 -04:00
CatFile.hs cat-file resource pool 2020-04-20 15:19:31 -04:00
ChangedRefs.hs more RawFilePath conversion 2020-10-29 14:20:57 -04:00
CheckAttr.hs more RawFilePath conversion 2020-10-28 17:25:59 -04:00
CheckIgnore.hs more RawFilePath conversion 2020-11-03 10:11:04 -04:00
Common.hs finally really add back custom-setup stanza 2017-12-31 16:36:39 -04:00
Concurrent.hs propagate signals to the transferrer process group 2020-12-11 15:32:00 -04:00
Content.hs make getViaTmpFrom no longer update location log 2020-12-11 11:50:13 -04:00
CurrentBranch.hs refactor getCurrentBranch 2018-10-19 17:29:18 -04:00
Difference.hs include git-annex-shell back in 2019-12-02 11:51:52 -04:00
DirHashes.hs Added http special remote, which is useful for accessing other remotes that publish content stored in them via http/https. 2020-09-01 15:16:35 -04:00
Drop.hs more RawFilePath conversion 2020-11-03 18:34:27 -04:00
Environment.hs include git-annex-shell back in 2019-12-02 11:51:52 -04:00
Export.hs update warnExportImportConflict for import-only remotes 2020-12-17 16:25:46 -04:00
ExternalAddonProcess.hs WIP 2020-11-17 17:31:08 -04:00
FileMatcher.hs improve readability and fix a warning 2020-12-14 17:48:30 -04:00
Fixup.hs more RawFilePath conversion 2020-10-29 10:50:29 -04:00
GitOverlay.hs more RawFilePath conversion 2020-10-29 12:03:50 -04:00
HashObject.hs more RawFilePath conversion 2020-10-28 17:25:59 -04:00
Hook.hs don't try to remove pre-commit-annex and post-update-annex-hooks 2020-10-19 13:13:49 -04:00
Import.hs add thirdPartyPopulated interface 2020-12-18 15:23:58 -04:00
Ingest.hs populate new field 2020-12-15 10:37:07 -04:00
Init.hs propagate git-annex -c on to transferrer child process 2020-12-15 11:36:25 -04:00
InodeSentinal.hs finished this stage of the RawFilePath conversion 2020-11-06 14:10:58 -04:00
Journal.hs more RawFilePath conversion 2020-10-29 12:03:50 -04:00
Link.hs more RawFilePath conversion 2020-11-05 18:45:37 -04:00
Locations.hs move old direct mode stuff out of Annex.Locations 2020-11-12 12:40:35 -04:00
LockFile.hs more RawFilePath conversion 2020-10-29 10:50:29 -04:00
LockPool.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Magic.hs Serialize use of C magic library, which is not thread safe. 2020-09-17 17:27:42 -04:00
MetaData.hs Windows: include= and exclude= containing '/' will also match filenames that are written using '\' 2020-12-15 12:39:34 -04:00
Multicast.hs use programPath consistently, not readProgramFile 2020-03-30 16:06:27 -04:00
Notification.hs wip RawFilePath 2x git-annex find speedup 2019-11-26 16:01:58 -04:00
NumCopies.hs more RawFilePath conversion 2020-10-30 15:55:59 -04:00
Path.hs propagate git-annex -c on to transferrer child process 2020-12-15 11:36:25 -04:00
Perms.hs more RawFilePath conversion 2020-11-05 18:45:37 -04:00
PidLock.hs propagate git-annex -c on to transferrer child process 2020-12-15 11:36:25 -04:00
Queue.hs more RawFilePath conversion 2020-10-29 14:20:57 -04:00
RemoteTrackingBranch.hs refactor 2019-11-11 19:10:52 -04:00
ReplaceFile.hs more RawFilePath conversion 2020-10-29 12:03:50 -04:00
SpecialRemote.hs cache remote.log 2020-09-22 13:52:26 -04:00
Ssh.hs propagate signals to the transferrer process group 2020-12-11 15:32:00 -04:00
TaggedPush.hs Ref ByteString conversion done 2020-04-07 17:41:09 -04:00
Tmp.hs propagate signals to the transferrer process group 2020-12-11 15:32:00 -04:00
Transfer.hs make getViaTmpFrom no longer update location log 2020-12-11 11:50:13 -04:00
TransferrerPool.hs simplify 2020-12-17 12:09:49 -04:00
UntrustedFilePath.hs importfeed: Fix reversion that caused some '.' in filenames to be replaced with '_' 2020-08-05 11:35:00 -04:00
UpdateInstead.hs v7 for all repositories 2019-08-30 14:09:14 -04:00
Url.hs Build with the http-client-restricted library when available 2020-06-22 11:31:31 -04:00
UUID.hs Fix a potential failure to parse git config 2020-04-13 13:05:41 -04:00
VariantFile.hs more RawFilePath 2019-12-18 17:10:28 -04:00
VectorClock.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
Version.hs jump right to v8, don't stop part way 2020-02-24 13:21:00 -04:00
View.hs Windows: include= and exclude= containing '/' will also match filenames that are written using '\' 2020-12-15 12:39:34 -04:00
Wanted.hs update licenses from GPL to AGPL 2019-03-13 15:48:14 -04:00
WorkerPool.hs move remoteList into dupState 2020-04-17 14:36:45 -04:00
WorkTree.hs more RawFilePath conversion 2020-10-30 13:07:41 -04:00
YoutubeDl.hs converted reads from stderr to use hGetLineUntilExitOrEOF 2020-11-19 16:21:17 -04:00