2010-12-31 17:39:30 +00:00
|
|
|
{- git-annex command
|
|
|
|
|
-
|
2015-01-21 16:50:09 +00:00
|
|
|
- Copyright 2010 Joey Hess <id@joeyh.name>
|
2010-12-31 17:39:30 +00:00
|
|
|
-
|
2019-03-13 19:48:14 +00:00
|
|
|
- Licensed under the GNU AGPL version 3 or higher.
|
2010-12-31 17:39:30 +00:00
|
|
|
-}
|
|
|
|
|
|
|
|
|
|
module Command.RecvKey where
|
|
|
|
|
|
|
|
|
|
import Command
|
2011-10-04 04:40:47 +00:00
|
|
|
import Annex.Content
|
2015-07-31 20:00:13 +00:00
|
|
|
import Annex.Action
|
2013-03-29 00:34:07 +00:00
|
|
|
import Annex
|
2012-09-19 18:28:32 +00:00
|
|
|
import Utility.Rsync
|
2016-08-03 16:37:12 +00:00
|
|
|
import Types.Transfer
|
2020-12-11 15:33:10 +00:00
|
|
|
import Logs.Location
|
2012-08-23 19:22:23 +00:00
|
|
|
import Command.SendKey (fieldTransfer)
|
2010-12-31 17:39:30 +00:00
|
|
|
|
2015-07-08 16:33:27 +00:00
|
|
|
cmd :: Command
|
2015-07-08 19:08:02 +00:00
|
|
|
cmd = noCommit $ command "recvkey" SectionPlumbing
|
|
|
|
|
"runs rsync in server mode to receive content"
|
|
|
|
|
paramKey (withParams seek)
|
2010-12-31 17:39:30 +00:00
|
|
|
|
2015-07-08 16:33:27 +00:00
|
|
|
seek :: CmdParams -> CommandSeek
|
2018-10-01 18:12:06 +00:00
|
|
|
seek = withKeys (commandAction . start)
|
2010-12-31 17:39:30 +00:00
|
|
|
|
2020-09-14 20:49:33 +00:00
|
|
|
start :: (SeekInput, Key) -> CommandStart
|
|
|
|
|
start (_, key) = fieldTransfer Download key $ \_p -> do
|
2018-06-21 17:34:11 +00:00
|
|
|
-- This matches the retrievalSecurityPolicy of Remote.Git
|
|
|
|
|
let rsp = RetrievalAllKeysSecure
|
disk free checking for unsized keys
Improve disk free space checking when transferring unsized keys to
local git remotes. Since the size of the object file is known, can
check that instead.
Getting unsized keys from local git remotes does not check the actual
object size. It would be harder to handle that direction because the size
check is run locally, before anything involving the remote is done. So it
doesn't know the size of the file on the remote.
Also, transferring unsized keys to other remotes, including ssh remotes and
p2p remotes don't do disk size checking for unsized keys. This would need a
change in protocol.
(It does seem like it would be possible to implement the same thing for
directory special remotes though.)
In some sense, it might be better to not ever do disk free checking for
unsized keys, than to do it only sometimes. A user might notice this
direction working and consider it a bug that the other direction does not.
On the other hand, disk reserve checking is not implemented for most
special remotes at all, and yet it is implemented for a few, which is also
inconsistent, but best effort. And so doing this best effort seems to make
some sense. Fundamentally, if the user wants the size to always be checked,
they should not use unsized keys.
Sponsored-by: Brock Spratlen on Patreon
2024-01-16 18:29:10 +00:00
|
|
|
ifM (getViaTmp rsp DefaultVerify key (AssociatedFile Nothing) Nothing go)
|
2014-08-04 13:16:47 +00:00
|
|
|
( do
|
2020-12-11 15:33:10 +00:00
|
|
|
logStatus key InfoPresent
|
avoid flushing keys db queue after each Annex action
The flush was only done Annex.run' to make sure that the queue was flushed
before git-annex exits. But, doing it there means that as soon as one
change gets queued, it gets flushed soon after, which contributes to
excessive writes to the database, slowing git-annex down.
(This does not yet speed git-annex up, but it is a stepping stone to
doing so.)
Database queues do not autoflush when garbage collected, so have to
be flushed explicitly. I don't think it's possible to make them
autoflush (except perhaps if git-annex sqitched to using ResourceT..).
The comment in Database.Keys.closeDb used to be accurate, since the
automatic flushing did mean that all writes reached the database even
when closeDb was not called. But now, closeDb or flushDb needs to be
called before stopping using an Annex state. So, removed that comment.
In Remote.Git, change to using quiesce everywhere that it used to use
stopCoProcesses. This means that uses on onLocal in there are just as
slow as before. I considered only calling closeDb on the local git remotes
when git-annex exits. But, the reason that Remote.Git calls stopCoProcesses
in each onLocal is so as not to leave git processes running that have files
open on the remote repo, when it's on removable media. So, it seemed to make
sense to also closeDb after each one, since sqlite may also keep files
open. Although that has not seemed to cause problems with removable
media so far. It was also just easier to quiesce in each onLocal than
once at the end. This does likely leave performance on the floor, so
could be revisited.
In Annex.Content.saveState, there was no reason to close the db,
flushing it is enough.
The rest of the changes are from auditing for Annex.new, and making
sure that quiesce is called, after any action that might possibly need
it.
After that audit, I'm pretty sure that the change to Annex.run' is
safe. The only concern might be that this does let more changes get
queued for write to the db, and if git-annex is interrupted, those will be
lost. But interrupting git-annex can obviously already prevent it from
writing the most recent change to the db, so it must recover from such
lost data... right?
Sponsored-by: Dartmouth College's Datalad project
2022-10-12 17:50:46 +00:00
|
|
|
_ <- quiesce True
|
2014-08-04 13:16:47 +00:00
|
|
|
return True
|
|
|
|
|
, return False
|
|
|
|
|
)
|
2013-01-11 19:43:09 +00:00
|
|
|
where
|
other 80% of avoding verification when hard linking to objects in shared repo
In c6632ee5c8e66c26ef18317f56ae02bae1e7e280, it actually only handled
uploading objects to a shared repository. To avoid verification when
downloading objects from a shared repository, was a lot harder.
On the plus side, if the process of downloading a file from a remote
is able to verify its content on the side, the remote can indicate this
now, and avoid the extra post-download verification.
As of yet, I don't have any remotes (except Git) using this ability.
Some more work would be needed to support it in special remotes.
It would make sense for tahoe to implicitly verify things downloaded from it;
as long as you trust your tahoe server (which typically runs locally),
there's cryptographic integrity. OTOH, despite bup being based on shas,
a bup repo under an attacker's control could have the git ref used for an
object changed, and so a bup repo shouldn't implicitly verify. Indeed,
tahoe seems unique in being trustworthy enough to implicitly verify.
2015-10-02 17:56:42 +00:00
|
|
|
go tmp = unVerified $ do
|
2013-03-30 23:05:51 +00:00
|
|
|
opts <- filterRsyncSafeOptions . maybe [] words
|
|
|
|
|
<$> getField "RsyncOptions"
|
2020-11-02 20:31:28 +00:00
|
|
|
liftIO $ rsyncServerReceive (map Param opts) (fromRawFilePath tmp)
|