2024-02-29 17:26:06 +00:00
|
|
|
{- git-annex "URL" and "VURL" backends -- keys whose content is
|
|
|
|
- available from urls.
|
2011-08-06 18:57:22 +00:00
|
|
|
-
|
2024-02-29 17:26:06 +00:00
|
|
|
- Copyright 2011-2024 Joey Hess <id@joeyh.name>
|
2011-08-06 18:57:22 +00:00
|
|
|
-
|
2019-03-13 19:48:14 +00:00
|
|
|
- Licensed under the GNU AGPL version 3 or higher.
|
2011-08-06 18:57:22 +00:00
|
|
|
-}
|
|
|
|
|
|
|
|
module Backend.URL (
|
|
|
|
backends,
|
|
|
|
fromUrl
|
|
|
|
) where
|
|
|
|
|
2016-01-20 20:36:33 +00:00
|
|
|
import Annex.Common
|
2017-02-24 19:16:56 +00:00
|
|
|
import Types.Key
|
2011-08-06 18:57:22 +00:00
|
|
|
import Types.Backend
|
Better sanitization of problem characters when generating URL and WORM keys.
FAT has a lot of characters it does not allow in filenames, like ? and *
It's probably the worst offender, but other filesystems also have
limitiations.
In 2011, I made keyFile escape : to handle FAT, but missed the other
characters. It also turns out that when I did that, I was also living
dangerously; any existing keys that contained a : had their object
location change. Oops.
So, adding new characters to escape to keyFile is out. Well, it would be
possible to make keyFile behave differently on a per-filesystem basis, but
this would be a real nightmare to get right. Consider that a rsync special
remote uses keyFile to determine the filenames to use, and we don't know
the underlying filesystem on the rsync server..
Instead, I have gone for a solution that is backwards compatable and
simple. Its only downside is that already generated URL and WORM keys
might not be able to be stored on FAT or some other filesystem that
dislikes a character used in the key. (In this case, the user can just
migrate the problem keys to a checksumming backend. If this became a big
problem, fsck could be made to detect these and suggest a migration.)
Going forward, new keys that are created will escape all characters that
are likely to cause problems. And if some filesystem comes along that's
even worse than FAT (seems unlikely, but here it is 2013, and people are
still using FAT!), additional characters can be added to the set that are
escaped without difficulty.
(Also, made WORM limit the part of the filename that is embedded in the key,
to deal with filesystem filename length limits. This could have already
been a problem, but is more likely now, since the escaping of the filename
can make it longer.)
This commit was sponsored by Ian Downes
2013-10-05 19:01:49 +00:00
|
|
|
import Backend.Utilities
|
add equivilant key log for VURL keys
When downloading a VURL from the web, make sure that the equivilant key
log is populated.
Unfortunately, this does not hash the content while it's being
downloaded from the web. There is not an interface in Backend currently
for incrementally hash generation, only for incremental verification of an
existing hash. So this might add a noticiable delay, and it has to show
a "(checksum...") message. This could stand to be improved.
But, that separate hashing step only has to happen on the first download
of new content from the web. Once the hash is known, the VURL key can have
its hash verified incrementally while downloading except when the
content in the web has changed. (Doesn't happen yet because
verifyKeyContentIncrementally is not implemented yet for VURL keys.)
Note that the equivilant key log file is formatted as a presence log.
This adds a tiny bit of overhead (eg "1 ") per line over just listing the
urls. The reason I chose to use that format is it seems possible that
there will need to be a way to remove an equivilant key at some point in
the future. I don't know why that would be necessary, but it seemed wise
to allow for the possibility.
Downloads of VURL keys from other special remotes that claim urls,
like bittorrent for example, does not popilate the equivilant key log.
So for now, no checksum verification will be done for those.
Sponsored-by: Nicholas Golder-Manning on Patreon
2024-02-29 19:41:57 +00:00
|
|
|
import Logs.EquivilantKeys
|
2011-08-06 18:57:22 +00:00
|
|
|
|
2011-12-31 08:11:39 +00:00
|
|
|
backends :: [Backend]
|
2024-02-29 17:26:06 +00:00
|
|
|
backends = [backendURL, backendVURL]
|
2011-08-06 18:57:22 +00:00
|
|
|
|
2024-02-29 17:26:06 +00:00
|
|
|
backendURL :: Backend
|
|
|
|
backendURL = Backend
|
2017-02-24 19:16:56 +00:00
|
|
|
{ backendVariety = URLKey
|
2020-07-20 18:06:05 +00:00
|
|
|
, genKey = Nothing
|
2015-10-01 17:28:49 +00:00
|
|
|
, verifyKeyContent = Nothing
|
2021-02-09 19:00:51 +00:00
|
|
|
, verifyKeyContentIncrementally = Nothing
|
2012-12-20 19:43:14 +00:00
|
|
|
, canUpgradeKey = Nothing
|
2014-07-10 21:06:04 +00:00
|
|
|
, fastMigrate = Nothing
|
2014-07-27 16:33:46 +00:00
|
|
|
-- The content of an url can change at any time, so URL keys are
|
|
|
|
-- not stable.
|
|
|
|
, isStableKey = const False
|
2024-02-29 20:14:13 +00:00
|
|
|
, isCryptographicallySecure = pure False
|
2012-06-05 23:51:03 +00:00
|
|
|
}
|
2011-08-06 18:57:22 +00:00
|
|
|
|
2024-02-29 17:26:06 +00:00
|
|
|
backendVURL :: Backend
|
|
|
|
backendVURL = Backend
|
|
|
|
{ backendVariety = VURLKey
|
|
|
|
, genKey = Nothing
|
|
|
|
, verifyKeyContent = Nothing -- TODO
|
|
|
|
, verifyKeyContentIncrementally = Nothing -- TODO
|
|
|
|
, canUpgradeKey = Nothing
|
|
|
|
, fastMigrate = Nothing
|
|
|
|
-- Even if a hash is recorded on initial download from the web and
|
|
|
|
-- is used to verify every subsequent transfer including other
|
|
|
|
-- downloads from the web, in a split-brain situation there
|
|
|
|
-- can be more than one hash and different versions of the content.
|
|
|
|
-- So the content is not stable.
|
|
|
|
, isStableKey = const False
|
2024-02-29 20:14:13 +00:00
|
|
|
, isCryptographicallySecure = pure False
|
2024-02-29 17:26:06 +00:00
|
|
|
-- TODO it is when all recorded hashes are
|
|
|
|
}
|
|
|
|
|
Better sanitization of problem characters when generating URL and WORM keys.
FAT has a lot of characters it does not allow in filenames, like ? and *
It's probably the worst offender, but other filesystems also have
limitiations.
In 2011, I made keyFile escape : to handle FAT, but missed the other
characters. It also turns out that when I did that, I was also living
dangerously; any existing keys that contained a : had their object
location change. Oops.
So, adding new characters to escape to keyFile is out. Well, it would be
possible to make keyFile behave differently on a per-filesystem basis, but
this would be a real nightmare to get right. Consider that a rsync special
remote uses keyFile to determine the filenames to use, and we don't know
the underlying filesystem on the rsync server..
Instead, I have gone for a solution that is backwards compatable and
simple. Its only downside is that already generated URL and WORM keys
might not be able to be stored on FAT or some other filesystem that
dislikes a character used in the key. (In this case, the user can just
migrate the problem keys to a checksumming backend. If this became a big
problem, fsck could be made to detect these and suggest a migration.)
Going forward, new keys that are created will escape all characters that
are likely to cause problems. And if some filesystem comes along that's
even worse than FAT (seems unlikely, but here it is 2013, and people are
still using FAT!), additional characters can be added to the set that are
escaped without difficulty.
(Also, made WORM limit the part of the filename that is embedded in the key,
to deal with filesystem filename length limits. This could have already
been a problem, but is more likely now, since the escaping of the filename
can make it longer.)
This commit was sponsored by Ian Downes
2013-10-05 19:01:49 +00:00
|
|
|
{- Every unique url has a corresponding key. -}
|
2024-02-29 17:26:06 +00:00
|
|
|
fromUrl :: String -> Maybe Integer -> Bool -> Key
|
|
|
|
fromUrl url size verifiable = mkKey $ \k -> k
|
2015-01-06 21:58:57 +00:00
|
|
|
{ keyName = genKeyName url
|
2024-02-29 17:26:06 +00:00
|
|
|
, keyVariety = if verifiable then VURLKey else URLKey
|
2015-01-06 21:58:57 +00:00
|
|
|
, keySize = size
|
|
|
|
}
|