mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
git-annex/Limit/Wanted.hs

60 lines
2 KiB
Haskell
Raw Normal View History

add --want-get and --want-drop options New --want-get and --want-drop options which can be used to test preferred content settings. For example, "git annex find --in . --want-drop"
2013-10-28 18:50:17 +00:00
{- git-annex limits by wanted status
-
seek: defer matcher check until more info is known Sped up seeking for files to operate on, when using options like --copies or --in, by around 20%. Benchmark showed an increase for --copies from 155 seconds to 121 seconds, and --in remote will be similar to that. For --in here, the speedup was less, 5-10% or so. (both warm cache) This commit was sponsored by Jack Hill on Patreon.
2020-09-24 21:59:05 +00:00
- Copyright 2012-2020 Joey Hess <id@joeyh.name>
add --want-get and --want-drop options New --want-get and --want-drop options which can be used to test preferred content settings. For example, "git annex find --in . --want-drop"
2013-10-28 18:50:17 +00:00
-
update licenses from GPL to AGPL This does not change the overall license of the git-annex program, which was already AGPL due to a number of sources files being AGPL already. Legally speaking, I'm adding a new license under which these files are now available; I already released their current contents under the GPL license. Now they're dual licensed GPL and AGPL. However, I intend for all my future changes to these files to only be released under the AGPL license, and I won't be tracking the dual licensing status, so I'm simply changing the license statement to say it's AGPL. (In some cases, others wrote parts of the code of a file and released it under the GPL; but in all cases I have contributed a significant portion of the code in each file and it's that code that is getting the AGPL license; the GPL license of other contributors allows combining with AGPL code.)
2019-03-13 19:48:14 +00:00
- Licensed under the GNU AGPL version 3 or higher.
add --want-get and --want-drop options New --want-get and --want-drop options which can be used to test preferred content settings. For example, "git annex find --in . --want-drop"
2013-10-28 18:50:17 +00:00
-}
module Limit.Wanted where
remove 163 lines of code without changing anything except imports
2016-01-20 20:36:33 +00:00
import Annex.Common
add --want-get and --want-drop options New --want-get and --want-drop options which can be used to test preferred content settings. For example, "git annex find --in . --want-drop"
2013-10-28 18:50:17 +00:00
import Annex.Wanted
import Limit
import Types.FileMatcher
seek: defer matcher check until more info is known Sped up seeking for files to operate on, when using options like --copies or --in, by around 20%. Benchmark showed an increase for --copies from 155 seconds to 121 seconds, and --in remote will be similar to that. For --in here, the speedup was less, 5-10% or so. (both warm cache) This commit was sponsored by Jack Hill on Patreon.
2020-09-24 21:59:05 +00:00
import Logs.PreferredContent
new matching options --want-get-by and --want-drop-by Sponsored-by: Graham Spencer on Patreon
2022-07-28 17:26:03 +00:00
import qualified Remote
add --want-get and --want-drop options New --want-get and --want-drop options which can be used to test preferred content settings. For example, "git annex find --in . --want-drop"
2013-10-28 18:50:17 +00:00
addWantGet :: Annex ()
initial implementation of --explain Currently it only displays explanations of options like --in and --copies. In the future, it should explain preferred content expression evaluation and other decisions. The explanations of a few things could be better. In particular, "standard" will just appear as-is (or as "!standard" if it doesn't match), rather than explaining why the standard preferred content expression for the group matches or not. Currently as implemented, it goes to stdout, and so commands like git-annex find that have custom output will not display --explain information. Perhaps that should change, dunno. Sponsored-by: Dartmouth College's DANDI project
2023-07-25 20:11:06 +00:00
addWantGet = addPreferredContentLimit "want-get" $
plumb in LiveUpdate (WIP) Each command that first checks preferred content (and/or required content) and then does something that can change the sizes of repositories needs to call prepareLiveUpdate, and plumb it through the preferred content check and the location log update. So far, only Command.Drop is done. Many other commands that don't need to do this have been updated to keep working. There may be some calls to NoLiveUpdate in places where that should be done. All will need to be double checked. Not currently in a compilable state.
2024-08-23 20:35:12 +00:00
checkWant $ wantGet NoLiveUpdate False Nothing
add --want-get and --want-drop options New --want-get and --want-drop options which can be used to test preferred content settings. For example, "git annex find --in . --want-drop"
2013-10-28 18:50:17 +00:00
new matching options --want-get-by and --want-drop-by Sponsored-by: Graham Spencer on Patreon
2022-07-28 17:26:03 +00:00
addWantGetBy :: String -> Annex ()
addWantGetBy name = do
u <- Remote.nameToUUID name
initial implementation of --explain Currently it only displays explanations of options like --in and --copies. In the future, it should explain preferred content expression evaluation and other decisions. The explanations of a few things could be better. In particular, "standard" will just appear as-is (or as "!standard" if it doesn't match), rather than explaining why the standard preferred content expression for the group matches or not. Currently as implemented, it goes to stdout, and so commands like git-annex find that have custom output will not display --explain information. Perhaps that should change, dunno. Sponsored-by: Dartmouth College's DANDI project
2023-07-25 20:11:06 +00:00
addPreferredContentLimit "want-get-by" $ checkWant $ \af ->
plumb in LiveUpdate (WIP) Each command that first checks preferred content (and/or required content) and then does something that can change the sizes of repositories needs to call prepareLiveUpdate, and plumb it through the preferred content check and the location log update. So far, only Command.Drop is done. Many other commands that don't need to do this have been updated to keep working. There may be some calls to NoLiveUpdate in places where that should be done. All will need to be double checked. Not currently in a compilable state.
2024-08-23 20:35:12 +00:00
wantGetBy NoLiveUpdate False Nothing af u
new matching options --want-get-by and --want-drop-by Sponsored-by: Graham Spencer on Patreon
2022-07-28 17:26:03 +00:00
add --want-get and --want-drop options New --want-get and --want-drop options which can be used to test preferred content settings. For example, "git annex find --in . --want-drop"
2013-10-28 18:50:17 +00:00
addWantDrop :: Annex ()
initial implementation of --explain Currently it only displays explanations of options like --in and --copies. In the future, it should explain preferred content expression evaluation and other decisions. The explanations of a few things could be better. In particular, "standard" will just appear as-is (or as "!standard" if it doesn't match), rather than explaining why the standard preferred content expression for the group matches or not. Currently as implemented, it goes to stdout, and so commands like git-annex find that have custom output will not display --explain information. Perhaps that should change, dunno. Sponsored-by: Dartmouth College's DANDI project
2023-07-25 20:11:06 +00:00
addWantDrop = addPreferredContentLimit "want-drop" $ checkWant $ \af ->
plumb in LiveUpdate (WIP) Each command that first checks preferred content (and/or required content) and then does something that can change the sizes of repositories needs to call prepareLiveUpdate, and plumb it through the preferred content check and the location log update. So far, only Command.Drop is done. Many other commands that don't need to do this have been updated to keep working. There may be some calls to NoLiveUpdate in places where that should be done. All will need to be double checked. Not currently in a compilable state.
2024-08-23 20:35:12 +00:00
wantDrop NoLiveUpdate False Nothing Nothing af (Just [])
new matching options --want-get-by and --want-drop-by Sponsored-by: Graham Spencer on Patreon
2022-07-28 17:26:03 +00:00
addWantDropBy :: String -> Annex ()
addWantDropBy name = do
u <- Remote.nameToUUID name
initial implementation of --explain Currently it only displays explanations of options like --in and --copies. In the future, it should explain preferred content expression evaluation and other decisions. The explanations of a few things could be better. In particular, "standard" will just appear as-is (or as "!standard" if it doesn't match), rather than explaining why the standard preferred content expression for the group matches or not. Currently as implemented, it goes to stdout, and so commands like git-annex find that have custom output will not display --explain information. Perhaps that should change, dunno. Sponsored-by: Dartmouth College's DANDI project
2023-07-25 20:11:06 +00:00
addPreferredContentLimit "want-drop-by" $ checkWant $ \af ->
plumb in LiveUpdate (WIP) Each command that first checks preferred content (and/or required content) and then does something that can change the sizes of repositories needs to call prepareLiveUpdate, and plumb it through the preferred content check and the location log update. So far, only Command.Drop is done. Many other commands that don't need to do this have been updated to keep working. There may be some calls to NoLiveUpdate in places where that should be done. All will need to be double checked. Not currently in a compilable state.
2024-08-23 20:35:12 +00:00
wantDrop NoLiveUpdate False (Just u) Nothing af (Just [])
seek: defer matcher check until more info is known Sped up seeking for files to operate on, when using options like --copies or --in, by around 20%. Benchmark showed an increase for --copies from 155 seconds to 121 seconds, and --in remote will be similar to that. For --in here, the speedup was less, 5-10% or so. (both warm cache) This commit was sponsored by Jack Hill on Patreon.
2020-09-24 21:59:05 +00:00
initial implementation of --explain Currently it only displays explanations of options like --in and --copies. In the future, it should explain preferred content expression evaluation and other decisions. The explanations of a few things could be better. In particular, "standard" will just appear as-is (or as "!standard" if it doesn't match), rather than explaining why the standard preferred content expression for the group matches or not. Currently as implemented, it goes to stdout, and so commands like git-annex find that have custom output will not display --explain information. Perhaps that should change, dunno. Sponsored-by: Dartmouth College's DANDI project
2023-07-25 20:11:06 +00:00
addPreferredContentLimit :: String -> (MatchInfo -> Annex Bool) -> Annex ()
addPreferredContentLimit desc a = do
seek: defer matcher check until more info is known Sped up seeking for files to operate on, when using options like --copies or --in, by around 20%. Benchmark showed an increase for --copies from 155 seconds to 121 seconds, and --in remote will be similar to that. For --in here, the speedup was less, 5-10% or so. (both warm cache) This commit was sponsored by Jack Hill on Patreon.
2020-09-24 21:59:05 +00:00
nfn <- introspectPreferredRequiredContent matchNeedsFileName Nothing
nfc <- introspectPreferredRequiredContent matchNeedsFileContent Nothing
nk <- introspectPreferredRequiredContent matchNeedsKey Nothing
fix some compile warnings left in yesterday at least 2 could have caused a crash in some circumstances This commit was sponsored by Brett Eisenberg on Patreon.
2020-09-25 14:55:03 +00:00
nl <- introspectPreferredRequiredContent matchNeedsLocationLog Nothing
avoid reposize database locking overhead when not needed Only when the preferred content expression being matched uses balanced preferred content is this overhead needed. It might be possible to eliminate the locking entirely. Eg, check the live changes before and after the action and re-run if they are not stable. For now, this is good enough, it avoids existing preferred content getting slow. If balanced preferred content turns out to be too slow to check, that could be tried later.
2024-08-28 14:52:34 +00:00
lsz <- introspectPreferredRequiredContent matchNeedsLiveRepoSize Nothing
treat "not present" in preferred content as invalid Detect when a preferred content expression contains "not present", which would lead to repeatedly getting and then dropping files, and make it never match. This also applies to "not balanced" and "not sizebalanced". --explain will tell the user when this happens Note that getMatcher calls matchMrun' and does not check for unstable negated limits. While there is no --present anyway, if there was, it would not make sense for --not --present to complain about instability and fail to match.
2024-09-03 17:49:34 +00:00
nu <- introspectPreferredRequiredContent matchNegationUnstable Nothing
seek: defer matcher check until more info is known Sped up seeking for files to operate on, when using options like --copies or --in, by around 20%. Benchmark showed an increase for --copies from 155 seconds to 121 seconds, and --in remote will be similar to that. For --in here, the speedup was less, 5-10% or so. (both warm cache) This commit was sponsored by Jack Hill on Patreon.
2020-09-24 21:59:05 +00:00
addLimit $ Right $ MatchFiles
plumb in LiveUpdate (WIP) Each command that first checks preferred content (and/or required content) and then does something that can change the sizes of repositories needs to call prepareLiveUpdate, and plumb it through the preferred content check and the location log update. So far, only Command.Drop is done. Many other commands that don't need to do this have been updated to keep working. There may be some calls to NoLiveUpdate in places where that should be done. All will need to be double checked. Not currently in a compilable state.
2024-08-23 20:35:12 +00:00
{ matchAction = const $ const a
seek: defer matcher check until more info is known Sped up seeking for files to operate on, when using options like --copies or --in, by around 20%. Benchmark showed an increase for --copies from 155 seconds to 121 seconds, and --in remote will be similar to that. For --in here, the speedup was less, 5-10% or so. (both warm cache) This commit was sponsored by Jack Hill on Patreon.
2020-09-24 21:59:05 +00:00
, matchNeedsFileName = nfn
, matchNeedsFileContent = nfc
, matchNeedsKey = nk
fix some compile warnings left in yesterday at least 2 could have caused a crash in some circumstances This commit was sponsored by Brett Eisenberg on Patreon.
2020-09-25 14:55:03 +00:00
, matchNeedsLocationLog = nl
avoid reposize database locking overhead when not needed Only when the preferred content expression being matched uses balanced preferred content is this overhead needed. It might be possible to eliminate the locking entirely. Eg, check the live changes before and after the action and re-run if they are not stable. For now, this is good enough, it avoids existing preferred content getting slow. If balanced preferred content turns out to be too slow to check, that could be tried later.
2024-08-28 14:52:34 +00:00
, matchNeedsLiveRepoSize = lsz
treat "not present" in preferred content as invalid Detect when a preferred content expression contains "not present", which would lead to repeatedly getting and then dropping files, and make it never match. This also applies to "not balanced" and "not sizebalanced". --explain will tell the user when this happens Note that getMatcher calls matchMrun' and does not check for unstable negated limits. While there is no --present anyway, if there was, it would not make sense for --not --present to complain about instability and fail to match.
2024-09-03 17:49:34 +00:00
, matchNegationUnstable = nu
initial implementation of --explain Currently it only displays explanations of options like --in and --copies. In the future, it should explain preferred content expression evaluation and other decisions. The explanations of a few things could be better. In particular, "standard" will just appear as-is (or as "!standard" if it doesn't match), rather than explaining why the standard preferred content expression for the group matches or not. Currently as implemented, it goes to stdout, and so commands like git-annex find that have custom output will not display --explain information. Perhaps that should change, dunno. Sponsored-by: Dartmouth College's DANDI project
2023-07-25 20:11:06 +00:00
, matchDesc = matchDescSimple desc
seek: defer matcher check until more info is known Sped up seeking for files to operate on, when using options like --copies or --in, by around 20%. Benchmark showed an increase for --copies from 155 seconds to 121 seconds, and --in remote will be similar to that. For --in here, the speedup was less, 5-10% or so. (both warm cache) This commit was sponsored by Jack Hill on Patreon.
2020-09-24 21:59:05 +00:00
}
improve matcher data type to allow matching Keys, instead of just files (no behavior changes)
2014-01-18 18:51:55 +00:00
AssociatedFile newtype To prevent any further mistakes like 301aff34c42d896038c42e0a0bc7b1cf71e0ad22 This commit was sponsored by Francois Marier on Patreon.
2017-03-10 17:12:24 +00:00
checkWant :: (AssociatedFile -> Annex Bool) -> MatchInfo -> Annex Bool
convert TopFilePath to use RawFilePath Adds a dependency on filepath-bytestring, an as yet unreleased fork of filepath that operates on RawFilePath. Git.Repo also changed to use RawFilePath for the path to the repo. This does eliminate some RawFilePath -> FilePath -> RawFilePath conversions. And filepath-bytestring's </> is probably faster. But I don't expect a major performance improvement from this. This is mostly groundwork for making Annex.Location use RawFilePath, which will allow for a conversion-free pipleline.
2019-12-09 17:49:05 +00:00
checkWant a (MatchingFile fi) = a (AssociatedFile (Just $ matchFile fi))
prep for fixing find --branch --unlocked Added LinkType to ProvidedInfo, and unified MatchingKey with ProvidedInfo. They're both used in the same way, so there was no real reason to keep separate. Note that addLocked and addUnlocked still set matchNeedsFileName, because to handle MatchingFile, they do need it. However, they don't use it when MatchingInfo is provided. This should be ok, the --branch case will be able skip checking matchNeedsFileName, since it will provide a filename in any case.
2021-03-02 16:47:23 +00:00
checkWant a (MatchingInfo p) = a (AssociatedFile (providedFilePath p))
split ProvidedInfo and UserProvidedInfo The latter is for git-annex matchexpression and matching against it can throw an exception. Splitting out the former reduces the potential for mistakes and avoids needing to worry about matching against that throwing an exception. This is more groundwork for matching largefiles while importing, without downloading content. This commit was sponsored by Graham Spencer on Patreon.
2020-09-28 16:06:10 +00:00
checkWant _ (MatchingUserInfo {}) = return False
Reference in a new issue Copy permalink