Commit graph

3590 commits

Author SHA1 Message Date
Joey Hess
711252331e
comment 2021-06-14 14:34:22 -04:00
Joey Hess
398f9decd4
comment 2021-06-14 14:32:38 -04:00
Joey Hess
78da00c7a6
Future proof activity log parsing
When the log has an activity that is not known, eg added by a future
version of git-annex, it used to be treated as no activity at all,
which would make git-annex expire think it should expire the repository,
despite it having some kind of recent activity.

Hopefully there will be no reason to add a new activity until enough
time has passed that this commit is in use everywhere.

Sponsored-by: Jake Vosloo on Patreon
2021-06-14 14:18:19 -04:00
Joey Hess
3ac9363c03
comment 2021-06-14 12:42:11 -04:00
Joey Hess
014dc63a55
avoid sometimes expensive operations when annex.supportunlocked = false
This will mostly just avoid a DB lookup, so things get marginally
faster. But in cases where there are many files using the same key, it
can be a more significant speedup.

Added overhead is one MVar lookup per call, which should be small
enough, since this happens after transferring or ingesting a file,
which is always a lot more work than that. It would be nice, though,
to move getGitConfig to AnnexRead, which there is an open todo about.
2021-06-14 12:40:41 -04:00
Joey Hess
6cb9113ff5
comments 2021-06-08 17:38:56 -04:00
Joey Hess
7b6deb1109
display scanning message whenever reconcileStaged has enough files to chew on
Clear visible progress bar first.

Removed showSideActionAfter because it can't be used in reconcileStaged
(import loop). Instead, it counts the number of files it
processes and displays it after it's seen a sufficient to know it's
taking a while.

Sponsored-by: Dartmouth College's Datalad project
2021-06-08 12:48:30 -04:00
Joey Hess
13b9a288d3
scanAnnexedFiles in smudge --update
This makes git checkout and git merge hooks do the work to catch up with
changes that they made to the tree. Rather than doing it at some later
point when the user is not thinking about that past operation.

Sponsored-by: Dartmouth College's Datalad project
2021-06-08 11:37:47 -04:00
Joey Hess
2467de4f9b
todo 2021-06-07 16:58:35 -04:00
Joey Hess
0f10f208a7
avoid double work in git-annex init
reconcileStaged was doing a redundant scan to scannAnnexedFiles.

It would probably make sense to move the body of scannAnnexedFiles
into reconcileStaged, the separation does not really serve any purpose.

Sponsored-by: Dartmouth College's Datalad project
2021-06-07 16:50:14 -04:00
Joey Hess
6ceb31a30a
optimise reconcileStaged with git cat-file streaming
Commit 428c91606b made it need to do more
work in situations like switching between very different branches.

Compare with seekFilteredKeys which has a similar optimisation. Might be
possible to factor out the common part from these?

Sponsored-by: Dartmouth College's Datalad project
2021-06-07 15:26:48 -04:00
Joey Hess
570e93abfd
comment 2021-06-07 13:28:36 -04:00
Joey Hess
1c35cacf8e
fix link 2021-06-07 13:06:16 -04:00
Joey Hess
b960ebf1b3
Merge branch 'master' of ssh://git-annex.branchable.com 2021-06-07 12:59:21 -04:00
Ilya_Shlyakhter
4d581ad6b4 Added a comment: deferring the keys-to-files scan 2021-06-07 16:11:01 +00:00
Joey Hess
a0bba3afad
comment 2021-06-07 11:49:28 -04:00
Ilya_Shlyakhter
5359f8bc14 added suggestion to match keys by file extension in the key 2021-06-07 15:08:51 +00:00
lucas.gautheron@09f1983993dfb0907d02ba268b3ca672f1dc3eea
b38dc11a37 Added a comment 2021-06-05 10:10:57 +00:00
Ilya_Shlyakhter
d39dfed2a7 Added a comment: "why all these wild ideas are being thrown out there" 2021-06-04 22:15:33 +00:00
Joey Hess
a2c9360905
Merge branch 'master' of ssh://git-annex.branchable.com 2021-06-04 16:45:02 -04:00
Joey Hess
8a13bbedd6
--size-limit exit 101
Sponsored-by: Mark Reidenbach on Patreon
2021-06-04 16:43:47 -04:00
Atemu
ee5f30ee6b 2021-06-04 20:26:27 +00:00
Joey Hess
771a122c9e
add --size-limit option
When this option is not used, there should be effectively no added
overhead, thanks to the optimisation in
b3cd0cc6ba.

When an action fails on a file, the size of the file still counts toward
the size limit. This was necessary to support concurrency, but also
generally seems like the right choice.

Most commands that operate on annexed files support the option.
export and import do not, and I don't know if it would make sense for
export to.. Why would you want an incomplete export? sync doesn't, and
while it would be easy to make it support it for transferring files,
it's not clear if dropping files should also take the size limit into
account. Commands like add that don't operate on annexed files don't
support the option either.

Exiting 101 not yet implemented.

Sponsored-by: Denis Dzyubenko on Patreon
2021-06-04 16:16:53 -04:00
Joey Hess
7868dbd5e0
comment 2021-06-04 13:53:24 -04:00
Joey Hess
327033c2e5
comment 2021-06-04 13:36:51 -04:00
Joey Hess
0434674c85
avoid displaying the scanning annexed files message when repo is not large
Avoids users thinking this scan is a big deal, when it's not in the
majority of repos.

showSideActionAfter has some ugly caveats, since it has to display in
the background of another action. I could not see a better way to do it
and it works fine in this particular case. It also doesn't really belong
in Annex.Concurrent, but cannot go in Messages due to an import loop.

Sponsored-by: Dartmouth College's Datalad project
2021-06-04 13:16:48 -04:00
Joey Hess
95cec1bdfe
comment 2021-06-04 13:14:29 -04:00
yarikoptic
b925ea2923 about "scanning for annexed" while in git-annex branch 2021-06-04 15:20:34 +00:00
Atemu
f70251d638 2021-06-03 13:36:20 +00:00
Ilya_Shlyakhter
de12aeb1a4 Added a comment: matching include/exclude based on file extension in the key 2021-06-02 17:02:58 +00:00
Ilya_Shlyakhter
6a2bfad192 Added a comment: keys db optimization 2021-06-02 16:53:03 +00:00
Joey Hess
6f3f972355
Merge branch 'master' of ssh://git-annex.branchable.com 2021-06-01 11:43:36 -04:00
Joey Hess
3155c0d03e
todo 2021-06-01 10:39:48 -04:00
Ilya_Shlyakhter
a7e8a630fb Added a comment: keys-to-paths db 2021-05-31 23:15:21 +00:00
Joey Hess
f00e365f41
comments 2021-05-31 17:54:17 -04:00
Ilya_Shlyakhter
2dac55978c Added a comment: startup scan for files 2021-05-31 20:50:36 +00:00
Joey Hess
8734f17bc5
comment 2021-05-31 15:15:09 -04:00
Joey Hess
988dbce27a
Merge branch 'master' of ssh://git-annex.branchable.com 2021-05-31 15:05:40 -04:00
Joey Hess
eb6f6ff9b8
speed up keys database writes
There seems to be no reason to check the time here. I think it was
inherited from code in Database.Fsck, which does have a reason to commit
every few minutes. Removing that syscall speeds up a git-annex init
in a repo with 100000 annexed files by about 3 seconds.

Sponsored-by: Dartmouth College's Datalad project
2021-05-31 15:01:00 -04:00
Atemu
6da7f26e2a 2021-05-31 18:59:15 +00:00
Atemu
ae129dc317 2021-05-31 18:42:56 +00:00
Joey Hess
0f54e5e0ae
speed up initial scanning for annexed files
Streaming through git this way speeds it up by around 25%. This is
similar to the optimisations of seeking annexed files.

Sponsored-by: Dartmouth College's Datalad project
2021-05-31 14:29:34 -04:00
Joey Hess
759e5a9903
todo 2021-05-31 10:50:22 -04:00
Joey Hess
3b7f28feca
comment 2021-05-31 10:43:59 -04:00
Joey Hess
57a0ef8d90
comment and reject todo 2021-05-27 12:19:35 -04:00
Atemu
0a0889e72e Added a comment 2021-05-26 07:11:20 +00:00
Joey Hess
13a6bfff49
comments 2021-05-25 16:37:32 -04:00
Joey Hess
f5dc06077d
Merge branch 'master' of ssh://git-annex.branchable.com 2021-05-25 13:10:34 -04:00
Joey Hess
b5f5475ed6
New matching options --excludesamecontent and --includesamecontent
The normalisation of filenames turns out to be the tricky part here,
because the associated files coming out of the keys db may look like
"./foo/bar" or "../bar". For the former to match a glob like "foo/*",
it needs to be normalised.

Note that, on windows, normalise "./foo/bar" = "foo\\bar"
which a glob like "foo/*" won't match. So the glob is matched a second
time, on the toInternalGitPath, so allowing the user to provide a glob
with the slashes in either direction. However, this still won't support
some wacky edge cases like the user providing a glob of "foo/bar\\*"

Sponsored-by: Dartmouth College's Datalad project
2021-05-25 13:08:18 -04:00
Lukey
2ccf525b7f Added a comment 2021-05-25 16:48:26 +00:00