git-annex/git-annex.cabal

1091 lines
26 KiB
Text
Raw Normal View History

Name: git-annex
Version: 10.20230926
Cabal-Version: 1.12
License: AGPL-3
Maintainer: Joey Hess <id@joeyh.name>
Author: Joey Hess
Stability: Stable
Copyright: 2010-2023 Joey Hess
2012-09-24 19:20:28 +00:00
License-File: COPYRIGHT
Homepage: http://git-annex.branchable.com/
Build-type: Custom
Category: Utility
Synopsis: manage files with git, without checking their contents into git
Description:
git-annex allows managing files with git, without checking the file
contents into git. While that may seem paradoxical, it is useful when
dealing with files larger than git can currently easily handle, whether due
2012-02-15 23:43:15 +00:00
to limitations in memory, time, or disk space.
.
It can store large files in many places, from local hard drives, to a
large number of cloud storage services, including S3, WebDAV,
and rsync, and many other usable via plugins.
Files can be stored encrypted with gpg, so that the cloud storage
provider cannot see your data. git-annex keeps track of where each file
is stored, so it knows how many copies are available, and has many
facilities to ensure your data is preserved.
.
git-annex can also be used to keep a folder in sync between computers,
noticing when files are changed, and automatically committing them
to git and transferring them to other computers. The git-annex webapp
makes it easy to set up and use git-annex this way.
-- The tarball uploaded to hackage does not include every non-haskell
-- file in the git repo. The website is left out, as are man pages,
-- so is build machinery for standalone apps, and packages.
-- Include only files that are needed make cabal install git-annex work.
Extra-Source-Files:
stack.yaml
stack-lts-18.13.yaml
README
CHANGELOG
NEWS
doc/license/GPL
doc/license/AGPL
doc/logo.svg
doc/logo_16x16.png
Assistant/WebApp/routes
static/activityicon.gif
static/css/bootstrap.css
static/css/bootstrap-theme.css
static/js/jquery.ui.core.js
static/js/longpolling.js
static/js/jquery.full.js
static/js/jquery.ui.sortable.js
static/js/jquery.ui.mouse.js
static/js/jquery.ui.widget.js
static/js/bootstrap.js
static/syncicon.gif
static/favicon.ico
static/fonts/glyphicons-halflings-regular.woff
static/fonts/glyphicons-halflings-regular.eot
static/fonts/glyphicons-halflings-regular.svg
static/fonts/glyphicons-halflings-regular.ttf
templates/sidebar/main.hamlet
templates/sidebar/alert.hamlet
templates/bootstrap.hamlet
templates/error.cassius
templates/README
templates/error.hamlet
templates/documentation/license.hamlet
templates/documentation/repogroup.hamlet
templates/documentation/about.hamlet
templates/dashboard/main.hamlet
templates/dashboard/transfers.cassius
templates/dashboard/transfers.hamlet
templates/dashboard/metarefresh.hamlet
templates/page.cassius
templates/page.hamlet
templates/control/repairrepository.hamlet
templates/control/repairrepository/done.hamlet
templates/control/notrunning.julius
templates/control/notrunning.hamlet
templates/control/repositoryswitcher.hamlet
templates/control/shutdown.hamlet
templates/control/log.hamlet
templates/page.julius
templates/repolist.julius
templates/configurators/adddrive/combine.hamlet
templates/configurators/adddrive/setupmodal.hamlet
templates/configurators/adddrive/encrypt.hamlet
templates/configurators/newrepository.hamlet
templates/configurators/needglaciercli.hamlet
templates/configurators/adds3.hamlet
templates/configurators/genkeymodal.hamlet
templates/configurators/main.hamlet
templates/configurators/needconnection.hamlet
templates/configurators/newrepository/form.hamlet
templates/configurators/newrepository/first.hamlet
templates/configurators/newrepository/combine.hamlet
templates/configurators/enablewebdav.hamlet
templates/configurators/pairing/local/inprogress.hamlet
templates/configurators/pairing/local/prompt.hamlet
templates/configurators/pairing/wormhole/prompt.hamlet
templates/configurators/pairing/wormhole/start.hamlet
templates/configurators/pairing/disabled.hamlet
templates/configurators/addglacier.hamlet
templates/configurators/fsck.cassius
templates/configurators/edit/nonannexremote.hamlet
templates/configurators/edit/webrepository.hamlet
templates/configurators/edit/repository.hamlet
templates/configurators/unused.hamlet
templates/configurators/ssh/testmodal.hamlet
templates/configurators/ssh/expiredpassword.hamlet
templates/configurators/ssh/error.hamlet
templates/configurators/ssh/combine.hamlet
templates/configurators/ssh/enable.hamlet
templates/configurators/ssh/add.hamlet
templates/configurators/ssh/setupmodal.hamlet
templates/configurators/ssh/confirm.hamlet
templates/configurators/enableia.hamlet
templates/configurators/fsck.hamlet
templates/configurators/addrepository/archive.hamlet
templates/configurators/addrepository/cloud.hamlet
templates/configurators/addrepository/connection.hamlet
templates/configurators/addrepository/ssh.hamlet
templates/configurators/addrepository/misc.hamlet
templates/configurators/addrepository/wormholepairing.hamlet
templates/configurators/rsync.net/add.hamlet
templates/configurators/rsync.net/encrypt.hamlet
templates/configurators/needgcrypt.hamlet
templates/configurators/needtor.hamlet
templates/configurators/needmagicwormhole.hamlet
templates/configurators/enabledirectory.hamlet
templates/configurators/fsck/status.hamlet
templates/configurators/fsck/form.hamlet
templates/configurators/fsck/preferencesform.hamlet
templates/configurators/fsck/formcontent.hamlet
templates/configurators/delete/finished.hamlet
templates/configurators/delete/start.hamlet
templates/configurators/delete/currentrepository.hamlet
templates/configurators/unused/form.hamlet
templates/configurators/adddrive.hamlet
templates/configurators/preferences.hamlet
templates/configurators/addia.hamlet
templates/configurators/enableaws.hamlet
templates/configurators/addrepository.hamlet
templates/actionbutton.hamlet
templates/repolist.hamlet
templates/controlmenu.hamlet
templates/notifications/longpolling.julius
Utility/libkqueue.h
Flag Assistant
Description: Enable git-annex assistant, webapp, and watch command
Default: True
2012-09-07 23:44:20 +00:00
Flag Pairing
Description: Enable pairing
2012-09-07 18:54:00 +00:00
2013-02-27 05:41:01 +00:00
Flag Production
Description: Enable production build (slower build; faster binary)
Flag ParallelBuild
Description: Enable production build (slower build; faster binary)
Default: False
Flag TorrentParser
Description: Use haskell torrent library to parse torrent files
Flag MagicMime
Description: Use libmagic to determine file MIME types
Flag Crypton
Description: Use the crypton library rather than the no longer maintained cryptonite
add database benchmark The benchmark shows that the database access is quite fast indeed! And, it scales linearly to the number of keys, with one exception, getAssociatedKey. Based on this benchmark, I don't think I need worry about optimising for cases where all files are locked and the database is mostly empty. In those cases, database access will be misses, and according to this benchmark, should add only 50 milliseconds to runtime. (NB: There may be some overhead to getting the database opened and locking the handle that this benchmark doesn't see.) joey@darkstar:~/src/git-annex>./git-annex benchmark setting up database with 1000 setting up database with 10000 benchmarking keys database/getAssociatedFiles from 1000 (hit) time 62.77 μs (62.70 μs .. 62.85 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 62.81 μs (62.76 μs .. 62.88 μs) std dev 201.6 ns (157.5 ns .. 259.5 ns) benchmarking keys database/getAssociatedFiles from 1000 (miss) time 50.02 μs (49.97 μs .. 50.07 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.09 μs (50.04 μs .. 50.17 μs) std dev 206.7 ns (133.8 ns .. 295.3 ns) benchmarking keys database/getAssociatedKey from 1000 (hit) time 211.2 μs (210.5 μs .. 212.3 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 211.0 μs (210.7 μs .. 212.0 μs) std dev 1.685 μs (334.4 ns .. 3.517 μs) benchmarking keys database/getAssociatedKey from 1000 (miss) time 173.5 μs (172.7 μs .. 174.2 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 173.7 μs (173.0 μs .. 175.5 μs) std dev 3.833 μs (1.858 μs .. 6.617 μs) variance introduced by outliers: 16% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (hit) time 64.01 μs (63.84 μs .. 64.18 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 64.85 μs (64.34 μs .. 66.02 μs) std dev 2.433 μs (547.6 ns .. 4.652 μs) variance introduced by outliers: 40% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (miss) time 50.33 μs (50.28 μs .. 50.39 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.32 μs (50.26 μs .. 50.38 μs) std dev 202.7 ns (167.6 ns .. 252.0 ns) benchmarking keys database/getAssociatedKey from 10000 (hit) time 1.142 ms (1.139 ms .. 1.146 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.142 ms (1.140 ms .. 1.144 ms) std dev 7.142 μs (4.994 μs .. 10.98 μs) benchmarking keys database/getAssociatedKey from 10000 (miss) time 1.094 ms (1.092 ms .. 1.096 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.095 ms (1.095 ms .. 1.097 ms) std dev 4.277 μs (2.591 μs .. 7.228 μs)
2016-01-12 17:01:44 +00:00
Flag Benchmark
Description: Enable benchmarking
Default: True
add database benchmark The benchmark shows that the database access is quite fast indeed! And, it scales linearly to the number of keys, with one exception, getAssociatedKey. Based on this benchmark, I don't think I need worry about optimising for cases where all files are locked and the database is mostly empty. In those cases, database access will be misses, and according to this benchmark, should add only 50 milliseconds to runtime. (NB: There may be some overhead to getting the database opened and locking the handle that this benchmark doesn't see.) joey@darkstar:~/src/git-annex>./git-annex benchmark setting up database with 1000 setting up database with 10000 benchmarking keys database/getAssociatedFiles from 1000 (hit) time 62.77 μs (62.70 μs .. 62.85 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 62.81 μs (62.76 μs .. 62.88 μs) std dev 201.6 ns (157.5 ns .. 259.5 ns) benchmarking keys database/getAssociatedFiles from 1000 (miss) time 50.02 μs (49.97 μs .. 50.07 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.09 μs (50.04 μs .. 50.17 μs) std dev 206.7 ns (133.8 ns .. 295.3 ns) benchmarking keys database/getAssociatedKey from 1000 (hit) time 211.2 μs (210.5 μs .. 212.3 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 211.0 μs (210.7 μs .. 212.0 μs) std dev 1.685 μs (334.4 ns .. 3.517 μs) benchmarking keys database/getAssociatedKey from 1000 (miss) time 173.5 μs (172.7 μs .. 174.2 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 173.7 μs (173.0 μs .. 175.5 μs) std dev 3.833 μs (1.858 μs .. 6.617 μs) variance introduced by outliers: 16% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (hit) time 64.01 μs (63.84 μs .. 64.18 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 64.85 μs (64.34 μs .. 66.02 μs) std dev 2.433 μs (547.6 ns .. 4.652 μs) variance introduced by outliers: 40% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (miss) time 50.33 μs (50.28 μs .. 50.39 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.32 μs (50.26 μs .. 50.38 μs) std dev 202.7 ns (167.6 ns .. 252.0 ns) benchmarking keys database/getAssociatedKey from 10000 (hit) time 1.142 ms (1.139 ms .. 1.146 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.142 ms (1.140 ms .. 1.144 ms) std dev 7.142 μs (4.994 μs .. 10.98 μs) benchmarking keys database/getAssociatedKey from 10000 (miss) time 1.094 ms (1.092 ms .. 1.096 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.095 ms (1.095 ms .. 1.097 ms) std dev 4.277 μs (2.591 μs .. 7.228 μs)
2016-01-12 17:01:44 +00:00
Flag DebugLocks
Description: Debug location of MVar/STM deadlocks
Default: False
Flag Dbus
Description: Enable dbus support
source-repository head
type: git
location: git://git-annex.branchable.com/
custom-setup
2023-08-01 19:47:05 +00:00
Setup-Depends:
base (>= 4.11.1.0 && < 5.0),
split,
filepath,
exceptions,
bytestring,
filepath-bytestring (>= 1.4.2.1.4),
process (>= 1.6.3),
time (>= 1.5.0),
directory (>= 1.2.7.0),
2023-08-01 19:47:05 +00:00
async,
utf8-string,
2023-08-01 19:47:05 +00:00
Cabal (< 4.0)
Executable git-annex
Main-Is: git-annex.hs
2015-05-10 21:03:20 +00:00
Build-Depends:
base (>= 4.11.1.0 && < 5.0),
network-uri (>= 2.6),
optparse-applicative (>= 0.14.2),
containers (>= 0.5.8),
2015-05-10 21:03:20 +00:00
exceptions (>= 0.6),
stm (>= 2.3),
mtl (>= 2),
uuid (>= 1.2.6),
process (>= 1.6.3),
data-default,
case-insensitive,
random,
dlist,
unix-compat (>= 0.5 && < 0.8),
SafeSemaphore,
async,
directory (>= 1.2.7.0),
disk-free-space,
filepath,
filepath-bytestring (>= 1.4.2.1.1),
IfElse,
monad-logger (>= 0.3.10),
free,
utf8-string,
bytestring,
text,
sandi,
monad-control,
transformers,
bloomfilter (>= 2.0.0),
edit-distance,
resourcet,
http-client (>= 0.5.3),
http-client-tls,
http-types (>= 0.7),
http-conduit (>= 2.3.0),
http-client-restricted (>= 0.0.2),
conduit,
time (>= 1.5.0),
old-locale,
persistent-sqlite (>= 2.8.1),
persistent (>= 2.8.1),
persistent-template,
unliftio-core,
microlens,
aeson,
vector,
tagsoup,
unordered-containers,
feed (>= 1.0.0),
regex-tdfa,
socks,
byteable,
stm-chans,
securemem,
crypto-api,
memory,
deepseq,
split,
attoparsec (>= 0.13.2.2),
concurrent-output (>= 1.10),
QuickCheck (>= 2.10.0),
tasty (>= 1.2),
tasty-hunit,
tasty-quickcheck,
tasty-rerun,
ansi-terminal >= 0.9,
aws (>= 0.20),
DAV (>= 1.0),
network (>= 3.0.0.0),
network-bsd,
git-lfs (>= 1.2.0)
CC-Options: -Wall
GHC-Options: -Wall -fno-warn-tabs -Wincomplete-uni-patterns
Default-Language: Haskell2010
Default-Extensions: LambdaCase
Other-Extensions: TemplateHaskell
-- Some things don't work with the non-threaded RTS.
GHC-Options: -threaded
2013-02-27 05:41:01 +00:00
-- Fully optimize for production.
if flag(Production)
-- Lower memory systems can run out of memory with -O2, so
-- optimise slightly less.
if arch(arm)
GHC-Options: -O2 -optlo-O2
else
GHC-Options: -O2
else
GHC-Options: -O0
if flag(ParallelBuild)
GHC-Options: -j
2023-03-14 02:39:16 +00:00
-- Avoid linking with unused dynamic libraries.
if os(linux) || os(freebsd)
GHC-Options: -optl-Wl,--as-needed
if flag(Crypton)
Build-Depends: crypton
CPP-Options: -DWITH_CRYPTON
else
Build-Depends: cryptonite (>= 0.23)
2013-12-10 05:18:04 +00:00
if (os(windows))
Build-Depends:
Win32 ((>= 2.6.1.0 && < 2.12.0.0) || >= 2.13.4.0),
setenv,
process (>= 1.6.2.0),
2019-08-08 15:58:38 +00:00
silently (>= 1.2.5.1)
2013-12-10 05:18:04 +00:00
else
Build-Depends: unix (>= 2.7.2)
if flag(Assistant) && ! os(solaris) && ! os(gnu)
CPP-Options: -DWITH_ASSISTANT -DWITH_WEBAPP
Build-Depends:
mountpoints,
yesod (>= 1.4.3),
yesod-static (>= 1.5.1),
yesod-form (>= 1.4.8),
yesod-core (>= 1.6.0),
path-pieces (>= 0.2.1),
warp (>= 3.2.8),
warp-tls (>= 3.2.2),
wai,
wai-extra,
blaze-builder,
clientsession,
template-haskell,
shakespeare (>= 2.0.11)
Other-Modules:
Assistant
Assistant.Alert
Assistant.Alert.Utility
Assistant.BranchChange
Assistant.Changes
Assistant.Commits
Assistant.Common
Assistant.CredPairCache
Assistant.DaemonStatus
Assistant.DeleteRemote
Assistant.Drop
Assistant.Fsck
Assistant.Gpg
Assistant.Install
Assistant.MakeRemote
Assistant.MakeRepo
Assistant.Monad
Assistant.NamedThread
Assistant.Pairing
Assistant.Pairing.MakeRemote
Assistant.Pairing.Network
Assistant.Pushes
Assistant.RemoteControl
Assistant.Repair
Assistant.RepoProblem
Assistant.Restart
Assistant.ScanRemotes
Assistant.Ssh
Assistant.Sync
Assistant.Threads.Committer
Assistant.Threads.ConfigMonitor
Assistant.Threads.Cronner
Assistant.Threads.DaemonStatus
Assistant.Threads.Exporter
Assistant.Threads.Glacier
Assistant.Threads.Merger
Assistant.Threads.MountWatcher
Assistant.Threads.NetWatcher
Assistant.Threads.PairListener
Assistant.Threads.ProblemFixer
Assistant.Threads.Pusher
Assistant.Threads.RemoteControl
Assistant.Threads.SanityChecker
Assistant.Threads.TransferPoller
Assistant.Threads.TransferScanner
Assistant.Threads.TransferWatcher
Assistant.Threads.Transferrer
Assistant.Threads.UpgradeWatcher
Assistant.Threads.Upgrader
Assistant.Threads.Watcher
Assistant.Threads.WebApp
Assistant.TransferQueue
Assistant.TransferSlots
Assistant.Types.Alert
Assistant.Types.BranchChange
Assistant.Types.Changes
Assistant.Types.Commits
Assistant.Types.CredPairCache
Assistant.Types.DaemonStatus
Assistant.Types.NamedThread
Assistant.Types.Pushes
Assistant.Types.RemoteControl
Assistant.Types.RepoProblem
Assistant.Types.ScanRemotes
Assistant.Types.ThreadName
Assistant.Types.ThreadedMonad
Assistant.Types.TransferQueue
Assistant.Types.TransferSlots
Assistant.Types.UrlRenderer
Assistant.Unused
Assistant.Upgrade
Assistant.WebApp
Assistant.WebApp.Common
Assistant.WebApp.Configurators
Assistant.WebApp.Configurators.AWS
Assistant.WebApp.Configurators.Delete
Assistant.WebApp.Configurators.Edit
Assistant.WebApp.Configurators.Fsck
Assistant.WebApp.Configurators.IA
Assistant.WebApp.Configurators.Local
Assistant.WebApp.Configurators.Pairing
Assistant.WebApp.Configurators.Preferences
Assistant.WebApp.Configurators.Ssh
Assistant.WebApp.Configurators.Unused
Assistant.WebApp.Configurators.Upgrade
Assistant.WebApp.Configurators.WebDAV
Assistant.WebApp.Control
Assistant.WebApp.DashBoard
Assistant.WebApp.Documentation
Assistant.WebApp.Form
Assistant.WebApp.Gpg
Assistant.WebApp.MakeRemote
Assistant.WebApp.Notifications
Assistant.WebApp.OtherRepos
Assistant.WebApp.Page
Assistant.WebApp.Pairing
Assistant.WebApp.Repair
Assistant.WebApp.RepoId
Assistant.WebApp.RepoList
Assistant.WebApp.SideBar
Assistant.WebApp.Types
Command.Assistant
Command.Watch
Command.WebApp
Utility.Mounts
Utility.Yesod
Utility.WebApp
if os(linux)
Build-Depends: hinotify (>= 0.3.10)
CPP-Options: -DWITH_INOTIFY
Other-Modules: Utility.DirWatcher.INotify
else
if os(darwin)
Build-Depends: hfsevents
CPP-Options: -DWITH_FSEVENTS
Other-Modules:
Utility.DirWatcher.FSEvents
else
if os(windows)
Build-Depends: Win32-notify
CPP-Options: -DWITH_WIN32NOTIFY
Other-Modules: Utility.DirWatcher.Win32Notify
else
if (! os(solaris) && ! os(gnu) && ! os(linux))
CPP-Options: -DWITH_KQUEUE
C-Sources: Utility/libkqueue.c
2017-10-24 16:54:33 +00:00
Includes: Utility/libkqueue.h
Other-Modules: Utility.DirWatcher.Kqueue
if flag(Dbus)
if (os(linux))
Build-Depends: dbus (>= 0.10.7), fdo-notify (>= 0.3)
CPP-Options: -DWITH_DBUS -DWITH_DESKTOP_NOTIFY -DWITH_DBUS_NOTIFICATIONS
Other-Modules: Utility.DBus
if flag(Pairing)
2012-09-07 23:44:20 +00:00
Build-Depends: network-multicast, network-info
CPP-Options: -DWITH_PAIRING
2012-09-07 18:54:00 +00:00
if flag(TorrentParser)
Build-Depends: torrent (>= 10000.0.0)
CPP-Options: -DWITH_TORRENTPARSER
if flag(MagicMime)
Build-Depends: magic
CPP-Options: -DWITH_MAGICMIME
add database benchmark The benchmark shows that the database access is quite fast indeed! And, it scales linearly to the number of keys, with one exception, getAssociatedKey. Based on this benchmark, I don't think I need worry about optimising for cases where all files are locked and the database is mostly empty. In those cases, database access will be misses, and according to this benchmark, should add only 50 milliseconds to runtime. (NB: There may be some overhead to getting the database opened and locking the handle that this benchmark doesn't see.) joey@darkstar:~/src/git-annex>./git-annex benchmark setting up database with 1000 setting up database with 10000 benchmarking keys database/getAssociatedFiles from 1000 (hit) time 62.77 μs (62.70 μs .. 62.85 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 62.81 μs (62.76 μs .. 62.88 μs) std dev 201.6 ns (157.5 ns .. 259.5 ns) benchmarking keys database/getAssociatedFiles from 1000 (miss) time 50.02 μs (49.97 μs .. 50.07 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.09 μs (50.04 μs .. 50.17 μs) std dev 206.7 ns (133.8 ns .. 295.3 ns) benchmarking keys database/getAssociatedKey from 1000 (hit) time 211.2 μs (210.5 μs .. 212.3 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 211.0 μs (210.7 μs .. 212.0 μs) std dev 1.685 μs (334.4 ns .. 3.517 μs) benchmarking keys database/getAssociatedKey from 1000 (miss) time 173.5 μs (172.7 μs .. 174.2 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 173.7 μs (173.0 μs .. 175.5 μs) std dev 3.833 μs (1.858 μs .. 6.617 μs) variance introduced by outliers: 16% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (hit) time 64.01 μs (63.84 μs .. 64.18 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 64.85 μs (64.34 μs .. 66.02 μs) std dev 2.433 μs (547.6 ns .. 4.652 μs) variance introduced by outliers: 40% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (miss) time 50.33 μs (50.28 μs .. 50.39 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.32 μs (50.26 μs .. 50.38 μs) std dev 202.7 ns (167.6 ns .. 252.0 ns) benchmarking keys database/getAssociatedKey from 10000 (hit) time 1.142 ms (1.139 ms .. 1.146 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.142 ms (1.140 ms .. 1.144 ms) std dev 7.142 μs (4.994 μs .. 10.98 μs) benchmarking keys database/getAssociatedKey from 10000 (miss) time 1.094 ms (1.092 ms .. 1.096 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.095 ms (1.095 ms .. 1.097 ms) std dev 4.277 μs (2.591 μs .. 7.228 μs)
2016-01-12 17:01:44 +00:00
if flag(Benchmark)
Build-Depends: criterion
add database benchmark The benchmark shows that the database access is quite fast indeed! And, it scales linearly to the number of keys, with one exception, getAssociatedKey. Based on this benchmark, I don't think I need worry about optimising for cases where all files are locked and the database is mostly empty. In those cases, database access will be misses, and according to this benchmark, should add only 50 milliseconds to runtime. (NB: There may be some overhead to getting the database opened and locking the handle that this benchmark doesn't see.) joey@darkstar:~/src/git-annex>./git-annex benchmark setting up database with 1000 setting up database with 10000 benchmarking keys database/getAssociatedFiles from 1000 (hit) time 62.77 μs (62.70 μs .. 62.85 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 62.81 μs (62.76 μs .. 62.88 μs) std dev 201.6 ns (157.5 ns .. 259.5 ns) benchmarking keys database/getAssociatedFiles from 1000 (miss) time 50.02 μs (49.97 μs .. 50.07 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.09 μs (50.04 μs .. 50.17 μs) std dev 206.7 ns (133.8 ns .. 295.3 ns) benchmarking keys database/getAssociatedKey from 1000 (hit) time 211.2 μs (210.5 μs .. 212.3 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 211.0 μs (210.7 μs .. 212.0 μs) std dev 1.685 μs (334.4 ns .. 3.517 μs) benchmarking keys database/getAssociatedKey from 1000 (miss) time 173.5 μs (172.7 μs .. 174.2 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 173.7 μs (173.0 μs .. 175.5 μs) std dev 3.833 μs (1.858 μs .. 6.617 μs) variance introduced by outliers: 16% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (hit) time 64.01 μs (63.84 μs .. 64.18 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 64.85 μs (64.34 μs .. 66.02 μs) std dev 2.433 μs (547.6 ns .. 4.652 μs) variance introduced by outliers: 40% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (miss) time 50.33 μs (50.28 μs .. 50.39 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.32 μs (50.26 μs .. 50.38 μs) std dev 202.7 ns (167.6 ns .. 252.0 ns) benchmarking keys database/getAssociatedKey from 10000 (hit) time 1.142 ms (1.139 ms .. 1.146 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.142 ms (1.140 ms .. 1.144 ms) std dev 7.142 μs (4.994 μs .. 10.98 μs) benchmarking keys database/getAssociatedKey from 10000 (miss) time 1.094 ms (1.092 ms .. 1.096 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.095 ms (1.095 ms .. 1.097 ms) std dev 4.277 μs (2.591 μs .. 7.228 μs)
2016-01-12 17:01:44 +00:00
CPP-Options: -DWITH_BENCHMARK
if flag(DebugLocks)
CPP-Options: -DDEBUGLOCKS
Other-Modules:
Annex
Annex.Action
Annex.AdjustedBranch
Annex.AdjustedBranch.Merge
Annex.AdjustedBranch.Name
Annex.AutoMerge
Annex.BloomFilter
Annex.Branch
Annex.Branch.Transitions
Annex.BranchState
Annex.CatFile
Annex.ChangedRefs
Annex.CheckAttr
Annex.CheckIgnore
Annex.Common
Annex.Concurrent
Annex.Concurrent.Utility
Annex.Content
Annex.Content.Presence
Annex.Content.Presence.LowLevel
2018-08-22 18:41:09 +00:00
Annex.Content.LowLevel
Annex.Content.PointerFile
Annex.CopyFile
Annex.CurrentBranch
Annex.Debug
Annex.Debug.Utility
Annex.Difference
Annex.DirHashes
Annex.Drop
Annex.Environment
Annex.Export
Annex.ExternalAddonProcess
Annex.FileMatcher
Annex.Fixup
Annex.GitOverlay
Annex.HashObject
Annex.Hook
Annex.Import
Annex.Ingest
Annex.Init
Annex.InodeSentinal
Annex.Journal
Annex.Link
Annex.Locations
Annex.LockFile
Annex.LockPool
2019-01-23 16:39:02 +00:00
Annex.Magic
Annex.MetaData
Annex.MetaData.StandardFields
Annex.Multicast
Annex.Notification
Annex.NumCopies
Annex.Path
Annex.Perms
2020-08-26 16:20:56 +00:00
Annex.PidLock
Annex.Queue
Annex.ReplaceFile
Annex.RemoteTrackingBranch
Annex.SpecialRemote
Annex.SpecialRemote.Config
Annex.Ssh
Annex.StallDetection
Annex.TaggedPush
Annex.Tmp
Annex.Transfer
Annex.TransferrerPool
Annex.UntrustedFilePath
Annex.UpdateInstead
Annex.UUID
Annex.Url
Annex.VariantFile
Annex.VectorClock
Annex.VectorClock.Utility
Annex.Verify
Annex.Version
Annex.View
Annex.View.ViewedFile
Annex.Wanted
Annex.WorkerPool
Annex.WorkTree
Annex.YoutubeDl
Assistant.Install.AutoStart
Assistant.Install.Menu
make my authorship explicit in the code This is intended to guard against LLM code theft, which is the current bubble technology de jour. Note that authorJoeyHess' with a year older than the year I began developing git-annex will behave badly, by intention. Eg, it will spin and eventually crash. This is not the first anti-LLM protection in git-annex. For example see 9562da790fece82d6dfa756b571c67d0fdf57468. That method, while much harder for an adversary to detect and remove, also complicates code somewhat significantly, and needs extensions to be enabled. There are also probably significantly fewer ways to implement that method in Haskell. This new approach, by contrast, will be easy to add throughout the code base, with very little effort, and without complicating reading or maintaining it any more than noticing that yes, I am the author of this code. An adversary could of course remove all calls to these functions before feeding code into their LLM-based laundry facility. I think this would need to be done manually, or with the help of some fairly advanced Haskell parsing though. In some cases, authorJoeyHess needs to be removed, while in other places it needs to be replaced with a value. Also a monadic use of authorJoeyHess' may involve other added monadic machinery which would need to be eliminated to keep the code compiling. Alternatively, an adversary could replace my name with something innocuous. This would be clear intent to remove author attribution from my code, even more than running it through an LLM laundry is. If you work for a large company that is laundering my code through an LLM, please do us a favor and use your immense privilege to quit and go do something socially beneficial. I will not explain further developments of this code in such detail, and you have better things to do than playing cat and mouse with me as I explore directions such as extending this approach to the type level. Sponsored-by: k0ld on Patreon
2023-11-20 16:07:07 +00:00
Author
Backend
Backend.External
Backend.Hash
Backend.URL
Backend.Utilities
Backend.WORM
Benchmark
Build.BundledPrograms
Build.Configure
Build.DesktopFile
Build.Mans
Build.TestConfig
Build.Version
BuildInfo
BuildFlags
CmdLine
CmdLine.Action
CmdLine.Batch
CmdLine.GitAnnex
CmdLine.GitAnnex.Options
CmdLine.GitAnnexShell
CmdLine.GitAnnexShell.Checks
CmdLine.GitAnnexShell.Fields
CmdLine.AnnexSetter
CmdLine.Option
CmdLine.GitRemoteTorAnnex
CmdLine.Seek
CmdLine.Usage
Command
Command.Add
Command.AddUnused
Command.AddUrl
Command.Adjust
Command.Assist
Command.Benchmark
Command.CalcKey
Command.CheckPresentKey
2017-01-31 21:27:44 +00:00
Command.Config
Command.ConfigList
Command.ConfigRemote
Command.ContentLocation
Command.Copy
Command.Dead
Command.Describe
Command.DiffDriver
Command.Direct
Command.Drop
Command.DropKey
Command.DropUnused
Command.EnableRemote
2016-11-30 18:35:24 +00:00
Command.EnableTor
Command.ExamineKey
Command.Expire
2017-09-13 18:21:08 +00:00
Command.Export
Command.FilterBranch
Command.FilterProcess
Command.Find
Command.FindKeys
Command.FindRef
Command.Fix
Command.Forget
Command.FromKey
Command.Fsck
Command.FuzzTest
Command.GCryptSetup
Command.Get
Command.Group
Command.GroupWanted
Command.Help
Command.Import
Command.ImportFeed
Command.InAnnex
Command.Indirect
Command.Info
Command.Init
Command.InitRemote
Command.Inprogress
Command.List
Command.Lock
Command.Log
Command.LookupKey
Command.Map
Command.MatchExpression
Command.Merge
Command.MetaData
Command.Migrate
Command.Mirror
Command.Move
Command.Multicast
Command.NotifyChanges
Command.NumCopies
Command.MinCopies
Command.OldKeys
2016-11-30 18:35:24 +00:00
Command.P2P
Command.P2PStdIO
Command.PostReceive
Command.PreCommit
Command.Proxy
Command.Pull
Command.Push
Command.ReKey
Command.ReadPresentKey
Command.RecvKey
Command.RegisterUrl
Command.Reinit
Command.Reinject
Command.RemoteDaemon
2019-04-15 17:05:44 +00:00
Command.RenameRemote
Command.Repair
Command.Required
Command.ResolveMerge
Command.Restage
Command.RmUrl
Command.Satisfy
Command.Schedule
Command.Semitrust
Command.SendKey
Command.SetKey
Command.SetPresentKey
Command.Smudge
Command.Status
Command.Sync
Command.Test
Command.TestRemote
Command.Transferrer
Command.TransferKey
Command.TransferKeys
Command.Trust
Command.Unannex
Command.Undo
Command.Ungroup
Command.Uninit
Command.Unlock
Command.UnregisterUrl
Command.Untrust
Command.Unused
Command.Upgrade
Command.VAdd
Command.VCycle
Command.VFilter
Command.VPop
Command.Version
Command.Vicfg
Command.View
Command.Wanted
Command.Whereis
Command.WhereUsed
Common
Config
Config.Cost
Config.Files
Config.Files.AutoStart
Config.DynamicConfig
Config.GitConfig
Config.Smudge
Creds
Crypto
Database.Benchmark
Database.ContentIdentifier
Database.Export
Database.Fsck
Database.Handle
Database.ImportFeed
Database.Init
Database.Keys
Database.Keys.Handle
Database.Keys.Tables
Database.Keys.SQL
Database.Queue
Database.Types
Database.Utility
Git
Git.AutoCorrect
Git.Branch
Git.BuildVersion
Git.CatFile
Git.CheckAttr
Git.CheckIgnore
Git.Command
Git.Command.Batch
Git.Config
Git.ConfigTypes
Git.Construct
2019-09-24 16:32:54 +00:00
Git.Credential
Git.CurrentRepo
Git.DiffTree
Git.DiffTreeItem
Git.Env
Git.FileMode
Git.FilePath
Git.FilterProcess
Git.Fsck
Git.GCrypt
Git.HashObject
2019-04-24 18:55:49 +00:00
Git.History
Git.Hook
Git.Index
Git.LockFile
Git.Log
Git.LsFiles
Git.LsTree
Git.Merge
Git.Objects
Git.PktLine
Git.Queue
2023-04-12 21:18:29 +00:00
Git.Quote
Git.Ref
Git.RefLog
Git.Remote
Git.Remote.Remove
Git.Repair
Git.Sha
Git.Ssh
Git.Status
Git.Tree
Git.Types
Git.UnionMerge
Git.UpdateIndex
Git.Url
Git.Version
2017-02-24 17:42:30 +00:00
Key
Limit
Limit.Wanted
Logs
Logs.Activity
sync: use log to track adjusted branch needs updating Speeds up sync in an adjusted branch by avoiding re-adjusting the branch unncessarily, particularly when it is adjusted with --hide-missing or --unlock-present. When there are a lot of files, that was the majority of the time of a --no-content sync. Uses a log file, which is updated when content presence changes. This adds a little bit of overhead to every file get/drop when on such an adjusted branch. The overhead is minimal for get of any size of file, but might be noticable for drop in some cases. It seems like a reasonable trade-off. It would be possible to update the log file only at the end, but then it would not happen if the command is interrupted. When not in an adjusted branch, there should be no additional overhead. (getCurrentBranch is an MVar read, and it avoids the MVar read of getGitConfig.) Note that this does not deal with situations such as: git checkout master, git-annex get, git checkout adjusted branch, git-annex sync. The sync won't know that the adjusted branch needs to be updated. Dealing with that would add overhead to operation in non-adjusted branches, which I don't like. Also, there are other situations like having two adjusted branches that both need to be updated like this, and switching between them and sync not updating. This does mean a behavior change to sync, since it did previously deal with those situations. But, the documentation did not say that it did. The man pages only talk about sync updating the adjusted branch after it transfers content. I did consider making sync keep track of content it transferred (and dropped) and only update the adjusted branch then, not to catch up to other changes made previously. That would perform better. But it seemed rather hard to implement, and also it would have problems with races with a concurrent get/drop, which this implementation avoids. And it seemed pretty likely someone had gotten used to get/drop followed by sync updating the branch. It seems much less likely someone is switching branches, doing get/drop, and then switching back and expecting sync to update the branch. Re-running git-annex adjust still does a full re-adjusting of the branch, for anyone who needs that. Sponsored-by: Leon Schuermann on Patreon
2023-06-08 18:35:26 +00:00
Logs.AdjustedBranchUpdate
Logs.Chunk
Logs.Chunk.Pure
2017-01-31 21:27:44 +00:00
Logs.Config
2019-02-20 21:22:56 +00:00
Logs.ContentIdentifier
Logs.ContentIdentifier.Pure
Logs.Difference
Logs.Difference.Pure
Logs.Export
Logs.Export.Pure
Logs.File
Logs.FsckResults
Logs.Group
Logs.Import
2016-05-27 16:15:01 +00:00
Logs.Line
Logs.Location
Logs.MapLog
Logs.MetaData
Logs.MetaData.Pure
Logs.Multicast
Logs.NumCopies
Logs.PreferredContent
Logs.PreferredContent.Raw
Logs.Presence
Logs.Presence.Pure
Logs.Remote
Logs.Remote.Pure
Logs.RemoteState
add restage log When pointer files need to be restaged, they're first written to the log, and then when the restage operation runs, it reads the log. This way, if the git-annex process is interrupted before it can do the restaging, a later git-annex process can do it. Currently, this lets a git-annex get/drop command be interrupted and then re-ran, and as long as it gets/drops additional files, it will clean up after the interrupted command. But more changes are needed to make it easier to restage after an interrupted process. Kept using the git queue to run the restage action, even though the list of files that it builds up for that action is not actually used by the action. This could perhaps be simplified to make restaging a cleanup action that gets registered, rather than using the git queue for it. But I wasn't sure if that would cause visible behavior changes, when eg dropping a large number of files, currently the git queue flushes periodically, and so it restages incrementally, rather than all at the end. In restagePointerFiles, it reads the restage log twice, once to get the number of files and size, and a second time to process it. This seemed better than reading the whole file into memory, since potentially a huge number of files could be in there. Probably the OS will cache the file in memory and there will not be much performance impact. It might be better to keep running tallies in another file though. But updating that atomically with the log seems hard. Also note that it's possible for calcRestageLog to see a different file than streamRestageLog does. More files may be added to the log in between. That is ok, it will only cause the filterprocessfaster heuristic to operate with slightly out of date information, so it may make the wrong choice for the files that got added and be a little slower than ideal. Sponsored-by: Dartmouth College's DANDI project
2022-09-23 18:38:59 +00:00
Logs.Restage
Logs.Schedule
Logs.SingleValue
Logs.SingleValue.Pure
Logs.Smudge
Logs.Transfer
Logs.Transitions
Logs.Trust
Logs.Trust.Basic
Logs.Trust.Pure
Logs.UUID
Logs.UUIDBased
Logs.Unused
Logs.Upgrade
Logs.View
Logs.Web
Messages
Messages.Concurrent
Messages.Internal
Messages.JSON
Messages.Progress
Messages.Serialized
P2P.Address
P2P.Annex
2016-11-22 18:37:19 +00:00
P2P.Auth
2016-11-22 18:34:49 +00:00
P2P.IO
P2P.Protocol
Remote
Remote.Adb
Remote.BitTorrent
Remote.Borg
Remote.Bup
Remote.Ddar
Remote.Directory
Remote.Directory.LegacyChunked
Remote.External
Remote.External.AsyncExtension
Remote.External.Types
Remote.GCrypt
Remote.Git
Remote.GitLFS
Remote.Glacier
Remote.Helper.AWS
Remote.Helper.Chunked
Remote.Helper.Chunked.Legacy
Remote.Helper.Encryptable
2019-02-20 19:55:01 +00:00
Remote.Helper.ExportImport
Remote.Helper.Git
Remote.Helper.Hooks
Remote.Helper.Http
Remote.Helper.Messages
Remote.Helper.P2P
Remote.Helper.Path
Remote.Helper.ReadOnly
Remote.Helper.ThirdPartyPopulated
Remote.Helper.Special
Remote.Helper.Ssh
Remote.HttpAlso
Remote.Hook
Remote.List
Remote.List.Util
Remote.P2P
Remote.Rsync
Remote.Rsync.RsyncUrl
Remote.S3
Remote.Tahoe
Remote.Web
Remote.WebDAV
Remote.WebDAV.DavLocation
RemoteDaemon.Common
RemoteDaemon.Core
RemoteDaemon.Transport
RemoteDaemon.Transport.GCrypt
2016-11-20 19:45:01 +00:00
RemoteDaemon.Transport.Tor
RemoteDaemon.Transport.Ssh
RemoteDaemon.Transport.Ssh.Types
RemoteDaemon.Types
Test
Test.Framework
Types
2016-08-08 14:41:54 +00:00
Types.ActionItem
Types.AdjustedBranch
Types.Availability
Types.Backend
Types.Benchmark
Types.BranchState
Types.CatFileHandles
Types.CleanupActions
Types.Command
Types.Concurrency
Types.Creds
Types.Crypto
Types.DeferredParse
Types.DesktopNotify
Types.Difference
Types.Direction
Types.Distribution
2017-09-15 20:34:45 +00:00
Types.Export
Types.FileMatcher
Types.GitConfig
Types.Group
Types.Import
Types.IndexFiles
Types.Key
Types.KeySource
Types.Link
Types.LockCache
Types.Messages
Types.MetaData
Types.Mime
Types.NumCopies
Types.ProposedAccepted
Types.RefSpec
Types.Remote
Types.RemoteConfig
Types.RemoteState
Types.RepoVersion
Types.ScheduledActivity
Types.StandardGroups
Types.StallDetection
Types.StoreRetrieve
Types.Test
2016-08-08 14:41:54 +00:00
Types.Transfer
2020-12-09 17:28:16 +00:00
Types.Transferrer
Types.TransferrerPool
Types.Transitions
Types.TrustLevel
Types.UUID
Types.Upgrade
Types.UrlContents
Types.VectorClock
Types.View
Types.WorkerPool
Upgrade
Upgrade.V0
Upgrade.V1
Upgrade.V2
Upgrade.V3
Upgrade.V4
Upgrade.V5
Upgrade.V5.Direct
Upgrade.V6
Upgrade.V7
Upgrade.V8
Upgrade.V9
Utility.Aeson
Utility.Android
Utility.Applicative
Utility.Attoparsec
Utility.AuthToken
Utility.Base64
Utility.Batch
Utility.Bloom
Utility.CoProcess
Utility.CopyFile
Utility.Daemon
Utility.Data
Utility.DataUnits
Utility.Debug
Utility.DebugLocks
Utility.DirWatcher
Utility.DirWatcher.Types
Utility.Directory
Utility.Directory.Create
Utility.Directory.Stream
Utility.DiskFree
Utility.Dot
Utility.DottedVersion
Utility.Env
Utility.Env.Basic
Utility.Env.Set
Utility.Exception
Utility.FileMode
Utility.FileSize
Utility.FileSystemEncoding
Utility.Format
Utility.FreeDesktop
Utility.Glob
Utility.Gpg
Utility.Hash
Utility.HtmlDetect
Utility.HumanNumber
Utility.HumanTime
Utility.InodeCache
Utility.IPAddress
Utility.LockFile
Utility.LockPool
Utility.LockPool.LockHandle
Utility.LockPool.STM
Utility.LogFile
Utility.Lsof
Utility.MagicWormhole
Utility.Matcher
Utility.MD5
Utility.Metered
Utility.Misc
Utility.Monad
Utility.MoveFile
Utility.Network
Utility.NotificationBroadcaster
Utility.OpenFd
Utility.OptParse
Utility.OSX
Utility.PID
Utility.PartialPrelude
Utility.Path
Utility.Path.AbsRel
Utility.Path.Max
Utility.Path.Tests
Utility.Path.Windows
Utility.Percentage
Utility.Process
Utility.Process.Shim
Utility.Process.Transcript
Utility.QuickCheck
Utility.RawFilePath
Utility.ResourcePool
Utility.Rsync
Utility.SafeCommand
Utility.SafeOutput
Utility.Scheduled
Utility.Scheduled.QuickCheck
Utility.Shell
Utility.ShellEscape
Utility.SimpleProtocol
Utility.Split
Utility.SshConfig
Utility.SshHost
Utility.Su
Utility.SystemDirectory
Utility.Terminal
2018-10-30 03:13:36 +00:00
Utility.TimeStamp
Utility.TList
Utility.Tense
Utility.ThreadLock
Utility.ThreadScheduler
Utility.Tmp
Utility.Tmp.Dir
Utility.Tor
Utility.Touch
Utility.Tuple
Utility.Url
Utility.Url.Parse
Utility.UserInfo
Utility.Verifiable
if (os(windows))
Other-Modules:
Utility.LockFile.Windows
Utility.LockPool.Windows
else
Other-Modules:
Utility.LockFile.Posix
Utility.LockPool.Posix
Annex.LockPool.PosixOrPid
2019-07-22 13:21:41 +00:00
Utility.LockFile.LockStatus
Utility.LockFile.PidLock
Utility.LockPool.PidLock
Utility.LinuxMkLibs