git-annex/git-annex.cabal

1167 lines
29 KiB
Text
Raw Normal View History

Name: git-annex
Version: 10.20221103
Cabal-Version: 1.12
License: AGPL-3
Maintainer: Joey Hess <id@joeyh.name>
Author: Joey Hess
Stability: Stable
Copyright: 2010-2022 Joey Hess
2012-09-24 19:20:28 +00:00
License-File: COPYRIGHT
Homepage: http://git-annex.branchable.com/
Build-type: Custom
Category: Utility
Synopsis: manage files with git, without checking their contents into git
Description:
git-annex allows managing files with git, without checking the file
contents into git. While that may seem paradoxical, it is useful when
dealing with files larger than git can currently easily handle, whether due
2012-02-15 23:43:15 +00:00
to limitations in memory, time, or disk space.
.
It can store large files in many places, from local hard drives, to a
large number of cloud storage services, including S3, WebDAV,
and rsync, with a dozen cloud storage providers usable via plugins.
Files can be stored encrypted with gpg, so that the cloud storage
provider cannot see your data. git-annex keeps track of where each file
is stored, so it knows how many copies are available, and has many
facilities to ensure your data is preserved.
.
git-annex can also be used to keep a folder in sync between computers,
noticing when files are changed, and automatically committing them
to git and transferring them to other computers. The git-annex webapp
makes it easy to set up and use git-annex this way.
-- The tarball uploaded to hackage does not include every non-haskell
-- file in the git repo. The website is left out, so is build machinary for
-- standalone apps, and packages. Include only files that are needed
-- make cabal install git-annex work.
Extra-Source-Files:
stack.yaml
README
CHANGELOG
NEWS
doc/license/GPL
doc/license/AGPL
doc/git-annex.mdwn
doc/git-annex-add.mdwn
doc/git-annex-addunused.mdwn
doc/git-annex-addurl.mdwn
doc/git-annex-adjust.mdwn
doc/git-annex-assistant.mdwn
doc/git-annex-backends.mdwn
doc/git-annex-calckey.mdwn
doc/git-annex-checkpresentkey.mdwn
doc/git-annex-config.mdwn
doc/git-annex-contentlocation.mdwn
doc/git-annex-copy.mdwn
doc/git-annex-dead.mdwn
doc/git-annex-describe.mdwn
doc/git-annex-diffdriver.mdwn
doc/git-annex-direct.mdwn
doc/git-annex-drop.mdwn
doc/git-annex-dropkey.mdwn
doc/git-annex-dropunused.mdwn
doc/git-annex-edit.mdwn
doc/git-annex-enableremote.mdwn
doc/git-annex-enable-tor.mdwn
doc/git-annex-examinekey.mdwn
doc/git-annex-expire.mdwn
2017-12-14 17:32:24 +00:00
doc/git-annex-export.mdwn
doc/git-annex-filter-branch.mdwn
doc/git-annex-find.mdwn
doc/git-annex-findref.mdwn
doc/git-annex-fix.mdwn
doc/git-annex-forget.mdwn
doc/git-annex-fromkey.mdwn
doc/git-annex-fsck.mdwn
doc/git-annex-fuzztest.mdwn
doc/git-annex-get.mdwn
doc/git-annex-group.mdwn
doc/git-annex-groupwanted.mdwn
doc/git-annex-import.mdwn
doc/git-annex-importfeed.mdwn
doc/git-annex-indirect.mdwn
doc/git-annex-info.mdwn
doc/git-annex-init.mdwn
doc/git-annex-initremote.mdwn
doc/git-annex-inprogress.mdwn
doc/git-annex-list.mdwn
doc/git-annex-lock.mdwn
doc/git-annex-log.mdwn
doc/git-annex-lookupkey.mdwn
doc/git-annex-map.mdwn
doc/git-annex-matchexpression.mdwn
doc/git-annex-matching-options.mdwn
doc/git-annex-merge.mdwn
doc/git-annex-metadata.mdwn
doc/git-annex-migrate.mdwn
doc/git-annex-mirror.mdwn
doc/git-annex-move.mdwn
doc/git-annex-multicast.mdwn
doc/git-annex-numcopies.mdwn
2016-11-30 18:35:24 +00:00
doc/git-annex-p2p.mdwn
doc/git-annex-pre-commit.mdwn
doc/git-annex-preferred-content.mdwn
doc/git-annex-proxy.mdwn
doc/git-annex-readpresentkey.mdwn
doc/git-annex-registerurl.mdwn
doc/git-annex-reinit.mdwn
doc/git-annex-reinject.mdwn
doc/git-annex-rekey.mdwn
doc/git-annex-remotedaemon.mdwn
2019-04-15 17:05:44 +00:00
doc/git-annex-renameremote.mdwn
doc/git-annex-repair.mdwn
doc/git-annex-required.mdwn
doc/git-annex-restage.mdwn
doc/git-annex-resolvemerge.mdwn
doc/git-annex-rmurl.mdwn
doc/git-annex-schedule.mdwn
doc/git-annex-semitrust.mdwn
doc/git-annex-setkey.mdwn
doc/git-annex-setpresentkey.mdwn
doc/git-annex-shell.mdwn
doc/git-annex-smudge.mdwn
doc/git-annex-status.mdwn
doc/git-annex-sync.mdwn
doc/git-annex-test.mdwn
doc/git-annex-testremote.mdwn
doc/git-annex-transferkey.mdwn
doc/git-annex-transferkeys.mdwn
doc/git-annex-trust.mdwn
doc/git-annex-unannex.mdwn
doc/git-annex-undo.mdwn
doc/git-annex-ungroup.mdwn
doc/git-annex-uninit.mdwn
doc/git-annex-unlock.mdwn
doc/git-annex-untrust.mdwn
doc/git-annex-unregisterurl.mdwn
doc/git-annex-unused.mdwn
doc/git-annex-upgrade.mdwn
doc/git-annex-vadd.mdwn
doc/git-annex-vcycle.mdwn
doc/git-annex-version.mdwn
doc/git-annex-vfilter.mdwn
doc/git-annex-vicfg.mdwn
doc/git-annex-view.mdwn
doc/git-annex-vpop.mdwn
doc/git-annex-wanted.mdwn
doc/git-annex-watch.mdwn
doc/git-annex-webapp.mdwn
doc/git-annex-whereis.mdwn
doc/git-annex-whereused.mdwn
doc/git-remote-tor-annex.mdwn
doc/logo.svg
doc/logo_16x16.png
Build/mdwn2man
Assistant/WebApp/routes
static/activityicon.gif
static/css/bootstrap.css
static/css/bootstrap-theme.css
static/js/jquery.ui.core.js
static/js/longpolling.js
static/js/jquery.full.js
static/js/jquery.ui.sortable.js
static/js/jquery.ui.mouse.js
static/js/jquery.ui.widget.js
static/js/bootstrap.js
static/syncicon.gif
static/favicon.ico
static/fonts/glyphicons-halflings-regular.woff
static/fonts/glyphicons-halflings-regular.eot
static/fonts/glyphicons-halflings-regular.svg
static/fonts/glyphicons-halflings-regular.ttf
templates/sidebar/main.hamlet
templates/sidebar/alert.hamlet
templates/bootstrap.hamlet
templates/error.cassius
templates/README
templates/error.hamlet
templates/documentation/license.hamlet
templates/documentation/repogroup.hamlet
templates/documentation/about.hamlet
templates/dashboard/main.hamlet
templates/dashboard/transfers.cassius
templates/dashboard/transfers.hamlet
templates/dashboard/metarefresh.hamlet
templates/page.cassius
templates/page.hamlet
templates/control/repairrepository.hamlet
templates/control/repairrepository/done.hamlet
templates/control/notrunning.julius
templates/control/notrunning.hamlet
templates/control/repositoryswitcher.hamlet
templates/control/shutdown.hamlet
templates/control/log.hamlet
templates/page.julius
templates/repolist.julius
templates/configurators/adddrive/combine.hamlet
templates/configurators/adddrive/setupmodal.hamlet
templates/configurators/adddrive/encrypt.hamlet
templates/configurators/newrepository.hamlet
templates/configurators/needglaciercli.hamlet
templates/configurators/adds3.hamlet
templates/configurators/genkeymodal.hamlet
templates/configurators/main.hamlet
templates/configurators/needconnection.hamlet
templates/configurators/newrepository/form.hamlet
templates/configurators/newrepository/first.hamlet
templates/configurators/newrepository/combine.hamlet
templates/configurators/enablewebdav.hamlet
templates/configurators/pairing/local/inprogress.hamlet
templates/configurators/pairing/local/prompt.hamlet
templates/configurators/pairing/wormhole/prompt.hamlet
templates/configurators/pairing/wormhole/start.hamlet
templates/configurators/pairing/disabled.hamlet
templates/configurators/addglacier.hamlet
templates/configurators/fsck.cassius
templates/configurators/edit/nonannexremote.hamlet
templates/configurators/edit/webrepository.hamlet
templates/configurators/edit/repository.hamlet
templates/configurators/unused.hamlet
templates/configurators/ssh/testmodal.hamlet
templates/configurators/ssh/expiredpassword.hamlet
templates/configurators/ssh/error.hamlet
templates/configurators/ssh/combine.hamlet
templates/configurators/ssh/enable.hamlet
templates/configurators/ssh/add.hamlet
templates/configurators/ssh/setupmodal.hamlet
templates/configurators/ssh/confirm.hamlet
templates/configurators/enableia.hamlet
templates/configurators/fsck.hamlet
templates/configurators/addrepository/archive.hamlet
templates/configurators/addrepository/cloud.hamlet
templates/configurators/addrepository/connection.hamlet
templates/configurators/addrepository/ssh.hamlet
templates/configurators/addrepository/misc.hamlet
templates/configurators/addrepository/wormholepairing.hamlet
templates/configurators/rsync.net/add.hamlet
templates/configurators/rsync.net/encrypt.hamlet
templates/configurators/needgcrypt.hamlet
templates/configurators/needtor.hamlet
templates/configurators/needmagicwormhole.hamlet
templates/configurators/enabledirectory.hamlet
templates/configurators/fsck/status.hamlet
templates/configurators/fsck/form.hamlet
templates/configurators/fsck/preferencesform.hamlet
templates/configurators/fsck/formcontent.hamlet
templates/configurators/delete/finished.hamlet
templates/configurators/delete/start.hamlet
templates/configurators/delete/currentrepository.hamlet
templates/configurators/unused/form.hamlet
templates/configurators/adddrive.hamlet
templates/configurators/preferences.hamlet
templates/configurators/addia.hamlet
templates/configurators/enableaws.hamlet
templates/configurators/addrepository.hamlet
templates/actionbutton.hamlet
templates/repolist.hamlet
templates/controlmenu.hamlet
templates/notifications/longpolling.julius
Utility/libkqueue.h
Flag Assistant
Description: Enable git-annex assistant, webapp, and watch command
Default: True
2012-09-07 23:44:20 +00:00
Flag Pairing
Description: Enable pairing
2012-09-07 18:54:00 +00:00
2013-02-27 05:41:01 +00:00
Flag Production
Description: Enable production build (slower build; faster binary)
Flag TorrentParser
Description: Use haskell torrent library to parse torrent files
Flag MagicMime
Description: Use libmagic to determine file MIME types
add database benchmark The benchmark shows that the database access is quite fast indeed! And, it scales linearly to the number of keys, with one exception, getAssociatedKey. Based on this benchmark, I don't think I need worry about optimising for cases where all files are locked and the database is mostly empty. In those cases, database access will be misses, and according to this benchmark, should add only 50 milliseconds to runtime. (NB: There may be some overhead to getting the database opened and locking the handle that this benchmark doesn't see.) joey@darkstar:~/src/git-annex>./git-annex benchmark setting up database with 1000 setting up database with 10000 benchmarking keys database/getAssociatedFiles from 1000 (hit) time 62.77 μs (62.70 μs .. 62.85 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 62.81 μs (62.76 μs .. 62.88 μs) std dev 201.6 ns (157.5 ns .. 259.5 ns) benchmarking keys database/getAssociatedFiles from 1000 (miss) time 50.02 μs (49.97 μs .. 50.07 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.09 μs (50.04 μs .. 50.17 μs) std dev 206.7 ns (133.8 ns .. 295.3 ns) benchmarking keys database/getAssociatedKey from 1000 (hit) time 211.2 μs (210.5 μs .. 212.3 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 211.0 μs (210.7 μs .. 212.0 μs) std dev 1.685 μs (334.4 ns .. 3.517 μs) benchmarking keys database/getAssociatedKey from 1000 (miss) time 173.5 μs (172.7 μs .. 174.2 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 173.7 μs (173.0 μs .. 175.5 μs) std dev 3.833 μs (1.858 μs .. 6.617 μs) variance introduced by outliers: 16% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (hit) time 64.01 μs (63.84 μs .. 64.18 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 64.85 μs (64.34 μs .. 66.02 μs) std dev 2.433 μs (547.6 ns .. 4.652 μs) variance introduced by outliers: 40% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (miss) time 50.33 μs (50.28 μs .. 50.39 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.32 μs (50.26 μs .. 50.38 μs) std dev 202.7 ns (167.6 ns .. 252.0 ns) benchmarking keys database/getAssociatedKey from 10000 (hit) time 1.142 ms (1.139 ms .. 1.146 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.142 ms (1.140 ms .. 1.144 ms) std dev 7.142 μs (4.994 μs .. 10.98 μs) benchmarking keys database/getAssociatedKey from 10000 (miss) time 1.094 ms (1.092 ms .. 1.096 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.095 ms (1.095 ms .. 1.097 ms) std dev 4.277 μs (2.591 μs .. 7.228 μs)
2016-01-12 17:01:44 +00:00
Flag Benchmark
Description: Enable benchmarking
Default: True
add database benchmark The benchmark shows that the database access is quite fast indeed! And, it scales linearly to the number of keys, with one exception, getAssociatedKey. Based on this benchmark, I don't think I need worry about optimising for cases where all files are locked and the database is mostly empty. In those cases, database access will be misses, and according to this benchmark, should add only 50 milliseconds to runtime. (NB: There may be some overhead to getting the database opened and locking the handle that this benchmark doesn't see.) joey@darkstar:~/src/git-annex>./git-annex benchmark setting up database with 1000 setting up database with 10000 benchmarking keys database/getAssociatedFiles from 1000 (hit) time 62.77 μs (62.70 μs .. 62.85 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 62.81 μs (62.76 μs .. 62.88 μs) std dev 201.6 ns (157.5 ns .. 259.5 ns) benchmarking keys database/getAssociatedFiles from 1000 (miss) time 50.02 μs (49.97 μs .. 50.07 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.09 μs (50.04 μs .. 50.17 μs) std dev 206.7 ns (133.8 ns .. 295.3 ns) benchmarking keys database/getAssociatedKey from 1000 (hit) time 211.2 μs (210.5 μs .. 212.3 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 211.0 μs (210.7 μs .. 212.0 μs) std dev 1.685 μs (334.4 ns .. 3.517 μs) benchmarking keys database/getAssociatedKey from 1000 (miss) time 173.5 μs (172.7 μs .. 174.2 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 173.7 μs (173.0 μs .. 175.5 μs) std dev 3.833 μs (1.858 μs .. 6.617 μs) variance introduced by outliers: 16% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (hit) time 64.01 μs (63.84 μs .. 64.18 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 64.85 μs (64.34 μs .. 66.02 μs) std dev 2.433 μs (547.6 ns .. 4.652 μs) variance introduced by outliers: 40% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (miss) time 50.33 μs (50.28 μs .. 50.39 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.32 μs (50.26 μs .. 50.38 μs) std dev 202.7 ns (167.6 ns .. 252.0 ns) benchmarking keys database/getAssociatedKey from 10000 (hit) time 1.142 ms (1.139 ms .. 1.146 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.142 ms (1.140 ms .. 1.144 ms) std dev 7.142 μs (4.994 μs .. 10.98 μs) benchmarking keys database/getAssociatedKey from 10000 (miss) time 1.094 ms (1.092 ms .. 1.096 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.095 ms (1.095 ms .. 1.097 ms) std dev 4.277 μs (2.591 μs .. 7.228 μs)
2016-01-12 17:01:44 +00:00
Flag DebugLocks
Description: Debug location of MVar/STM deadlocks
Default: False
Flag Dbus
Description: Enable dbus support
Flag GitLfs
Description: Build with git-lfs library (rather than vendored copy)
Default: True
source-repository head
type: git
location: git://git-annex.branchable.com/
custom-setup
Setup-Depends: base (>= 4.11.1.0), split, unix-compat,
filepath, exceptions, bytestring, IfElse, data-default,
filepath-bytestring (>= 1.4.2.1.4),
process (>= 1.6.3),
time (>= 1.5.0),
directory (>= 1.2.7.0),
async, utf8-string, transformers, Cabal
Executable git-annex
Main-Is: git-annex.hs
2015-05-10 21:03:20 +00:00
Build-Depends:
base (>= 4.11.1.0 && < 5.0),
network-uri (>= 2.6),
optparse-applicative (>= 0.14.1),
containers (>= 0.5.8),
2015-05-10 21:03:20 +00:00
exceptions (>= 0.6),
stm (>= 2.3),
mtl (>= 2),
uuid (>= 1.2.6),
process (>= 1.6.3),
data-default,
case-insensitive,
random,
dlist,
2020-01-02 16:18:38 +00:00
unix-compat (>= 0.5),
SafeSemaphore,
async,
directory (>= 1.2.7.0),
disk-free-space,
filepath,
filepath-bytestring (>= 1.4.2.1.1),
IfElse,
monad-logger (>= 0.3.10),
free,
utf8-string,
bytestring,
text,
sandi,
monad-control,
transformers,
bloomfilter (>= 2.0.0),
edit-distance,
resourcet,
connection (>= 0.2.6),
http-client (>= 0.5.3),
http-client-tls,
http-types (>= 0.7),
http-conduit (>= 2.3.0),
http-client-restricted (>= 0.0.2),
conduit,
time (>= 1.5.0),
old-locale,
persistent-sqlite (>= 2.8.1),
persistent (>= 2.8.1),
persistent-template,
unliftio-core,
microlens,
aeson,
vector,
tagsoup,
unordered-containers,
feed (>= 1.0.0),
regex-tdfa,
socks,
byteable,
stm-chans,
securemem,
crypto-api,
cryptonite (>= 0.23),
memory,
deepseq,
split,
attoparsec (>= 0.13.2.2),
concurrent-output (>= 1.10),
QuickCheck (>= 2.10.0),
tasty (>= 1.2),
tasty-hunit,
tasty-quickcheck,
tasty-rerun,
ansi-terminal >= 0.9,
aws (>= 0.20),
DAV (>= 1.0),
network (>= 3.0.0.0),
network-bsd,
mountpoints,
yesod (>= 1.4.3),
yesod-static (>= 1.5.1),
yesod-form (>= 1.4.8),
yesod-core (>= 1.6.0),
path-pieces (>= 0.2.1),
warp (>= 3.2.8),
warp-tls (>= 3.2.2),
wai,
wai-extra,
blaze-builder,
clientsession,
template-haskell,
shakespeare (>= 2.0.11)
CC-Options: -Wall
GHC-Options: -Wall -fno-warn-tabs -Wincomplete-uni-patterns
Default-Language: Haskell2010
Default-Extensions: LambdaCase
Other-Extensions: TemplateHaskell
-- Some things don't work with the non-threaded RTS.
GHC-Options: -threaded
2013-02-27 05:41:01 +00:00
-- Fully optimize for production.
if flag(Production)
-- Lower memory systems can run out of memory with -O2, so
-- optimise slightly less.
if arch(arm)
GHC-Options: -O2 -optlo-O2
else
GHC-Options: -O2
else
GHC-Options: -O0
-- Avoid linking with unused dynamic libaries.
if os(linux) || os(freebsd)
GHC-Options: -optl-Wl,--as-needed
2013-12-10 05:18:04 +00:00
if (os(windows))
Build-Depends:
Win32 ((>= 2.6.1.0 && < 2.12.0.0) || >= 2.13.3.1),
setenv,
process (>= 1.6.2.0),
2019-08-08 15:58:38 +00:00
silently (>= 1.2.5.1)
2013-12-10 05:18:04 +00:00
else
Build-Depends: unix (>= 2.7.2)
if flag(GitLfs)
Build-Depends: git-lfs (>= 1.2.0)
CPP-Options: -DWITH_GIT_LFS
else
Other-Modules: Utility.GitLFS
if flag(Assistant) && ! os(solaris) && ! os(gnu)
CPP-Options: -DWITH_ASSISTANT -DWITH_WEBAPP
Other-Modules:
Assistant
Assistant.Alert
Assistant.Alert.Utility
Assistant.BranchChange
Assistant.Changes
Assistant.Commits
Assistant.Common
Assistant.CredPairCache
Assistant.DaemonStatus
Assistant.DeleteRemote
Assistant.Drop
Assistant.Fsck
Assistant.Gpg
Assistant.Install
Assistant.Install.AutoStart
Assistant.Install.Menu
Assistant.MakeRemote
Assistant.MakeRepo
Assistant.Monad
Assistant.NamedThread
Assistant.Pairing
Assistant.Pairing.MakeRemote
Assistant.Pairing.Network
Assistant.Pushes
Assistant.RemoteControl
Assistant.Repair
Assistant.RepoProblem
Assistant.Restart
Assistant.ScanRemotes
Assistant.Ssh
Assistant.Sync
Assistant.Threads.Committer
Assistant.Threads.ConfigMonitor
Assistant.Threads.Cronner
Assistant.Threads.DaemonStatus
Assistant.Threads.Exporter
Assistant.Threads.Glacier
Assistant.Threads.Merger
Assistant.Threads.MountWatcher
Assistant.Threads.NetWatcher
Assistant.Threads.PairListener
Assistant.Threads.ProblemFixer
Assistant.Threads.Pusher
Assistant.Threads.RemoteControl
Assistant.Threads.SanityChecker
Assistant.Threads.TransferPoller
Assistant.Threads.TransferScanner
Assistant.Threads.TransferWatcher
Assistant.Threads.Transferrer
Assistant.Threads.UpgradeWatcher
Assistant.Threads.Upgrader
Assistant.Threads.Watcher
Assistant.Threads.WebApp
Assistant.TransferQueue
Assistant.TransferSlots
Assistant.Types.Alert
Assistant.Types.BranchChange
Assistant.Types.Changes
Assistant.Types.Commits
Assistant.Types.CredPairCache
Assistant.Types.DaemonStatus
Assistant.Types.NamedThread
Assistant.Types.Pushes
Assistant.Types.RemoteControl
Assistant.Types.RepoProblem
Assistant.Types.ScanRemotes
Assistant.Types.ThreadName
Assistant.Types.ThreadedMonad
Assistant.Types.TransferQueue
Assistant.Types.TransferSlots
Assistant.Types.UrlRenderer
Assistant.Unused
Assistant.Upgrade
Assistant.WebApp
Assistant.WebApp.Common
Assistant.WebApp.Configurators
Assistant.WebApp.Configurators.AWS
Assistant.WebApp.Configurators.Delete
Assistant.WebApp.Configurators.Edit
Assistant.WebApp.Configurators.Fsck
Assistant.WebApp.Configurators.IA
Assistant.WebApp.Configurators.Local
Assistant.WebApp.Configurators.Pairing
Assistant.WebApp.Configurators.Preferences
Assistant.WebApp.Configurators.Ssh
Assistant.WebApp.Configurators.Unused
Assistant.WebApp.Configurators.Upgrade
Assistant.WebApp.Configurators.WebDAV
Assistant.WebApp.Control
Assistant.WebApp.DashBoard
Assistant.WebApp.Documentation
Assistant.WebApp.Form
Assistant.WebApp.Gpg
Assistant.WebApp.MakeRemote
Assistant.WebApp.Notifications
Assistant.WebApp.OtherRepos
Assistant.WebApp.Page
Assistant.WebApp.Pairing
Assistant.WebApp.Repair
Assistant.WebApp.RepoId
Assistant.WebApp.RepoList
Assistant.WebApp.SideBar
Assistant.WebApp.Types
Command.Assistant
Command.Watch
Command.WebApp
Utility.Mounts
Utility.OSX
Utility.Yesod
Utility.WebApp
if os(linux)
Build-Depends: hinotify (>= 0.3.10)
CPP-Options: -DWITH_INOTIFY
Other-Modules: Utility.DirWatcher.INotify
else
if os(darwin)
Build-Depends: hfsevents
CPP-Options: -DWITH_FSEVENTS
Other-Modules:
Utility.DirWatcher.FSEvents
else
if os(windows)
Build-Depends: Win32-notify
CPP-Options: -DWITH_WIN32NOTIFY
Other-Modules: Utility.DirWatcher.Win32Notify
else
if (! os(solaris) && ! os(gnu) && ! os(linux))
CPP-Options: -DWITH_KQUEUE
C-Sources: Utility/libkqueue.c
2017-10-24 16:54:33 +00:00
Includes: Utility/libkqueue.h
Other-Modules: Utility.DirWatcher.Kqueue
if flag(Dbus)
if (os(linux))
Build-Depends: dbus (>= 0.10.7), fdo-notify (>= 0.3)
CPP-Options: -DWITH_DBUS -DWITH_DESKTOP_NOTIFY -DWITH_DBUS_NOTIFICATIONS
Other-Modules: Utility.DBus
if flag(Pairing)
2012-09-07 23:44:20 +00:00
Build-Depends: network-multicast, network-info
CPP-Options: -DWITH_PAIRING
2012-09-07 18:54:00 +00:00
if flag(TorrentParser)
Build-Depends: torrent (>= 10000.0.0)
CPP-Options: -DWITH_TORRENTPARSER
if flag(MagicMime)
Build-Depends: magic
CPP-Options: -DWITH_MAGICMIME
add database benchmark The benchmark shows that the database access is quite fast indeed! And, it scales linearly to the number of keys, with one exception, getAssociatedKey. Based on this benchmark, I don't think I need worry about optimising for cases where all files are locked and the database is mostly empty. In those cases, database access will be misses, and according to this benchmark, should add only 50 milliseconds to runtime. (NB: There may be some overhead to getting the database opened and locking the handle that this benchmark doesn't see.) joey@darkstar:~/src/git-annex>./git-annex benchmark setting up database with 1000 setting up database with 10000 benchmarking keys database/getAssociatedFiles from 1000 (hit) time 62.77 μs (62.70 μs .. 62.85 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 62.81 μs (62.76 μs .. 62.88 μs) std dev 201.6 ns (157.5 ns .. 259.5 ns) benchmarking keys database/getAssociatedFiles from 1000 (miss) time 50.02 μs (49.97 μs .. 50.07 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.09 μs (50.04 μs .. 50.17 μs) std dev 206.7 ns (133.8 ns .. 295.3 ns) benchmarking keys database/getAssociatedKey from 1000 (hit) time 211.2 μs (210.5 μs .. 212.3 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 211.0 μs (210.7 μs .. 212.0 μs) std dev 1.685 μs (334.4 ns .. 3.517 μs) benchmarking keys database/getAssociatedKey from 1000 (miss) time 173.5 μs (172.7 μs .. 174.2 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 173.7 μs (173.0 μs .. 175.5 μs) std dev 3.833 μs (1.858 μs .. 6.617 μs) variance introduced by outliers: 16% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (hit) time 64.01 μs (63.84 μs .. 64.18 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 64.85 μs (64.34 μs .. 66.02 μs) std dev 2.433 μs (547.6 ns .. 4.652 μs) variance introduced by outliers: 40% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (miss) time 50.33 μs (50.28 μs .. 50.39 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.32 μs (50.26 μs .. 50.38 μs) std dev 202.7 ns (167.6 ns .. 252.0 ns) benchmarking keys database/getAssociatedKey from 10000 (hit) time 1.142 ms (1.139 ms .. 1.146 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.142 ms (1.140 ms .. 1.144 ms) std dev 7.142 μs (4.994 μs .. 10.98 μs) benchmarking keys database/getAssociatedKey from 10000 (miss) time 1.094 ms (1.092 ms .. 1.096 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.095 ms (1.095 ms .. 1.097 ms) std dev 4.277 μs (2.591 μs .. 7.228 μs)
2016-01-12 17:01:44 +00:00
if flag(Benchmark)
Build-Depends: criterion
add database benchmark The benchmark shows that the database access is quite fast indeed! And, it scales linearly to the number of keys, with one exception, getAssociatedKey. Based on this benchmark, I don't think I need worry about optimising for cases where all files are locked and the database is mostly empty. In those cases, database access will be misses, and according to this benchmark, should add only 50 milliseconds to runtime. (NB: There may be some overhead to getting the database opened and locking the handle that this benchmark doesn't see.) joey@darkstar:~/src/git-annex>./git-annex benchmark setting up database with 1000 setting up database with 10000 benchmarking keys database/getAssociatedFiles from 1000 (hit) time 62.77 μs (62.70 μs .. 62.85 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 62.81 μs (62.76 μs .. 62.88 μs) std dev 201.6 ns (157.5 ns .. 259.5 ns) benchmarking keys database/getAssociatedFiles from 1000 (miss) time 50.02 μs (49.97 μs .. 50.07 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.09 μs (50.04 μs .. 50.17 μs) std dev 206.7 ns (133.8 ns .. 295.3 ns) benchmarking keys database/getAssociatedKey from 1000 (hit) time 211.2 μs (210.5 μs .. 212.3 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 211.0 μs (210.7 μs .. 212.0 μs) std dev 1.685 μs (334.4 ns .. 3.517 μs) benchmarking keys database/getAssociatedKey from 1000 (miss) time 173.5 μs (172.7 μs .. 174.2 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 173.7 μs (173.0 μs .. 175.5 μs) std dev 3.833 μs (1.858 μs .. 6.617 μs) variance introduced by outliers: 16% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (hit) time 64.01 μs (63.84 μs .. 64.18 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 64.85 μs (64.34 μs .. 66.02 μs) std dev 2.433 μs (547.6 ns .. 4.652 μs) variance introduced by outliers: 40% (moderately inflated) benchmarking keys database/getAssociatedFiles from 10000 (miss) time 50.33 μs (50.28 μs .. 50.39 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 50.32 μs (50.26 μs .. 50.38 μs) std dev 202.7 ns (167.6 ns .. 252.0 ns) benchmarking keys database/getAssociatedKey from 10000 (hit) time 1.142 ms (1.139 ms .. 1.146 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.142 ms (1.140 ms .. 1.144 ms) std dev 7.142 μs (4.994 μs .. 10.98 μs) benchmarking keys database/getAssociatedKey from 10000 (miss) time 1.094 ms (1.092 ms .. 1.096 ms) 1.000 R² (1.000 R² .. 1.000 R²) mean 1.095 ms (1.095 ms .. 1.097 ms) std dev 4.277 μs (2.591 μs .. 7.228 μs)
2016-01-12 17:01:44 +00:00
CPP-Options: -DWITH_BENCHMARK
if flag(DebugLocks)
CPP-Options: -DDEBUGLOCKS
Other-Modules:
Annex
Annex.Action
Annex.AdjustedBranch
Annex.AdjustedBranch.Merge
Annex.AdjustedBranch.Name
Annex.AutoMerge
Annex.BloomFilter
Annex.Branch
Annex.Branch.Transitions
Annex.BranchState
Annex.CatFile
Annex.ChangedRefs
Annex.CheckAttr
Annex.CheckIgnore
Annex.Common
Annex.Concurrent
Annex.Concurrent.Utility
Annex.Content
Annex.Content.Presence
Annex.Content.Presence.LowLevel
2018-08-22 18:41:09 +00:00
Annex.Content.LowLevel
Annex.Content.PointerFile
Annex.CopyFile
Annex.CurrentBranch
Annex.Debug
Annex.Debug.Utility
Annex.Difference
Annex.DirHashes
Annex.Drop
Annex.Environment
Annex.Export
Annex.ExternalAddonProcess
Annex.FileMatcher
Annex.Fixup
Annex.GitOverlay
Annex.HashObject
Annex.Hook
Annex.Import
Annex.Ingest
Annex.Init
Annex.InodeSentinal
Annex.Journal
Annex.Link
Annex.Locations
Annex.LockFile
Annex.LockPool
2019-01-23 16:39:02 +00:00
Annex.Magic
Annex.MetaData
Annex.MetaData.StandardFields
Annex.Multicast
Annex.Notification
Annex.NumCopies
Annex.Path
Annex.Perms
2020-08-26 16:20:56 +00:00
Annex.PidLock
Annex.Queue
Annex.ReplaceFile
Annex.RemoteTrackingBranch
Annex.SpecialRemote
Annex.SpecialRemote.Config
Annex.Ssh
Annex.StallDetection
Annex.TaggedPush
Annex.Tmp
Annex.Transfer
Annex.TransferrerPool
Annex.UntrustedFilePath
Annex.UpdateInstead
Annex.UUID
Annex.Url
Annex.VariantFile
Annex.VectorClock
Annex.VectorClock.Utility
Annex.Verify
Annex.Version
Annex.View
Annex.View.ViewedFile
Annex.Wanted
Annex.WorkerPool
Annex.WorkTree
Annex.YoutubeDl
Backend
Backend.External
Backend.Hash
Backend.URL
Backend.Utilities
Backend.WORM
Benchmark
Build.BundledPrograms
Build.Configure
Build.DesktopFile
Build.Mans
Build.TestConfig
Build.Version
BuildInfo
BuildFlags
CmdLine
CmdLine.Action
CmdLine.Batch
CmdLine.GitAnnex
CmdLine.GitAnnex.Options
CmdLine.GitAnnexShell
CmdLine.GitAnnexShell.Checks
CmdLine.GitAnnexShell.Fields
CmdLine.AnnexSetter
CmdLine.Option
CmdLine.GitRemoteTorAnnex
CmdLine.Seek
CmdLine.Usage
Command
Command.Add
Command.AddUnused
Command.AddUrl
Command.Adjust
Command.Benchmark
Command.CalcKey
Command.CheckPresentKey
2017-01-31 21:27:44 +00:00
Command.Config
Command.ConfigList
Command.ContentLocation
Command.Copy
Command.Dead
Command.Describe
Command.DiffDriver
Command.Direct
Command.Drop
Command.DropKey
Command.DropUnused
Command.EnableRemote
2016-11-30 18:35:24 +00:00
Command.EnableTor
Command.ExamineKey
Command.Expire
2017-09-13 18:21:08 +00:00
Command.Export
Command.FilterBranch
Command.FilterProcess
Command.Find
Command.FindRef
Command.Fix
Command.Forget
Command.FromKey
Command.Fsck
Command.FuzzTest
Command.GCryptSetup
Command.Get
Command.Group
Command.GroupWanted
Command.Help
Command.Import
Command.ImportFeed
Command.InAnnex
Command.Indirect
Command.Info
Command.Init
Command.InitRemote
Command.Inprogress
Command.List
Command.Lock
Command.Log
Command.LookupKey
Command.Map
Command.MatchExpression
Command.Merge
Command.MetaData
Command.Migrate
Command.Mirror
Command.Move
Command.Multicast
Command.NotifyChanges
Command.NumCopies
Command.MinCopies
2016-11-30 18:35:24 +00:00
Command.P2P
Command.P2PStdIO
Command.PostReceive
Command.PreCommit
Command.Proxy
Command.ReKey
Command.ReadPresentKey
Command.RecvKey
Command.RegisterUrl
Command.Reinit
Command.Reinject
Command.RemoteDaemon
2019-04-15 17:05:44 +00:00
Command.RenameRemote
Command.Repair
Command.Required
Command.ResolveMerge
Command.Restage
Command.RmUrl
Command.Schedule
Command.Semitrust
Command.SendKey
Command.SetKey
Command.SetPresentKey
Command.Smudge
Command.Status
Command.Sync
Command.Test
Command.TestRemote
Command.Transferrer
Command.TransferKey
Command.TransferKeys
Command.Trust
Command.Unannex
Command.Undo
Command.Ungroup
Command.Uninit
Command.Unlock
Command.UnregisterUrl
Command.Untrust
Command.Unused
Command.Upgrade
Command.VAdd
Command.VCycle
Command.VFilter
Command.VPop
Command.Version
Command.Vicfg
Command.View
Command.Wanted
Command.Whereis
Command.WhereUsed
Common
Config
Config.Cost
Config.Files
Config.Files.AutoStart
Config.DynamicConfig
Config.GitConfig
Config.Smudge
Creds
Crypto
Database.Benchmark
Database.ContentIdentifier
Database.Export
Database.Fsck
Database.Handle
Database.Init
Database.Keys
Database.Keys.Handle
Database.Keys.Tables
Database.Keys.SQL
Database.Queue
Database.Types
Git
Git.AutoCorrect
Git.Branch
Git.BuildVersion
Git.CatFile
Git.CheckAttr
Git.CheckIgnore
Git.Command
Git.Command.Batch
Git.Config
Git.ConfigTypes
Git.Construct
2019-09-24 16:32:54 +00:00
Git.Credential
Git.CurrentRepo
Git.DiffTree
Git.DiffTreeItem
Git.Env
Git.FileMode
Git.FilePath
Git.Filename
Git.FilterProcess
Git.Fsck
Git.GCrypt
Git.HashObject
2019-04-24 18:55:49 +00:00
Git.History
Git.Hook
Git.Index
Git.LockFile
Git.LsFiles
Git.LsTree
Git.Merge
Git.Objects
Git.PktLine
Git.Queue
Git.Ref
Git.RefLog
Git.Remote
Git.Remote.Remove
Git.Repair
Git.Sha
Git.Ssh
Git.Status
Git.Tree
Git.Types
Git.UnionMerge
Git.UpdateIndex
Git.Url
Git.Version
2017-02-24 17:42:30 +00:00
Key
Limit
Limit.Wanted
Logs
Logs.Activity
Logs.Chunk
Logs.Chunk.Pure
2017-01-31 21:27:44 +00:00
Logs.Config
2019-02-20 21:22:56 +00:00
Logs.ContentIdentifier
Logs.ContentIdentifier.Pure
Logs.Difference
Logs.Difference.Pure
Logs.Export
Logs.Export.Pure
Logs.File
Logs.FsckResults
Logs.Group
2016-05-27 16:15:01 +00:00
Logs.Line
Logs.Location
Logs.MapLog
Logs.MetaData
Logs.MetaData.Pure
Logs.Multicast
Logs.NumCopies
Logs.PreferredContent
Logs.PreferredContent.Raw
Logs.Presence
Logs.Presence.Pure
Logs.Remote
Logs.Remote.Pure
Logs.RemoteState
add restage log When pointer files need to be restaged, they're first written to the log, and then when the restage operation runs, it reads the log. This way, if the git-annex process is interrupted before it can do the restaging, a later git-annex process can do it. Currently, this lets a git-annex get/drop command be interrupted and then re-ran, and as long as it gets/drops additional files, it will clean up after the interrupted command. But more changes are needed to make it easier to restage after an interrupted process. Kept using the git queue to run the restage action, even though the list of files that it builds up for that action is not actually used by the action. This could perhaps be simplified to make restaging a cleanup action that gets registered, rather than using the git queue for it. But I wasn't sure if that would cause visible behavior changes, when eg dropping a large number of files, currently the git queue flushes periodically, and so it restages incrementally, rather than all at the end. In restagePointerFiles, it reads the restage log twice, once to get the number of files and size, and a second time to process it. This seemed better than reading the whole file into memory, since potentially a huge number of files could be in there. Probably the OS will cache the file in memory and there will not be much performance impact. It might be better to keep running tallies in another file though. But updating that atomically with the log seems hard. Also note that it's possible for calcRestageLog to see a different file than streamRestageLog does. More files may be added to the log in between. That is ok, it will only cause the filterprocessfaster heuristic to operate with slightly out of date information, so it may make the wrong choice for the files that got added and be a little slower than ideal. Sponsored-by: Dartmouth College's DANDI project
2022-09-23 18:38:59 +00:00
Logs.Restage
Logs.Schedule
Logs.SingleValue
Logs.SingleValue.Pure
Logs.Smudge
Logs.Transfer
Logs.Transitions
Logs.Trust
Logs.Trust.Basic
Logs.Trust.Pure
Logs.UUID
Logs.UUIDBased
Logs.Unused
Logs.Upgrade
Logs.View
Logs.Web
Messages
Messages.Concurrent
Messages.Internal
Messages.JSON
Messages.Progress
Messages.Serialized
P2P.Address
P2P.Annex
2016-11-22 18:37:19 +00:00
P2P.Auth
2016-11-22 18:34:49 +00:00
P2P.IO
P2P.Protocol
Remote
Remote.Adb
Remote.BitTorrent
Remote.Borg
Remote.Bup
Remote.Ddar
Remote.Directory
Remote.Directory.LegacyChunked
Remote.External
Remote.External.AsyncExtension
Remote.External.Types
Remote.GCrypt
Remote.Git
Remote.GitLFS
Remote.Glacier
Remote.Helper.AWS
Remote.Helper.Chunked
Remote.Helper.Chunked.Legacy
Remote.Helper.Encryptable
2019-02-20 19:55:01 +00:00
Remote.Helper.ExportImport
Remote.Helper.Git
Remote.Helper.Hooks
Remote.Helper.Http
Remote.Helper.Messages
Remote.Helper.P2P
Remote.Helper.ReadOnly
Remote.Helper.ThirdPartyPopulated
Remote.Helper.Special
Remote.Helper.Ssh
Remote.HttpAlso
Remote.Hook
Remote.List
Remote.List.Util
Remote.P2P
Remote.Rsync
Remote.Rsync.RsyncUrl
Remote.S3
Remote.Tahoe
Remote.Web
Remote.WebDAV
Remote.WebDAV.DavLocation
RemoteDaemon.Common
RemoteDaemon.Core
RemoteDaemon.Transport
RemoteDaemon.Transport.GCrypt
2016-11-20 19:45:01 +00:00
RemoteDaemon.Transport.Tor
RemoteDaemon.Transport.Ssh
RemoteDaemon.Transport.Ssh.Types
RemoteDaemon.Types
Test
Test.Framework
Types
2016-08-08 14:41:54 +00:00
Types.ActionItem
Types.AdjustedBranch
Types.Availability
Types.Backend
Types.Benchmark
Types.BranchState
Types.CatFileHandles
Types.CleanupActions
Types.Command
Types.Concurrency
Types.Creds
Types.Crypto
Types.DeferredParse
Types.DesktopNotify
Types.Difference
Types.Distribution
2017-09-15 20:34:45 +00:00
Types.Export
Types.FileMatcher
Types.GitConfig
Types.Group
Types.Import
Types.IndexFiles
Types.Key
Types.KeySource
Types.Link
Types.LockCache
Types.Messages
Types.MetaData
Types.Mime
Types.NumCopies
Types.ProposedAccepted
Types.RefSpec
Types.Remote
Types.RemoteConfig
Types.RemoteState
Types.RepoVersion
Types.ScheduledActivity
Types.StandardGroups
Types.StallDetection
Types.StoreRetrieve
Types.Test
2016-08-08 14:41:54 +00:00
Types.Transfer
2020-12-09 17:28:16 +00:00
Types.Transferrer
Types.TransferrerPool
Types.Transitions
Types.TrustLevel
Types.UUID
Types.Upgrade
Types.UrlContents
Types.VectorClock
Types.View
Types.WorkerPool
Upgrade
Upgrade.V0
Upgrade.V1
Upgrade.V2
Upgrade.V3
Upgrade.V4
Upgrade.V5
Upgrade.V5.Direct
Upgrade.V6
Upgrade.V7
Upgrade.V8
Upgrade.V9
Utility.Aeson
Utility.Android
Utility.Applicative
Utility.Attoparsec
Utility.AuthToken
Utility.Base64
Utility.Batch
Utility.Bloom
Utility.CoProcess
Utility.CopyFile
Utility.Daemon
Utility.Data
Utility.DataUnits
Utility.Debug
Utility.DebugLocks
Utility.DirWatcher
Utility.DirWatcher.Types
Utility.Directory
Utility.Directory.Create
Utility.Directory.Stream
Utility.DiskFree
Utility.Dot
Utility.DottedVersion
Utility.Env
Utility.Env.Basic
Utility.Env.Set
Utility.Exception
Utility.FileMode
Utility.FileSize
Utility.FileSystemEncoding
Utility.Format
Utility.FreeDesktop
Utility.Glob
Utility.Gpg
Utility.Hash
Utility.HtmlDetect
Utility.HumanNumber
Utility.HumanTime
Utility.InodeCache
Utility.IPAddress
Utility.LockFile
Utility.LockPool
Utility.LockPool.LockHandle
Utility.LockPool.STM
Utility.LogFile
Utility.Lsof
Utility.MagicWormhole
Utility.Matcher
Utility.MD5
Utility.Metered
Utility.Misc
Utility.Monad
Utility.MoveFile
Utility.Network
Utility.NotificationBroadcaster
Utility.OptParse
Utility.PID
Utility.PartialPrelude
Utility.Path
Utility.Path.AbsRel
Utility.Path.Max
Utility.Path.Tests
Utility.Percentage
Utility.Process
Utility.Process.Shim
Utility.Process.Transcript
Utility.QuickCheck
Utility.RawFilePath
Utility.ResourcePool
Utility.Rsync
Utility.SafeCommand
Utility.Scheduled
Utility.Scheduled.QuickCheck
Utility.Shell
Utility.ShellEscape
Utility.SimpleProtocol
Utility.Split
Utility.SshConfig
Utility.SshHost
Utility.Su
Utility.SystemDirectory
2018-10-30 03:13:36 +00:00
Utility.TimeStamp
Utility.TList
Utility.Tense
Utility.ThreadLock
Utility.ThreadScheduler
Utility.Tmp
Utility.Tmp.Dir
Utility.Tor
Utility.Touch
Utility.Tuple
Utility.Url
Utility.UserInfo
Utility.Verifiable
if (os(windows))
Other-Modules:
Utility.LockFile.Windows
Utility.LockPool.Windows
else
Other-Modules:
Utility.LockFile.Posix
Utility.LockPool.Posix
Annex.LockPool.PosixOrPid
2019-07-22 13:21:41 +00:00
Utility.LockFile.LockStatus
Utility.LockFile.PidLock
Utility.LockPool.PidLock
Utility.LinuxMkLibs