From 899334223f4c46307af6c34ae103971b5f37ce8f Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 6 Jun 2012 14:33:12 -0400 Subject: [PATCH 01/12] fixed 2 races! Only 1 serious race to go! --- doc/design/assistant/inotify.mdwn | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/doc/design/assistant/inotify.mdwn b/doc/design/assistant/inotify.mdwn index e7c61c68b1..079941f597 100644 --- a/doc/design/assistant/inotify.mdwn +++ b/doc/design/assistant/inotify.mdwn @@ -82,16 +82,14 @@ Many races need to be dealt with by this code. Here are some of them. * File is added and then replaced with another file before the annex add stages the symlink in git. - **Currently unfixed**; `git add` will be run on the new file, which is - not at all good when it's big. Could be dealt with by using `git - update-index` to manually put the symlink into the index without git + Now fixed; `git annex watch` avoids running `git add` because of this + race. Instead, it stages symlinks directly into the index, without looking at what's currently on disk. * Link is moved, fixed link is written by fix event, but then that is removed by the user and replaced with a file before the event finishes. - **Currently unfixed**: `git add` will be run on the file. Basically same - effect as previous race above. + Now fixed; same fix as previous race above. * File is removed and then re-added before the removal event starts. From 81d4991fa3b9b9a303d37eef407cc7dcdc997a94 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 6 Jun 2012 15:13:18 -0400 Subject: [PATCH 02/12] thoughts --- doc/design/assistant/inotify.mdwn | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/doc/design/assistant/inotify.mdwn b/doc/design/assistant/inotify.mdwn index 079941f597..28b3130e33 100644 --- a/doc/design/assistant/inotify.mdwn +++ b/doc/design/assistant/inotify.mdwn @@ -67,10 +67,18 @@ Many races need to be dealt with by this code. Here are some of them. **Currently unfixed**; This changes content in the annex, and fsck will later catch the inconsistency. - Possible fixes: Somehow track or detect if a file is open for write - by any processes. Or, when possible, making a copy on write copy - before adding the file would avoid this. Or, as a last resort, make - an expensive copy of the file and add that. + Possible fixes: + + * Somehow track or detect if a file is open for write by any processes. + * Or, when possible, making a copy on write copy before adding the file + would avoid this. + * Or, as a last resort, make an expensive copy of the file and add that. + * Tracking file opens and closes with inotify could tell if any other + processes have the file open. But there are problems.. It doesn't + seem to differentiate between files opened for read and for write. + And there would still be a race after the last close and before it's + injected into the annex, where it could be opened for write again. + Would need to detect that and undo the annex injection or something. * File is added and then replaced with another file before the annex add makes its symlink. From e17bc40c312ee04019bdd65cf992e0caa1819df1 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 6 Jun 2012 15:24:51 -0400 Subject: [PATCH 03/12] update --- doc/design/assistant/inotify.mdwn | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/doc/design/assistant/inotify.mdwn b/doc/design/assistant/inotify.mdwn index 28b3130e33..b963597c83 100644 --- a/doc/design/assistant/inotify.mdwn +++ b/doc/design/assistant/inotify.mdwn @@ -22,9 +22,7 @@ really useful, it needs to: - periodically auto-commit staged changes (avoid autocommitting when lots of changes are coming in) - tunable delays before adding new files, etc -- Coleasce related add/rm events. See commit - cbdaccd44aa8f0ca30afba23fc06dd244c242075 for some details of the problems - with doing this. +- coleasce related add/rm events for speed and less disk IO - don't annex `.gitignore` and `.gitattributes` files, but do auto-stage changes to them - configurable option to only annex files meeting certian size or From 8aa194bf806c4620f0794ff7a13084e328061cc8 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 6 Jun 2012 15:51:34 -0400 Subject: [PATCH 04/12] update --- doc/design/assistant/inotify.mdwn | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/design/assistant/inotify.mdwn b/doc/design/assistant/inotify.mdwn index b963597c83..0f4e8d28db 100644 --- a/doc/design/assistant/inotify.mdwn +++ b/doc/design/assistant/inotify.mdwn @@ -23,10 +23,10 @@ really useful, it needs to: lots of changes are coming in) - tunable delays before adding new files, etc - coleasce related add/rm events for speed and less disk IO -- don't annex `.gitignore` and `.gitattributes` files, but do auto-stage - changes to them +- don't annex `.gitignore` and `.gitattributes` files **done** - configurable option to only annex files meeting certian size or filename criteria +- option to check files not meeting annex criteria into git directly - honor .gitignore, not adding files it excludes (difficult, probably needs my own .gitignore parser to avoid excessive running of git commands to check for ignored files) From a79aebbe2a15a05f7475b08d98279bb88ef07305 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 6 Jun 2012 16:54:39 -0400 Subject: [PATCH 05/12] update --- doc/design/assistant/inotify.mdwn | 5 +++-- doc/design/assistant/webapp.mdwn | 2 ++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/doc/design/assistant/inotify.mdwn b/doc/design/assistant/inotify.mdwn index 0f4e8d28db..ab88210b2b 100644 --- a/doc/design/assistant/inotify.mdwn +++ b/doc/design/assistant/inotify.mdwn @@ -19,6 +19,9 @@ really useful, it needs to: - notice deleted files and stage the deletion (tricky; there's a race with add since it replaces the file with a symlink..) **done** +- Gracefully handle when the default limit of 8192 inotified directories + is exceeded. This can be tuned by root, so help the user fix it. + **done** - periodically auto-commit staged changes (avoid autocommitting when lots of changes are coming in) - tunable delays before adding new files, etc @@ -32,8 +35,6 @@ really useful, it needs to: to check for ignored files) - Possibly, when a directory is moved out of the annex location, unannex its contents. -- Gracefully handle when the default limit of 8192 inotified directories - is exceeded. This can be tuned by root, so help the user fix it. - Support OSes other than Linux; it only uses inotify currently. OSX and FreeBSD use the same mechanism, and there is a Haskell interface for it, diff --git a/doc/design/assistant/webapp.mdwn b/doc/design/assistant/webapp.mdwn index abf7b38c94..598c1ff3a4 100644 --- a/doc/design/assistant/webapp.mdwn +++ b/doc/design/assistant/webapp.mdwn @@ -23,6 +23,8 @@ The webapp is a web server that displays a shiny interface. * there could be a UI to export a file, which would make it be served up over http by the web app +* Display any relevant warning messages. One is the `inotify max_user_watches` + exceeded message. ## implementation From baf9c7102ec58b3b31ef9d6a9424cf5e749bcc04 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 6 Jun 2012 16:56:48 -0400 Subject: [PATCH 06/12] blog for the day --- .../assistant/blog/day_3__more_races.mdwn | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 doc/design/assistant/blog/day_3__more_races.mdwn diff --git a/doc/design/assistant/blog/day_3__more_races.mdwn b/doc/design/assistant/blog/day_3__more_races.mdwn new file mode 100644 index 0000000000..9c11828420 --- /dev/null +++ b/doc/design/assistant/blog/day_3__more_races.mdwn @@ -0,0 +1,26 @@ +Today I worked on the race conditions, and fixed two of them. Both +were fixed by avoiding using `git add`, which looks at the files currently +on disk. Instead, `git annex watch` injects symlinks directly into git's +index, using `git update-index`. + +There is one bad race condition remaining. If multiple processes have a +file open for write, one can close it, and it will be added to the annex. +But then the other can still write to it. + +---- + +Getting away from race conditions for a while, I made `git annex watch` +not annex `.gitignore` and `.gitattributes` files. + +And, I made it handle running out of inotify descriptors. By default, +`/proc/sys/fs/inotify/max_user_watches` is 8192, and that's how many +directories inotify can watch. Now when it needs more, it will print +a nice message showing how to increase it with `sysctl`. + +FWIW, DropBox also uses inotify and has the same limit. It seems to not +tell the user how to fix it when it goes over. Here's what `git annex +watch` will say: + + Too many directories to watch! (Not watching ./dir4299) + Increase the limit by running: + echo fs.inotify.max_user_watches=81920 | sudo tee -a /etc/sysctl.conf; sudo sysctl -p From 9ee59f62d56ae8bdbe6a4600a6274108d89b0187 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawkmtR6oVColYKoU0SjBORLDGrwR10G-mKo" Date: Wed, 6 Jun 2012 22:03:29 +0000 Subject: [PATCH 07/12] Added a comment: Dropbox Inotify --- .../comment_1_d6015338f602b574a3805de5481fc45e._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/design/assistant/blog/day_3__more_races/comment_1_d6015338f602b574a3805de5481fc45e._comment diff --git a/doc/design/assistant/blog/day_3__more_races/comment_1_d6015338f602b574a3805de5481fc45e._comment b/doc/design/assistant/blog/day_3__more_races/comment_1_d6015338f602b574a3805de5481fc45e._comment new file mode 100644 index 0000000000..2d330f3327 --- /dev/null +++ b/doc/design/assistant/blog/day_3__more_races/comment_1_d6015338f602b574a3805de5481fc45e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkmtR6oVColYKoU0SjBORLDGrwR10G-mKo" + nickname="Jo-Herman" + subject="Dropbox Inotify" + date="2012-06-06T22:03:29Z" + content=""" +Actually, Dropbox giver you a warning via libnotify inotify. It tends to go away too quickly to properly read though, much less actually copy down the command... +"""]] From 54f5462ad4a6f3e95a76d008f5eabbff988e7f9f Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 6 Jun 2012 23:25:57 +0000 Subject: [PATCH 08/12] Added a comment --- .../comment_2_4d6b23fc6442e0ee0303523cb69d0fba._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/design/assistant/blog/day_3__more_races/comment_2_4d6b23fc6442e0ee0303523cb69d0fba._comment diff --git a/doc/design/assistant/blog/day_3__more_races/comment_2_4d6b23fc6442e0ee0303523cb69d0fba._comment b/doc/design/assistant/blog/day_3__more_races/comment_2_4d6b23fc6442e0ee0303523cb69d0fba._comment new file mode 100644 index 0000000000..523e6d85ff --- /dev/null +++ b/doc/design/assistant/blog/day_3__more_races/comment_2_4d6b23fc6442e0ee0303523cb69d0fba._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.252.8.36" + subject="comment 2" + date="2012-06-06T23:25:57Z" + content=""" +When I work on the [[webapp]], I'm planning to make it display this warning, and any other similar warning messages that might come up. +"""]] From c7efb2888cb612ea7635d8840d22ab1b38a8cb80 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawnBJ6Dv1glxzzi4qIzGFNa6F-mfHIvv9Ck" Date: Thu, 7 Jun 2012 03:43:19 +0000 Subject: [PATCH 09/12] Added a comment: Wording --- ...ent_3_03f5b2344c2a47dea60086f217d60f9b._comment | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/design/assistant/blog/day_3__more_races/comment_3_03f5b2344c2a47dea60086f217d60f9b._comment diff --git a/doc/design/assistant/blog/day_3__more_races/comment_3_03f5b2344c2a47dea60086f217d60f9b._comment b/doc/design/assistant/blog/day_3__more_races/comment_3_03f5b2344c2a47dea60086f217d60f9b._comment new file mode 100644 index 0000000000..92f5dcbd62 --- /dev/null +++ b/doc/design/assistant/blog/day_3__more_races/comment_3_03f5b2344c2a47dea60086f217d60f9b._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnBJ6Dv1glxzzi4qIzGFNa6F-mfHIvv9Ck" + nickname="Jim" + subject="Wording" + date="2012-06-07T03:43:19Z" + content=""" +For the unfamiliar, it's hard to tell if a command like that would persist. I'd suggest being as clear as possible, e.g.: + + Increase the limit for now by running: + sudo sysctl fs.inotify.max_user_watches=81920 + Increase the limit now and automatically at every boot by running: + echo fs.inotify.max_user_watches=81920 | sudo tee -a /etc/sysctl.conf; sudo sysctl -p + +"""]] From 2478aca3c531d6d60fed87d9aa5eb42f6dbe6517 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Thu, 7 Jun 2012 04:48:15 +0000 Subject: [PATCH 10/12] Added a comment --- .../comment_4_860e90e989ec022100001c65e353a91e._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/design/assistant/blog/day_3__more_races/comment_4_860e90e989ec022100001c65e353a91e._comment diff --git a/doc/design/assistant/blog/day_3__more_races/comment_4_860e90e989ec022100001c65e353a91e._comment b/doc/design/assistant/blog/day_3__more_races/comment_4_860e90e989ec022100001c65e353a91e._comment new file mode 100644 index 0000000000..05b601eafe --- /dev/null +++ b/doc/design/assistant/blog/day_3__more_races/comment_4_860e90e989ec022100001c65e353a91e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.252.8.36" + subject="comment 4" + date="2012-06-07T04:48:15Z" + content=""" +Good thought Jim. I've done something like that. +"""]] From 8408a91082f440e139884117b2698bb8e0bd3fe9 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Thu, 7 Jun 2012 10:37:06 -0400 Subject: [PATCH 11/12] fixie --- doc/tips/using_box.com_as_a_special_remote.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tips/using_box.com_as_a_special_remote.mdwn b/doc/tips/using_box.com_as_a_special_remote.mdwn index 917c7a93bd..cafbc033c4 100644 --- a/doc/tips/using_box.com_as_a_special_remote.mdwn +++ b/doc/tips/using_box.com_as_a_special_remote.mdwn @@ -24,7 +24,7 @@ With a little setup, git-annex can use Box as a * Create `~/.davfs2/davfs2.conf` with some important settings: mkdir ~/.davfs2/ - echo use_locks 0 >> ~/.davfs2/davfs2.conf + echo use_locks 0 > ~/.davfs2/davfs2.conf echo cache_size 1 >> ~/.davfs2/davfs2.conf echo delay_upload 0 >> ~/.davfs2/davfs2.conf From 4d1c114e4d27ae339f6fb408d398945fa68c2435 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Thu, 7 Jun 2012 11:16:48 -0400 Subject: [PATCH 12/12] initremote: Automatically describe a remote when creating it. This ensures that all special remotes show up in git annex status. Before, a special remote that was not manually described, and was not a current git remote, did not show up there, although initremote did list it. --- Command/InitRemote.hs | 2 ++ Remote.hs | 7 ++----- debian/changelog | 1 + 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Command/InitRemote.hs b/Command/InitRemote.hs index 698d604552..a78505a197 100644 --- a/Command/InitRemote.hs +++ b/Command/InitRemote.hs @@ -15,6 +15,7 @@ import qualified Remote import qualified Logs.Remote import qualified Types.Remote as R import Annex.UUID +import Logs.UUID def :: [Command] def = [command "initremote" @@ -60,6 +61,7 @@ findByName name = do where generate = do uuid <- liftIO genUUID + describeUUID uuid name return (uuid, M.insert nameKey name M.empty) findByName' :: String -> M.Map UUID R.RemoteConfig -> Maybe (UUID, R.RemoteConfig) diff --git a/Remote.hs b/Remote.hs index e9e66990c5..839c6ddb09 100644 --- a/Remote.hs +++ b/Remote.hs @@ -54,9 +54,9 @@ remoteMap :: (Remote -> a) -> Annex (M.Map UUID a) remoteMap c = M.fromList . map (\r -> (uuid r, c r)) . filter (\r -> uuid r /= NoUUID) <$> remoteList -{- Map of UUIDs and their descriptions. +{- Map of UUIDs of remotes and their descriptions. - The names of Remotes are added to suppliment any description that has - - been set for a repository. -} + - been set for a repository. -} uuidDescriptions :: Annex (M.Map UUID String) uuidDescriptions = M.unionWith addName <$> uuidMap <*> remoteMap name @@ -101,9 +101,6 @@ nameToUUID n = byName' n >>= go double (a, _) = (a, a) {- Pretty-prints a list of UUIDs of remotes, for human display. - - - - Shows descriptions from the uuid log, falling back to remote names, - - as some remotes may not be in the uuid log. - - When JSON is enabled, also generates a machine-readable description - of the UUIDs. -} diff --git a/debian/changelog b/debian/changelog index 9a010327df..8a734e0aa7 100644 --- a/debian/changelog +++ b/debian/changelog @@ -2,6 +2,7 @@ git-annex (3.20120606) UNRELEASED; urgency=low * add: Prevent (most) modifications from being made to a file while it is being added to the annex. + * initremote: Automatically describe a remote when creating it. -- Joey Hess Tue, 05 Jun 2012 20:25:51 -0400