From 764340e7a21b010e7367cbea601a8a7ebcab1549 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 11 Jun 2014 18:59:49 +0000 Subject: [PATCH 01/13] Added a comment --- ...omment_14_5b3bb068b62b12c7cc7504836a8acf32._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/bugs/FAT:_Date_resolution_for_mtime_2s--__62___implications/comment_14_5b3bb068b62b12c7cc7504836a8acf32._comment diff --git a/doc/bugs/FAT:_Date_resolution_for_mtime_2s--__62___implications/comment_14_5b3bb068b62b12c7cc7504836a8acf32._comment b/doc/bugs/FAT:_Date_resolution_for_mtime_2s--__62___implications/comment_14_5b3bb068b62b12c7cc7504836a8acf32._comment new file mode 100644 index 0000000000..e3efdb6d0e --- /dev/null +++ b/doc/bugs/FAT:_Date_resolution_for_mtime_2s--__62___implications/comment_14_5b3bb068b62b12c7cc7504836a8acf32._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="108.236.230.124" + subject="comment 14" + date="2014-06-11T18:59:49Z" + content=""" +Opened a separate bug for [[Windows_file_timestamp_timezone_madness]]. + +Fixed FAT issue on Linux as discussed in my previous comment. +"""]] From 398640f35e8884bbcc2e296cba4825249d3ecfd2 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 11 Jun 2014 19:09:17 +0000 Subject: [PATCH 02/13] Added a comment --- ...mment_1_9db59ab2242186a23a47337a1597f4e2._comment | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/bugs/Windows_file_timestamp_timezone_madness/comment_1_9db59ab2242186a23a47337a1597f4e2._comment diff --git a/doc/bugs/Windows_file_timestamp_timezone_madness/comment_1_9db59ab2242186a23a47337a1597f4e2._comment b/doc/bugs/Windows_file_timestamp_timezone_madness/comment_1_9db59ab2242186a23a47337a1597f4e2._comment new file mode 100644 index 0000000000..fbda5c2858 --- /dev/null +++ b/doc/bugs/Windows_file_timestamp_timezone_madness/comment_1_9db59ab2242186a23a47337a1597f4e2._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="108.236.230.124" + subject="comment 1" + date="2014-06-11T19:09:17Z" + content=""" +Rather than getting the timezone, another approach might be to look at the inode sentinal file. Its timestamp will also appear to have changed. If the delta is exactly some number of hours, and the inode sential's other data is unchanged, a Windows-specific hack could apply that same delta to all inode cache timestamps. + +Except, time zones are not all actually on hour boundaries. Some are half hours, some may be 15 minutes, and next week some crazy country might legislate a 3 minute delta for all I know. + +Well, could just say if the inode sentinal's mtime has changed at all (delta < 3600 seconds), and it's otherwise unchanged, and we're on windows, assume this is a time zone change. When would that fail? Only if the repository is copied to someplace, and the mtime is not preserved. +"""]] From aaf6b642551bb451ed5aa2a4b28389241fc4a12b Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawk7iPiqWr3BVPLWEDvJhSSvcOqheLEbLNo" Date: Wed, 11 Jun 2014 19:47:35 +0000 Subject: [PATCH 03/13] Added a comment --- ...ent_4_975d2631faa17d257a6fce40e24a6e3b._comment | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/bugs/git-annex:_createSession:_permission_denied___40__Operation_not_permitted__41__/comment_4_975d2631faa17d257a6fce40e24a6e3b._comment diff --git a/doc/bugs/git-annex:_createSession:_permission_denied___40__Operation_not_permitted__41__/comment_4_975d2631faa17d257a6fce40e24a6e3b._comment b/doc/bugs/git-annex:_createSession:_permission_denied___40__Operation_not_permitted__41__/comment_4_975d2631faa17d257a6fce40e24a6e3b._comment new file mode 100644 index 0000000000..5048cad9b2 --- /dev/null +++ b/doc/bugs/git-annex:_createSession:_permission_denied___40__Operation_not_permitted__41__/comment_4_975d2631faa17d257a6fce40e24a6e3b._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawk7iPiqWr3BVPLWEDvJhSSvcOqheLEbLNo" + nickname="Dirk" + subject="comment 4" + date="2014-06-11T19:47:35Z" + content=""" +For me the autobuilt tarball works. Go to this page: http://git-annex.branchable.com/install/Linux_standalone/. Right before the comments start is a list of the different autobuilt tar balls. + +I assume that this fix will appear in the regular tarballs with the next release. + +Dirk + + +"""]] From c09a2e9f0fe84c655c89465a0ad1195e96ed59fd Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 11 Jun 2014 19:51:15 +0000 Subject: [PATCH 04/13] Added a comment --- ..._a6a3871747306913b69abcd73d13305e._comment | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 doc/bugs/Windows_file_timestamp_timezone_madness/comment_2_a6a3871747306913b69abcd73d13305e._comment diff --git a/doc/bugs/Windows_file_timestamp_timezone_madness/comment_2_a6a3871747306913b69abcd73d13305e._comment b/doc/bugs/Windows_file_timestamp_timezone_madness/comment_2_a6a3871747306913b69abcd73d13305e._comment new file mode 100644 index 0000000000..c213aa0a92 --- /dev/null +++ b/doc/bugs/Windows_file_timestamp_timezone_madness/comment_2_a6a3871747306913b69abcd73d13305e._comment @@ -0,0 +1,26 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="108.236.230.124" + subject="comment 2" + date="2014-06-11T19:51:15Z" + content=""" +Note that multiple time zone changes complicate this. I think that means that the delta can't be simply applied when comparing inode caches. Instead, probably it needs to be applied when generating inode caches. + +A scenario: + +1. Time zone is at +1h when the inode sential is written. +2. Time zone changes to +2h +3. File F is added (with a current timestamp of T) +4. Time zone changes to +5h + +I am a little confused by which way windows moves the timestamps when the time zone changes. Let's assume I might get the sign wrong. + +Let F's timestamp after step 4, F4 = T+-3h. + +Let the delta after step 4, D4 = +-4h +And, let the delta after step 2, D2 = +-1h + +If step 3 writes the current timestamp to the inode cache, then the cache still has T in it after step 4. F4+D4 /= T (T +-3h +-4h /= T). So comparison doesn't work. + +If instead the current delta is applied when generating inode caches (both for storing on disk, and for immediate comparison), then the inode cache will have T+D2 in it. Then after step 4, generating a new inode cache for F will yield F4+D4. So, does F4+D4 == T+D2? T +-3h +-4h == T +-1h YES! +"""]] From 2c34730850f1e0315f2ed8fe4e37aa10781419f8 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 11 Jun 2014 19:56:49 +0000 Subject: [PATCH 05/13] Added a comment --- .../comment_4_c710e27db41311b157d8caaafc32dc7e._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/videos/git-annex_assistant_lan/comment_4_c710e27db41311b157d8caaafc32dc7e._comment diff --git a/doc/videos/git-annex_assistant_lan/comment_4_c710e27db41311b157d8caaafc32dc7e._comment b/doc/videos/git-annex_assistant_lan/comment_4_c710e27db41311b157d8caaafc32dc7e._comment new file mode 100644 index 0000000000..1035f6348c --- /dev/null +++ b/doc/videos/git-annex_assistant_lan/comment_4_c710e27db41311b157d8caaafc32dc7e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="108.236.230.124" + subject="comment 4" + date="2014-06-11T19:56:49Z" + content=""" +Running git-annex will also register it on OSX. The registration just consists of making a ~/.ssh/git-annex-shell that runs the real git-annex-shell. The assistant detects when it needs to use that wrapper when setting up a repository. +"""]] From fee38c834bef1adee411a3c498d572fd7bf96c0e Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 11 Jun 2014 20:09:14 +0000 Subject: [PATCH 06/13] Added a comment --- .../comment_4_62be3dd4092b15cdf85cf9a231b2863a._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/On_Ubuntu_14.04_assistant_fails_to_create_new_setup/comment_4_62be3dd4092b15cdf85cf9a231b2863a._comment diff --git a/doc/bugs/On_Ubuntu_14.04_assistant_fails_to_create_new_setup/comment_4_62be3dd4092b15cdf85cf9a231b2863a._comment b/doc/bugs/On_Ubuntu_14.04_assistant_fails_to_create_new_setup/comment_4_62be3dd4092b15cdf85cf9a231b2863a._comment new file mode 100644 index 0000000000..748b57df5c --- /dev/null +++ b/doc/bugs/On_Ubuntu_14.04_assistant_fails_to_create_new_setup/comment_4_62be3dd4092b15cdf85cf9a231b2863a._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="108.236.230.124" + subject="comment 4" + date="2014-06-11T20:09:14Z" + content=""" +What happens if you run `git-annex assistant --autostart` ? +"""]] From e48a363c3014f177f5a668510f65716b8ef88427 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 11 Jun 2014 22:11:39 +0000 Subject: [PATCH 07/13] Added a comment --- ..._7fe149bedb8ceab75953996ac8e20f0f._comment | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/bugs/Windows_file_timestamp_timezone_madness/comment_3_7fe149bedb8ceab75953996ac8e20f0f._comment diff --git a/doc/bugs/Windows_file_timestamp_timezone_madness/comment_3_7fe149bedb8ceab75953996ac8e20f0f._comment b/doc/bugs/Windows_file_timestamp_timezone_madness/comment_3_7fe149bedb8ceab75953996ac8e20f0f._comment new file mode 100644 index 0000000000..1a25d3c829 --- /dev/null +++ b/doc/bugs/Windows_file_timestamp_timezone_madness/comment_3_7fe149bedb8ceab75953996ac8e20f0f._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="108.236.230.124" + subject="comment 3" + date="2014-06-11T22:11:38Z" + content=""" +I have implemented this on the loststamp git branch. It seems to work! + +However, it has a big problem: + + If the timezone changes while the assistant (or a long-running command) + runs, it won't notice, since it only checks the inode cache once, and + so will use the old delta for all new inode caches it generates for new + files it's added. Which will result in them seeming changed the next time + it runs. + +So, it would be really nice to be able to check the actual timezone instead. +But I suppose I can make the assistant poll the inode cache file, or check +it when adding a new file, or something like that. Bleagh. +"""]] From 0c40bf5a94ec33c7ed18529b180ce7759b167c17 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 11 Jun 2014 18:20:58 -0400 Subject: [PATCH 08/13] confirmed --- doc/bugs/Windows_file_timestamp_timezone_madness.mdwn | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/bugs/Windows_file_timestamp_timezone_madness.mdwn b/doc/bugs/Windows_file_timestamp_timezone_madness.mdwn index a90c58f24e..8b1f0ef8f9 100644 --- a/doc/bugs/Windows_file_timestamp_timezone_madness.mdwn +++ b/doc/bugs/Windows_file_timestamp_timezone_madness.mdwn @@ -14,3 +14,5 @@ Unfortunately, Data.Time.LocalTime.getCurrentTimeZone doesn't seem to really work on windows. It always returns a time zone 60 minutes from UTS in my tests, no matter what the zone really is. I need to test this more widely and file a GHC bug if appropriate. + +[[!tag confirmed]] From 2cc8a538158a8efd73e756ac5aff69f1b7a2c3bb Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 11 Jun 2014 22:54:48 +0000 Subject: [PATCH 09/13] Added a comment --- ...comment_4_c9e8c9997b7c3a82c14fc34af319382d._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/bugs/Windows_file_timestamp_timezone_madness/comment_4_c9e8c9997b7c3a82c14fc34af319382d._comment diff --git a/doc/bugs/Windows_file_timestamp_timezone_madness/comment_4_c9e8c9997b7c3a82c14fc34af319382d._comment b/doc/bugs/Windows_file_timestamp_timezone_madness/comment_4_c9e8c9997b7c3a82c14fc34af319382d._comment new file mode 100644 index 0000000000..d06df7b779 --- /dev/null +++ b/doc/bugs/Windows_file_timestamp_timezone_madness/comment_4_c9e8c9997b7c3a82c14fc34af319382d._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="108.236.230.124" + subject="comment 4" + date="2014-06-11T22:54:48Z" + content=""" +Getting the actual time zone on windows works better if you unset TZ first. + +But, a haskell program that polls the time zone fails to notice when it's changed. It only notices after being restarted. I have contacted the time library maintainer about this. +"""]] From 4b525889f1c185a550d83ff023b5fbd884e04436 Mon Sep 17 00:00:00 2001 From: "http://joeyh.name/" Date: Wed, 11 Jun 2014 23:02:28 +0000 Subject: [PATCH 10/13] Added a comment --- ..._0739426403f5bf9954acbc86ca0d11ea._comment | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 doc/bugs/Windows_file_timestamp_timezone_madness/comment_5_0739426403f5bf9954acbc86ca0d11ea._comment diff --git a/doc/bugs/Windows_file_timestamp_timezone_madness/comment_5_0739426403f5bf9954acbc86ca0d11ea._comment b/doc/bugs/Windows_file_timestamp_timezone_madness/comment_5_0739426403f5bf9954acbc86ca0d11ea._comment new file mode 100644 index 0000000000..ac1f090f51 --- /dev/null +++ b/doc/bugs/Windows_file_timestamp_timezone_madness/comment_5_0739426403f5bf9954acbc86ca0d11ea._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="108.236.230.124" + subject="comment 5" + date="2014-06-11T23:02:28Z" + content=""" +I've developed a fix for the time library. This patch has been sent to the author, hopefully it will get applied and then I can use getCurrentTImeZone. Note that git-annex would need to unset TZ first, which might be hard on windows. + +
+diff --git a/cbits/HsTime.c b/cbits/HsTime.c
+index cfafb27..86ca92a 100644
+--- a/cbits/HsTime.c
++++ b/cbits/HsTime.c
+@@ -8,6 +8,7 @@ long int get_current_timezone_seconds (time_t t,int* pdst,char const* * pname)
+     tzset();
+     struct tm* ptm = localtime_r(&t,&tmd);
+ #else
++    tzset();
+     struct tm* ptm = localtime(&t);
+ #endif
+     if (ptm)
+
+"""]] From d6098906f231af89a59fe1a5558b685c9ed36162 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Wed, 11 Jun 2014 19:06:20 -0400 Subject: [PATCH 11/13] devblog --- .../day_183__rubbing_sticks_together.mdwn | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 doc/devblog/day_183__rubbing_sticks_together.mdwn diff --git a/doc/devblog/day_183__rubbing_sticks_together.mdwn b/doc/devblog/day_183__rubbing_sticks_together.mdwn new file mode 100644 index 0000000000..ecab379167 --- /dev/null +++ b/doc/devblog/day_183__rubbing_sticks_together.mdwn @@ -0,0 +1,23 @@ +Spent all day on some horrible timestamp issues on legacy systems. + +On FAT, timestamps have a 2s granularity, which is ok, but then Linux adds +a temporary higher resolution cache, which is lost on unmount. This +confused git-annex since the mtimes seemed to change and it had to +re-checksum half the files to get unconfused, which was not good. +I found a way to use the inode sentinal file to detect when on FAT +and put in a workaround, without degrading git-annex everywhere else. + +On Windows, time zones are a utter disaster; it changes the mtime it reports +for files after the time zone has changed. Also there's a bug in the +haskell time library which makes it return old time zone data after a time +zone change. (I just finished developing a fix for that bug..) + +Left with nothing but a few sticks, I rubbed them together, and +actually found a way to deal with this problem too. Scary details in +[[bugs/Windows_file_timestamp_timezone_madness]]. While I've implemented +it, it's stuck on a branch until I find a way to make git-annex notice when +the timezone changes while it's running. + +---- + +Today's work was sponsored by Svenne Krap. From 5c5c5cda8572fc9ac3f2e5ca309c2336f55aadb2 Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Thu, 12 Jun 2014 14:21:53 -0400 Subject: [PATCH 12/13] unset TZ on Windows TZ gets set when opening a cygwin terminal. What I'm oberving is strange.. when TZ is set, even if it's set to the same thing as the system time zone, it seems to result in files showing with different mtimes than when TZ is not set. Having TZ set also prevents seeing the real system timezone, so let's kill it. --- git-annex.hs | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/git-annex.hs b/git-annex.hs index a96dd8cbd3..f1af0eea51 100644 --- a/git-annex.hs +++ b/git-annex.hs @@ -48,29 +48,38 @@ main = do isshell n = takeFileName n == "git-annex-shell" #ifdef mingw32_HOST_OS -{- On Windows, if HOME is not set, probe it and set it, re-execing - - git-annex with the new environment. - - +{- On Windows, if HOME is not set, probe it and set it. - This is a workaround for some Cygwin commands needing HOME to be set, - and for there being no known way to set environment variables on - Windows, except by passing an environment in each call to a program. - While ugly, this workaround is easier than trying to ensure HOME is set - in all calls to the affected programs. + - + - If TZ is set, unset it. + - TZ being set can interfere with workarounds for Windows timezone + - horribleness, and prevents getCurrentTimeZone from seeing the system + - time zone. + - + - Due to Windows limitations, have to re-exec git-annex with the new + - environment. -} winEnv :: ([String] -> IO ()) -> [String] -> IO () -winEnv a ps = go =<< getEnv "HOME" +winEnv a ps = do + e <- getEnvironment + home <- myHomeDir + let e' = wantedenv e home + if (e' /= e) + then do + cmd <- readProgramFile + (_, _, _, pid) <- createProcess (proc cmd ps) + { env = Just e' } + exitWith =<< waitForProcess pid + else a ps where - go (Just _) = a ps - go Nothing = do - home <- myHomeDir - putStrLn $ "** Windows hack; overrideing HOME to " ++ home - e <- getEnvironment - let eoverride = + wantedenv e home = delEntry "TZ" $ case lookup "HOME" e of + Nothing -> e + Just _ -> addEntries [ ("HOME", home) , ("CYGWIN", "nodosfilewarning") - ] - cmd <- readProgramFile - (_, _, _, pid) <- createProcess (proc cmd ps) - { env = Just $ e ++ eoverride } - exitWith =<< waitForProcess pid + ] e #endif From 9dd380cf3b70727c313ea4d16a516a169176c92e Mon Sep 17 00:00:00 2001 From: Joey Hess Date: Thu, 12 Jun 2014 15:17:32 -0400 Subject: [PATCH 13/13] this just went from horrible to insanely weird --- ...ndows_file_timestamp_timezone_madness.mdwn | 30 ++++++++++++------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/doc/bugs/Windows_file_timestamp_timezone_madness.mdwn b/doc/bugs/Windows_file_timestamp_timezone_madness.mdwn index 8b1f0ef8f9..b92935f4fb 100644 --- a/doc/bugs/Windows_file_timestamp_timezone_madness.mdwn +++ b/doc/bugs/Windows_file_timestamp_timezone_madness.mdwn @@ -5,14 +5,24 @@ appear to change. This means that after such a change, git-annex will see new mtimes, and want to re-checksum every file in the repo. -The best way to fix this seems to be to normalize the timestamp returned by -getFileStatus, which is relative to the current time zone, to be relative -to UTC. (As is always the case on Unix, of course). -However, to do that, I need to know the current timezone. - -Unfortunately, Data.Time.LocalTime.getCurrentTimeZone doesn't seem to really -work on windows. It always returns a time zone 60 minutes from UTS in my tests, -no matter what the zone really is. I need to test this more widely and file -a GHC bug if appropriate. - [[!tag confirmed]] + +> Update: Actually, I seem to have been getting confused by behavior of +> cygwin terminal setting TZ. That indeed led to timestamp changes when the +> time zone changed. I have made git-annex unset TZ to avoid this. +> +> Without TZ set, time stamps are actually stable across time zone changes. +> Ie, a simple program to read the time stamp of a file and print it +> always shows the same thing, before and after a timezone change. +> +> However, and here's where it gets truely ghastly: A program that stats a +> file in a loop will see its timestamp change when the timezone changes. +> I suspect this might be a bug in the Haskell RTS caching something it +> should not. Stopping and re-running the program gets back to the +> original timestamp. +> +> I have not tested DST changes, but it's hard to imagine it being any +> worse than the above behavior. +> +> So, that's insane then. We can't trust timestamps to be stable on windows +> when git-annex is running for a long period of time. --[[Joey]]