Merge branch 'master' into windows

Conflicts:
	Annex/Environment.hs
	Build/Configure.hs
	Git/Construct.hs
	Utility/FileMode.hs
This commit is contained in:
Joey Hess 2013-05-14 15:37:24 -04:00
commit dc66b1f27d
51 changed files with 678 additions and 87 deletions

View file

@ -82,17 +82,15 @@ testCp k option = TestCase cmd $ testCmd k cmdline
cmd = "cp " ++ option
cmdline = cmd ++ " " ++ testFile ++ " " ++ testFile ++ ".new"
{- For release builds, VERSION_FROM_CHANGELOG makes it use just the most
- recent version from the changelog. -}
isReleaseBuild :: IO Bool
isReleaseBuild = isJust <$> catchMaybeIO (getEnv "VERSION_FROM_CHANGELOG")
isReleaseBuild = isJust <$> catchMaybeIO (getEnv "RELEASE_BUILD")
{- Version is usually based on the major version from the changelog,
- plus the date of the last commit, plus the git rev of that commit.
- This works for autobuilds, ad-hoc builds, etc.
-
- If git or a git repo is not available, or something goes wrong,
- just use the version from the changelog. -}
- or this is a release build, just use the version from the changelog. -}
getVersion :: Test
getVersion = do
changelogversion <- getChangelogVersion

View file

@ -52,15 +52,20 @@ upgradableKey backend key = isNothing (Types.Key.keySize key) || backendupgradab
{- Store the old backend's key in the new backend
- The old backend's key is not dropped from it, because there may
- be other files still pointing at that key. -}
- be other files still pointing at that key.
-
- To ensure that the data we have for the old key is valid, it's
- fscked here. First we generate the new key. This ensures that the
- data cannot get corrupted after the fsck but before the new key is
- generated.
-}
perform :: FilePath -> Key -> Backend -> Backend -> CommandPerform
perform file oldkey oldbackend newbackend = do
ifM (Command.Fsck.checkBackend oldbackend oldkey (Just file))
( maybe stop go =<< genkey
, stop
)
perform file oldkey oldbackend newbackend = go =<< genkey
where
go newkey = stopUnless (Command.ReKey.linkKey oldkey newkey) $
go Nothing = stop
go (Just newkey) = stopUnless checkcontent $ finish newkey
checkcontent = Command.Fsck.checkBackend oldbackend oldkey $ Just file
finish newkey = stopUnless (Command.ReKey.linkKey oldkey newkey) $
next $ Command.ReKey.cleanup file oldkey newkey
genkey = do
content <- calcRepo $ gitAnnexLocation oldkey

View file

@ -23,6 +23,8 @@ module Git.Construct (
checkForRepo,
) where
{-# LANGUAGE CPP #-}
#ifndef __WINDOWS__
import System.Posix.User
#endif
@ -143,6 +145,9 @@ fromRemoteLocation :: String -> Repo -> IO Repo
fromRemoteLocation s repo = gen $ calcloc s
where
gen v
#ifdef __WINDOWS__
| dosstyle v = fromRemotePath v repo
#endif
| scpstyle v = fromUrl $ scptourl v
| urlstyle v = fromUrl v
| otherwise = fromRemotePath v repo
@ -176,6 +181,11 @@ fromRemoteLocation s repo = gen $ calcloc s
| "/" `isPrefixOf` d = d
| "~" `isPrefixOf` d = '/':d
| otherwise = "/~/" ++ d
#ifdef __WINDOWS__
-- git on Windows will write a path to .git/config with "drive:",
-- which is not to be confused with a "host:"
dosstyle = hasDrive
#endif
{- Constructs a Repo from the path specified in the git remotes of
- another Repo. -}

3
debian/changelog vendored
View file

@ -1,6 +1,8 @@
git-annex (4.20130502) UNRELEASED; urgency=low
* Android: The webapp is ported and working.
* Windows: There is a very rough Windows port. Do not trust it with
important data.
* git-annex-shell: Ensure that received files can be read. Files
transferred from some Android devices may have very broken permissions
as received.
@ -21,6 +23,7 @@ git-annex (4.20130502) UNRELEASED; urgency=low
* rsync special remotes: When sending from a crippled filesystem, use
the destination's default file permissions, as the local ones can
be arbitrarily broken. (Ie, ----rwxr-x for files on Android)
* migrate: Detect if a file gets corrupted while it's being migrated.
-- Joey Hess <joeyh@debian.org> Thu, 02 May 2013 20:39:19 -0400

2
debian/rules vendored
View file

@ -4,7 +4,7 @@
export CABAL=./Setup
# Do use the changelog's version number, rather than making one up.
export VERSION_FROM_CHANGELOG=1
export RELEASE_BUILD=1
%:
dh $@

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://openid.fmarier.org/"
nickname="fmarier"
subject="Make sure your network is correctly set up in the first place!"
date="2013-05-12T08:37:38Z"
content="""
It turns out that my internal network had badly broken IPv6 configs between the boxes. That was interfering with the multicast packets but not with anything else since all of my other internal traffic is over IPv4.
Getting rid of these static IPv6 addresses has solved my problem. Local pairing is now working as advertised :)
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 8"
date="2013-05-13T17:22:17Z"
content="""
I don't understand how IPv6 could affect local pairing. The assistant does not currently do pairing over ipv6 due to a limitation in the multicast library it's using.
"""]]

View file

@ -0,0 +1,34 @@
I have some git repositories I don't edit often that I'd like to back up. I'd like to add these to my git annex, so I don't have to resort to a time-consuming hack (such as setting up a proper submodule, or bundling the repositories).
But when I try to add a .git directory to git annex, I get a bunch of errors of the form
git-annex: user error (xargs ["-0","git","--git-dir=/tmp/tmp.LhGN3nT9uM/annex/.git","--work-tree=/tmp/tmp.LhGN3nT9uM/annex","add","--"] exited 123)
failed
git-annex: add: 1 failed
add repo/.git/hooks/pre-push.sample ok
(Recording state in git...)
error: Invalid path 'repo/.git/hooks/pre-push.sample'
error: unable to add repo/.git/hooks/pre-push.sample to index
fatal: adding files failed
STR:
$ mkdir annex
$ cd annex
annex$ git init
annex$ git annex init
annex$ cd ..
$ mkdir repo
$ cd repo
repo$ git init
repo$ cd ..
$ mv repo annex
$ cd annex
annex$ find repo | xargs -n1 git annex add
# Lots of errors of the form above.
You can't simply do `git annex add repo` because that will ignore the .git directory. Similarly,` git annex add .git` (which I'd think really should try to add the contents of the .git directory) ignores everything.
I don't know what this error means. Is there a right way to work around this?

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 1"
date="2013-05-13T17:36:31Z"
content="""
It's a fairly fundamental limitation of git that you cannot check `.git` directories into a git repository.
For backups and sneakernet transfers, `git bundle` is easy to use..
"""]]

View file

@ -0,0 +1,14 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
nickname="Justin"
subject="comment 2"
date="2013-05-13T17:53:06Z"
content="""
Be that as it may, the whole reason git-annex exists is to work around fundamental limitations of git!
The issue is that I don't want to treat a folder which I happen to have applied version control to differently than a folder which happens not to be version controlled (aside from committing to the version-controlled folder, of course!). Both folders are in my git annex; I shouldn't have to worry about it. (My whole \"documents\" folder is in git annex, and it contains many small git repositories.)
I guess I could write a script to unbundle and re-bundle on command. In fact, one could imagine integrating these scripts into git annex somehow.
Is that something you'd consider taking upstream?
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 3"
date="2013-05-13T18:08:36Z"
content="""
Well, git-annex is all about working around *1* limitation of git. There are several other limitations that it would be nice to have tools to deal with better, including storing metadata, and this one. I can't take everything on.
I mostly worry about these limitations in the context of naive users using the git-annex assistant for file sync. I have heard that some people put their git repositories in dropbox.
You don't sound at all naive, so I'll suggest another tool I wrote, mr <http://joeyh.name/code/mr>. With mr you can run a single command and operate on all repositories at or under a directory.
"""]]

View file

@ -0,0 +1,50 @@
### Please describe the problem.
I have a git annex repo which cannot find the files with whereis, even though the files and contents are there. I have changed ownership of all the files. I am not sure, but I think that is when the problem was introduced. The current user that is invoking git annex owns and can access all files in the repository/annex)
Creating a new repository from scratch works just fine.
### What steps will reproduce the problem?
# (in my current, somehow corrupt annex)
$ echo hello > testfile
$ git annex add testfile
add testfile (checksum...) ok
(Recording state in git...)
$ git commit -am testfile
[master 73ed120] testfile
1 file changed, 1 insertion(+)
create mode 120000 testfile
$ git annex whereis testfile
whereis testfile (0 copies) failed
git-annex: whereis: 1 failed
# The contents exists though
$ ls -l testfile
lrwxrwxrwx 1 ftp ftp 176 May 13 09:43 testfile -> .git/annex/objects/P5/4q/SHA256-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03/SHA256-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03
$ cat .git/annex/objects/P5/4q/SHA256-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03/SHA256-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03
hello
$ sha256sum testfile
5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03 testfile
# the file can be found when unlocking/locking
$ git annex unlock testfile
unlock testfile (copying...) ok
$ git annex lock testfile
lock testfile ok
(Recording state in git...)
### What version of git-annex are you using? On what operating system?
I ran Debian squeeze, with git annex 3.20120629~bpo60+2 when the problem was introduced. I just upgraded to wheezy, but the same problem exists with 3.20120629 from wheezy.
I also manually installed 4.20130501 from unstable, which also showed the same problem.
### Please provide any additional information below.
I am not sure what information to supply, please provide pointers on what information might be useful.
> [[done]] per comment --[[Joey]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawljcHBnBhazpDpk5k_9Wk9S_zA0hnjqdLQ"
nickname="Andreas"
subject="comment 1"
date="2013-05-13T08:24:50Z"
content="""
Sorry for the noise - it turned out the local annex repository somehow had been marked as \"dead\" (as shown by git status).
The fix was simply to git annex semitrust <repo uuid> and it went back to life.
I am not sure how this bug tracker works, but this bug report can be closed as invalid. Thanks for git annex!
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 1"
date="2013-05-13T19:02:38Z"
content="""
I can't think of a way to make the assistant detect that `~/.gitconfig` contains settings not supported by the version of git bundled with git-annex. Even if I could detect it, what could it do? It could perhaps prevent the config file being used by git-annex, but that could have other unwanted consequences.
An error message like this seems about as good as this situation can be handled. However, the version of git bundled with git-annex should be kept up-to-date to prevent this kind of problem as much as possible. I don't currently have a good way to ensure that happens; it takes whatever version of git is installed on the build system.
I'm going to try to get the autobuilders all updated to git 1.8.2, at least.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 2"
date="2013-05-13T19:08:09Z"
content="""
Done for both Linux autobuilds. OSX still needs updating; emailed the owners of those builds.
"""]]

View file

@ -0,0 +1,25 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA"
nickname="Laszlo"
subject="comment 11"
date="2013-05-11T05:36:48Z"
content="""
rechecksuming: it seem like it is indeed fixed in the newest (2013.05.01) version downloaded from here:
http://downloads.kitenet.net/git-annex/linux/
I tried to add the big stress test dir as a secondary repository into git-annex (along with my real data dir), but
seems like some library is not matching on my system, so some curl is complaining:
curl: /lib/tls/i686/cmov/libc.so.6: version `GLIBC_2.12' not found (required by /home/user/Desktop/down/git-annex.linux//usr/lib/i386-linux-gnu/libldap_r-2.4.so.2)
I'm on ubuntu 10.04.
And the log file is starting to fill up, so maybe once a problem occur, it should only write into the log file once.
I will redone this stress test next week, without combining with any repository.
Thank you very much for your response, I do appreciate you are bothering/dealing with my complains!:)
Best,
Laszlo
"""]]

View file

@ -0,0 +1,19 @@
[[!comment format=mdwn
username="Xyem"
ip="87.194.19.134"
subject="comment 8"
date="2013-05-12T22:59:37Z"
content="""
Just reproduced this issue on 4.20130417-g4bb97d5. Looks like the digesting is the failing part, but no indication as to why.
[2013-05-12 23:23:21 BST] Watcher: file deleted test/SleepIsDeath_v16_UnixSource.tar.gz-{b8c941eb-baf0-46de-81fa-19d25aca05fb}.dtapart
[2013-05-12 23:23:21 BST] read: git [\"--git-dir=/home/xyem/annex/.git\",\"--work-tree=/home/xyem/annex\",\"ls-files\",\"--others\",\"--exclude-standard\",\"-z\",\"--\",\"test/SleepIsDeath_v16_UnixSource.tar.gz-{b8c941eb-baf0-46de-81fa-19d25aca05fb}.dtapart\",\"test/SleepIsDeath_v16_UnixSource.tar.gz\"]
[2013-05-12 23:23:21 BST] read: lsof [\"-F0can\",\"+d\",\"/home/xyem/annex/.git/annex/tmp/\"]
[2013-05-12 23:23:22 BST] Committer: Adding SleepIsDe..ce.tar.gz
add test/SleepIsDeath_v16_UnixSource.tar.gz (checksum...) [2013-05-12 23:23:22 BST] read: sha256sum [\"/home/xyem/annex/.git/annex/tmp/SleepIsDeath_v16_UnixSource.tar6479.gz\"]
failed
[2013-05-12 23:23:22 BST] Committer: committing 1 changes
[2013-05-12 23:23:22 BST] Committer: Committing changes to git
(Recording state in git...)
"""]]

View file

@ -0,0 +1,44 @@
[[!comment format=mdwn
username="Xyem"
ip="87.194.19.134"
subject="comment 9"
date="2013-05-12T23:34:51Z"
content="""
Whoops. Upgraded to 4.20130501-g4a5bfb3 and reproduced it again. Looks like the file is being changed between the file being closed and git-annex trying to add it.
Though I'm using DownThemAll, I have it set to \"1 segment\" so the file should only have 1 writer.
Looks like the file gets retried immediately and is being added okay though.
Would it be worth me trying to log filesystem accesses with a passthrough FUSE filesystem to try and figure out what it going on?
O'Reilly/Mind Performance Hacks/urn_x-domain_oreilly.com_product_9780596153113.EBOOK-{711cd6ac-12b9-4d4e-92d6-66caa9825cf4}.dtapart still has writers, not adding
[2013-05-13 00:18:29 BST] read: git [\"--git-dir=/home/xyem/annex/.git\",\"--work-tree=/home/xyem/annex\",\"ls-tree\",\"-z\",\"--\",\"refs/heads/git-annex\",\"uuid.log\",\"remote.log\",\"trust.log\",\"group.log\",\"preferred-content.log\"]
[2013-05-13 00:18:33 BST] Watcher: file deleted O'Reilly/Mind Performance Hacks/urn_x-domain_oreilly.com_product_9780596153113.EBOOK-{711cd6ac-12b9-4d4e-92d6-66caa9825cf4}.dtapart
[2013-05-13 00:18:33 BST] read: git [\"--git-dir=/home/xyem/annex/.git\",\"--work-tree=/home/xyem/annex\",\"ls-files\",\"--others\",\"--exclude-standard\",\"-z\",\"--\",\"O'Reilly/Mind Performance Hacks/urn_x-domain_oreilly.com_product_9780596153113.EBOOK-{711cd6ac-12b9-4d4e-92d6-66caa9825cf4}.dtapart\",\"O'Reilly/Mind Performance Hacks/Mind_Performance_Hacks.mobi\"]
[2013-05-13 00:18:33 BST] read: lsof [\"-F0can\",\"+d\",\"/home/xyem/annex/.git/annex/tmp/\"]
[2013-05-13 00:18:33 BST] Committer: Adding Mind_Perf..acks.mobi
add O'Reilly/Mind Performance Hacks/Mind_Performance_Hacks.mobi (checksum...) [2013-05-13 00:18:33 BST] read: sha256sum [\"/home/xyem/annex/.git/annex/tmp/Mind_Performance_Hacks13351.mobi\"]
O'Reilly/Mind Performance Hacks/Mind_Performance_Hacks.mobi changed while it was being added
[2013-05-13 00:18:33 BST] Committer: delaying commit of 1 changes
failed
[2013-05-13 00:18:33 BST] Committer: committing 1 changes
[2013-05-13 00:18:33 BST] Committer: Committing changes to git
(Recording state in git...)
[2013-05-13 00:18:33 BST] feed: git [\"--git-dir=/home/xyem/annex/.git\",\"--work-tree=/home/xyem/annex\",\"update-index\",\"-z\",\"--index-info\"]
[2013-05-13 00:18:33 BST] read: git [\"--git-dir=/home/xyem/annex/.git\",\"--work-tree=/home/xyem/annex\",\"commit\",\"--allow-empty-message\",\"--no-edit\",\"-m\",\"\",\"--quiet\",\"--no-verify\"]
[2013-05-13 00:18:33 BST] read: git [\"--git-dir=/home/xyem/annex/.git\",\"--work-tree=/home/xyem/annex\",\"symbolic-ref\",\"HEAD\"]
[2013-05-13 00:18:33 BST] read: git [\"--git-dir=/home/xyem/annex/.git\",\"--work-tree=/home/xyem/annex\",\"show-ref\",\"refs/heads/master\"]
[2013-05-13 00:18:34 BST] read: git [\"--git-dir=/home/xyem/annex/.git\",\"--work-tree=/home/xyem/annex\",\"ls-files\",\"--others\",\"--exclude-standard\",\"-z\",\"--\",\"O'Reilly/Mind Performance Hacks/Mind_Performance_Hacks.mobi\"]
[2013-05-13 00:18:34 BST] read: lsof [\"-F0can\",\"+d\",\"/home/xyem/annex/.git/annex/tmp/\"]
[2013-05-13 00:18:34 BST] Committer: Adding Mind_Perf..acks.mobi
add O'Reilly/Mind Performance Hacks/Mind_Performance_Hacks.mobi (checksum...) [2013-05-13 00:18:34 BST] read: sha256sum [\"/home/xyem/annex/.git/annex/tmp/Mind_Performance_Hacks13351.mobi\"]
[2013ok
-05-13 00:18:34 BST] Watcher: file deleted O'Reilly/Mind Performance Hacks/Mind_Performance_Hacks.mobi
[2013-05-13 00:18:34 BST] Watcher: add symlink O'Reilly/Mind Performance Hacks/Mind_Performance_Hacks.mobi
[2013-05-13 00:18:34 BST] Committer: committing 2 changes
[2013-05-13 00:18:34 BST] chat: git [\"--git-dir=/home/xyem/annex/.git\",\"--work-tree=/home/xyem/annex\",\"hash-object\",\"-t\",\"blob\",\"-w\",\"--stdin\"]
[2013-05-13 00:18:34 BST] Committer: Committing changes to git
(Recording state in git...)
"""]]

View file

@ -26,7 +26,7 @@ We are, approximately, here:
* [[OSX]] port is in fairly good shape, but still has some room for improvement
* [[android]] port is zooming along
* Windows port does not exist yet
* [[Windows]] port is barely getting started
## not yet on the map:

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkGCmVc5qIJaQQgG82Hc5zzBdAVdhe2JEM"
nickname="Bruno"
subject="comment 1"
date="2013-05-10T23:52:43Z"
content="""
I have a 'Connection timed out' when I try to set-up my Gtalk account on my Nexus 4 (4.2.2). I tested it at home without a web proxy.
Here's a screen capture: [http://i.imgur.com/RBcNjSv.png](http://i.imgur.com/RBcNjSv.png).
"""]]

View file

@ -0,0 +1,46 @@
Set up my Windows development environment. For future reference, I've
installed:
* haskell platform for windows
* cygwin
* gcc and a full C toolchain in cygwin
* git from upstream (probably git-annex will use this)
* git in cygwin (the other git was not visible inside cygwin)
* vim in cygwin
* vim from upstream, as the cygwin vim is not very pleasant to use
* openssh in cygwin (seems to be missing a ssh server)
* rsync in cygwin
* Everything that `cabal install git-annex` is able to install successfully.
This includes all the libraries needed to build regular git-annex,
but not the webapp. Good start though.
Result basically feels like a linux system that can't decide which way
slashes in paths go. :P I've never used Cygwin before (I last used a
Windows machine in 2003 for that matter), and it's a fairly impressive hack.
----
Fixed up git-annex's configure program to run on Windows (or, at least, in
Cygwin), and have started getting git-annex to build.
For now, I'm mostly stubbing out functions that use unix stuff. Gotten the
first 44 of 300 source files to build this way.
Once I get it to build, if only with stubs, I'll have a good
idea about all the things I need to find Windows equivilants of.
Hopefully most of it will be provided by
<http://hackage.haskell.org/package/unix-compat-0.3.0.1>.
----
So that's the plan. There is a possible shortcut, rather than doing a full
port. It seems like it would probably not be too hard to rebuild ghc inside
Cygwin, and the resulting ghc would probably have a full POSIX emulation
layer going through cygwin. From ghc's documentation, it looks like that's
how ghc used to be built at some point in the past, so it would probably
not be too hard to build it that way. With such a cygwin ghc, git-annex
would probably build with little or no changes. However, it would be a
git-annex targeting Cygwin, and not really a native Windows port. So
it'd see Cygwin's emulated POSIX filesystem paths, etc. That
seems probably not ideal for most windows users.. but if I get really stuck
I may go back and try this method.

View file

@ -0,0 +1,29 @@
After working on it all day, git-annex now builds on Windows!
Even better, `git annex init` works. So does `git annex status`, and
probably more. Not `git annex add` yet, so I wasn't able to try much more.
I didn't have to add many stubs today, either. Many of the missing Windows
features were only used in code paths that made git-annex faster, but I
could fall back to a slower code path on Windows.
The things that are most problimatic so far:
* POSIX file locking. This is used in git-annex in several places to
make it safe when multiple git-annex processes are running. I put in
really horrible dotfile type locking in the Windows code paths, but I
don't trust it at all of course.
* There is, apparently, no way to set an environment variable in Windows
from Haskell. It is only possible to set up a new process' environment
before starting it. Luckily most of the really crucial environment
variable stuff in git-annex is of this latter sort, but there were
a few places I had to stub out code that tries to manipulate git-annex's
own environment.
The `windows` branch has a diff of 2089 lines. It add 88 ifdefs to the code
base. Only 12 functions are stubbed out on Windows. This could be so much
worse.
Next step: Get the test suite to build. Currently ifdefed out because it
uses some stuff like `setEnv` and `changeWorkingDirectory` that I don't know
how to do in Windows yet.

View file

@ -0,0 +1,14 @@
It's remarkable that a bad decision made in 1982 can cause me to waste an
entire day in 2013. Yes, `/` vs `\` fun time. Even though I long ago
converted git-annex to use the haskell `</>` operator wherever it builds
up paths (which transparently handles either type of separator), I still
spent most of today dealing with it. Including some libraries I use that
get it wrong. Adding to the fun is that git uses `/` internally, even on
Windows, so Windows separated paths have to be converted when being fed
into git.
Anyway, `git annex add` now works on Windows. So does `git annex find`,
and `git annex whereis`, and probably most query stuff.
Today was very un-fun and left me with a splitting headache, so I will
certainly *not* be working on the Windows port tomorrow.

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://jamesjustjames.wordpress.com/"
nickname="purpleidea"
subject="Isn't windows deprecated?"
date="2013-05-13T05:51:20Z"
content="""
I feel bad for you, however I respect you for keeping your promise to try and hack on Windows. I had to port over some code than ran beautifully on GNU/Linux and it was more trouble than it was worth. In the end it was never used :P
Cheers!
"""]]

View file

@ -0,0 +1,9 @@
Spent some time today to get caught up on bug reports and website traffic.
Fixed a few things.
Did end up working on Windows for a while too. I got `git annex drop`
working. But nothing that moves content quite works yet..
I've run into a stumbling block with `rsync`. It thinks that
`C:\repo` is a path on a ssh server named "C". Seems I will need to translate
native windows paths to unix-style paths when running rsync.

View file

@ -8,10 +8,7 @@ Stackoverflow has some details.
NTFS supports symbolic links two different ways: an [[!wikipedia NTFS symbolic link]] and an [[!wikipedia NTFS_junction_point]]. The former seems like the closest analogue to POSIX symlinks.
Make git use them, as it (apparently) does not yet.
Currently, on Windows, git checks out symlinks as files containing the symlink
target as their contents.
The windows port will not use symlinks. It will only support direct mode.
## POSIX

View file

@ -14,6 +14,7 @@ exceptions:
* Repositories on FAT and other less than stellar filesystems
that don't support things like symlinks will be automatically put
into direct mode.
* Windows always uses direct mode.
## enabling (and disabling) direct mode

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://edheil.wordpress.com/"
ip="173.162.44.162"
subject="comment 10"
date="2013-05-13T04:08:14Z"
content="""
I'm not sure what's up. I hope Joey can weigh in.
\"git annex sync\" ought to go get the symlinks from your remote, and merge them into your master branch, which is currently checked out, so you should see them. :(
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 11"
date="2013-05-13T17:33:26Z"
content="""
When the assistant (or git annex sync) pushes to a repository, it creates a `synced/master` branch. So your bare repository has no regular `master` branch. So when you clone it, you get a repository with no branch checked out, which is what git-annex sync complains about.
There are several solutions. The easiest is to run `git merge origin/synced/master`; then your checkout will have a master branch and you can use `git annex sync` from then on.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://me.yahoo.com/a/bBy7WkgQicYHIiiyj.Vm0TcMbxi2quzbPFef#6f9f7"
nickname="Frederik Vanrenterghem"
subject="comment 7"
date="2013-05-12T14:04:29Z"
content="""
That all sounded very plausible, but for some reason it failed to work for me. I went back to basics and tried getting there by replicating steps 1 to 3 of the walkthrough, but also to no aval. I'll still try one more thing - setting up a local repository on the VPS and syncing it to my main one over XMPP. That seems rather convoluted given the bare repository is on the same server, but who knows...
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="http://edheil.wordpress.com/"
ip="173.162.44.162"
subject="comment 8"
date="2013-05-12T19:06:14Z"
content="""
So what *does* show up? Is your repo an empty directory? Or are the symlinks that should point to content there?
as far as git goes, are you on the master branch? *is* there a master branch? If you do \"git status\" what do you see?
There's no reason this shouldn't work for you....
"""]]

View file

@ -0,0 +1,37 @@
[[!comment format=mdwn
username="https://me.yahoo.com/a/bBy7WkgQicYHIiiyj.Vm0TcMbxi2quzbPFef#6f9f7"
nickname="Frederik Vanrenterghem"
subject="comment 9"
date="2013-05-12T23:51:16Z"
content="""
I get the following:
frederik@niihau:~/Documents$ git status
# On branch master
#
# Initial commit
#
nothing to commit (create/copy files and use \"git add\" to track)
And this is the config:
frederik@niihau:~/Documents$ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote \"origin\"]
fetch = +refs/heads/*:refs/remotes/origin/*
url = /home/frederik/annex
annex-uuid = 1648a298-27aa-4cb1-xx-xxx
[annex]
uuid = 4406bd35-9ff7-4008-yy-yyy
version = 3
[remote \"niihau\"]
url = /home/frederik/annex
fetch = +refs/heads/*:refs/remotes/niihau/*
annex-uuid = 1648a298-27aa-4cb1-xx-xxx
Not having worked with git before, all this is still very new for me. I have 33M of data in the .git directory, mostly in objects, so it does seem to have synced something already.
"""]]

View file

@ -0,0 +1,13 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 3"
date="2013-05-13T18:00:42Z"
content="""
This sounds like one or more bugs that I should address, but I'm having trouble understanding exactly what you did, from your description, at the level of detail I need to replicate a problem.
Do you still have a copy of the corrupted repository? Or any error messages to show me?
It sounds like at least one problem might have been this bug: [[bugs/Unable_to_switch_back_to_direct_mode]]
At least that is fixed in current git.
"""]]

View file

@ -0,0 +1,11 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 1"
date="2013-05-13T18:28:01Z"
content="""
I actually didn't originally consider this, but it turns out I have wonderful users, and you're at least the second to consider this case.
So the released git-annex version does already verify the checksum when doing a migration. However, as I was checking this, I noticed
I had left a window between the verification and the generation of the new key for the migration. It's pretty unlikely a file would get corrupted just as it was being migrated, but that's no excuse. I've committed a change that reverses the order; so it generates a new key and then verifies the old one.
"""]]

View file

@ -0,0 +1,3 @@
`git annex dropunused` is simple enough.
But how do I perform the equivalent procedure on an rsync remote? I'd presume that `git annex copy --to remote` is not sufficient.

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
nickname="Justin"
subject="comment 1"
date="2013-05-12T23:06:34Z"
content="""
Oh, sheesh, it's right there at the bottom of http://git-annex.branchable.com/special_remotes/. Oops.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 2"
date="2013-05-13T18:03:23Z"
content="""
Well, in your defense it's documented there because I can't think of a better place to put it. Maybe you looked in some other places first before finding it there, and I should add a pointer to those places?
(For the curious, the magic run is just: git annex unused --from remote)
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
nickname="Justin"
subject="comment 3"
date="2013-05-13T18:56:32Z"
content="""
I think in general the `git annex command --help` bits could be more thorough. If `git annex dropunused --help` indicated that it took `--from`, that would probably have helped me.
"""]]

View file

@ -0,0 +1,15 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 4"
date="2013-05-13T18:59:08Z"
content="""
You might be using an older version. Here's the output of that command in the current version:
<pre>
Usage: git-annex dropunused NUM|RANGE ... [option ...]
-f REMOTE --from=REMOTE drop content from a remote
To see additional options common to all commands, run: git annex help options
</pre>
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c"
nickname="Justin"
subject="comment 5"
date="2013-05-13T19:07:11Z"
content="""
Ah, rebuilding got me that. That's pretty good!
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 1"
date="2013-05-13T17:56:35Z"
content="""
From what you've described, the box.com special remote is the only remote you have connecting your two computers.
But special remotes are only used to transfer the contents of files. They do not have the content of the git repository.
You need to set up a git remote that both computers can access. Or you can use the webapp to set up XMPP to sync between the computers directly. If you start the webapp, it will detect you're missing a git remote, and suggest setting up XMPP. [[/assistant/xmppnudge.png]]
"""]]

View file

@ -0,0 +1,13 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 1"
date="2013-05-13T18:39:59Z"
content="""
Sorry it took so long to get back to you.
You do not necessarily have to have git-annex installed on your web server. But it will open up one of the nicest ways to use git-annex with a server, which is to put a bare git repository on the server, and let git-annex send the contents of large files to that repository. It's fine to install any old version of git-annex on the server, they're all forwards and backwards compatable.
In any case, you need git-annex installed on any computers where you want to access the repository, certainly.
"""]]

View file

@ -0,0 +1,20 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 1"
date="2013-05-13T18:49:25Z"
content="""
Well, it would be possible to add a configuration setting in the repository that tells the assistant what address:port to listen on.
But maybe it would be better to not use `assistant --autostart` in your situation? That command is basically the same as this shell script:
[[!format sh "
#!/bin/sh
for dir in $(cat $HOME/.config/git-annex/autostart); do
cd $dir
git annex webapp &
done
"]]
So you can write similar shell scripts that start the webapp with whatever options you like.
"""]]

View file

@ -14,19 +14,13 @@ detailed instructions | quick install
[[Gentoo]] | `emerge git-annex`
[[ScientificLinux5]] | (and other RHEL5 clones like CentOS5)
[[openSUSE]] |
Windows | [[sorry, Windows not supported yet|todo/windows_support]]
[[Windows]] | **alpha**
"""]]
## Using cabal
As a haskell package, git-annex can be installed using cabal.
Start by installing the [Haskell Platform](http://hackage.haskell.org/platform/),
and then:
cabal install git-annex --bindir=$HOME/bin
That installs the latest release. Alternatively, you can [[download]]
git-annex yourself and [[manually_build_with_cabal|install/cabal]].
As a haskell package, git-annex can be installed from source pretty easily
[[using cabal|cabal]].
## Installation from scratch

10
doc/install/Windows.mdwn Normal file
View file

@ -0,0 +1,10 @@
git-annex has recently been ported to Windows.
This port is in an alpha state; many things will not work. See
[[todo/windows_support]] for current status. Note especially that
git-annex always uses [[direct_mode]] on Windows.
To build git-annex from source on Windows, you need to install
the Haskell Platform and Cygwin. When installing Cygwin, select
to install gcc, make, rsync, and ssh. Then, within Cygwin, git-annex
can be compiled following the instructions for [[using cabal|cabal]].

View file

@ -1,4 +1,7 @@
As a haskell package, git-annex can be installed using cabal. For example:
As a haskell package, git-annex can be installed using cabal.
Start by installing the [Haskell Platform](http://hackage.haskell.org/platform/),
and then:
cabal update
PATH=$HOME/bin:$PATH

View file

@ -0,0 +1,18 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawlJemqsekZTC5dvc-MAByUWaBvsYE-mFUo"
nickname="Gábor"
subject="Cabal dependencies"
date="2013-05-12T12:52:20Z"
content="""
After finishing the installation the cabal way, here are the packages I installed. It is possible that there are other packages I installed previously as dependency for other packages.
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.2 LTS
Release: 12.04
Codename: precise
$ apt-get install cabal-install libgnutls28-dev libgsasl7-dev c2hs libghc-libxml-sax-dev zlib1g-dev libghc-zlib-dev
$ cabal install git-annex --bindir=$HOME/bin
"""]]

View file

@ -21,7 +21,7 @@ quite a lot.
* [SafeSemaphore](http://hackage.haskell.org/package/SafeSemaphore)
* [UUID](http://hackage.haskell.org/package/uuid)
* [regex-tdfa](http://hackage.haskell.org/package/regex-tdfa)
* Optional haskell stuff, used by the [[assistant]] and its webapp (edit Makefile to disable)
* Optional haskell stuff, used by the [[assistant]] and its webapp
* [stm](http://hackage.haskell.org/package/stm)
(version 2.3 or newer)
* [hinotify](http://hackage.haskell.org/package/hinotify)

View file

@ -0,0 +1,15 @@
[[!comment format=mdwn
username="diepes"
ip="41.160.8.49"
subject="rsync.net support git and rsync, not git-annex"
date="2013-05-13T14:44:53Z"
content="""
how would i use rsync.net as a full repository ? (annex files and git repo)
It support's rsync, and I discovered now git as well.
I had a look at the webapp and can't figure out how to set it up, so I can have multiple remote's sync to rsync.net for annex files and git sync.
Link: http://www.rsync.net/resources/howto/github_integration.html
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
nickname="joey"
subject="comment 7"
date="2013-05-13T18:42:14Z"
content="""
@Chris I guess there's no really easy way because searching for a given key is not something many people need to do.
However, git does provide a way. Try `git log --stat -S $KEY`
"""]]

View file

@ -1,61 +1,14 @@
Can it be built on Windows?
The git-annex Windows port is not ready for prime time. But it does exist
now! --[[Joey]]
short answer: not yet
## status
First, you need to get some unix utilities for windows. Git of course.
Also rsync, and a `cp` command that understands at least `cp -p`, and
`uuid`, and `xargs` and `sha1sum`. Note that some of these could be
replaced with haskell libraries to some degree.
There are probably still some places where it assumes / as a path
separator, although I fixed probably almost all by now.
Then windows versions of these functions could be found,
which are all the ones that need POSIX, I think. A fair amount of this,
the stuff to do with signals and users, could be empty stubs in windows.
The file manipulation, particularly symlinks, would probably be the main
challenge.
<pre>
addSignal
blockSignals
changeWorkingDirectory
createLink
createSymbolicLink
emptySignalSet
executeFile
fileMode
fileSize
forkProcess
getAnyProcessStatus
getEffectiveUserID
getEnvDefault
getFileStatus
getProcessID
getProcessStatus
getSignalMask
getSymbolicLinkStatus
getUserEntryForID
getUserEntryForName
groupWriteMode
homeDirectory
installHandler
intersectFileModes
isRegularFile
isSymbolicLink
modificationTime
otherWriteMode
ownerWriteMode
readSymbolicLink
setEnv
setFileMode
setSignalMask
sigCHLD
sigINT
unionFileModes
</pre>
A good starting point is
<http://hackage.haskell.org/package/unix-compat-0.3.0.1>. However, note
that its implementations of stuff like `createSymbolicLink` are stubs.
--[[Joey]]
* Does not work with Cygwin's build of git (that git does not consistently
support use of DOS style paths, which git-annex uses on Windows).
Must use the upstream build of git for Windows.
* OTOH, you do need to install Cygwin's build of rsync (unless you can
find another one), and Cygwin's build of ssh.
* test suite doesn't work
* Bad file locking, it's probably not safe to run more than one git-annex
process at the same time on Windows.
* No support for the assistant or webapp.