Merge branch 'master' into assistant

This commit is contained in:
Joey Hess 2012-07-26 14:28:19 -04:00
commit 6de38a2ca8
45 changed files with 877 additions and 8 deletions

View file

@ -0,0 +1,26 @@
Utility/WebApp.hs, didn't quite have the right definition to use 'open' instead of 'xdg-open' on OSX, the follow fixes that
<pre>
diff --git a/Utility/WebApp.hs b/Utility/WebApp.hs
index 6936c66..0593dda 100644
--- a/Utility/WebApp.hs
+++ b/Utility/WebApp.hs
@@ -42,7 +42,7 @@ localhost = "localhost"
runBrowser :: String -> IO Bool
runBrowser url = boolSystem cmd [Param url]
where
-#if MAC
+#if OSX
cmd = "open"
#else
cmd = "xdg-open"
</pre>
> [[done]], thanks
I guess I should really clone the repo and submit a stream of minor changes
:P, @joeyh please let me know if you're getting annoyed with copy and
pasting the small fixes from the bug/forums section.
> If you're going to be writing some patches, a git repo I can pull from
> would make my life easier. --[[Joey]]

View file

@ -0,0 +1,37 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 4"
date="2012-07-04T13:17:05Z"
content="""
In relation to the system limits,
laplace:~ jtang$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 10240
Also, the maxfiles for the whole system is
laplace:~ jtang$ sysctl kern.maxfiles
kern.maxfiles: 12288
the above was the defaults as far as I recall. What you probably would be interested is the ulimits that the user see
laplace:~ jtang$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
I would imagine the limit that you are looking for is 256. Hope this helps.
----
On the point about deletions not being propagated, it does do a commit. I suspect that the kqueue code is just not picking up the changes and pushing the changes out. The watch command on a single annex with no remotes functions as expected.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 2"
date="2012-06-25T22:36:39Z"
content="""
On the system limits side, I think if you want to make it more approachable by more users then adjusting system limits might scare users away. On the note of the ssh-agents spawning like no tomorrow on my machine, it turned out that i had a symlink from my .bashrc to .bash_profile, I guess I should not be too lazy and have two seperate files.
"""]]

View file

@ -0,0 +1,35 @@
Seems commit 6cecc26206c4a539999b04664136c6f785211a41 missed on dependancy, that is blaze-markup
<pre>
Assistant/Threads/WebApp.hs:25:8:
Could not find module `Text.Blaze.Renderer.String'
It is a member of the hidden package `blaze-markup-0.5.1.0'.
Perhaps you need to add `blaze-markup' to the build-depends in your .cabal file.
Use -v to see a list of the files searched for.
cabal: Error: some packages failed to install:
git-annex-3.20120721 failed during the building phase. The exception was:
ExitFailure 1
</pre>
This should fix it
<pre>
x00:git-annex jtang$ git diff
diff --git a/git-annex.cabal b/git-annex.cabal
index c7d9bf5..4f98d2a 100644
--- a/git-annex.cabal
+++ b/git-annex.cabal
@@ -76,7 +76,7 @@ Executable git-annex
if flag(Webapp)
Build-Depends: yesod, yesod-static, case-insensitive, http-types,
transformers, wai, wai-logger, warp, blaze-builder, blaze-html,
- crypto-api, hamlet
+ blaze-markup, crypto-api, hamlet
CPP-Options: -DWITH_WEBAPP
if (os(darwin))
</pre>
> [[done]].. interestingly, cabal had not complained about there here,
> as in my version, it's in blaze, not blaze-markup. Added it anyway.
> --[[Joey]]

View file

@ -0,0 +1,27 @@
After fixing a few things - see [[bugs/the tip at commit 6cecc26206c4a539999b04664136c6f785211a41 disables the watch command on OSX]], [[bugs/Missing dependancy in commit 6cecc26206c4a539999b04664136c6f785211a41]] and [[bugs/Fix for opening a browser on a mac (or xdg-open on linux/bsd?)]] I tried the watch command on my ~180gig annex of stuff. This might be yet again related to the issue of [[bugs/Issue on OSX with some system limits]]
the watch command segfaults
<pre>
x00:annex jtang$ git annex watch --foreground -d
watch . [2012-07-26 12:27:16 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","show-ref","git-annex"]
[2012-07-26 12:27:16 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","show-ref","--hash","refs/heads/git-annex"]
[2012-07-26 12:27:16 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","log","refs/heads/git-annex..38d3f769ef004b96b6d640cfb59a45f7b4edf5f6","--oneline","-n1"]
[2012-07-26 12:27:16 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","log","refs/heads/git-annex..ebabe9c92516c350a30126037173080648f5930b","--oneline","-n1"]
[2012-07-26 12:27:16 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","log","refs/heads/git-annex..d36d8d88847decc2320f0be22892ad94a8abe594","--oneline","-n1"]
[2012-07-26 12:27:16 IST] read: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","log","refs/heads/git-annex..aaa62a8191b3c964fdf546077049f626e8561b22","--oneline","-n1"]
[2012-07-26 12:27:16 IST] chat: git ["--git-dir=/Users/jtang/annex/.git","--work-tree=/Users/jtang/annex","cat-file","--batch"]
(scanning...) error: git-annex died of signal 11
</pre>
The above was done on the usual OSX 10.7 system that I have.
---
I'll try and bisect it and find out where the problem first appeared, does the tests currently test the watch command? (also my comments seem to get moderated whether i use my openid account with google or with the native ikiwiki account, so some comments might be hidden)
> The test suite does not currently test the watch command, unfortunatly.
>
> Wow, I had not noticed the 30 pending moderated comments.. Let them all
> thru, and I guess I'll turn off comment spam filtering for now, since
> there has apparently been none. --[[Joey]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="4.153.2.189"
subject="not good.."
date="2012-07-26T17:09:16Z"
content="""
Do you see the segfault in a repo with fewer files?
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="4.153.2.189"
subject="comment 2"
date="2012-07-26T17:09:59Z"
content="""
Also, you might try bisecting to find whatever commit it first started segfaulting on.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="jtang"
ip="79.97.135.214"
subject="comment 3"
date="2012-07-26T17:11:55Z"
content="""
It fails on repos with either no files or smaller repos.
"""]]

View file

@ -0,0 +1,62 @@
_git annex initremote_ without a complete command set still adds an entry to the uuid.log etc... and thus clutters up the state of the annex. I would not have expected this behaviour as a user.
_initremote_ should fail and not do anything if the commands that it has been given are incomplete or incorrect. I was initialising a few rsync repos and noticed that i ended up having mutiple rsync remotes with the same name but different uuid's. I know its hard if not impossible to remove these uuid's so I have just marked them as "dead" in my live annexes.
Here's a transcript of the problem
<pre>
x00:sandbox jtang$ mkdir atest
x00:sandbox jtang$ cd atest/
x00:atest jtang$ git init
Initialized empty Git repository in /Users/jtang/sandbox/atest/.git/
x00:atest jtang$ git annex init
init ok
(Recording state in git...)
x00:atest jtang$ git annex status
supported backends: SHA256 SHA1 SHA512 SHA224 SHA384 SHA256E SHA1E SHA512E SHA224E SHA384E WORM URL
supported remote types: git S3 bup directory rsync web hook
trusted repositories: 0
semitrusted repositories: 2
00000000-0000-0000-0000-000000000001 -- web
cbb58e1c-d737-11e1-b682-83239d5ff2e0 -- here
untrusted repositories: 0
dead repositories: 0
transfers in progress: none
available local disk space: 185 gigabytes (+1 megabyte reserved)
local annex keys: 0
local annex size: 0 bytes
known annex keys: 0
known annex size: 0 bytes
bloom filter size: 16 mebibytes (0% full)
backend usage:
x00:atest jtang$ git annex initremote foo
git-annex: Specify the type of remote with type=
x00:atest jtang$ git annex initremote foo type=rsync
(Recording state in git...)
initremote foo git-annex: Specify encryption=key or encryption=none or encryption=shared
x00:atest jtang$ git annex initremote foo type=rsync
(Recording state in git...)
initremote foo git-annex: Specify encryption=key or encryption=none or encryption=shared
x00:atest jtang$ git annex status
supported backends: SHA256 SHA1 SHA512 SHA224 SHA384 SHA256E SHA1E SHA512E SHA224E SHA384E WORM URL
supported remote types: git S3 bup directory rsync web hook
trusted repositories: (Recording state in git...)
0
semitrusted repositories: 5
00000000-0000-0000-0000-000000000001 -- web
cbb58e1c-d737-11e1-b682-83239d5ff2e0 -- here
d3adfcd0-d737-11e1-b15b-b7032388f8aa -- foo
d6d8e1e0-d737-11e1-956a-0b3d3451226a -- foo
d78d795c-d737-11e1-ac98-4fe3d6fdfd54 -- foo
untrusted repositories: 0
dead repositories: 0
transfers in progress: none
available local disk space: 185 gigabytes (+1 megabyte reserved)
local annex keys: 0
local annex size: 0 bytes
known annex keys: 0
known annex size: 0 bytes
bloom filter size: 16 mebibytes (0% full)
backend usage:
x00:atest jtang$
</pre>

View file

@ -0,0 +1,113 @@
Building commit 805d50c69d40be97baa28735371778df63b5fed6
<pre>
x00:git-annex jtang$ cabal install
Resolving dependencies...
Configuring dbus-0.10...
Building dbus-0.10...
Preprocessing library dbus-0.10...
[1 of 9] Compiling DBus.Types ( lib/DBus/Types.hs, dist/build/DBus/Types.o )
[2 of 9] Compiling DBus.Message ( lib/DBus/Message.hs, dist/build/DBus/Message.o )
[3 of 9] Compiling DBus.Wire ( lib/DBus/Wire.hs, dist/build/DBus/Wire.o )
[4 of 9] Compiling DBus.Address ( lib/DBus/Address.hs, dist/build/DBus/Address.o )
[5 of 9] Compiling DBus ( lib/DBus.hs, dist/build/DBus.o )
[6 of 9] Compiling DBus.Introspection ( lib/DBus/Introspection.hs, dist/build/DBus/Introspection.o )
[7 of 9] Compiling DBus.Transport ( lib/DBus/Transport.hs, dist/build/DBus/Transport.o )
lib/DBus/Transport.hs:196:72: Not in scope: `getPeerCred'
cabal: Error: some packages failed to install:
dbus-0.10 failed during the building phase. The exception was:
ExitFailure 1
git-annex-3.20120721 depends on dbus-0.10 which failed to install.
</pre>
The above isn't a git-annex problem but a dbus problem, at first I thought I didn't have the network package installed, but did. I should probably report this problem to the haskell dbus author.
On a slightly different note, based on the makefile DBUS is not enabled on OSX/BSD so I did not expect the cabal file to use dbus as well, I'm currently interested in poking at the webapp stuff ;)
Although DBUS is available on OSX from macports I get the feeling that the haskell-dbus package might need some poking before it works properly.
To continue, pulling, installing the dependancies (dbus is still boned) and building commit 6cecc26206c4a539999b04664136c6f785211a41
<pre>
[ 92 of 205] Compiling Utility.Url ( Utility/Url.hs, tmp/Utility/Url.o )
Utility/Url.hs:39:14: Not in scope: `parseURI'
Utility/Url.hs:73:14: Not in scope: `parseURI'
Utility/Url.hs:88:12: Not in scope: type constructor or class `URI'
Utility/Url.hs:91:30: Not in scope: type constructor or class `URI'
Utility/Url.hs:107:38: Not in scope: `parseURIReference'
Utility/Url.hs:111:95: Not in scope: `relativeTo'
make: *** [git-annex] Error 1
</pre>
Which then lead me to doing a "cabal install -f-DBus" which spits out the following when trying to link the binary
<pre>
[206 of 206] Compiling Main ( git-annex.hs, dist/build/git-annex/git-annex-tmp/Main.o )
Linking dist/build/git-annex/git-annex ...
Undefined symbols for architecture x86_64:
"_addfds_kqueue", referenced from:
_s16v6_info in Kqueue.o
"_init_kqueue", referenced from:
_s16v3_info in Kqueue.o
"_waitchange_kqueue", referenced from:
_UtilityziKqueue_zdwa1_info in Kqueue.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
cabal: Error: some packages failed to install:
git-annex-3.20120721 failed during the building phase. The exception was:
ExitFailure 1
</pre>
I then just tried to build commit with 6cecc26206c4a539999b04664136c6f785211a41 (i have the needed dependancies installed), gives me this...
<pre>
x00:git-annex jtang$ make
ghc -O2 -threaded -Wall -ignore-package monads-fd -ignore-package monads-tf -outputdir tmp -IUtility -DWITH_ASSISTANT -DWITH_S3 -DWITH_WEBAPP --make git-annex Utility/libdiskfree.o Utility/libmounts.o Utility/libkqueue.o
Assistant/Threads/MountWatcher.hs:39:0:
warning: #warning Building without dbus support; will use mtab polling
[ 92 of 205] Compiling Utility.Url ( Utility/Url.hs, tmp/Utility/Url.o )
Utility/Url.hs:98:65:
Couldn't match expected type `network-2.3.0.13:Network.URI.URI'
with actual type `URI'
In the second argument of `mkRequest', namely `u'
In the expression: mkRequest requesttype u :: Request_String
In an equation for `req':
req = mkRequest requesttype u :: Request_String
make: *** [git-annex] Error 1
</pre>
The latest version of the network package in hackage is network-2.3.0.14 which I have installed, this might also be the reason why dbus is broken. removing network-2.3.0.14 at least makes it happy again.
to remove the network-2.3.0.14 package
<pre>
ghc-pkg unregister network-2.3.0.14
</pre>
Hope the above isn't too random of bug/issue report.
----
going through <http://hackage.haskell.org/packages/archive/network/2.3.0.14/doc/html/Network-Socket.html> shows that getPeerCred is only available on systems where SO_PEERCRED is supported, *sigh* OSX isn't supported and thus haskell-dbus is broken. Apparently getpeerid is more portable but it isnt supported in the network package. It looks like dbus support on OSX isn't really going to work too well till haskell-dbus gets fixed on OSX (or BSD?)
> Does OSX acually come with dbus by default, and can you
> use something like `dbus-monitor` to see events when
> plugging in removable drives? If so, this might be worth spending time
> on.
>> No OSX does not come with dbus by default, the user must install it
>
> Currently though, dbus is not supposed to be built on non-Linux systems.
> (Well, it might work on Freebsd or something, but I've not tried it.)
> I've fixed the cabal file to only enable it on Linux.

View file

@ -0,0 +1,22 @@
The recent commit 6cecc26206c4a539999b04664136c6f785211a41 seems to have disabled the watch command on OSX, this certainly is the case when I try to run the webapp.
The following fixes the makefile
<pre>
x00:git-annex jtang$ git diff
diff --git a/Makefile b/Makefile
index 9f312dc..4a74e71 100644
--- a/Makefile
+++ b/Makefile
@@ -27,7 +27,7 @@ endif
endif
PREFIX=/usr
-GHCFLAGS=-O2 $(BASEFLAGS) $(FEATURES)
+GHCFLAGS=-O2 $(BASEFLAGS) $(FEATURES) $(OPTFLAGS)
ifdef PROFILE
GHCFLAGS=-prof -auto-all -rtsopts -caf-all -fforce-recomp $(BASEFLAGS) $(FEATURES) $(OPTFLAGS)
</pre>
> [[done]] --[[Joey]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 2"
date="2012-07-20T19:31:11Z"
content="""
Glad that I can help ;) Alas if it weren't for the learning curve of haskell, I'd fix it myself.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 1"
date="2012-06-19T06:53:26Z"
content="""
heh, yea, it's detecting changes on OSX ;)
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 1"
date="2012-06-19T07:01:26Z"
content="""
issues with the watch command on OSX, it seems that there is a race condition somewhere. I dumped a few iso's into an annex and it only annexed the smaller files (checksums) and the bigger ones (the iso's) just got made read only. also do you want these bugs to be logged here or in the bugs section?
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
nickname="Justin"
subject="comment 1"
date="2012-06-27T12:46:31Z"
content="""
can X and Y be the names of the git-annex remotes?
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://joeyh.name/"
ip="4.153.2.189"
subject="comment 2"
date="2012-07-26T17:27:39Z"
content="""
That's a good question. Unfortunatly they cannot; X and Y need to be stable across repositories, and git remotes can have different names in different repositories.
Even using the description that git-annex stores for each repository for X and Y is problimatic, since that description can change, and so could be different in two repos that are each trying to resolve the same merge conflict.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 2"
date="2012-07-19T18:43:30Z"
content="""
Joey, yes dbus is available from macports and homebrew, it's not installed by default (or as a dependancy) for most packages in macports.
"""]]

View file

@ -0,0 +1,83 @@
After an all-nighter, I have `git annex webapp` launching a WebApp!
It doesn't do anything useful yet, just uses Yesod to display a couple of
hyperlinked pages and a favicon, securely.
The binary size grew rather alarmingly, BTW. :) Indeed, it's been growing
for months..
-rwxr-xr-x 1 root root 9.4M Jul 21 16:59 git-annex-no-assistant-stripped
-rwxr-xr-x 1 joey joey 12M Jul 25 20:54 git-annex-no-webapp-stripped
-rwxr-xr-x 1 joey joey 17M Jul 25 20:52 git-annex-with-webapp-stripped
----
Along the way, some Not Invented Here occurred:
I didn't use the yesod scaffolded site, because it's a lot of what
seems mostly to be cruft in this use case. And because I don't like
code generated from templates that people are then expected to edit. Ugh.
That's my least favorite part of Yesod. This added some pain, since
I had to do everything the hard way.
I didn't use [wai-handler-launch](http://hackage.haskell.org/package/wai-handler-launch)
because:
* It seems broken on IPv6 capable machines (it always opens
`http://127.0.0.1:port/` even though it apparently doesn't always
listen there.. I think it was listening on my machine's ipv6 address
instead. I know, I know; I should file a bug about this..)
* It always uses port 4587, which is **insane**. What if you have two
webapps?
* It requires javascript in the web browser, which
is used to ping the server, and shut it down when the web browser closes
(which behavior is wrong for git-annex anyway, since the daemon should
stay running across browser closes).
* It opens the webapp on web server startup, which is wrong for git-annex;
instead the command `git annex webapp` will open the webapp,
after `git annex assistant` started the web server.
Instead, I rolled my own WAI webapp laucher, that binds to any free port
on localhost, It does use `xdg-open` to launch the web browser,
like wai-handler-launch (or just `open` on OS X).
Also, I wrote my own WAI logger, which logs using System.Log.Logger,
instead of to stdout, like `runDebug` does.
----
The webapp only listens for connections from localhost, but that's
not sufficient "security". Instead, I added a secret token to
every url in the webapp, that only `git annex webapp` knows about.
But, if that token is passed to `xdg-open` on its command line,
it will be briefly visible to local attackers in the parameters of
`xdg-open`.. And if the web browser's not already running, it'll run
with it as a parameter, and be *very* visible.
So instead, I used a nasty hack. On startup, the assistant
will create a html file, readably only by the user, that redirects
the user to the real site url. Then `git annex webapp` will run
xdg-open on that file.
----
Making Yesod check the `auth=` parameter (to verify that the secret token
is right) is when using Yesod started to pay off. Yesod has a simple
`isAuthorized` method that can be overridden to do your own authentication
like this.
But Yesod really started to shine when I went to add the `auth=` parameter
to every url in the webapp. There's a `joinPath` method can can be used
to override the default url builder. And every type-safe url in the
application goes through there, so it's perfect for this.
I just had to be careful to make it not add `auth=` to the url for the
favicon, which is included in the "Permission Denied" error page. That'd be
an amusing security hole..
----
Next up: Doing some AJAX to get a dynamic view of the state of the daemon,
including currently running transfers, in the webapp. AKA stuff I've never
done before, and that, unlike all this heavy Haskell Yesod, scares me. :)

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawn7Oyqusvn0oONFtVhCx5gRAcvPjyRMcBI"
nickname="Michaël"
subject="is ftp an option?"
date="2012-05-30T10:44:12Z"
content="""
for people only having ftp-access to there storage.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://www.klomp.eu/"
ip="95.91.241.82"
subject="Watch also possible with git?"
date="2012-06-15T17:25:30Z"
content="""
Hi,
it seems that you put a lot of efforts in handling race conditions. Thats great. I wonder if the watch can also be used with git (i.e. changes are commited into git and not as annex)? I know that other projects follow this idea but why using different tools if the git-annex assistant could handle both...
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="http://wiggy.net/"
nickname="Wichert"
subject="macports"
date="2012-06-12T13:00:34Z"
content="""
The average OSX user has a) no idea what macports is, and b) will not be able to install it. Anything that requires a user to do anything with a commandline (or really anything other than using a GUI installer) is effectively a dealbreaker. For our use cases OSX is definitely a requirement, but it must only use standard OSX installation methods in order to be usable. Being in the appstore would be ideal, but standard dmg/pkg installers are still common enough that they are also acceptable.
FWIW this is the same reason many git GUIs were not usable for our OSX users: they required separate installation of the git commandline tools.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 5"
date="2012-06-17T21:42:59Z"
content="""
okay, I've gotten gitbuilder to poll the git repo every minute for changes, gitbuilder doesn't build every commit. It doesn't work like that, it checks out the master and builds that. If there is a failure it automatically bisects to find out where the problem first got introduced. Hope the change to the builder helps!
"""]]

View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawnV2c63kDc6X21a1H81me1mIenUCScd2Gs"
nickname="Emanuele"
subject="watch branch?"
date="2012-06-01T19:19:17Z"
content="""
Hello there? Where can I find more info about this git watch branch?
Keep up the good work!
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawnBl7cA6wLDxVNUyLIHvAyCkf8ir3alYpk"
nickname="Tyson"
subject="Bridging LANs"
date="2012-07-10T10:20:59Z"
content="""
Why rely on the cloud when you can instead use XMPP and jingle to perform NAT traversal for you? AFAIKT, it also means that traffic won't leave your router if the two endpoints are behind the same router.
"""]]

View file

@ -0,0 +1,80 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 1"
date="2012-07-03T08:26:43Z"
content="""
On \"git syncing\" point number 9, on OSX you could potentially do this on a semi-regular basis
<pre>
system_profiler SPNetworkVolumeDataType
Volumes:
net:
Type: autofs
Mount Point: /net
Mounted From: map -hosts
Automounted: Yes
home:
Type: autofs
Mount Point: /home
Mounted From: map auto_home
Automounted: Yes
</pre>
and
<pre>
x00:~ jtang$ system_profiler SPUSBDataType
USB:
USB High-Speed Bus:
Host Controller Location: Built-in USB
Host Controller Driver: AppleUSBEHCI
PCI Device ID: 0x0aa9
PCI Revision ID: 0x00b1
PCI Vendor ID: 0x10de
Bus Number: 0x26
Hub:
Product ID: 0x2504
Vendor ID: 0x0424 (SMSC)
Version: 0.01
Speed: Up to 480 Mb/sec
Location ID: 0x26200000 / 3
Current Available (mA): 500
Current Required (mA): 2
USB to ATA/ATAPI Bridge:
Capacity: 750.16 GB (750,156,374,016 bytes)
Removable Media: Yes
Detachable Drive: Yes
BSD Name: disk1
Product ID: 0x2338
Vendor ID: 0x152d (JMicron Technology Corp.)
Version: 1.00
Serial Number: 313541813001
Speed: Up to 480 Mb/sec
Manufacturer: JMicron
Location ID: 0x26240000 / 5
Current Available (mA): 500
Current Required (mA): 2
Partition Map Type: MBR (Master Boot Record)
S.M.A.R.T. status: Not Supported
Volumes:
Porta-Disk:
Capacity: 750.16 GB (750,156,341,760 bytes)
Available: 668.42 GB (668,424,208,384 bytes)
Writable: Yes
File System: ExFAT
....
</pre>
I think its possible to programatically get this information either from the CLI (it dumps out XML output if required) or some development library. There is also DBUS in macports, but I have never had much interaction with it, so I don't know if its good or bad on OSX.
"""]]

View file

@ -7,6 +7,9 @@ The webapp is a web server that displays a shiny interface.
token. This guards against other users on the same system. **done**
(I would like to avoid passwords or other authentication methods,
it's your local system.)
* Don't pass the url with secret token directly to the web browser,
as that exposes it to `ps`. Instead, write a html file only the user can read,
that redirects to the webapp. **done**
* Alternative for Linux at least would be to write a small program using
GTK+ Webkit, that runs the webapp, and can know what user ran it, avoiding
needing authentication.
@ -28,11 +31,12 @@ The webapp is a web server that displays a shiny interface.
## implementation
Hope to use Yesod.
TODO: Ensure that Yesod will work on arm. Necessary for later Android port.
Will its template haskell cause a problem? Does new GHC support TH on ARM?
Will it use too much memory or be too slow?
Hopefully Yesod comes with some good UI widgets. Otherwise, need to use
Jquery or similar.
* use `addStaticContent` to make /favicon.ico work. Return `Right (route, query)`
and I think the route can be `favicon_ico`.
* perhaps define a custom `errorHandler`, which could avoid the potential
of leaking auth tokens on error pages
* possibly lose the ugly auth= token past the first page,
and use a client-side session. It could be encrypted using the token
as the `encryptKey`. Note: Would need to set the session duration
to infinite (how?)
* look up "server-sent events" sent using `sendWaiResponse`

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="yatesa"
ip="171.25.193.21"
subject="Secret URL token"
date="2012-06-19T03:41:16Z"
content="""
> Instruct the user's web browser to open an url that contains a secret token. This guards against other users on the same system.
How will you implement that? Running \"sensible-browser URL\" would be the obvious way, but the secret URL would show up in a well timed ps listing. (And depending on the browser, ps may show the URL the entire time it's running.)
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="jtang"
ip="79.97.135.214"
subject="comment 3"
date="2012-07-26T17:35:18Z"
content="""
Using twitter-bootstrap for the webapp - this might be a wishlist item, but would it be possible to ensure that the webapp's css uses twitter-bootstrap classes. It would make theming much easier in the long run and it would give you a nice modern look with a low amount of effort.
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawlYu7QmD7wrbHWkoxuriaA9XcijM-g5vrQ"
nickname="Royal"
subject="comment 2"
date="2012-06-05T17:19:16Z"
content="""
Hi,
I want to replace rsync with aspera-rsync. Whenever there is file transfer between 2 repositories which are in two different hosts, git-annex will use rsync protocol. I am trying to replace that rsync call with aspera-rsync so that transfer can be more faster. Since I am new to Haskell I am finding difficulties to understand the flow of execution. Is there any way I can debug so that I can get the flow?
Thanks
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 2"
date="2012-07-25T06:52:32Z"
content="""
Ah I was looking at the walkthrough on how to fix the issue, I had not thought about looking at the tips section. That tip fixed the issue for me, thanks.
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="https://me.yahoo.com/a/IAg3idYGk.joxsJb2WCxl20gig.0.8hS#d5165"
nickname="Kelly"
subject="comment 2"
date="2012-05-10T15:01:15Z"
content="""
I think my comment a couple days ago got caught in the spam filter, so I'm reposting.
What were the ideas to avoid parameterisation? What were the problems of parameterisation, other than just the current hardcoded assumptions?
Speaking of hash insecurity, http://static.usenix.org/events/hotos03/tech/full_papers/henson/henson_html/node8.html says compare-by-hash is a bad idea. As I understand, git doesn't have an option of verifying content matches when the hash matches when adding data to the object store (like zfs's \"dedup=verify\" option, which you can use even when using sha256), because the assumption is that the risk of collision (or at least just the risk of accidental collision) is negligible. Would it be worthwhile to add this option to git-annex?
"""]]

View file

@ -0,0 +1,11 @@
[[!comment format=mdwn
username="https://me.yahoo.com/a/IAg3idYGk.joxsJb2WCxl20gig.0.8hS#d5165"
nickname="Kelly"
subject="comment 2"
date="2012-05-09T01:22:13Z"
content="""
What were the ideas to avoid parameterisation? What were the problems of parameterisation, other than just the current hardcoded assumptions?
Speaking of hash insecurity, http://static.usenix.org/events/hotos03/tech/full_papers/henson/henson_html/node8.html says compare-by-hash is a bad idea. As I understand, git doesn't have an option of verifying content matches when the hash matches when adding data to the object store (like zfs's \"dedup=verify\" option, which you can use even when using sha256), because the assumption is that the risk of collision (or at least just the risk of accidental collision) is negligible. Would it be worthwhile to add this option to git-annex?
"""]]

View file

@ -0,0 +1,18 @@
[[!comment format=mdwn
username="http://christian.amsuess.com/"
nickname="chrysn"
subject="comment 2"
date="2012-05-11T17:40:20Z"
content="""
from my layman's standpoint, i think it would be feasible. i've suggested this previously, but not pushed it too much. quoting from [[my user page|users/chrysn]]:
* **would like git-annex to**: not be required any more as git itself learns to use cow filesystems to avoid abundant disk usage and gets better with sparser checkouts (git-annex might then still be a simpler tool that watches over what can be safely dropped for a sparser checkout)
*concerning hash sizes or parameterized hashes*: the problems with hash sizes could be avoided if instead of putting the objects in the \"normal\" object dir, barefiles would be managed in a similar way as packs are. when a new files gets added, they'd be cow-copied to ``.git/objects/bare/${HA}/${SH}``, and ``.git/objects/bareprefix/${HA}/${SH}`` would contain the \"blob ${SIZE}\0\" prefix that gets concatenated to the object body to form the object itself.
(maybe it'd even be sufficient to *just store the size* in the bareprefix, as all those objects would be blobs, but then again, some flexibility won't hurt.)
if the *pack file format* is flexible enough, the bareprefix files can get packed too. for the adventerous user who modifies bigfiles, the pack file mechanisms should be made aware of their presence, and be able to store deltas between them. the operations for applying those deltas would be difficult to optimize, and could be added at a later stage. a typical example could be storing a pdf file -- the pdf file format is designed for appending, so chances are the new version is just the old version plus several k at the end.
neither of that would affect git's *wire protocol*, so no compatibility problems. (it would be advisable to find a reasonable way to do sparse checkouts, though; something like \"server, pack and send your master, but make it sparse and don't include blobs >1mb\").
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawlYu7QmD7wrbHWkoxuriaA9XcijM-g5vrQ"
nickname="Royal"
subject="Resolving conflict"
date="2012-04-24T03:59:31Z"
content="""
Hi,
Now I am able to resolve the conflict.
Thank you.
"""]]

View file

@ -0,0 +1,41 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawlYu7QmD7wrbHWkoxuriaA9XcijM-g5vrQ"
nickname="Royal"
subject="Resolving conflict"
date="2012-04-23T15:49:30Z"
content="""
Thanks for the reply.
I am executing the following commands.
git init main
cd main
git annex init main
echo a > a
git annex add a
git commit -m Initial
git annex unlock a
echo aa > a
git annex add a
git commit -m first
git annex unlock a
echo aaa > a
git annex add a
git commit -m second
git log
git cherry-pick <Hash of first commit>
--------------------
Error:
error: could not apply 2be8f38... first
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
How can resolve the the above conflict.
If I see the content of the file I will get the content of second commit.
Is there any way I can get the content for first commit(Like in git we have 'theirs' option.)
Thank you.
"""]]

View file

@ -0,0 +1,18 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo"
nickname="Justin"
subject="comment 2"
date="2012-06-27T12:45:42Z"
content="""
I have a hacked up version of sharebox that does this.. I need to fix it up and push it to github..
the short of it is that you can do
def calculate_size(path):
annexfile = os.path.basename(os.readlink(path))
#SHA256-s2007550713--....
size = annexfile.split(\"-\")[1]
return int(size[1:])
to get the size of files.. a 'git-annex du' should be pretty straightforward...
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 1"
date="2012-06-23T08:00:12Z"
content="""
actually, scratch that, i found it. it was in _.git/annex/daemon.log_ along with the other bits and pieces
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 2"
date="2012-07-03T14:48:16Z"
content="""
Adding a date and timestamp would be a nice start to improving things.
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 1"
date="2012-07-02T16:25:55Z"
content="""
I've some binaries for OSX which can be found at <http://www.sgenomics.org/~jtang/gitbuilder-git-annex-x00-x86_64-apple-darwin10.8.0-binary/dist/> its just the master branch, and it's built on a system that runs macports. Binaries are built and updated whenever there are changes made to the master branch of git-annex.
"""]]

View file

@ -0,0 +1,16 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0"
nickname="Tobias"
subject="Thats a fair solution"
date="2012-07-22T13:51:25Z"
content="""
Until you don't have(access to) an existing repository to clone from.
I really hope you revisit this when you come to the encryption part of the assistant.
Btw, I also run FreeBSD if you need a tester on that at some point.
Sincerely
Tobias Ussing
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="hannes"
ip="130.226.142.243"
subject="original repo git annex version"
date="2012-07-08T11:55:42Z"
content="""
is 3.20120629
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus"
nickname="Jimmy"
subject="comment 17"
date="2012-07-24T06:33:13Z"
content="""
@a-or-b that issue is logged here [[bugs/subtle build issue on OSX 10.7 and Haskell Platform (if you have the 32bit version installed)]], you can use cabal to build and install git-annex and it will detect if its 32 or 64bit automatically.
"""]]

View file

@ -0,0 +1,19 @@
[[!comment format=mdwn
username="https://www.google.com/accounts/o8/id?id=AItOawmURXBzaYE1gmVc-X9eLAyDat_6rHPl670"
nickname="Bram"
subject="Build failure using Cabal"
date="2012-06-24T19:46:13Z"
content="""
Unfortunately I get a build failure when building this version using Cabal:
[113 of 183] Compiling Assistant.Changes ( Assistant/Changes.hs, dist/build/git-annex/git-annex-tmp/Assistant/Changes.o )
Assistant/Changes.hs:73:30:
Not in scope: `tryReadTChan'
Perhaps you meant `readTChan' (imported from Control.Concurrent.STM)
cabal: Error: some packages failed to install:
git-annex-3.20120624 failed during the building phase. The exception was:
ExitFailure 1
The complete build log is at <http://paste.debian.net/176125/>. I'm looking forward to trying out the new watch behaviour!
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://me.yahoo.com/a/2djv2EYwk43rfJIAQXjYt_vfuOU-#a11a6"
nickname="Olivier R"
subject="It doesn't work 100%"
date="2012-05-03T21:42:54Z"
content="""
When you remove tracked files... it doesn't show the new status. it's like if the file was ignored.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="https://rmunn.myopenid.com/"
nickname="rmunn"
subject="comment 1"
date="2012-06-12T15:52:35Z"
content="""
* One way to handle the configuration might be with regular expressions. If the URL matches regex A, handle it with downloader A' (with option set A''). If the URL matches regex B, handle it with downloader B' and option set B''. And so on. Then if nothing is matched, the default downloader is wget/curl.
* In my experience, youtube-dl breakages are fixed relatively quickly; a much more serious problem from a trust standpoint is that Youtube videos often disappear. Sometimes due to a legitimate copyright claim, sometimes due to illegitimate copyright claims. (I've seen both happen). Or because the video uploader decided to upload *other* videos that violated copyright, and Youtube closed his/her account, thereby removing *all* his/her videos from the Web. Youtube is definitely an untrustworthy repository as far as \"the file will still be there later on\" is concerned. Perhaps a default trust relationship could go along with the regexes? URLs matching regex A are semitrusted, while URLs matching regex B are untrusted.
"""]]