From e0e40964ab2d0a305fe2503215f306e1228aac24 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU" Date: Thu, 8 Dec 2011 22:42:29 +0000 Subject: [PATCH 1/3] --- ..._no_fixed_hostname_and_optimising_ssh.mdwn | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn diff --git a/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn b/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn new file mode 100644 index 0000000000..db108f48b9 --- /dev/null +++ b/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn @@ -0,0 +1,72 @@ +## Intro + +This tip is based on my experience of using `git annex` with my out-and-about netbook which hits many different wifi networks and has no fixed home or address. + +I'm not using a bare repository that allows pushing (an alternate way to sort the issue) nor did I fancy allowing `git push` to run against my desktop checked out repository. + +None of this stuff is really `git annex` specific but is useful to know... + +## Dealing with no fixed hostname + +Essentially set up two repos as per the [[walkthrough]]. + +Desktop as follows: + + cd ~/annex + git init + git annex init "desktop" + +And the laptop like this + + git clone ssh://desktop/annex + git init + git annex init "laptop" + +Now we want to add the the repos as remotes of each other. + +For the laptop it is easy: + + git remote add desktop ssh://desktop/~/annex + +However for the desktop to add an ever changing laptops hostname it's a little tricky. We make use of remote SSH tunnels to do this. Essentially we have the laptop which always knows it's own name and address and that of the desktop create a tunnel starting on an arbitrary port on the desktop and heading back to itself on it's own SSH server port (22). + +To do this make part of your laptops SSH config look like this: + + Host desktop + User matt + HostName desktop.example.org + RemoteForward 2222 localhost:22 + +Now on the desktop to connect over the tunnel to the laptops SSH port you need this: + + Host laptop + User matt + HostName localhost + port 2222 + +So to add the desktops remote: + +a) From the laptop ensure the tunnel is up + + ssh desktop + +b) From the desktop add the remote + + git remote add laptop ssh://laptop/~/annex + +So now you can work on the train, pop on the wifi at work upon arrival and sync up with a `git pull && git annex get`. + +An alternate solution maybe to use direct tunnels over Openvpn. + +## Optimising SSH + +Running a `git annex get .`, at least in the version I have, creates a new SSH connection for every file transfer (maybe this is a feature request?) + +Lot's of new small files in an _annex_ cause lot's of connections to be made quickly: this is an relatively expensive overhead and is enough for connection limiting to start in my case. The process can be made much faster by using SSH's connection sharing capabilities. An SSH config like this should do it: + + # Global Settings + ControlMaster auto + ControlPersist 30 + ControlPath ~/.ssh/master-%r@%h:%p + +This will create a master connection for sharing if one isn't present, maintain it for 30 seconds after closing down the connection (just-in-cases') and automatically use the master connection for subsequent connections. Wins all round! From b3ac4af6b0722db35b8295d1131b41c827cdd708 Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU" Date: Thu, 8 Dec 2011 22:50:45 +0000 Subject: [PATCH 2/3] --- ...h_no_fixed_hostname_and_optimising_ssh.mdwn | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn b/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn index db108f48b9..3f02ea47d3 100644 --- a/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn +++ b/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn @@ -2,9 +2,9 @@ This tip is based on my experience of using `git annex` with my out-and-about netbook which hits many different wifi networks and has no fixed home or address. -I'm not using a bare repository that allows pushing (an alternate way to sort the issue) nor did I fancy allowing `git push` to run against my desktop checked out repository. +I'm not using a bare repository that allows pushing (an alternative solution) nor do I fancy allowing `git push` to run against my desktop checked out repository (perhaps I worry over nothing?) -None of this stuff is really `git annex` specific but is useful to know... +None of this is really `git annex` specific but I think it is useful to know... ## Dealing with no fixed hostname @@ -28,23 +28,23 @@ For the laptop it is easy: git remote add desktop ssh://desktop/~/annex -However for the desktop to add an ever changing laptops hostname it's a little tricky. We make use of remote SSH tunnels to do this. Essentially we have the laptop which always knows it's own name and address and that of the desktop create a tunnel starting on an arbitrary port on the desktop and heading back to itself on it's own SSH server port (22). +However for the desktop to add an ever changing laptops hostname it's a little tricky. We make use of remote SSH tunnels to do this. Essentially we have the laptop (which always knows it's own name and address and knows the address of the desktop) create a tunnel starting on an arbitrary port at the desktop and heads back to the laptop on it's own SSH server port (22). -To do this make part of your laptops SSH config look like this: +To do this make part of your laptop's SSH config look like this: Host desktop User matt HostName desktop.example.org RemoteForward 2222 localhost:22 -Now on the desktop to connect over the tunnel to the laptops SSH port you need this: +Now on the desktop to connect over the tunnel to the laptop's SSH port you need this: Host laptop User matt HostName localhost port 2222 -So to add the desktops remote: +So to add the desktop's remote: a) From the laptop ensure the tunnel is up @@ -54,13 +54,13 @@ b) From the desktop add the remote git remote add laptop ssh://laptop/~/annex -So now you can work on the train, pop on the wifi at work upon arrival and sync up with a `git pull && git annex get`. +So now you can work on the train, pop on the wifi at work upon arrival, and sync up with a `git pull && git annex get`. -An alternate solution maybe to use direct tunnels over Openvpn. +An alternative solution may be to use direct tunnels over Openvpn. ## Optimising SSH -Running a `git annex get .`, at least in the version I have, creates a new SSH connection for every file transfer (maybe this is a feature request?) +Running a `git annex get .`, at least in the version I have, creates a new SSH connection for every file transfer (maybe this should be a feature request?) Lot's of new small files in an _annex_ cause lot's of connections to be made quickly: this is an relatively expensive overhead and is enough for connection limiting to start in my case. The process can be made much faster by using SSH's connection sharing capabilities. An SSH config like this should do it: From e664af5d8cf310ac3dcb7e27a5a6846360917fbf Mon Sep 17 00:00:00 2001 From: "https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU" Date: Fri, 9 Dec 2011 10:03:20 +0000 Subject: [PATCH 3/3] --- ...ing_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn b/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn index 3f02ea47d3..8fb2bf9db1 100644 --- a/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn +++ b/doc/tips/using_git_annex_with_no_fixed_hostname_and_optimising_ssh.mdwn @@ -1,6 +1,6 @@ ## Intro -This tip is based on my experience of using `git annex` with my out-and-about netbook which hits many different wifi networks and has no fixed home or address. +This tip is based on my (Matt Ford) experience of using `git annex` with my out-and-about netbook which hits many different wifi networks and has no fixed home or address. I'm not using a bare repository that allows pushing (an alternative solution) nor do I fancy allowing `git push` to run against my desktop checked out repository (perhaps I worry over nothing?)