Commit graph

2618 commits

Author SHA1 Message Date
Joey Hess
7ae16bb6f7 Revert "let url claims optionally include a suggested filename"
This reverts commit 85df9c30e9.

Putting filename in the claim was a bad idea.
2014-12-11 14:09:57 -04:00
Joey Hess
85df9c30e9 let url claims optionally include a suggested filename 2014-12-11 12:47:57 -04:00
Joey Hess
2f6150b48d typo 2014-12-08 19:27:35 -04:00
Joey Hess
30bf112185 Urls can now be claimed by remotes. This will allow creating, for example, a external special remote that handles magnet: and *.torrent urls. 2014-12-08 19:15:07 -04:00
Joey Hess
ee27298b91 implement CLAIMURL for external special remote 2014-12-08 13:57:13 -04:00
Joey Hess
8093008ef4 External special remote protocol now includes commands for setting and getting the urls associated with a key. 2014-12-08 13:32:46 -04:00
Joey Hess
c66b658eaa fix language 2014-12-05 13:40:44 -04:00
Yaroslav Halchenko
0efe9825d0 DOC: minor typos and rewording in few docs 2014-12-04 22:28:07 -05:00
2001:41d0:fe61:2600:96de:80ff:feb2:56d
5469f7d21d poll vote (/sdcard/annex) 2014-11-09 00:35:20 +00:00
2003:51:4f3b:6200:b00d:4d41:366d:5b54
f2357222f0 poll vote (/sdcard/annex) 2014-11-08 23:33:52 +00:00
Joey Hess
fa9d31cc91 this is either spam, or indistinguishable from spam 2014-10-21 12:30:53 -04:00
debryd
ec28d14ccf Added a comment 2014-10-21 01:39:10 +00:00
Joey Hess
7b50b3c057 fix some mixed space+tab indentation
This fixes all instances of " \t" in the code base. Most common case
seems to be after a "where" line; probably vim copied the two space layout
of that line.

Done as a background task while listening to episode 2 of the Type Theory
podcast.
2014-10-09 15:09:11 -04:00
https://www.google.com/accounts/o8/id?id=AItOawlM_DRhi_5pJrTA0HbApHR25iAgy-NBXTY
9b7bbe334d Added a comment 2014-10-01 23:35:39 +00:00
https://www.google.com/accounts/o8/id?id=AItOawlM_DRhi_5pJrTA0HbApHR25iAgy-NBXTY
faf19e1253 Added a comment: Can tags/metadata be used for preferred content? 2014-10-01 22:45:36 +00:00
https://www.google.com/accounts/o8/id?id=AItOawlM_DRhi_5pJrTA0HbApHR25iAgy-NBXTY
16fd5c00c5 Added a comment 2014-10-01 22:43:40 +00:00
http://joeyh.name/
27f10e6081 Added a comment 2014-09-16 18:36:18 +00:00
https://www.google.com/accounts/o8/id?id=AItOawmH7o6q2l99M-PQolOfbR3_i5B_jtTIcAE
d3aa8223b0 mark annex.genmetadata as code. 2014-09-12 19:53:33 +00:00
Joey Hess
eb26f00c04 devblog and look back over the past year 2014-09-11 16:15:16 -04:00
71.207.150.202
c4bfd38877 poll vote (/sdcard/annex) 2014-09-09 22:38:22 +00:00
Joey Hess
b4cf22a388 pushed checkPresent exception handling out of Remote implementations
I tend to prefer moving toward explicit exception handling, not away from
it, but in this case, I think there are good reasons to let checkPresent
throw exceptions:

1. They can all be caught in one place (Remote.hasKey), and we know
   every possible exception is caught there now, which we didn't before.
2. It simplified the code of the Remotes. I think it makes sense for
   Remotes to be able to be implemented without needing to worry about
   catching exceptions inside them. (Mostly.)
3. Types.StoreRetrieve.Preparer can only work on things that return a
   Bool, which all the other relevant remote methods already did.
   I do not see a good way to generalize that type; my previous attempts
   failed miserably.
2014-08-06 13:45:19 -04:00
Joey Hess
b261df735d convert bup to new ChunkedEncryptable API (but do not support chunking)
bup already splits files and does rolling deltas, so there is no reason to
use chunking here.

The new API made it easier to add progress support for storeKey, so that's
done. Unfortunately, bup-split still outputs its own progress with -q,
so a little ugly, but not too bad.

Made dropping remove the branch for an object, for two reasons:

1. The new API calls removeKey to roll back a storeKey when the content
   changed unexpectedly.
2. So that testremote will be happy.

Also, fixed a bug that caused a crash when removing the branch for an
object in rollback.
2014-08-02 18:48:49 -04:00
Joey Hess
0f7b8de91b update roadmap 2014-08-01 18:22:13 -04:00
Joey Hess
5aa2286e7b Merge branch 'newchunks'
I am happy enough with this to make it live!
2014-08-01 18:00:47 -04:00
Joey Hess
83c58f17e0 mention possibility of parallel chunk upload/download 2014-07-30 10:43:21 -04:00
Joey Hess
b39506d1cf expand to rolling hash based design 2014-07-28 17:11:37 -04:00
Joey Hess
2474cf0032 make explicit the implicit requirement that CHECKPRESENT not say a key is present until it's all done being stored 2014-07-28 14:37:22 -04:00
Joey Hess
e47182920c chunk log format should be extensible to allow for eg, logging when rolling hash chunks are used 2014-07-28 13:00:46 -04:00
Joey Hess
85d17a698d Merge branch 'master' into newchunks
Conflicts:
	doc/design/assistant/chunks.mdwn
2014-07-27 12:24:03 -04:00
Joey Hess
729d38a763 update 2014-07-27 12:23:28 -04:00
Joey Hess
1755c5de40 thought about chunk key hashing 2014-07-25 15:12:51 -04:00
Joey Hess
e2c44bf656 implement chunk logs
Slightly tricky as they are not normal UUIDBased logs, but are instead maps
from (uuid, chunksize) to chunkcount.

This commit was sponsored by Frank Thomas.
2014-07-24 16:23:36 -04:00
Joey Hess
d751591ac8 add chunk metadata to Key
Added new fields for chunk number, and chunk size. These will not appear
in normal keys ever, but will be used for chunked data stored on special
remotes.

This commit was sponsored by Jouni K Seppanen.
2014-07-24 13:36:23 -04:00
Joey Hess
937197842e update 2014-07-24 12:41:34 -04:00
Joey Hess
ca1d80d708 chunk then encrypt 2014-07-23 22:38:14 -04:00
Joey Hess
a354bf488b link 2014-07-23 17:56:26 -04:00
Joey Hess
f15c1fdc8f minor 2014-07-23 17:55:28 -04:00
Joey Hess
20627e9fab 4 designs for better chunking
Having a hard time finding a way to totally obscure file sizes, but
otherwise happy with design #4.

This commit was sponsored by Michael Alan Dorman.
2014-07-23 17:41:48 -04:00
134.102.206.230
2ed3679809 poll vote (/sdcard/annex) 2014-07-21 12:40:30 +00:00
Joey Hess
eb015a1c96 update month 2014-07-08 12:39:07 -04:00
66.87.139.102
00cda7d582 poll vote (OpenStack SWIFT) 2014-07-07 05:44:00 +00:00
202.6.117.189
3cfcd54dad poll vote (OpenStack SWIFT) 2014-07-03 00:30:26 +00:00
86.132.35.224
91617d4a69 poll vote (Box.com (done)) 2014-06-28 19:32:13 +00:00
212.79.181.167
15133688c9 poll vote (My phone (or MP3 player)) 2014-06-28 00:23:11 +00:00
https://www.google.com/accounts/o8/id?id=AItOawnwfqF4wL6l_O26RyzoBowUMvQ_955Vpao
0a478056dd Added a comment 2014-06-14 06:29:55 +00:00
Joey Hess
a7c15d0e51 update 2014-06-12 18:46:07 -04:00
Joey Hess
5903f438f0 change month 2014-06-04 15:16:26 -04:00
Joey Hess
a3be4c8f09 fix dead link 2014-06-04 15:16:03 -04:00
Joey Hess
0d70e36466 fix broken link 2014-05-30 15:32:42 -04:00
Joey Hess
970605964c
remove spam comment (will ban this spammer's openid) 2014-05-29 13:20:34 -04:00
https://www.google.com/accounts/o8/id?id=AItOawlAx-pYJ036y2lgKtLJvXm7s-HFRBiJlZY
730710b097 Added a comment 2014-05-28 13:27:20 +00:00
92.192.126.78
7a10cec87e poll vote (Same as for regular git-annex. ~/annex/) 2014-05-28 08:55:36 +00:00
Joey Hess
80e4d10afb update 2014-05-23 12:09:02 -04:00
http://joeyh.name/
8c92e0fb2e Added a comment 2014-05-19 15:47:19 +00:00
2a02:898:109::201:1
8544dd752f poll vote (/sdcard/annex) 2014-05-16 18:57:18 +00:00
Joey Hess
88d2a61947 update some comments re ssh password entry on Android 2014-05-15 15:20:24 -04:00
Joey Hess
dc72ea4ab5 deal with ssh key expiry
Not a perfect solution, but good enough, few users will wait 10 minutes in
the middle and see it expire, I hope.
2014-05-15 15:16:37 -04:00
Joey Hess
acbac3b8af Merge branch 'master' into sshpassword 2014-05-15 14:42:33 -04:00
Joey Hess
8cf3534194 gcrypt needs a new version, which I am releaseing 2014-05-15 14:36:39 -04:00
Joey Hess
a11176bab3 avoid clobbering existing env
This is necessary on windows, ssh couldn't resolve hostnames without env
for whatever reason.
2014-05-14 16:36:03 -04:00
Joey Hess
85e9e8c0cf webapp: Better ssh password prompting.
When setting up a remote on a ssh server, prompt for a password inside the
webapp, rather than relying on ssh's own password prompting in the terminal
the webapp was started from, or ssh-askpass.

Avoids double prompting for the ssh password (and triple-prompting on
windows for rsync.net), since the entered password is cached for 10 minutes
and this cached password is reused when setting up the repository, after
the initial probe.

When the user has an existing ssh key set up, they can choose to use it,
rather than entering a password. The webapp used to probe for this case
automatically, so this is a little harder, but it's an advanced user thing.

Note that this commit is known to break enabling existing rsync
repositories. It hs not been tested with gcrypt repositories. It's not been
successfully tested yet on Windows.

This commit was sponsored by Ralph Mayer.
2014-05-14 15:38:32 -04:00
212.194.32.144
51ea854367 poll vote (Google Drive) 2014-05-11 21:19:09 +00:00
Joey Hess
6abc6fe9d6 keep track of satisfied requests, and summarize 2014-05-09 16:41:05 -03:00
Joey Hess
4a99d835ba example preferred content expressions 2014-05-09 13:34:36 -03:00
Joey Hess
309fd966e7 update 2014-05-09 12:47:08 -03:00
http://johan.kiviniemi.name/
4d2a244a22 Added a comment 2014-05-07 06:27:11 +00:00
Joey Hess
312d82a2fd
devblog 2014-05-06 17:53:22 -03:00
Joey Hess
217774f72e Merge branch 'master' of ssh://git-annex.branchable.com 2014-05-06 16:33:06 -03:00
Joey Hess
5230c78bb6 reorg and add a start of a design for requests and ad-hoc routing with TTL in git-annex branch 2014-05-06 16:32:41 -03:00
50.56.228.64
52173d97c0 poll vote (OpenStack SWIFT) 2014-05-05 17:14:06 +00:00
Joey Hess
f380085e26 improved reporting 2014-05-03 11:55:01 -03:00
Joey Hess
a5c5704b54 add mocambos network data 2014-05-03 10:42:13 -03:00
Joey Hess
28959686b0 use a map so immobile nodes have names 2014-05-03 09:49:39 -03:00
Joey Hess
f2186ee6ae add routing simulation 2014-05-02 15:09:46 -03:00
tdussa
1e446b2d50 Added a comment: Why not automatically add the whole date? 2014-04-30 20:41:21 +00:00
Joey Hess
79cf404e75 support being run by ssh as ssh-askpass replacement
To use, set GIT_ANNEX_SSHASKPASS to point to a fifo or regular file
(FIFO is better, avoids touching disk or multiple readers) that contains
the password. Then set SSH_ASKPASS=git-annex, and when ssh runs it, it will
tell ssh the password.

This is not yet used..
2014-04-29 18:08:10 -04:00
https://www.google.com/accounts/o8/id?id=AItOawnXybLxkPMYpP3yw4b_I6IdC3cKTD-xEdU
4a37557ff4 Added a comment 2014-04-29 10:10:04 +00:00
Joey Hess
b188561ba2 update 2014-04-26 18:58:06 -04:00
Joey Hess
5f521f6952
devblog 2014-04-25 16:34:33 -04:00
Joey Hess
2adeef61a1 design 2014-04-25 16:28:58 -04:00
https://id.koumbit.net/anarcat
19f8b57ace typo 2014-04-21 21:27:17 +00:00
Joey Hess
7261fac77e update 2014-04-21 17:18:40 -04:00
Mesar Hameed
698ef8ed26 fix typos. 2014-04-16 10:41:07 +02:00
78.49.117.156
4365d1faf1 poll vote (My phone (or MP3 player)) 2014-04-16 00:13:19 +00:00
Joey Hess
55102467ac
fix RESUME 2014-04-14 14:20:10 -04:00
Joey Hess
138d25518d Merge branch 'master' into remotecontrol
Conflicts:
	doc/devblog/day_152__more_ssh_connection_caching.mdwn
2014-04-14 13:38:35 -04:00
Joey Hess
da253358b7 ssh transport seems ready, needs testing 2014-04-12 18:39:50 -04:00
Joey Hess
ff192ace2f detect wicd network disconnection events 2014-04-12 18:36:48 -04:00
Joey Hess
bb44df206d NetWatcher: detect when networkmanager has lost network connection
This is a better approach to finding both when NM has lost a network
connection, and when a new network connection is made by NM.

Tested with network-manager 0.9.8.8.

This commit was sponsored by Cedric Staub.
2014-04-12 17:58:19 -04:00
Joey Hess
a33b30d0c4 remotedaemon: When network connection is lost, close all cached ssh connections.
This commit was sponsored by Cedric Staub.
2014-04-12 16:32:59 -04:00
2601:0:b100:7c0::1
1b3a48f992 poll vote (DCIM directory (photos and videos only)) 2014-04-12 19:38:11 +00:00
Joey Hess
96ce2812e0 Merge branch 'master' into remotecontrol
Conflicts:
	debian/changelog
2014-04-12 14:20:41 -04:00
Joey Hess
4cca9cef0c
update 2014-04-09 16:43:43 -04:00
Joey Hess
33b8cff433 webapp: Show a network signal icon next to ssh remotes that it's currently connected with. 2014-04-09 15:26:41 -04:00
Joey Hess
356eec08a6
typo 2014-04-09 14:22:59 -04:00
Joey Hess
fb73792f72 make git-remote-daemon ssh transport robust
* Remote system might be available, and connection get lost. Should
  reconnect, but needs to avoid bad behavior (ie, constant reconnect
  attempts.) Use exponential backoff.

* Detect if old system had a too old git-annex-shell, and show the user
  a nice message in the webapp. Required parsing error messages, so perhaps
  this code shoudl be removed once enough time has passed..

* Switch the protocol to using remote URI's, rather than remote names.
  Names change. Also avoids issues with serialization of names containing
  whitespace.

This is nearly ready for merge into master now. I'd still like to make the ssh
transport smarter about reusing ssh connection caching during git pull.

This commit was sponsored by Jim Paris.
2014-04-09 14:14:55 -04:00
Joey Hess
ccfceb887c update 2014-04-08 15:30:32 -04:00
Joey Hess
fac7bca05b assistant: Now detects immediately when other repositories push changes to
a ssh remote, and pulls.

XMPP is no longer needed in this configuration!

Requires the remote server have git-annex-shell with notifychanges support.

(untested)

This commit was sponsored by Geog Wechslberger.
2014-04-08 15:23:50 -04:00
Joey Hess
9a4a3bfb43 fix STOP 2014-04-08 13:51:49 -04:00
Joey Hess
fa0cf81b26 cleaned up AnnexState handling in transports 2014-04-08 13:42:57 -04:00
Joey Hess
b3b07ab330 clarify 2014-04-08 12:09:24 -04:00
https://id.koumbit.net/anarcat
f9824f0f55 clarify that this is mostly done (i think?) 2014-04-07 04:41:56 +00:00
Joey Hess
43909723b3 added git-annex remotedaemon
So far, handling connecting to git-annex-shell notifychanges, and
pulling immediately when a change is pushed to a remote.

A little bit buggy (crashes after the first pull), but it already works!

This commit was sponsored by Mark Sheppard.
2014-04-06 19:10:23 -04:00
Joey Hess
0fbbec261d git-annex-shell: Added notifychanges command.
This will be used by the remote-daemon to quickly tell when changes have
been pushed from some other repository into a ssh remote.

Adjusted the remote-daemon protocol to communicate changed shas, rather
than git branch refs. This way, it can easily check if a sha is new.

This commit was sponsored by Carlos Trijueque Albarran.
2014-04-05 16:10:39 -04:00
http://johan.kiviniemi.name/
0c48ba389e Added a comment: Rolling hash chunking 2014-04-04 14:16:25 +00:00
Joey Hess
18b85e6caf
update 2014-04-03 15:28:36 -04:00
Joey Hess
5735f73af6
update 2014-04-03 15:25:14 -04:00
Joey Hess
4d5c01bc52
big todo 2014-04-03 15:23:53 -04:00
Joey Hess
dde5f8527c
update 2014-04-03 15:07:34 -04:00
Joey Hess
160abff4f6
reload 2014-04-03 14:56:29 -04:00
Joey Hess
32f44e7b83
add design for git-remote-daemon 2014-04-03 14:53:09 -04:00
Joey Hess
9b707f0116
windows fix try 2 2014-04-02 18:02:32 -04:00
Joey Hess
8e21d26cd1 update roadmap month 2014-04-02 17:14:19 -04:00
216.115.165.4
4e90050b34 poll vote (Google Drive) 2014-03-21 20:46:26 +00:00
131.130.79.56
c6b714c8df poll vote (Google Drive) 2014-03-21 12:07:22 +00:00
Joey Hess
fac8198a2f fix up old comments that used format=txt, which is no longer enabled 2014-03-20 12:09:09 -04:00
Joey Hess
07678409c7 link to another place this could be used, perhaps 2014-03-18 15:53:06 -04:00
Joey Hess
21b8461483 link to another item 2014-03-18 15:31:41 -04:00
192.171.33.111
f58b37daa2 poll vote (/sdcard/annex) 2014-03-17 16:08:53 +00:00
93.128.10.19
bac2bd7d49 poll vote (/sdcard/annex) 2014-03-16 12:38:41 +00:00
78.53.214.75
7ad37ffd6f poll vote (DCIM directory (photos and videos only)) 2014-03-16 10:18:12 +00:00
194.197.235.109
a109d50684 poll vote (/sdcard/annex) 2014-03-16 07:53:27 +00:00
http://joeyh.name/
593d9a7537 Added a comment 2014-03-15 21:30:52 +00:00
bremner
739f3d56a1 Added a comment: convenient way to query metadata? 2014-03-15 20:58:28 +00:00
bremner
747193b590 removed 2014-03-15 20:54:57 +00:00
bremner
664a45b082 Added a comment: How to read metadata conveniently? 2014-03-15 20:49:00 +00:00
Joey Hess
05336116a4 implemntation plan 2014-03-13 19:37:41 -04:00
Joey Hess
69a1aaa40f
doubled speed with esqeleto 2014-03-13 11:09:05 -04:00
Joey Hess
3e86d35e84
fixed slow query on normalized table; still 10x slower than current .map files 2014-03-13 09:38:20 -04:00
Joey Hess
46aab35eb0 add 2014-03-12 18:11:35 -04:00
166.70.207.2
eb7e77bf34 poll vote (Tahoe-LAFS) 2014-03-11 20:35:18 +00:00
Joey Hess
ba4df0f112 update roadmap month 2014-03-05 13:27:18 -04:00
Joey Hess
d0fce426c4 pre-commit-annex hook script to automatically extract metadata from lots of types of files
Using the extract(1) program to do the heavy lifting.

Decided to make git-annex run pre-commit-annex when committing. Since
git-annex pre-commit also runs it, it'll be run when git commit is run too,
via the pre-commit hook. This basically gives back the pre-commit hook
that git-annex took away. The implementation avoids repeatedly looking
for the hook script when the assistant is running and committing
repeatedly; only checks if the hook is available once.

To make the script simpler, made git-annex metadata -s field?=value
only set a field when it's not already got a value.

This commit was sponsored by bak.
2014-03-02 20:11:58 -04:00
Joey Hess
34059c09e2 update 2014-03-02 16:27:04 -04:00
Joey Hess
06e39e6a79 update 2014-03-02 15:53:31 -04:00
Joey Hess
06e9080f01 metadata: FIeld names are now case insensative. 2014-02-25 18:45:09 -04:00
https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus
e52f64f202 Added a comment 2014-02-25 09:57:09 +00:00
https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus
5ebabc0841 Added a comment 2014-02-25 09:51:17 +00:00
Joey Hess
0506764fc3 update 2014-02-24 14:45:19 -04:00
Joey Hess
a26ae550a5 more todos 2014-02-23 00:25:35 -04:00
Joey Hess
5758f34e72 Merge branch 'master' of ssh://git-annex.branchable.com 2014-02-23 00:13:55 -04:00
Joey Hess
7498c5dd96 annex.genmetadata can be set to make git-annex automatically set metadata (year and month) when adding files 2014-02-23 00:08:29 -04:00
https://www.google.com/accounts/o8/id?id=AItOawm3vKzS4eOWYpKMoYXqMIjNsIg_nYF-loU
bf0d1d6017 Added a comment: Already existing metadata implementation 2014-02-22 21:45:26 +00:00
Joey Hess
c6fae24f7c etoolong 2014-02-22 16:37:13 -04:00
Joey Hess
2d2a480325 Merge branch 'master' of ssh://git-annex.branchable.com 2014-02-22 16:35:48 -04:00
Joey Hess
079b35a1a8 views: add automatically constructed file location metadata
When constructing views, metadata is available about the location of the
file in the view's reference branch. Allows incorporating parts of the
directory hierarchy in a view.

For example `git annex view tag=* podcasts/=*` makes a view in the form
tag/showname.

Performance impact: I benchmarked git annex view tag=* in the conference
proceedings repo to take 6.459s before this change, and 6.544s after.

FWIW, I considered making the syntax for this be podcasts/*, which might
be easier for the user to learn. However, I think it's not as good:

* The user has to then juggle two different syntaxes, and podcasts/* will
  be expanded by the shell so they also need to quote it, while podcasts/=*
  is unlikely to be expanded by the shell.
* It would allow for things like podcasts/*/* and *.mp3 which do not
  map well into views.

This commit was sponsored by Aurélien Pinceaux.
2014-02-22 16:27:53 -04:00
https://id.koumbit.net/anarcat
505db554f6 add missing traillink, so that we don't jump from disaster recovery to telehash.. 2014-02-22 05:19:52 +00:00
Joey Hess
ddba85bf72
bold 2014-02-20 16:42:24 -04:00
Joey Hess
991f0ad5c2 update roadmap 2014-02-20 16:38:49 -04:00
Joey Hess
19faebd962 better syntax for directory metadata 2014-02-20 16:36:30 -04:00