Commit graph

32465 commits

Author SHA1 Message Date
Joey Hess
d65a081f3f
improve message 2018-09-02 16:17:50 -04:00
Joey Hess
d0ef049cca
comment typo 2018-09-02 16:16:08 -04:00
Joey Hess
813ee2357c
improve message 2018-09-02 16:08:00 -04:00
Joey Hess
522c5cce58
devblog 2018-08-31 14:18:38 -04:00
Joey Hess
8eb944ea11
close todo, open todo 2018-08-31 14:01:24 -04:00
Joey Hess
308f49e9ae
update 2018-08-31 13:56:32 -04:00
Joey Hess
53d839d543
more efficient encoding 2018-08-31 13:49:08 -04:00
Joey Hess
24b76cb8e0
fix prefixing 2018-08-31 13:39:50 -04:00
Joey Hess
b3d42283ad
use per-remote metadata storage for S3 version ID
Since the same key can be stored in a versioned S3 bucket multiple times
with different version IDs, this allows tracking them all. Not currently
needed, but if we ever want to drop from a versioned S3 bucket, we'll
need to know them all.

This commit was supported by the NSF-funded DataLad project.
2018-08-31 13:27:29 -04:00
Joey Hess
5c99f6247e
per-remote metadata storage
Actually very straightforward reuse of the metadata log file code.
Although I had to add a todo item as git-annex forget won't clean up
dead remote's metadata yet.

This would be worth adding to the external special remote interface
sometime. Have not opened a todo though, guess I'll wait until something
needs it.

This commit was supported by the NSF-funded DataLad project.
2018-08-31 12:23:22 -04:00
Joey Hess
9d78a4387f
update 2018-08-31 12:23:04 -04:00
Joey Hess
6b75b9c448
turn on appendonly when versioning is enabled 2018-08-31 10:53:07 -04:00
Joey Hess
2cdcb1ea25
Merge branch 'master' of ssh://git-annex.branchable.com 2018-08-30 15:53:09 -04:00
Joey Hess
69d4c8dce6
devblog 2018-08-30 15:52:44 -04:00
Joey Hess
3a5d0402ba
update 2018-08-30 15:49:21 -04:00
Joey Hess
19dcff2b71
use S3 version ID for retrieval
Have to store the S3 object along with the version ID, so retrieval can
use the same object.

This commit was supported by the NSF-funded DataLad project.
2018-08-30 15:37:08 -04:00
kyle
5567496518 'annex add' locks unmodified file in V5 but not V6 2018-08-30 19:07:03 +00:00
Joey Hess
794e9a7a44
store S3 version IDs
Only done when versioning=yes is configured. It could always do it when
S3 sends back a version id, but there may be buckets that have
versioning enabled by accident, so it seemed better to honor the
configuration.

S3's docs say version IDs are "randomly generated", so presumably
storing the same content twice gets two different ones not the same one.
So I considered storing a list of version IDs for a key. That would
allow removing the key completely. But.. The way Logs.RemoteState works,
when there are multiple writers, the last writer wins. So storing a list
would need a different log format that merges, which seemed overkill to support
removing a key from an append-only remote.

Note that Logs.RemoteState for S3 is now dedicated to version IDs.
If something else needs to be stored, a new log will be needed to do it.

This commit was supported by the NSF-funded DataLad project.
2018-08-30 14:30:56 -04:00
Joey Hess
0ff5a41311
S3 versioning=yes config
Not yet used.

This commit was supported by the NSF-funded DataLad project.
2018-08-30 13:45:28 -04:00
yarikoptic
2969d93283 Added a comment: clear docker for ubuntu 18:04 - the same 2018-08-30 16:07:51 +00:00
kyle
d74c6ebf1f Added a comment: v5 2018-08-30 16:05:01 +00:00
yarikoptic
dd0283e985 Added a comment: reproduced more 2018-08-30 16:02:39 +00:00
yarikoptic
4b05d4179d Added a comment: confirming 2018-08-30 15:57:39 +00:00
Joey Hess
358178fbfb
don't untrust appendonly exports
Make exporttree=yes remotes that are appendonly not be untrusted, and not force
verification of content, since the usual concerns about losing data when an
export is updated by someone else don't apply.

Note that all the remote operations on keys are left as usual for
appendonly export remotes, except for storing content.

This commit was supported by the NSF-funded DataLad project.
2018-08-30 11:48:04 -04:00
Joey Hess
76f32012af
avoid sync/assistant drop from appendonly
Make git-annex sync and the assistant skip trying to drop from appendonly
remotes since it's just going to fail.

git-annex drop and similar commands will still try to drop from
appendonly, so the user will see failure messages when they try to do
that. To do otherwise would be confusing since the user has explicitly
asked for a drop with those commands.

This commit was supported by the NSF-funded DataLad project.
2018-08-30 11:23:57 -04:00
Joey Hess
8b39db20b5
export appendonly support
Make `git annex export` check appendonly when removing a file from an
export, and not update the location log, since the remote still contains
the content.

This commit was supported by the NSF-funded DataLad project.
2018-08-30 11:18:20 -04:00
Joey Hess
02630b39ee
add Remote.readonly
Does nothing yet.

Considered making bup readonly, but while the content can't be removed,
it is able to delete a branch, so didn't.

This commit was supported by the NSF-funded DataLad project.
2018-08-30 11:12:18 -04:00
Joey Hess
a3fa9ea0b5
followup 2018-08-30 11:08:46 -04:00
83.175.76.211
41baab276e poll vote (/sdcard/annex) 2018-08-30 05:03:01 +00:00
kyle@9c8cc228b2254d7f0f8c42be54a2ac1b81edf416
92be1cc8c7 2018-08-29 21:20:26 +00:00
Joey Hess
dad627fa9e
remove false starts, simplify 2018-08-29 14:12:18 -04:00
Joey Hess
5b78952f78
misunderstood some code; simplify 2018-08-29 14:09:18 -04:00
Joey Hess
e216c18318
new much improved plan 2018-08-29 13:59:52 -04:00
Joey Hess
d3c9d72245
followup 2018-08-29 12:20:24 -04:00
Joey Hess
f54c72d2e1
Fix build on FreeBSD
This must have been broken for years..

This commit was sponsored by Jack Hill on Patreon.
2018-08-29 12:09:03 -04:00
Joey Hess
084e88964d
close 2018-08-29 12:04:46 -04:00
Joey Hess
332c1513f9
fix link 2018-08-29 11:02:16 -04:00
Joey Hess
b99d60172d
Merge branch 'master' of ssh://git-annex.branchable.com 2018-08-29 10:56:26 -04:00
Joey Hess
3874c5c88d
further thoughts 2018-08-29 10:56:02 -04:00
Joey Hess
44658d80ef
clarify comment 2018-08-29 10:55:52 -04:00
77.119.239.37
587e0983ac poll vote (/sdcard/annex) 2018-08-29 13:38:24 +00:00
77.119.239.37
661f603a62 poll vote (/sdcard/annex) 2018-08-29 13:38:17 +00:00
77.119.239.37
0f37b9c7ec poll vote (/sdcard/annex) 2018-08-29 13:38:10 +00:00
77.119.239.37
db0f98a619 poll vote (/sdcard/annex) 2018-08-29 13:38:03 +00:00
Joey Hess
0a896bdf39
fix syntax 2018-08-29 09:20:25 -04:00
Joey Hess
2b0cc71fbf
don't configure smudge filter in bare repo
Pointless to do it there.

Also, the git status call would fail in such a repo.
2018-08-29 09:16:50 -04:00
Joey Hess
c565340adc
stop using external hash programs, since cryptonite is faster
In 2013, I wrote "Cryptohash benchmarks 90 to 101% faster than external
hashers". Re-benchmarking today, I found cryptonite's sha256 consistently
outperformed coreutils by 10% for large files. Tested 10 mb, 100 mb, 1 gb
files with both sha256 and sha512. And for smaller files, the external
process startup time swamps the hash time.

Perhaps cryptonite has improved. Or it could just do better on my
current CPU Intel(R) Pentium(R) CPU 4410Y @ 1.50GHz). Anyway, even if cryptonite
is slower in some situations, seems likely it would only be marginally slower;
it's got the same class of highly optimised C code under the hood as coreutils.
The main difference between the two sha256 implementations seems to be
how much of the inner loop they unroll..

This commit was sponsored by Henrik Riomar on Patreon.
2018-08-28 18:10:58 -04:00
Joey Hess
6001b3cf45
fix build warning 2018-08-28 13:17:06 -04:00
Joey Hess
759a87ad70
fix git command queue to be concurrency safe
Probably not noticed until now because the queue is large enough that two
threads each filling theirs at the same time and flushing is unlikely to
happen.

Also made explicit that each worker thread gets its own queue.
I think that was the case before, but if something was put in the queue
before worker threads were forked off, they could have each inherited the
same queue.

Could have gone with a single shared queue, but per-worker queues is more
efficient, because a worker can add lots of stuff to its own queue without
any locking.

This commit was sponsored by Ole-Morten Duesund on Patreon.
2018-08-28 13:16:33 -04:00
Joey Hess
3a71f3a4a9
reproduced 2018-08-28 12:35:15 -04:00