http://bugs.debian.org/807341
* Fix insecure temporary permissions when git-annex repair is used in
in a corrupted git repository.
Other calls to withTmpDir didn't leak any potentially private data,
but repair clones the git repository to a temp directory which is made
using the user's umask. Thus, it might expose a git repo that is
otherwise locked down.
* Fix potential denial of service attack when creating temp dirs.
Since withTmpDir used easily predictable temporary directory names,
an attacker could create foo.0, foo.1, etc and as long as it managed to
keep ahead of it, could prevent it from ever returning.
I'd rate this as a low utility DOS attack. Most attackers in a position
to do this could just fill up the disk /tmp is on to prevent anything
from writing temp files. And few parts of git-annex use withTmpDir
anyway, so DOS potential is quite low.
Examined all callers of withTmpDir and satisfied myself that
switching to mkdtmp and so getting a mode 700 temp dir wouldn't break any
of them.
Note that withTmpDirIn continues to not force temp dir to 700.
But it's only used for temp directories inside .git/annex/wherever/
so that is not a problem.
Also re-audited all other uses of temp files and dirs in git-annex.
The Keys database can hold multiple inode caches for a given key. One for
the annex object, and one for each pointer file, which may not be hard
linked to it.
Inode caches for a key are recorded when its content is added to the annex,
but only if it has known pointer files. This is to avoid the overhead of
maintaining the database when not needed.
When the smudge filter outputs a file's content, the inode cache is not
updated, because git's smudge interface doesn't let us write the file. So,
dropping will fall back to doing an expensive verification then. Ideally,
git's interface would be improved, and then the inode cache could be
updated then too.
Had everything available, just didn't combine the progress meter with the
other places progress is sent to update it. (And to a remote repo already
did show progress.)
Most special remotes should already display progress meters with -J,
same as without it. One exception to this is the web, since it relies on
wget/curl progress display without -J. Still todo..
There's a potential race, but it's detected and just results in the other
process failing to take the side lock, so possibly retrying one second
later on. The race window is quite narrow so the extra delay is minor.
Left the side lock files mode 666 because an interruption can leave a side
lock file created by another user for a shared repository. When this
happens, the non-owning user can't delete it (+t) but can still lock it,
and so the code falls back to acting as it did before this commit.
This is less portable, since currently sidelocks rely on /dev/shm.
But, I've seen crazy lustre inconsistencies that make me not trust the
link() method at all, so what can you do.
I have a strace taken on a lustre filesystem on which link() returned 0,
but didn't actually succeed, since the file already existed.
One of the linux man pages recommended using link followed by checking like
this. I was reading it yesterday, but cannot find it now.
Fixes a recent-ish build warning on about 64 bit vs non.
This is the method used by the disk-free-space library, and I tested it to
yield the same results on even 10 tb drives on OSX -- so it's getting 64
bit values.
Also, rename lockContent to lockContentExclusive
inAnnexSafe should perhaps be eliminated, and instead use
`lockContentShared inAnnex`. However, I'm waiting on that, as there are
only 2 call sites for inAnnexSafe and it's fiddly.
On Solaris, using f_bsize provided a value that is apparently much larger
than the real block size. The solaris docs for statvfs say
f_bsize is the "preferred" file system block size, and I guess the
filesystem prefers larger blocks, but uses smaller ones or something.
The docs also say that f_frsize is the "fundamental" block size.
Switched to using f_frsize on Linux and kFreeBSD too, since I guess
f_bsize could in theory vary the same way there too. Assuming that Solaris
is not violating the posix spec, I guess the linux man page for statvfs
is not as well written and I misunderstood it.
This is the kind of annoying thing that makes me not want to use a library.
conduitManagerSettings was a perfectly fine name and could have been kept
forever.
Since I want git-annex to keep building on debian stable, I need to still
support the old http-client, which required explicit calls to
closeManager, or use of withManager to get Managers to close at appropriate
times. This is not needed in the new version, and so they added a
deprecation warning. IMHO much too early, because look at the mess I had to
go through to avoid that deprecation warning while supporting both
versions..
When gpg.program is configured, it's used to get the command to run for
gpg. Useful on systems that have only a gpg2 command or want to use it
instead of the gpg command.
It was failing at link time, some problem with terminatePID.
Re-implemented that to not use a C wrapper function, which cleared up the
problem. Removed old EvilLinker hack with must have been related to the
same problem.
Note that I have not tested this with older ghc's. In
f11f7520b5 I mention having tried this
approach before, and getting segfaults.. So, who knows. It seems to work
fine with ghc 7.10 at least.
This is mostly to be able to see how long a command took to run. Also exit
code may be useful.
Unofrtunately, I can't put the command name in there, because it's not
available at this point, and it would be a much larger change to wrap the
ProcessHandle data type to add that. However, it's generally pretty obvious
which process exited from context.
Since _encodeFilePath generates a String that doesn't use the filesystem
encoding, when this exception is caught, we know we already have such a
String, and can just return it as-is.
This was caused by 23e9d3bb77
an Arbitrary String is not necessarily encoded using the filesystem
encoding, and in a non-utf8 locale, encodeBS throws an exception on such a
string. All I could think to do is limit test data to ascii.
This shouldn't be a problem in practice, because the all Strings in
git-annex that are not generated by Arbitrary should be loaded in a way
that does apply the filesystem encoding.
Oh boy, not again. So, another place that the filesystem encoding needs to
be applied. Yay.
In passing, I changed decodeBS so if a NUL is embedded in the input, the
resulting FilePath doesn't get truncated at that NUL. This was needed to
make prop_b64_roundtrips pass, and on reviewing the callers of decodeBS, I
didn't see any where this wouldn't make sense. When a FilePath is used to
operate on the filesystem, it'll get truncated at a NUL anyway, whereas if
a String is being used for something else, it might conceivably have a NUL
in it, and we wouldn't want it to get truncated when going through
decodeBS.
(NB: There may be a speed impact from this change.)
While cryptohash has SHA3 support, it has not been updated for the final
version of the spec. Note that cryptonite has not been ported to all arches
that cryptohash builds on yet.
This fixes a reversion introduced by relative path changes back last winter.
The root cause is simplifyPath "../foo" was incorrectly yielding "foo".
absPathFrom seems quite horrible. Probably most things that use it should
use </> instead.
This is needed because when preferred content matches on files,
the second pass would otherwise want to drop all keys. Using a bloom filter
avoids this, and in the case of a false positive, a key will be left
undropped that preferred content would allow dropping. Chances of that
happening are a mere 1 in 1 million.
I want this as fast as possible, so it can be added to code paths without
slowing them down.
Avoid the set lookup, and rely on laziness,
drops runtime from 14.37 ns to 11.03 ns according to this criterion benchmark:
import Criterion.Main
import qualified Types.Difference as New
import qualified Types.DifferenceOld as Old
main :: IO ()
main = defaultMain
[ bgroup "hasDifference"
[ bench "new" $ whnf (New.hasDifference New.OneLevelObjectHash) new
, bench "old" $ whnf (Old.hasDifference Old.OneLevelObjectHash) old
]
]
where
s = "fromList [ObjectHashLower, OneLevelObjectHash, OneLevelBranchHash]"
new = New.readDifferences s
old = Old.readDifferences s
A little bit of added boilerplate, but I suppose it's worth it to not
need to worry about set lookup overhead. Note that adding more differences
would slow down the old implementation; the new implementation will run
the same speed.
This reverts commit ef0e3ac22e.
Sebastian thinks best to revert this:
It seems to me the reason I needed to look at activatable sockets
might actually be a networkd bug, and I was in error about patch 0001.
On my machines (without DHCP), networkd quits after configuring the
links. I thought this had to do with network activation, but that was
probably mistaken. This was obscured by my testing the change by doing
systemctl stop/start on networkd; now that I actually unplugged the
network cable, I noticed no DBus messages are triggered by this on
this machine. Your test case might have had a similar problem
(networkd quitting on idle). Might be related to [1].
On another machine (with DHCP) networkd remains active all the time,
and patch 0002 works there. You might want to revert 0001, though:
Suppose someone’s running no manager at all, so that polling would be
required. Because networkd is still listed as activable, we would
refrain from polling – by mistake, because networkd doesn’t seem to
actually go active if we listen on its bus, and it’s listed as
activable even when it’s not configured. Connectivity-related messages
will come in when stopping/starting the service, but not when
unplugging the cable.
This removes a bit of complexity, and should make things faster
(avoids tokenizing Params string), and probably involve less garbage
collection.
In a few places, it was useful to use Params to avoid needing a list,
but that is easily avoided.
Problems noticed while doing this conversion:
* Some uses of Params "oneword" which was entirely unnecessary
overhead.
* A few places that built up a list of parameters with ++
and then used Params to split it!
Test suite passes.
