This fixes behavior in this situation:
l1 <- lockShared Nothing "lck"
l2 <- lockShared Nothing "lck"
dropLock l1
dropLock l2
Before, the lock was dropped upon the second dropLock call, but the fd
remained open, and would never be closed while the program was running.
Fixed by a rather round-about method, but it should work well enough.
It would have been simpler to open open the shared lock once, and not open
it again in the second call to lockShared. But, that's difficult to do
atomically.
This also affects Windows and PID locks, not just posix locks.
In the case of pid locks, multiple calls to waitLock within the same
process are allowed because the side lock is locked using a posix lock,
and so multiple exclusive locks can be taken in the same process. So,
this change fixes a similar problem with pid locks.
l1 <- waitLock (Seconds 1) "lck"
l2 <- waitLock (Seconds 1) "lck"
dropLock l1
dropLock l2
Here the l2 side lock fd remained open but not locked,
although the pid lock file was removed. After this change, the second
dropLock will close both fds to the side lock, and delete the pidlock.
* add, addurl, import, importfeed: When in a v6 repository on a crippled
filesystem, add files unlocked.
* annex.addunlocked: New configuration setting, makes files always be
added unlocked. (v6 only)
This is a mostly backwards compatable change. I broke backwards
compatability in the case where a filename starts with double-quote.
That seems likely to be very rare, and v6 unlocked files are a new feature
anyway, and fsck needs to fix missing associated file mappings anyway. So,
I decided that is good enough.
The encoding used is to just show the String when it contains a problem
character. While that adds some overhead to addAssociatedFile and
removeAssociatedFile, those are not called very often. This approach has
minimal decode overhead, because most filenames won't be encoded that way,
and it only has to look for the leading double-quote to skip the expensive
read. So, getAssociatedFiles remains fast.
I did consider using ByteString instead, but getting a FilePath converted
with all chars intact, even surrigates, is difficult, and it looks like
instance PersistField ByteString uses Text, which I don't trust for problem
encoded data. It would probably be slower too, and it would make the
database less easy to inspect manually.
Instead -J will behave as if it was built without concurrent-output support
in this situation. Ie, it will be mostly quiet, except when there's an
error.
Note that it's not a problem for a filename to contain invalid utf-8 when
in a utf-8 locale. That is handled ok by concurrent-output. It's only
displaying unicode characters in a non-unicode locale that doesn't work.
This lets readonly repos be used. If a repo is readonly, we can ignore the
keys database, because nothing that we can do will change the state of the
repo anyway.
That trailing slash is needed for legacy chunked mode, because it puts the
chunks in a subdir under the key. But, outside legacy chunked mode, it's BS
and it's amazing it worked at all with some webdav servers.
I needed BUILDEROPTIONS to allow passing flags to stack build, but it also
lets me move the -j1 out of the normal build path, and to debian/rules
which has the goal of having a reproducible build
The type checker should have noticed this, but the changes to mapM
that make it accept any Traversable hid the fact that it was not being
passed a list at all. Thus, what should have returned an empty list most
of the time instead returned [""] which was treated as the name of the
associated file, with disasterout consequences.
When I have time, I should add a test case checking what sync --content
drops. I should also consider replacing mapM with one re-specialized to
lists.
* Removed the webapp-secure build flag, rolling it into the webapp build
flag.
* Removed the quvi and tahoe build flags, which only adds aeson to
the core dependencies.
* Removed the feed build flag, which only adds feed to the core
dependencies.
Build flags have cost in both code complexity and also make Setup configure
have to work harder to find a usable set of build flags when some
dependencies are missing.
When Config.setConfig runs, it throws away the old Repo and loads a new
one. So, add an action to adjust the Repo so that -c settings will persist
across that.
The homomorphs are back, just encoded such that it doesn't crash in LANG=C
However, I noticed a bug in the old escaping; [pseudoSlash] was escaped the
same as ['/','/']. Fixed by using '%' to escape pseudoSlash. Which requires
doubling '%' to escape it, but that's already done in the escaping of
worktree filenames in a view, so is probably ok.
So, we need to look at both the file on disk to see if it's a annex link,
and the file in the index too. lookupFile doesn't look in the index if the file
is not present on disk.
Now available on mips, mipsel, but temporarily removed armel since build is
failing there.
If armel would just get caught up, I could remove the per-arch specs
entirely.
Maybe time to turn maint of this over to richih?
Decided it's too scary to make v6 unlocked files have 1 copy by default,
but that should be available to those who need it. This is consistent with
git-annex not dropping unused content without --force, etc.
* Added annex.thin setting, which makes unlocked files in v6 repositories
be hard linked to their content, instead of a copy. This saves disk
space but means any modification of an unlocked file will lose the local
(and possibly only) copy of the old version.
* Enable annex.thin by default on upgrade from direct mode to v6, since
direct mode made the same tradeoff.
* fix: Adjusts unlocked files as configured by annex.thin.
http://bugs.debian.org/807341
* Fix insecure temporary permissions when git-annex repair is used in
in a corrupted git repository.
Other calls to withTmpDir didn't leak any potentially private data,
but repair clones the git repository to a temp directory which is made
using the user's umask. Thus, it might expose a git repo that is
otherwise locked down.
* Fix potential denial of service attack when creating temp dirs.
Since withTmpDir used easily predictable temporary directory names,
an attacker could create foo.0, foo.1, etc and as long as it managed to
keep ahead of it, could prevent it from ever returning.
I'd rate this as a low utility DOS attack. Most attackers in a position
to do this could just fill up the disk /tmp is on to prevent anything
from writing temp files. And few parts of git-annex use withTmpDir
anyway, so DOS potential is quite low.
Examined all callers of withTmpDir and satisfied myself that
switching to mkdtmp and so getting a mode 700 temp dir wouldn't break any
of them.
Note that withTmpDirIn continues to not force temp dir to 700.
But it's only used for temp directories inside .git/annex/wherever/
so that is not a problem.
Also re-audited all other uses of temp files and dirs in git-annex.
Note that the implementation uses replaceFile, so that the actual
replacement of the work tree file is atomic. This seems a good property to
have!
It would be possible for unlock in v6 mode to be run on files that do not
have their content present. However, that would be a behavior change from
before, and I don't see any immediate need to support it, so I didn't
implement it.
Was not putting it inside the temp dir, but next to it!
This was just wrong, and it led to a longer filename that desired being
used, leading to some bug reports.
Note that this changes the default behavior of git add in a newly
initialized repository; it will add files to the annex.
Don't like that this could break workflows, but it's necessary in order for
any pointer files in the repo to be handled by git-annex.
Since all places where a repo is used in direct mode need to have git-annex
upgraded before the repo can safely be converted to v6, the upgrade needs
to be manual for now.
I suppose that at some point I'll want to drop all the direct mode support
code. At that point, will stop supporting v5, and will need to auto-upgrade
any remaining v5 repos. If possible, I'd like to carry the direct mode
support for say, a year or so, to give people plenty of time to upgrade and
avoid disruption.
importfeed just calls addurl functions, so inherits this from it.
Note that addurl still generates a temp file, and uses that key to download
the file. It just adds it to the work tree at the end when the file is small.
When core.sharedRepository is set, annex object files are not made mode
444, since that prevents a user other than the file owner from locking
them. Instead, a mode such as 664 is used in this case.
Had everything available, just didn't combine the progress meter with the
other places progress is sent to update it. (And to a remote repo already
did show progress.)
Most special remotes should already display progress meters with -J,
same as without it. One exception to this is the web, since it relies on
wget/curl progress display without -J. Still todo..
This was in the cabal file earlier, and was removed because it broke the
android cross build. Moving to the git-annex target of the Makefile
will make it be used for Debian packages etc but not android cross builds
or make fast or when users build with cabal.
replaceFile created a temp file, which was guaranteed to not overlap with
another temp file. However, makeAnnexLink then deleted that file, in
preparation for making the symlink in its place. This caused a race, since
some other replaceFile could create a temp file, using the same name!
I was able to reproduce the race easily running git-annex add -J10 in a
directory with 100 files (all with different contents). Some files would
get ingested into the annex, but their annex links would fail to be added.
There could be other situations where this same problem could occur.
Perhaps when the assistant is adding a file, if the user manually also ran
git-annex add. Perhaps in cases not involving adding a file.
The new replaceFile makes a temprary directory, which is guaranteed to be
unique, and doesn't make a temp file in there. makeAnnexLink can thus
create the symlink without problem and the race is avoided.
Audited all calls to replaceFile to make sure that the old behavior of
providing an empty temp file was not relied on.
The general problem of asking for a temp file and deleting it as part of
the process of using it could reach beyond replaceFile. Did some quick
audits and didn't find other cases of it. Probably only symlink creation
stuff would tend to make that mistake, mostly.
