This bug caused broken tree objects to get built by a later git annex sync.
This is a somewhat unlikely but not impossible situation, and the test
suite's union_merge_regression test tickled it when it was run on FAT.
Mostly the username is only used for the git committer or other display
purposes, and we can just fall back to a dummy value in these cases.
The only remaining place where an error is thrown is when starting local
pairing, which needs the username to be known.
The queue could potentially contain changes from before withAltRepo, and
get flushed inside the call, which would apply the changes to the modified
repo.
Or, changes could be queued in withAltRepo that were intended to affect
the modified repo, but don't get flushed until later.
I don't know of any cases where either happens, but better safe than sorry.
Note that this affect withIndexFile, which is used in git-annex branch
updates. So, it potentially makes things slower. Should not be by much;
the overhead consists only of querying the current queue a couple of times,
and potentially flushing changes queued within withAltRepo earlier, that
could have maybe been bundled with other later changes.
Notice in particular that the existing queue is not flushed when calling
withAltRepo. So eg when git annex add needs to stage files in the index,
it will still bundle them together efficiently.
Added guard in Annex.Transfer to prevent this problem at a deeper level.
I'm unhappy ith NoUUID, but having Maybe UUID instead wouldn't help either
if nothing checked that there was a UUID. Since there legitimately need to
be Remotes that do not have a UUID, I can't see a way to fix it at the type
level, short making there be two separate types of Remotes.
Removed the instance LensGpgEncParams RemoteConfig because it encouraged
code that does not take the RemoteGitConfig into account.
RemoteType's setup was changed to take a RemoteGitConfig,
although the only place that is able to provide a non-empty one is
enableremote, when it's changing an existing remote. This led to several
folow-on changes, and got RemoteGitConfig plumbed through.
This is actually worse than I thought; when git is being run with a
detached work tree, GIT_INDEX_FILE is treated as a path relative to CWD,
instead of the normal behavior of relative the top of the work tree.
This seems to make it basically impossible for any program that wants to
use GIT_INDEX_FILE to use anything other than an absolute path to it; there
are too many configurations to keep straight that can change how git
interprets what should be a simple relative path to a file.
(I have complained to the git developers.)
This affected git annex view. It turns out that some other places
that use GIT_INDEX_FILE were already working around the bug. I removed the
workaround from Annex.Branch since the new workaround will do.
Could not think of a foolproof way to detect if the old adjusted branch was
just behind the current branch. It's possible that the user amended the
adjusting commit at the head of the adjusted branch, for example.
I decided to bail in this situation, instead of just entering the old
branch, so that if git annex adjust succeeds the user is always in a
*current* adjusted branch, not some old and out of date one.
What could perhaps be done is enter the old branch and then update it. But
that seems too magical; the user may have rebased master or something or
may not want to propigate the changes from the old branch. Best to error
out.
It started exporting a isSymbolicLink which supports windows. But,
git-annex does no use symlinks on windows yet and this conflicts with the
function by the same name from unix-compat, so hide it.
git 2.8.1 (or perhaps 2.9.0) is going to prevent git merge from merging in
unrelated branches. Since the webapp's pairing etc features often combine
together repositories with unrelated histories, work around this behavior
change by setting GIT_MERGE_ALLOW_UNRELATED_HISTORIES when the assistant
merges.
Note though that this is not done for git annex sync's merges, so
it will follow git's default or configured behavior.
When git-annex is used with a git version older than 2.2.0, disable support for
adjusted branches, since GIT_COMMON_DIR is needed to update them and was first
added in that version of git.
Made all Annex.Perms file mode changing functions ignore errors when
core.sharedRepository is set, because the file might be owned by someone
else. I don't fancy getting bug reports about crashes due to set modes in
this configuration, which is a very foot-shooty configuration in the first
place.
The fsck warning is necessary because old repos kept files mode 444, which
doesn't allow locking them, and so if the mode remains 444 due to the file
being owned by someone else, the user should be told about it.
When annex.thin is set, adding an object will add the execute bits to the
work tree file, and this does mean that the annex object file ends up
executable.
This doesn't add any complexity that wasn't already present, because git
annex add of an executable file has always ingested it so that the annex
object ends up executable.
But, since an annex object file can be executable or not, when populating
an unlocked file from one, the executable bit is always added or removed
to match the mode of the pointer file.
This is how direct mode does it too, and somehow, for reasons that
currently escape me, this makes git merge not care if it's run with an
empty work tree.
Was using L.readFile, so the Handle would remain open until the garbage
collector got around to it. Changed to explicit open and close, so we know
it's always closed when the function returns.
This makes the direct mode to v6 upgrade able to be performed in one clone
of a repository without affecting other clones, which can continue using v5
and direct mode.
This does mean that it has to write out temp files containing updated
objects for the merge. So may use more disk space, and disk IO, but that
should generally win out over needing to launch N separate
git hash-object processes.
An unlocked present file does not have a pointer file in the worktree, so
info skipped counting it.
It may be that unused was also affected by the problem, but it seemed not
to be in my tests. I think because of the use of the associatedFilesFilter.
This fix slows down both info and unused a little bit, since they have to
query the contents of files from git, but only when handling unlocked files.
Only reverse adjust the changes in the commit, which means that adjustments
do not need to be generally cleanly reversable.
For example, an adjustment can unlock all locked files, but does not need
to worry about files that were originally unlocked when reversing, because
it will only ever be run on files that have been changed. So, it's ok
if it locks all files when reversed, or even leaves all files as-is when
reversed.
Using adjusted/unlocked/master made lots of git stuff dealing with "master"
complain that it was ambiguous. This new appoach is more like view branch
names, and shows the adjustment right there in the branch display even if
only the basename of the branch is shown.
There's a race here, but entering an adjusted branch for the first time is
not something to do when a commit is being made at the same time. Although,
may want to prevent the assistant from committing while entering the
adjusted branch.
So, it will pull and push the original branch, not the adjusted one.
And, for merging, it will use updateAdjustedBranch (not implemented yet).
Note that remaining uses of Git.Branch.current need to be checked too;
for things that should act on the original branch, and not the adjusted
branch.
"git annex adjust" may be a temporary interface, but works for a proof of
concept.
It is pretty fast at creating the adjusted branch. The main overhead is
injecting pointer files. It might be worth optimising that by reusing the
symlink target as the pointer file content. When I tried to do that,
the problem was that the clean filter doesn't use that same format, and so
git thought files had changed. Could be dealt with, perhaps make the clean
filter use symlink format for pointer files when on an adjusted branch?
But the real overhead is in checking out the branch, when git runs the
smudge filter once per file. That is perhaps too slow to be usable,
although it may only affect initial checkout of the branch, and not
updates. TBD.
* add, addurl, import, importfeed: When in a v6 repository on a crippled
filesystem, add files unlocked.
* annex.addunlocked: New configuration setting, makes files always be
added unlocked. (v6 only)
The problem with having the slashes unescaped is, it broke parsing, since
the parser takes the filename to get the part containing the key.
That particularly affected URL keys.
This makes the format be the same as symlinks point to, which keeps things
simple.
Existing pointer files will continue to work ok.
Before, the call to mkProgressUpdater created the directory as a
side-effect, but since that ignored failure to create it, this led to
a "does not exist" exception when the transfer lock file was created,
rather than a permissions error.
So, make sure the directory exists before trying to lock the file in it.
When a PermissionDenied exception is caught, skip making the transfer lock.
This lets downloads from readonly remotes happen.
If an upload is being tried, and the lock file can't be written due to
permissions, then probably the actual transfer will fail for the same
reason, so I think it's ok that it continues w/o taking the lock in that
case.
The type checker should have noticed this, but the changes to mapM
that make it accept any Traversable hid the fact that it was not being
passed a list at all. Thus, what should have returned an empty list most
of the time instead returned [""] which was treated as the name of the
associated file, with disasterout consequences.
When I have time, I should add a test case checking what sync --content
drops. I should also consider replacing mapM with one re-specialized to
lists.
Previously, it only flushed when the queue got larger than 1.
Also, make the queue auto-flush when items are added, rather than needing
to be flushed as a separate step. This simplifies the code and make it more
efficient too, as it avoids needing to read the queue out of the state to
check if it should be flushed.
The homomorphs are back, just encoded such that it doesn't crash in LANG=C
However, I noticed a bug in the old escaping; [pseudoSlash] was escaped the
same as ['/','/']. Fixed by using '%' to escape pseudoSlash. Which requires
doubling '%' to escape it, but that's already done in the escaping of
worktree filenames in a view, so is probably ok.
Linking the file to the tmp dir was not necessary in the clean
filter, and it caused the ctime to change, which caused git to think
the file was changed. This caused git status to get slow as it kept
re-cleaning unchanged files.
Fixes several bugs with updates of pointer files. When eg, running
git annex drop --from localremote
it was updating the pointer file in the local repository, not the remote.
Also, fixes drop ../foo when run in a subdir, and probably lots of other
problems. Test suite drops from ~30 to 11 failures now.
TopFilePath is used to force thinking about what the filepath is relative
to.
The data stored in the sqlite db is still just a plain string, and
TopFilePath is a newtype, so there's no overhead involved in using it in
DataBase.Keys.
WorkTree.lookupFile was finding a key for a file that's deleted from the
work tree, which is different than the v5 behavior (though perhaps the same
as the direct mode behavior). Fix by checking that the work tree file exists
before catting its key.
Hopefully this won't slow down much, probably the catKey is much more expensive.
I can't see any way to optimise this, except perhaps to make Command.Unused
check if work tree files exist before/after calling lookupFile. But,
it seems better to make lookupFile really only find keys for worktree files;
that's what it's intended to do.
Else, queued file stages won't have reached the index, and it won't find
everthing.
This evidently fixes a reversion in my work today, although I don't see how
I broke it. It didn't use to flush the queue first, before, and worked
somehow.
Test suite for v5 is back to 100% green now.
The smudge filter does need to be run, because if the key is in the local
annex already (due to renaming, or a copy of a file added, or a new file
added and its content has already arrived), git merge smudges the file and
this should provide its content.
This does probably mean that in merge conflict resolution, git smudges the
existing file, re-copying all its content to it, and then the file is
deleted. So, not efficient.
This is a behavior change for merge conflicts between locked files
that both pointed to the same key, in different ways.
Before, the conflict was resolved, but the file was renamed to .variant.
This was unnecessary, because there was only one variant.
Of course, this also handles conflicts between unlocked and locked, or even
two unlocked files with different pointer contents.
Since the file was present and locked, its annex object was not in the
inode cache. So, despite not needing to update the annex object when the
clean filter is run on the content by git merge, it does need to record the
inode cache of the annex object. Otherwise, the annex object will be
assumed to be bad, since its inode is not cached.
Several tricky parts:
* When the conflict is just between the same key being locked and unlocked,
the unlocked version wins, and the file is not renamed in this case.
* Need to update associated file map when conflict resolution renames
an unlocked file.
* git merge runs the smudge filter on the conflicting file, and actually
overwrites the file with the same content it had before, and so
invalidates its inode cache. This makes it difficult to know when it's
safe to remove such files as conflict cruft, without going so far as to
compare their entire contents.
Dealt with this by preventing the smudge filter from populating the file
when a merge is run. However, that also prevents the smudge filter being
run for non-conflicting files, so eg moving a file won't put its new
content into place.
* Ideally, if a merge or a merge conflict resolution renames an unlocked
file, the file in the work tree can just be moved, rather than copying
the content to a new worktree file.
This is attempted to be done in merge conflict resolution, but
due to git merge's behavior of running smudge filters, what actually
seems to happen is the old worktree file with the content is deleted and
rewritten as a pointer file, so doesn't get reused.
So, this is probably not as efficient as it optimally could be.
If that becomes a problem, could look into running the merge in a separate
worktree and updating the real worktree more efficiently, similarly to the
direct mode merge. However, the direct mode merge had a lot of bugs, and
I'd rather not use that more error-prone method unless really needed.
Decided it's too scary to make v6 unlocked files have 1 copy by default,
but that should be available to those who need it. This is consistent with
git-annex not dropping unused content without --force, etc.
* Added annex.thin setting, which makes unlocked files in v6 repositories
be hard linked to their content, instead of a copy. This saves disk
space but means any modification of an unlocked file will lose the local
(and possibly only) copy of the old version.
* Enable annex.thin by default on upgrade from direct mode to v6, since
direct mode made the same tradeoff.
* fix: Adjusts unlocked files as configured by annex.thin.
