mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

remove old closed bugs and todo items to speed up wiki updates and reduce size

Browse source 
Remove closed bugs and todos that were last edited or commented before 2024.

Except for ones tagged projects/* since projects like datalad want to keep
around records of old deleted bugs longer.

Command line used:

	for f in $(grep -l '|done\]\]' -- ./*.mdwn); do if ! grep -q "projects/" "$f"; then d="$(echo "$f" | sed 's/.mdwn$//')"; if [ -z "$(git log --since=01-01-2024 --pretty=oneline -- "$f")" -a -z "$(git log --since=01-01-2024 --pretty=oneline -- "$d")" ]; then git rm -- "./$f" ; git rm -rf "./$d"; fi; fi; done
	for f in $(grep -l '\[\[done\]\]' -- ./*.mdwn); do if ! grep -q "projects/" "$f"; then d="$(echo "$f" | sed 's/.mdwn$//')"; if [ -z "$(git log --since=01-01-2024 --pretty=oneline -- "$f")" -a -z "$(git log --since=01-01-2024 --pretty=oneline -- "$d")" ]; then git rm -- "./$f" ; git rm -rf "./$d"; fi; fi; done
This commit is contained in:
Joey Hess 2025-01-01 13:09:36 -04:00
parent 0000c3f325
commit 2fe36b35a2
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
871 changed files with 0 additions and 27861 deletions
Download patch file Download diff file Expand all files Collapse all files

26
doc/bugs/10.20220525_cannot_probe_for_untrusted_remote_key.mdwn
View file

@ -1,26 +0,0 @@
### Please describe the problem.
10.20220525 forcefully untrusts `exporttree=yes`, causing `fsck` to unconditionally fail when probing for a key that only exists on such a remote.
The full story is at https://github.com/datalad/datalad-next/issues/72
The summary is that an application needs to know whether a particular key is on such a remote, and it can only be on that remote (there is no other). Because the availability info from `whereis` cannot be trusted (same rational as the one causing the change in git-annex implementation) it uses `annex fsck --fast --key -f ...` which now fails.
Parsing the JSON output of the failing `fsck` call seems to be the only way left to accomplish this goal.
```
% git -C myclone/.git/dl-repoannex/origin/repoannex -c diff.ignoreSubmodules=none annex fsck -f origin --fast --key XDLRA--refs -c annex.dotfiles=true --json
Only these untrusted locations may have copies of XDLRA--refs
d8735795-663e-4702-8f7e-8c684264a9df -- [origin]
Back it up to trusted locations with git-annex copy.
{"dead":[],"command":"fsck","note":"(Avoid this check by running: git annex dead --key )","success":false,"input":[],"untrusted":[{"here":false,"uuid":"d8735795-663e-4702-8f7e-8c684264a9df","description":"[origin]"}],"key":"XDLRA--refs","error-messages":[],"file":null}
fsck: 1 failed
```
That however, is substantially more complex than looking at the exit code -- given that the entire call is about a single key.
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
99.9% are just splendid. Wondering here whether this particular consequence of the change was intentional and avoidable?
> [[notabug|done]] --[[Joey]]

15
doc/bugs/10.20220525_cannot_probe_for_untrusted_remote_key/comment_1_e35bef8fb0b9a40938022c43c87df597._comment
View file

@ -1,15 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2022-06-09T17:19:07Z"
content="""
That was a fix for an old reversion, so trying to use fsck for that would
have also failed before 8.20201129.
Fsck is not the right tool for the job. Fscking can fail for any of a
variety of reasons, including probably reasons that have not been thought
of or implemented yet, so taking failure to mean only one possible thing is
doomed.
A better tool exists: `git-annex checkpresentkey`
"""]]

20
doc/bugs/Adjust_--unlock_not_using_--reflink__63__.mdwn
View file

@ -1,20 +0,0 @@
### Please describe the problem.
Running adjust --unlock is unexpectedly slow and seems to use a lot of space, even on BTRFS, suggesting it probably does not use --reflink=auto like most other commands.
### What steps will reproduce the problem?
Run adjust --unlock with very large files.
### What version of git-annex are you using? On what operating system?
6.20170101-1+deb9u1 on Debian Stretch
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Yes I have! I've used it manage lots of video editing disks before, and am now migrating several slightly different copies of 15TB sized documentary footage from random USB3 disks and LTO tapes to a RAID server with BTRFS.
> Yay, this got fixed some time ago, by making the smudge filter not output
> the file content at all, but instead the link, and having a git hook
> later run git-annex to populate the files, which it does use reflink for
> when possible. [[done]] --[[Joey]]

18
doc/bugs/Adjust_--unlock_not_using_--reflink__63__/comment_1_c42743828ddc62f7df7f612c713ee309._comment
View file

@ -1,18 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2017-12-05T17:24:24Z"
content="""
What happens is `git annex adjust` creates a branch and runs `git checkout`
to check it out. Then git calls `git-annex smudge` on files, and
unfortunately the git smudge interface requires that git-annex output the
whole content of the file, to stdout, which git then writes to disk.
So yes, this doesn't use reflinks, but it's worse than that, a better
interface would let git-annex simply move or hard-link the file into place,
even on filesystems not supporting reflinks.
This is discussed in detail in [[todo/smudge]]. Unfortunately it will need
a better interface in git to be addressed, and that's why v6 is still an
experimental feature.
"""]]

16
doc/bugs/Android_termux_install_fails_with_Bad_system_call.mdwn
View file

@ -1,16 +0,0 @@
Following the installation instructions for termux fails on multiple
phones (including my own) with "Bad system call" when it comes to running
git-annex.
My guess as to what has happened is that the arm64 build has been updated
to use a newer version of glibc etc, and that is trying to use a syscall
that the Android kernel is too old to support. (Kernel 4.19 in my phone's
case.)
Unsure when it broke, but the arm64 build is built on Debian testing
for whatever reason, so it's will need a reasonably recent kernel. Perhaps
there should be an equivilant of the i386-ancient build for older arm64
systems? arm64 has by now been around as long as the Debian jessie used for
i386-ancient. --[[Joey]]
> [[done]] --[[Joey]]

37
doc/bugs/Broken_symlinks_in_directory_remote_causes_crash.mdwn
View file

@ -1,37 +0,0 @@
### Please describe the problem.
I have a directory remote with importtree=yes. In that remote, I have some symlinks that are broken. (Long story; this is a file server and they work on the system that has mounted them, but are broken here.)
### What steps will reproduce the problem?
I've added it with `git config remote.source.annex-tracking-branch main:$REPO`. When I run `git annex sync`, I get:
```
commit
On branch adjusted/main(unlocked)
nothing to commit, working tree clean
ok
list source
git-annex: Unable to list contents of source: [redacted]: getFileStatus: does not exist (No such file or directory)
failed
git-annex: sync: 1 failed
```
### What version of git-annex are you using? On what operating system?
8.20210223-2 on Debian
### Please provide any additional information below.
I would like git-annex to either:
1. Store the symlink as a symlink, or
2. Ignore bad symlinks
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Loading in other parts of my photo collection as we speak!
> [[fixed|done]], by ignoring the broken symlink. (There is a todo
> about importing symlinks,
>[[todo/import_symlinks_when_importing_from_directory]])--[[Joey]]

64
doc/bugs/CRLF_breaks_interoperability_Win-Ux__58__..post-receive.mdwn
View file

@ -1,64 +0,0 @@
### Please describe the problem.
I have created a git-annex via Windows. If I use sshfs to mount this
onto a Linux server, and then setup a new remote, I encounter the
error:
fatal: cannot run hooks/post-receive: No such file or directory
I've already identified that, due to the Windows created repository,
the script uses crlf line-endings. And of course, the Linux kernel
interprets the carriage-return as a trailing component of the shebang
#!/bin/sh^M
and obviously can't find the executable.
I've already worked around this by using a binfmt_misc approach
workaround, registering the shebang with ^M to trigger a wrapper
script. But I'm seeking the permanent fix of git-annex creating these
hooks with unix line-endings only. I think it's fair to say, there's
currently no approach that allows hooks to be in any other form than
unix scripts, and I'm happy to stand corrected, that it makes little
sense to have any other line-endings.
### What steps will reproduce the problem?
Run git annex sync against a Windows created remote git-annex, however
that is mounted, on a Linux/unix host.
### What version of git-annex are you using? On what operating system?
$ git annex version
git-annex version: 8.20210223
on a Debian bullseye operating system.
### Please provide any additional information below.
[[!format sh """
$ git annex sync
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
(use "git push" to publish your local commits)
It took 3.28 seconds to enumerate untracked files. 'status -uno'
may speed it up, but you have to be careful not to forget to add
new files yourself (see 'git help status').
nothing to commit, working tree clean
ok
pull origin
ok
push origin
fatal: cannot run hooks/post-receive: No such file or directory
To /var/tmp/mnt/winhost-w10-5920/cygdrive/e/my.gitannex/
a0ebd1826d..15cfd91da4 git-annex -> synced/git-annex
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Yes. I think git-annex is a hidden gem of the open source
community.
> [[fixed|done]] --[[Joey]]

17
doc/bugs/CRLF_breaks_interoperability_Win-Ux__58__..post-receive/comment_1_a9501e2f347b957e7dc53a06fe59cff4._comment
View file

@ -1,17 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2022-11-28T17:57:19Z"
content="""
You can edit the hooks that git-annex installs however you need to. It will
not overwrite modified hook files. If you edit this hook to have unix line
endings, does the hook still work when using this repository on windows?
Actually, I'm having difficulty seeing how the CR could have gotten into
that hook. `mkHookScript` uses `unlines` which behaves the same on Windows
as on Linux, so in either case lines should be separated only by `\n`.
This makes me wonder if something else modified the hook after git-annex
wrote it. If so, `git-annex init` should complain that the hook is
modified.
"""]]

25
doc/bugs/CRLF_breaks_interoperability_Win-Ux__58__..post-receive/comment_2_e4cac0d18fd6cde1084e93d0fe997f40._comment
View file

@ -1,25 +0,0 @@
[[!comment format=mdwn
username="beryllium@5bc3c32eb8156390f96e363e4ba38976567425ec"
avatar="http://cdn.libravatar.org/avatar/62b67d68e918b381e7e9dd6a96c16137"
subject="comment 2"
date="2023-01-05T06:48:58Z"
content="""
Apologies for the delay in response.
> You can edit the hooks that git-annex installs however you need to. It will not overwrite modified hook files. If you edit this hook to have unix line endings, does the hook still work when using this repository on windows?
Thanks, that aligns with what I surmised. I didn't apply that type of workaround for the sake of the report being as linear as possible (no alterations outside of running the git-annex commands).
Yes, of course, if I run dos2unix on the hook file, the repository works for both Linux and Windows. The latter, I understand because it is using msys to run the script, which is flexible with both forms of line-endings.
> Actually, I'm having difficulty seeing how the CR could have gotten into that hook. mkHookScript uses unlines which behaves the same on Windows as on Linux, so in either case lines should be separated only by \n.
My understanding has always been... native implementations on Windows (ie. no posix emulation linkage), if you don't call open() for \"binary\", your line-feeds indicate line termination only, and then the native line ending is written to the file, ie. Windows text files. So Windows native Python, Java, etc implementations, behave this way.
If I understand correctly, git-annex is relying on a native build of GHC? And it's library behaves that way too? Is that right?
> This makes me wonder if something else modified the hook after git-annex wrote it. If so, git-annex init should complain that the hook is modified.
As per above, I doubt that there was any alterations here. I tried again with a clean scenario, and indeed, the hook files were written as described.
"""]]

17
doc/bugs/CRLF_breaks_interoperability_Win-Ux__58__..post-receive/comment_3_a3a702176a316d6b599eb2a149a293ad._comment
View file

@ -1,17 +0,0 @@
[[!comment format=mdwn
username="beryllium@5bc3c32eb8156390f96e363e4ba38976567425ec"
avatar="http://cdn.libravatar.org/avatar/62b67d68e918b381e7e9dd6a96c16137"
subject="comment 3"
date="2023-01-05T07:16:30Z"
content="""
I should add, and this is not a commentary on the implementation of git-annex, etc... but having limited experience with BSD, and understanding that it uses different line endings for text files (???)... and other considerations... the git hooks mechanism is powerful, but seemingly presents challenges for interoperability.
Maybe the right approach is actually... as I did... let the end-user find their own workaround. It seems it's not really a git-annex fault after all.
I initially thought that git-annex should write the file out correctly... but it kind of is. It's writing it out for the local sensibilities.
In this particular case, it's my challenge to solve, not the tools... if you agree, I'd be content to have this closed with no action.
But I leave it to your better judgement.
"""]]

22
doc/bugs/CRLF_breaks_interoperability_Win-Ux__58__..post-receive/comment_4_82103077cde68c3b6a70dcf70c4ed837._comment
View file

@ -1,22 +0,0 @@
[[!comment format=mdwn
username="sb-beryllium@6e2c477eac63b823bd315ef8aaf5f93173c1f15b"
nickname="sb-beryllium"
avatar="http://cdn.libravatar.org/avatar/ef62105380b73ef91d760ec327c14e22"
subject="comment 4"
date="2023-01-07T13:05:23Z"
content="""
Apologies, this is beryllium (by alias)... due to unforeseen circumstances, I have had to register with a different email address. I am hoping this is a temporary situation.
Further apologies... I acknowledge that I am vacillating somewhat, but I've reverted to thinking that the hooks should be generated with unix line-endings (LF).
The reason I say this is, it does appear that the pseudo-standard for files under the .git directory is that they use unix line-endings only (perhaps an actual, documented standard... I'm not sure where to look/ask to confirm it).
This is the case for files such as .git/config, and .git/refs/** etc... under Git for Windows, and even JGit running with Windows native Java.
So to me.. it seems to make sense that hook files should have unix line-endings exclusively.
There are possible other mitigations... but that's where I stand with that finally. I'm still not fussed if no actions is determined to be the best course of action.
Thanks regardless.
"""]]

11
doc/bugs/CRLF_breaks_interoperability_Win-Ux__58__..post-receive/comment_5_c22f9e3c5e438b2cc05008bc3f91b7fc._comment
View file

@ -1,11 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2023-11-29T17:29:08Z"
content="""
[[!commit 4e35067325022e4d8790d45557156fd1166484df]] fixed this.
Previously, git-annex did write hook scripts with CRLF line endings.
Existing hook scripts won't be changed by git-annex, but if you delete
them, and re-run `git-annex init` it will reinstall it without the CRLF.
"""]]

39
doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X.mdwn
View file

@ -1,39 +0,0 @@
I have some git repositories I don't edit often that I'd like to back up. I'd like to add these to my git annex, so I don't have to resort to a time-consuming hack (such as setting up a proper submodule, or bundling the repositories).
But when I try to add a .git directory to git annex, I get a bunch of errors of the form
git-annex: user error (xargs ["-0","git","--git-dir=/tmp/tmp.LhGN3nT9uM/annex/.git","--work-tree=/tmp/tmp.LhGN3nT9uM/annex","add","--"] exited 123)
failed
git-annex: add: 1 failed
add repo/.git/hooks/pre-push.sample ok
(Recording state in git...)
error: Invalid path 'repo/.git/hooks/pre-push.sample'
error: unable to add repo/.git/hooks/pre-push.sample to index
fatal: adding files failed
STR:
$ mkdir annex
$ cd annex
annex$ git init
annex$ git annex init
annex$ cd ..
$ mkdir repo
$ cd repo
repo$ git init
repo$ cd ..
$ mv repo annex
$ cd annex
annex$ find repo | xargs -n1 git annex add
# Lots of errors of the form above.
You can't simply do `git annex add repo` because that will ignore the .git directory. Similarly,` git annex add .git` (which I'd think really should try to add the contents of the .git directory) ignores everything.
I don't know what this error means. Is there a right way to work around this?
> [[!meta title="cannot add .git/ to a git repository. even when using git-annex."]]
> [[!taglink confirmed]] (but may be out of scope for git-annex) --[[Joey]]
> >
> > [[closing|done]] as out of scope

8
doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X/comment_1_5d23b1f02c6a0db0b43939210c427994._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="andy"
avatar="http://cdn.libravatar.org/avatar/1f1c630290d8b02d7faddc9fa4482c8d"
subject="comment 1"
date="2017-02-16T06:34:50Z"
content="""
I've had the same problem; I solve it by packing the repositories into a compressed file of some sort (currently a 7z archive) and committing that to Git Annex. I have a script that makes that a one-line job. If I need to make changes to the repo, I can unpack the archive and work on the repo as usual.
"""]]

8
doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X/comment_2_6ede677f8113fcd6215ecd68de237f06._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="Lukey"
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
subject="comment 2"
date="2022-01-13T18:10:22Z"
content="""
I found an even better workaround, see [[tips/using_nested_git_repositories/]]
"""]]

8
doc/bugs/Can__39__t_add_a_git_repo_to_git_annex__58_____34__Invalid_path_repo__47__.git__47__X__34___for_many_X/comment_3_3377120e8e246f5ec74d372015590a6c._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2022-01-25T16:39:05Z"
content="""
I think this is out of scope for git-annex. It's not its place to change
what files git allows to be version controlled.
"""]]

68
doc/bugs/Changing_sync_to_include_content_breaks_UX.mdwn
View file

@ -1,68 +0,0 @@
### Please describe the problem.
After upgrading to git-annex 10.20230626, running `git annex sync` reports:
git-annex sync will change default behavior to operate on --content in a future version of git-annex. Recommend you explicitly use --no-content (or -g) to prepare for that change. (Or you can configure annex.synccontent)
which appears to be a barely documented change plan (at least I cannot find it in [the git-annex dev blog](https://git-annex.branchable.com/devblog/), only in the [latest change log]((https://hackage.haskell.org/package/git-annex-10.20230626/changelog)).
From the little that is said in [the 10.20230626 changelog](https://hackage.haskell.org/package/git-annex-10.20230626/changelog), it appears the intention is to -- **after 10 years** -- fairly quickly switch from `git annex sync` just syncing metadata (allowing git annexes to easily hold partial subsets of content), to doing a full content sync bidirectionally (apparently not allowing git annexes to hold partial subsets of content without explicit countermeasures for this behaviour breaking change).
I can understand why users might want a `git annex sync` that syncs content. And even maybe why it might want to be the default for those users who expect, eg, "Dropbox like behaviour".
But **changing the git annex sync default after 10 years** seems extremely user hostile.
Especially so when changing it from something that does not copy much data (default `git annex sync --no-content`) to something which (a) potentially copies a lot of data (over what might be a slow/expensive link), and (b) will potentially fill up drives due to repopulating entire large annexes which have historically relied on having only a subset of the content available locally, if the change in behaviour (after 10 years) is not caught in time.
The idea that users should go around every single git annex (I have dozens, with copies of those on dozens of machines and a bunch of offline drives), and make sure each one has `annex.synccontent` set, or every script that runs `git annex sync` has `git annex sync --no-content` on it, just to *restore the default of 10 years* is a pretty rough transition, and not a great user experience.
I would really strongly suggest that you do not change the behaviour of `sync` in this way *after 10 years*. And if you want a full sync option for user friendliness, then create, eg, `git annex fullsync` which is an alias for `git annex sync --content`.
If you no longer want to support the user model of having "incomplete annexes" (ie, all copies of a git annex must contain local copies of all data except changes made since the last sync), then the deprecation should be explicitly documented with advanced warning.
At minimum this signficant behaviour breaking change needs to be communicated *way* better than a random change log entry, and suddenly appearing in `git annex` output as a warning the world is going to break. And it shouldn't be necessary to, eg, trawl through the git source history to try to find any context for a major planned change.
Some slight saving graces:
* Fortunately `git annex sync --no-content` seems to be accepted at least back to git-annex 6.x, so at least it can be added into scripts without having to also check which git annex version is running; it's just a "no op" option in everything prior to git-annex 10.20030626.
* It looks like [`git annex config --set annex.synccontent false`](https://git-annex.branchable.com/git-annex-config/) might be carried with the repository (across syncs), reducing the number of places the new "override, back to the old default behaviour" setting has to be set (but it has to be set on every existing and new git annex, just to restore the 10 year historical behaviour)
* The [git annex source commit](https://git.kitenet.net/index.cgi/git-annex.git/commit/?id=f93a7fce1d5272c3282ce234053d26b10dd44198) has a tiny bit more context about there needing to be a "Debian Stable release" before the default changes, which doesn't seem to be documented anywhere else; if true, then since [Debian Bookworm just released, with git-annex 10.20230126](https://packages.debian.org/bookworm/git-annex) then the change in behaviour is at least 2-3 years away, at Debian's normal stable release schedule. But this doesn't seem to be documented anywhere else.
If the plan is that this change in default behaviour will be, eg, in 2H 2025, then I'd suggest (a) putting that planned date in the warning being issued on every run, and (b) putting that date in the documentation for [git annex sync](https://git-annex.branchable.com/git-annex-sync/) which currently just says "will change in a future version of git-annex" (which is very vague, and could be next month or a decade away). However as stated above, I'd really really strongly suggest just creating a new command, like `fullsync` for the new default behaviour, and *not* breaking backwards compatibility in behaviour.
### What steps will reproduce the problem?
Upgrade to git-annex 10.20230626, run `git annex sync` in a git annex repository, without having set `annex.synccontent` (i the git-annex config, or the git config).
Try to find documentation on this pending change; find nothing other than the changelog note foreshadowing a major change in behaviour after 10 years, and some comments in the source code.
### What version of git-annex are you using? On what operating system?
git-annex 10.20230626, on macOS, installed from Home Brew.
```
ewen@basadi:~$ git annex version
git-annex version: 10.20230626
build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.24.1 bloomfilter-2.0.1.0 cryptonite-0.30 DAV-1.3.4 feed-1.3.2.1 ghc-9.4.4 http-client-0.7.13.1 persistent-sqlite-2.13.1.1 torrent-10000.1.3 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: darwin x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
ewen@basadi:~$
```
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Absolutely. I've been using git-annex since nearly the beginning; and use it extensively to maintain partial copies of large annexes on laptops/desktops (and other space constrained systems). Hence this foreshadowed sudden change in behaviour being extremely surprising, and somewhat alarming.
Thanks for writing git-annex. It's the reason I've been sponsoring you on Patreon for years.
Please don't break backwards compatibility. Even in the user experience :-)
> [[closing|done]] per new plan --[[Joey]]

51
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_10_ec80bdff87c7b70e4235cd5b2ba6ed05._comment
View file

@ -1,51 +0,0 @@
[[!comment format=mdwn
username="jkniiv"
avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d"
subject="comment 10"
date="2023-07-12T16:25:48Z"
content="""
@nobodyinperson, @ewen: Thank you for a balanced discussion! I'm not as elegant with words
but here is my take on your ideas.
@nobodyinperson:
> How about:
> - `git annex assist` (exists, works):
Yes, I understand the need for `assist` -- not in any way opposed to it. However, I think both
`assist` and `sync` ought learn a new option called `--edit` (or, `--edit-message` but that's wordy)
to be used in a similar fashion to `git commit -m msg --edit`, ie. it would open the user's preferred editor
to edit the commit message.
> - `git annex metasync` (new proposal):
You mention that this command would be \"Option-wise as 'dumb' as git annex assist\" but I disagree: as a true
replacement for `sync` it definitely should have at least the `--no-commit` flag as an additional option so that those
used to a `git annex add; git commit -m foo [--edit]; git annex sync --no-content --no-commit` workflow could
migrate to it. Also, no auto-adding (please!) if this is supposed be a replacement for `sync --no-content`.
> - `git annex sync` (exists, has many options and configs):
You mention that this command ought to be \"deprecated eventually (in many years), with a big warning upon execution now already\".
Yes, I agree but only if we get a `metasync` with a `--no-commit` flag for reasons above.
---
@ewen:
> Firstly, if `git annex assist` has been added to create the magical \"Dropbox like experience\", I don't see the need for changing the default behaviour of `git annex sync` at all. (For the record I'm glad `git annex assist` exists; I don't need it, or want it, but it clearly solves a bunch of UI problems for some users, and that's great. This is entirely about backwards incompatible UX changes in existing commands.)
I think you have a point there that the situation pre-10.20230626 would be preferable.
But I can live with Yann's (@nobodyinperson) suggestions if there is some amount of compromising.
> Secondly, your (@nobodyinperson) suggestion for \"improving\" the current situation seems to be:
> 1. Entirely remove `git annex sync` (entirely breaking backwards compatiblity) and/or have it permanently display a warning that cannot be removed (annoying, incompatible with anything looking at output); and
> 2. Add a new (ie, not in any older version) `git annex metasync` which has some of the existing functionality, and some random other (unwanted by me at least) additional functionality (auto-adding any files in a directory where git annex is run, which I definitely do not want)
Yes, a replacement for `sync` should not in any case auto-add files. We want more control, not less.
(continued...)
"""]]

25
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_11_82e32aca2c23a1e27553bcd9708870b2._comment
View file

@ -1,25 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 11"""
date="2023-07-12T17:03:27Z"
content="""
Please don't engage in unwarranted speculation about strange desires of mine
to eliminate features that are crucial to git-annex. In general, avoid
inflammatory language on this website. Thank you.
It's important that git-annex be able to make changes that make it better.
Backwards compatability is not everything. Improvements can make git-annex
more cohesive and easier to understand for new users.
This kind of pre-warned transition is commonly used by git when making
similarly impactful changes.
annex.synccontent can be set in your global git config. It can also be set
with `git-annex config` which will affect all clones of that repo. This
will be a minor amount of work for most users to deal with.
BTW, this transition plan actually started 6 years ago when I added
annex.syncontent configuration and --no-content.
[[!commit b77903af48e650dbb777f29e98d0c7b388353ebd]]
As you note, it's important that's already available in all commonly used
old versions of git-annex.
"""]]

48
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_12_4e0746d00d8a596d878c1fe12ec68849._comment
View file

@ -1,48 +0,0 @@
[[!comment format=mdwn
username="jkniiv"
avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d"
subject="comment 12"
date="2023-07-12T17:30:52Z"
content="""
(...continued)
@ewen:
> My suggestion for preserving backwards compatibilty would be:
> 1. Move the behaviour change out of `git annex sync` and into `git annex init` (rationale: it's run once per annex/machine pair, not many times per day/week, *and* it makes the change \"new annexes only\" which is much closer to \"opt in\")
> 2. From git annex version N + 1, have git annex automatically set `annex.synccontent=true` at `git annex init` of a *new* repo (ie, not just naming the local copy of the annex), and for version N+1 to N+3 (or higher) have `git annex init` issue a warning it has defaulted sync content on now (unlike earlier versions), and describe how to find out how to turn it off (unless a global git setting is making `annex.synccontent=true` the user preferred default anyway, in which case the user doesn't need a warning of a setting matching their preferred option).
> 3. For any annex where `annex.synccontent` is *not* set, assume it's an older annex, and use the backwards compatible, historical, default (false) *without* issuing any warnings about \"this is going to change\" (and never change that behaviour for historical annexes)
> 4. Retain `git annex sync` forever, with it obeying `annex.synccontent` (default: false, but set to true in newly created annexes by default)
> 5. Remove the newly added warning in `git annex sync` entirely, and just keep the functionality of the version before 10.20230623
Sounds good. I like the concept of an opt-in the way you laid it out. It's well thought out.
> This would:
> 1. Allow changing the `git annex sync` default from `metadata only` to `full content` in a \"new usage, opt in only\" way
> 2. Provide a clear path to opt out, and a clear path to opt in existing repos to match
> 3. Avoid needing to issue warnings to users on every run of a very core command until they set config
> 4. Avoid breaking existing usage, and use cases
(Yes, that warning is annoying to say the least.)
I'm willing to accept some breakage to my workflows but I'd rather still see a separate
`git annex add`, `git commit -m ...` and \"sync metadata with one command\" workflow being supported.
The name of that sync command is not that important for me but I understand why it is for you.
> The git annex \"meta data sync\" dance (which effectively allows `git push` into a repo that has a working directory attached, something that is normally otherwise difficult) was great, and the thing which made git annex attractive to me over other options. It'd be a shame to render that great functionality unreachable without unwanted \"some users wanted this, so all users must have this\" functionality.
I agree to the strongest degree. I'm not a very good git user, but the core parts of git-annex
like `add` and `sync` have made me feel comfortable to a greater extent than the porcelain of core git.
I'm sure I can adapt to newfangled ways of doing my routine things with git-annex but as on \"old dog\" I'd
rather not adopt those new ways if I can avoid it.
"""]]

32
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_13_80ba2e4241372ff3ce1505900e17c22f._comment
View file

@ -1,32 +0,0 @@
[[!comment format=mdwn
username="ewen"
avatar="http://cdn.libravatar.org/avatar/605b2981cb52b4af268455dee7a4f64e"
subject="Plumbing commands vs Porcelain commands"
date="2023-07-13T00:26:20Z"
content="""
@joey (re [comment 11](http://git-annex.branchable.com/bugs/Changing_sync_to_include_content_breaks_UX/#comment-e4557f089ec22304ff506f09f791bb81)): thank you for replying. I apologise if my tone sounded accusatory; in hindsight writing late at night after a stressful year probably wasn't the best idea.
I don't know what your intentions are here, which is part of the problem. From the outside, it feels like the git-annex assistant functionality is now the dominant use case which is guiding the development changes. (And that's understandable if it's most/all of your funding to work on the project.) That seems to be leading towards the \"plumbing\" low level commands being transitioned into higher level (\"porcelain\" in git terminology) commands, which do multiple things; and access to the low level functionality being decreased or removed.
Yes, git has done a few \"UX breaking\" transitions in the past. But in all the cases I can think of the change in functionality was accompanied with obvious documentation on why the change was being made, and how to adapt. For instance the [git safe directory behaviour change (CVE-2022-24765)](https://github.blog/2022-04-12-git-security-vulnerability-announced/) prompted a lot of documentation about why it was changed (eg, [git config manpage](https://git-scm.com/docs/git-config#Documentation/git-config.txt-safedirectory)), and specifically how users could restore the existing functionality where that was appropriate (and, eg, it gave an exact command to run in the warning output). I've been running into that problem repeatedly for a year (including supporting other users), but at least knowing why, and that the recommended \"fix command\" is the ideal solution, makes it quick to explain/fix/carry on.
In the case of this git-annex change to `sync` (from a \"just plumbing\" meta data sync command, to a \"porcelain\" git annex assistant like command) there doesn't seem to be any clear documentation on why it is changing, or the recommended process to preserve (or recreate) the existing default functionality (meta data sync only), particularly in a backwards compatible way.
The reference to the [six year old commit](http://source.git-annex.branchable.com/?p=source.git;a=commitdiff;h=b77903af48e650dbb777f29e98d0c7b388353ebd) which you see as \"starting this change\" adds some more context, but almost all of the \"documentation\" about this change appears to be scattered between some source code commits, some short changelog entries, and a bug that started being about something else. As someone who has followed your personal blog and the git-annex dev blog for years, the change of \"sync\" from a low level plumbing command (with optional \"please do more\") to a high level porcelain command (with optional \"please skip most of the things you want to do) still seemed to appear very suddenly, without any foreshadowing it was going to change. And I remain unclear on the intended timeline for the (fairly fundamental, IMHO) change in default behaviour.
For my purposes I think from here on I'll be doing *all* of the following:
1. On any system I install git-annex, running `git config --global annex.synccontent false` to set my per-user, per-machine state to equivalent to now
2. On any git-annex wrapper scripts I have that run `sync`, explicitly calling `git annex sync --no-content` instead of a bare `git-annex sync` as before
3. On any new git-annex repo that I set up, running `git annex config --set annex.synccontent false`
4. On the existing git-annex repos, as I interact with them again and remember, run `git annex config --set annex.synccontent false` to maintain the existing default
5. On new/existing git-annex repos, consider also running `git annex wanted . present` as well, since that seems to effectively match my current policy (ie, what is there is what is supposed to be there, because I put it there).
(I appreciate that mostly any one of those \"should be sufficient\". But with a lot of existing git annex repos, spread across lots of machines/drives, some of which are offline, the risks of overlooking one or more of them later on is non-trivial. So belt and braces here.)
Ewen
"""]]

17
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_14_08f3c3d26d13fd44ae9c26791b90a8fc._comment
View file

@ -1,17 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 14"""
date="2023-07-14T00:01:27Z"
content="""
> From the outside, it feels like the git-annex assistant functionality is
> now the dominant use case which is guiding the development changes.
Based on what, the changes in the last release?
You're seeing a pattern where none exists.
> (And that's understandable if it's most/all of your funding to work on the project.)
No such funding, not that I allow funding to determine overall direction of
this project.
"""]]

18
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_15_38a0eafe91e05a7784dd5e305f38a8f3._comment
View file

@ -1,18 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 15"""
date="2023-07-15T16:09:46Z"
content="""
My current thinking is that when a repository does not have a preferred
content expression, a content sync could avoid adding content to it.
This would avoid surprising behavior after the transition, except for
repositories that did at some time have a preferred content expression set,
but one that no longer reflects the use of the repository.
However, current users of `sync --content` (or of annex.synccontent) do
expect it to fill their repositories that don't have preferred content
configured. So would a sync that defaults to content syncing then need to
behave differently than an explicit `sync --content`? Or should changing
behavior in that case also be part of the transition?
"""]]

12
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_16_ee529ac7b3d27f1cdc7edf5154c0da8f._comment
View file

@ -1,12 +0,0 @@
[[!comment format=mdwn
username="nobodyinperson"
avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5"
subject="comment 16"
date="2023-07-15T17:04:38Z"
content="""
> My current thinking is that when a repository does not have a preferred content expression, a content sync could avoid adding content to it.
Technically, that would make the default preferred content expression `present` instead of `include=*`, right? I'd like that. Less unwanted content after the initial clone when running `git annex assist` or `git annex sync --content`. Goes into the direction of [specifying the default preferred content expression](https://git-annex.branchable.com/todo/Setting_default_preferred_content_expressions/). Users having freshly cloned a repo can then explicitly `get` files or set up their preferred content expression.
Many annexed repos will be huge in size and rarely one will want *everything* from it initially. Also great for DataLad datasets.
"""]]

24
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_17_d60adea3f0c0030a403c33c57f27abda._comment
View file

@ -1,24 +0,0 @@
[[!comment format=mdwn
username="ewen"
avatar="http://cdn.libravatar.org/avatar/605b2981cb52b4af268455dee7a4f64e"
subject="unspecified default preferred content = present"
date="2023-07-17T00:50:09Z"
content="""
I would also be in favour of changing the default preferred content expression to be \"present\" at the same time as `sync` was changed to automatically sync content as well as metadata. The effective user experience of a user with the existing default out of box configuration would be basically equivalent (now: `sync` does not sync content by default; then: `sync` will sync content if a policy to indicate desired content is set, but the default policy is \"whatever is here is correct\").
If that change were made (default preferred content expression is \"present\") then:
* I think the current warning on `sync` without `--no-content` could be removed (as the effective default change from the historical defaults is small)
* There should probably be a warning on `git annex sync` *with* a preferred content policy explicitly set (*other than* `present`, the new default), that the `sync` without `--no-content` will \"soon\" automatically include that preferred content
* There should probably be a warning on `git annex sync --content` where there is no preferred content policy explicitly set, that the default preferred content policy is changing, and the user should set an explicit one
That should result in \"I do everything manually\" users (who haven't set a preferred content policy, or set it to `present`) having basically \"no effective change\". And the special cases of users (relying on the default `sync` not including their preferred content policy; relying on the existing default preferred content policy to work with `git annex sync --content`) get a reminder they'll need a smaller tweak to their config.
I'd also be fine if *new* git annexes (ie `git annex init` after new release) were to set an explicit preferred content policy to the current default (just leaving \"no content policy set\" as the implied `present` suggested above). So new annexes would still get the \"gets everything by default\" behaviour that it seems some users want. (I do *not* want everything by default as almost always I use annexes for content that is too large to keep on my laptop/desktop.)
Thanks for considering reducing the size of the UX change here.
Ewen
"""]]

12
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_18_594ae4ce426c7fd9963a22def58904d8._comment
View file

@ -1,12 +0,0 @@
[[!comment format=mdwn
username="nobodyinperson"
avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5"
subject="comment 18"
date="2023-07-17T06:12:26Z"
content="""
> I think the current warning on sync without --no-content could be removed (as the effective default change from the historical defaults is small)
The effect in term of file location will be the same after a `git annex symc` (with `present` as new defsult - or let's call it fallback - preferred content expression) and a `git annex sync --content`. **But**: There's one thing that `--content` also changes in terms of UX: speed/runtime. In my large annex repos with many files (~5 figures), a `git annex sync --content` takes *much* longer, even if no content is eventually synced. Git Annex apparently needs to go through all files whether they want to be copied somewhere. **Maybe a short-circuit path could be introduced here when all preferred content expressions effectively turn out to be `present`**. Users might wonder why `git annex sync` suddenly takes so long now.
In general, **a progress bar or if that's impossible another indication of what's going on between metadata syncing and content syncing** would be helpful. With this, I could agree that the warning can go away for a simple `git annex sync` with no preferred content expressions set.
"""]]

21
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_19_bdc211b026d3b05de127f6ed5a3d19b0._comment
View file

@ -1,21 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 19"""
date="2023-08-14T16:56:48Z"
content="""
Changing the default preferred content expression to `present` rather than
`anything` would affect other things like `get --auto` and the assistant. I
don't think that is a good idea.
My idea in comment #15 is a special case for sync and not a wider change.
It will have a special case either way. The current special case means
that users who have set preferred content need to remember to use `sync
--content`. The new special case will only manifest as a difference in
behavior between `sync` and `sync --content` (and `get --auto` etc)
for users with no preferred content configured.
I do think that's a valuable narrowing of the special case.
Gone ahead and changed the warning.
"""]]

12
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_1_1541a9323009b20f527501ebedc9d395._comment
View file

@ -1,12 +0,0 @@
[[!comment format=mdwn
username="nobodyinperson"
avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5"
subject="comment 1"
date="2023-07-12T06:49:46Z"
content="""
Simultaneously, a new command `git annex assist` has been added that does basically a single step of the assistant, so roughly `git annex add;git add -A;git annex sync --content`. That goes into the direction of your `fullsync` suggestion.
You can read more about the reasoning for `git annex sync` syncing content by default [here](https://git-annex.branchable.com/todo/Having___39__git_annex_sync__39___optionally_add/#comment-fc813a22c713490156234567ed211277), where I suggested to add a config to `git annex sync` that would optionally add new files as well. That turned out to launch an avalanche of new commands and changes (git annex pull, git annex push, git annex assist, sync syncing conent, etc.). 😅
IIUC, Joeys reasoning is that `git annex sync` was incomplete/inconsistent from the start and did too many configurable things. `git annex assist` syncs the entire repo state - as the assistant. Preferred content expressions can be used to specify what files a repo wants. If you set that to `git annex wanted REPO present`, a content sync won't affect it anyway. I still think that a configurable default preferred content expression for new repos is very important, now that syncinc content by default might get the default: https://git-annex.branchable.com/todo/Setting_default_preferred_content_expressions/
"""]]

21
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_2_c06017f123f1ee34c7310f923086d09a._comment
View file

@ -1,21 +0,0 @@
[[!comment format=mdwn
username="jkniiv"
avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d"
subject="I have to agree that this change to `sync` is annoying"
date="2023-07-12T07:18:54Z"
content="""
Frankly, as a fellow user of \"incomplete annexes\" I also find it rather jarring that having gotten used to git committing my own changes
on the command line and not relying on mere `sync` to do that (or the assistant -- I haven't found a use
for the assistant yet) but then also preferring to use invocations of `git annex sync --no-commit` (which I've abbreviated to
`git annex-sync` by way of a git alias) rather freely to record my git-level changes across remotes,
I would from here on out be forced to add the `-g` flag to all my invocations just to make sure that I don't get
an annoying warning or my then current repo doesn't cause possibly huge content to flow across the remotes because I haven't
remembered to set `annex.synccontent` appropriately. I know the ideal is to have preferred content settings
for all annexes but not every git-annex user have an innate sense of all the intricacies involved in doing that
and for them to be forced to do so is a bit too much, in my humble opinion.
@Joey, I have a suggestion: instead of breaking the UX of `sync`, why not do that on a lesser used command `mirror` and change it
to sync git-level data in addition to files (ie. I suggest that `mirror` could be a better/shorter name for OP's `fullsync`).
Then add a flag called `--only-content` to `mirror` to restore its old behavior. I bet that would have a smaller impact on git-annex's
users than making us count our blessings with `-g`s and all.
"""]]

17
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_3_c03b45321728155eae9470cd8bb2cc93._comment
View file

@ -1,17 +0,0 @@
[[!comment format=mdwn
username="jkniiv"
avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d"
subject="comment 3"
date="2023-07-12T07:33:16Z"
content="""
> IIUC, Joeys reasoning is that git annex sync was incomplete/inconsistent from the start and did too many configurable things.
That's not a reason to break backwards compatibility for such a prominent command as `sync`. Just no.
> git annex assist syncs the entire repo state - as the assistant.
I don't want to use the assistant -- or it's CLI alter ego assist -- I want my own manicured commit messages, puh-lease! The assistant
is a whole another use case, just not mine at the moment.
Just give us our old `sync` back. Rename it to `synk` (or, `sc`/`sk` -- as in \"sync for [k]urmudgeons\") if you want but let us have it. :)
"""]]

22
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_4_bb6972e9870a68a98d90c6847228ab31._comment
View file

@ -1,22 +0,0 @@
[[!comment format=mdwn
username="nobodyinperson"
avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5"
subject="comment 4"
date="2023-07-12T09:21:55Z"
content="""
Just for the record:
- `git annex assist -m MESSAGE` works, you can make own messages.
- `git annex assist` has nothing to do with the assistant and can be used without. It's just `git annex sync --content` plus adding new files. A very important command for newcomers to 'just sync it'.
To get 'the old sync' back, you have several options:
- `git config --global annex.synccontent false` for your entire local machine, overrides the next option
- `git annex config --set annex.synccontent false` for all participants of a git annex repo (yes, it syncs between machines), still overrideable by users' local git configs above
- properly set up preferred content expressions (effectively), e.g. `git annex wanted . present`.
- `git annex sync --no-content`
If you don't use preferred content expressions to decide which repo should get which files and you want only partial checkouts with some files, then I indeed don't see why you would ever need `git annex sync --content` as you probably manually `get` the files you want. In that case one could argue that `sync` is a misleading word, you are actually only looking for a metadata sync (`msync`, `metasync`, ...?) strictly without content.
[I was also surprised](https://git-annex.branchable.com/todo/Having___39__git_annex_sync__39___optionally_add/#comment-c8c3138128a684080e2aaafc48aedfcf) that joey opted for introducing `assist` instead of teaching `sync` how to (optionally) add new files as well and keeping the rest as it was. But I can understand the direction of his reasoning.
"""]]

14
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_5_da9812dd37de7bba4fe2b1785bdca1ae._comment
View file

@ -1,14 +0,0 @@
[[!comment format=mdwn
username="ewen"
avatar="http://cdn.libravatar.org/avatar/605b2981cb52b4af268455dee7a4f64e"
subject="History of change"
date="2023-07-12T10:27:43Z"
content="""
Thanks for the [pointer to the earlier bug discussing changing sync entirely](https://git-annex.branchable.com/todo/Having___39__git_annex_sync__39___optionally_add) @nobodyinperson; it's at least useful to have a reference for where all this breaking change suddenly came from given it's otherwise undocumented. I've added a back pointer comment to that bug pointing at this one, as they're clearly related discussion (and it wasn't obvious the other \"sync\" bug was at all related to the change).
I still strongly disagree with changing \"sync\" to do something pretty different from what it's done for 10 years being a good idea. But would certainly support there being *another* command that did a \"Dropbox like full sync\".
(Personally I'd really really prefer that \"sync\" stayed doing the same thing it's done for a decade: meta data sync, only. If we're going to have to run a new different command just to retain core \"meta data sync\" behaviour, I guess \"git annex sync --no-content\" is not much worse than any other command we'd have to remember to suddenly use instead starting version N + 1.)
Ewen
"""]]

16
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_6_bc8154b171c460b4590d2b32401f24a2._comment
View file

@ -1,16 +0,0 @@
[[!comment format=mdwn
username="ewen"
avatar="http://cdn.libravatar.org/avatar/605b2981cb52b4af268455dee7a4f64e"
subject="Preferred content settings not really an alternative"
date="2023-07-12T10:53:41Z"
content="""
To echo @jkniiv's \"preferred content settings for all annexes\", as a git annex user for at least 8 years (the annex I found this issue in dates back to 2015), I've still never figured out how \"preferred content settings\" are supposed to help in git annex -- let alone for my use cases.
As an example, the git annex I spotted this new warning in is one I use for downloading/collecting podcasts. The downloads happen on my desktop, or sometimes a laptop. All content gets *pushed* to two of my NASes as soon as possible (for an \"archival copy\" -- immediately after download if I'm at home), and then once I've listened to the podcast I drop it from my desktop/laptop, so it's gone from the \"to listen\" queue (and historically, to recover space on smaller laptop drives). I don't want the files suddenly appearing again, unless I need to refer to it for some reason, then I do \"git annex get ...\".
There's around 300GB of podcasts in the annex (around 7000 files) on my NASes; but the checked out version on my desktop is 9GB (about 200 files). 300GB is more space than some of my laptop internal drives have in total; and definitely more free space than any of them had, which is one of the reasons for using git annex for this purpose.
The \"preferred content policy\" is \"if I just downloaded it I want it until I drop it, then I don't want it unless I explicitly request it again, but the two NASes should always have copies\". From [setting git annex preferred content policy](https://git-annex.branchable.com/preferred_content/), and the [preferred content policy syntax](https://git-annex.branchable.com/git-annex-preferred-content/), AFAICT \"present\" is the closest thing to the \"content policy\" I want on my desktop/laptops, ie \"trust the user, I know what I'm doing, don't automatically move files\". But up to version \"10.20230626 + N\" that was also the default without a content policy, as file content didn't automagically move around unless some other policy was explicitly set.
Ewen
"""]]

33
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_7_20febad4aa06d5b39668a858d2b118b1._comment
View file

@ -1,33 +0,0 @@
[[!comment format=mdwn
username="nobodyinperson"
avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5"
subject="There is already something like fullsync: `git annex assist`, suggestion: new `metasync` command"
date="2023-07-12T11:15:26Z"
content="""
> But would certainly support there being another command that did a \"Dropbox like full sync\".
You might have missed it above, but `git annex assist` is **exactly** that new command that does a 'Dropbox-like sync' (adding new files, adding changes, metadata sync, content sync) and as OP suggested with `fullsync`. It is there, it works, it's wonderful. I think we can put that part of the discussion to rest.
> Personally I'd really really prefer that \"sync\" stayed doing the same thing it's done for a decade: meta data sync, only.
Actually, `git annex sync` isn't always **just** metadata sync. If you're in a repo where someone (anyone!) decided to run `git annex config --set annex.synccontent true`, there flies your Terabytes of content around when you do your `git annex sync`, assuming it's metadata-only. At first I was a fan of teaching `git annex sync` everything with flags and configs, but I think joey is right that this is inconsistent and leads to surprises.
If we're at it, we could get rid of the ambiguous `sync` command altogether (this is what joey also proposed for the very long run) and introduce a metadata-only sync command (`metasync`?) that **can't** sync content. Not with a setting, not with a flag, it just can't. Then that's perfect for those who only want to sync metadata and `get` files they want manually and without preferred content expressions (I suppose that's how you do it?).
How about:
- `git annex assist` (exists, works):
- syncs everything: metadata&content, leaves the repo completely clean and all participating remotes updated to the same state
- for those who don't want to run the assistant, but still have 'one command to sync it all' whenever they feel like it or want to specify a custom commit message.
- Preferred content expressions manage what files land where.
- Very important for newcomers as this will effectively be the only command they'll need to know to get a workflow started.
- `git annex metasync` (new proposal):
- basically `git annex sync --no-content` that **can't** ever sync content. No surprises through configs.
- Leaves the repo in a clean state and all connected remotes at the same state of metadata (the git stuff). Also adds new files! People who want more control should use the manual commands `git annex add|push|pull|etc.`.
- Option-wise as 'dumb' as `git annex assist'
- for those who never want automatic content syncing but `get|copy|move|...` stuff manually and don't want to fiddle with preferred content expressions
- current users of 'old' `git annex sync` without content syncing will eventually switch to this.
- `git annex sync` (exists, has many options and configs):
- roll back to non-content-syncing as before, for those that don't want to change their workflows yet
- deprecated eventually (in many years), with a big warning upon execution now already.
"""]]

43
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_8_84d31b8fdde0f0ab0355a5d30760e90d._comment
View file

@ -1,43 +0,0 @@
[[!comment format=mdwn
username="ewen"
avatar="http://cdn.libravatar.org/avatar/605b2981cb52b4af268455dee7a4f64e"
subject="Alternative proposal for better backwards compatibility"
date="2023-07-12T12:48:03Z"
content="""
Firstly, if `git annex assist` has been added to create the magical \"Dropbox like experience\", I don't see the need for changing the default behaviour of `git annex sync` at all. (For the record I'm glad `git annex assist` exists; I don't need it, or want it, but it clearly solves a bunch of UI problems for some users, and that's great. This is entirely about backwards incompatible UX changes in existing commands.)
Secondly, your (@nobodyinperson) suggestion for \"improving\" the current situation seems to be:
1. Entirely remove `git annex sync` (entirely breaking backwards compatiblity) and/or have it permanently display a warning that cannot be removed (annoying, incompatible with anything looking at output); and
2. Add a new (ie, not in any older version) `git annex metasync` which has some of the existing functionality, and some random other (unwanted by me at least) additional functionality (auto-adding any files in a directory where git annex is run, which I definitely do not want)
Which, to me, is *even more* breaking the user experience of using `git annex` for the last 8 years, than is currently happening. AFAICT there would then be *no way* at all to get the `git annex` \"just sync metadata without other unwanted behaviour\" that existed before. *And* it'd then require git annex version detection in scripts to tell if `git annex metadata --no-add-really-just-copy-metadata-honest` was available or not, or if one had to fall back to an old version of the command, which is a really user-hostile way to \"retain\" existing behaviour. (FTR in my case at least there is *zero* chance that `git annex sync` would suddenly encounter `annex.synccontent=true`, as it's just me, and I haven't set that; I don't need a \"guarantees\" it won't copy content option, except as a defense against the git annex defaults suddenly changing after 8 years.)
My suggestion for preserving backwards compatibilty would be:
1. Move the behaviour change out of `git annex sync` and into `git annex init` (rationale: it's run once per annex/machine pair, not many times per day/week, *and* it makes the change \"new annexes only\" which is much closer to \"opt in\")
2. From git annex version N + 1, have git annex automatically set `annex.synccontent=true` at `git annex init` of a *new* repo (ie, not just naming the local copy of the annex), and for version N+1 to N+3 (or higher) have `git annex init` issue a warning it has defaulted sync content on now (unlike earlier versions), and describe how to find out how to turn it off (unless a global git setting is making `annex.synccontent=true` the user preferred default anyway, in which case the user doesn't need a warning of a setting matching their preferred option).
3. For any annex where `annex.synccontent` is *not* set, assume it's an older annex, and use the backwards compatible, historical, default (false) *without* issuing any warnings about \"this is going to change\" (and never change that behaviour for historical annexes)
4. Retain `git annex sync` forever, with it obeying `annex.synccontent` (default: false, but set to true in newly created annexes by default)
5. Remove the newly added warning in `git annex sync` entirely, and just keep the functionality of the version before 10.20230623
This would:
1. Allow changing the `git annex sync` default from `metadata only` to `full content` in a \"new usage, opt in only\" way
2. Provide a clear path to opt out, and a clear path to opt in existing repos to match
3. Avoid needing to issue warnings to users on every run of a very core command until they set config
4. Avoid breaking existing usage, and use cases
The git annex \"meta data sync\" dance (which effectively allows `git push` into a repo that has a working directory attached, something that is normally otherwise difficult) was great, and the thing which made git annex attractive to me over other options. It'd be a shame to render that great functionality unreachable without unwanted \"some users wanted this, so all users must have this\" functionality.
Ewen
PS: `git annex push` / `git annex pull` do not appear to be meta data only tools to replace sync; they appear to be very explicitly targetted at copying content, and to be one remote at at a time as well (where as sync is \"all known remotes\"). Historically it looks like my wrapper scripts use `git annex get ...` to retrieve content, and `git annex copy --to=REMOTE ...` to put content onto a specific remote (ie, a `put`, when there is no `put`).
"""]]

12
doc/bugs/Changing_sync_to_include_content_breaks_UX/comment_9_17a75fa68bf3718230d1d523b7eb0250._comment
View file

@ -1,12 +0,0 @@
[[!comment format=mdwn
username="nobodyinperson"
avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5"
subject="comment 9"
date="2023-07-12T13:10:26Z"
content="""
Fine with me to keep `git annex sync` at its old behaviour. I'm happy about `assist|push|pull` and would also be happy with a `metasync`.
These are all decisions that joey will have to decide himself. He has written this amazing piece of software that I'll be forever grateful for (Thanks joey!!). If he decides that `git annex sync` syncs content, so be it.
Your idea with moving the content-syncing into `init` sounds reasonable from a backwards-compatibility point of view. But I doubt joey will want to make `init` (or any other command) accumulate patchy stuff.
"""]]

66
doc/bugs/Debian_and_MacOSX_cannot_read_eachother__39__s_S3_credentials.mdwn
View file

@ -1,66 +0,0 @@
### Please describe the problem.
I seem to be hitting a problem with the way git annex encrypts the credentials for S3 remotes. I am creating an S3 remote with shared encryption and embedded credentials. If I create the remote on MacOSX, enableremote fails on Debian. If I create the remote on Debian, enableremote fails on MacOSX with a similar error. I can enable MacOSX created remotes on other MacOSX systems and Debian created remotes on other Debian systems.
The enableremote creates a .git/annex/creds/uuid file, but the contents are garbled.
Could this be an issue with different versions of GPG on the systems?
### What steps will reproduce the problem?
- New git annex remote on Debian.
- Clone and git annex init on MacOSX
- Create S3 remote on MacOSX with encryption=shared embedcreds=yes
- Git annex sync
- On Debian enable remote
Alternatively, create the S3 remote on Debian, sync and enable remote on MacOSX
### What version of git-annex are you using? On what operating system?
MacOSX
gpg (GnuPG/MacGPG2) 2.0.30
libgcrypt 1.6.6
git-annex version: 6.20170101
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents ConcurrentOutput TorrentParser MagicMime Feeds Quvi
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
local repository version: 5
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: darwin x86_64
Debian 7
gpg (GnuPG) 1.4.12
git-annex version: 6.20170101+gitg93d69b1-1~ndall+1
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
local repository version: 5
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: linux x86_64
### Please provide any additional information below.
[[!format sh """
# Debian 7
$ git annex enableremote S3
enableremote S3 gpg: packet(2) with unknown version 13
gpg error above was caused by an old git-annex bug in credentials storage. Working around it..
git-annex: bad creds
CallStack (from HasCallStack):
error, called at ./Creds.hs:116:22 in main:Creds
# MacOSX
git annex enableremote S34
enableremote S34 gpg: [don't know]: invalid packet (ctb=6b)
gpg error above was caused by an old git-annex bug in credentials storage. Working around it..
git-annex: bad creds
CallStack (from HasCallStack):
error, called at ./Creds.hs:116:22 in main:Creds
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Yes! git annex has been enormously helpful. Thanks so much for this tool.
> I guess this was caused by an old gpg, and so no point in keeping it
> open. [[done]] --[[Joey]]

28
doc/bugs/Debian_and_MacOSX_cannot_read_eachother__39__s_S3_credentials/comment_1_e307e5d0ec02ed76dc7c3f0894770539._comment
View file

@ -1,28 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2017-02-13T18:27:02Z"
content="""
The .git/annex/creds/uuid files for S3 special remotes are supposed to
contain the S3 access credentials, the same values you set
`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to. Since it's just some
high-entropy credentials, perhaps that's why you thought it looked garbled?
I doubt that the error message about the cause of the gpg error is
right; that is a workaround for an old bug in git-annex 5.20140916. Unless
you used such an old version to set up the special remote, it's probably
not related to that problem at all.
The actual problem seems to be:
gpg: packet(2) with unknown version 13
gpg: [don't know]: invalid packet (ctb=6b)
This probably points to an incompatability between the gpg versions.
The easiest fix is probably to upgrade Debian to gpg 2.
With embedcreds, the S3 creds are symmetrically encrypted/decrypted
using gpg --symmetric. This is a less common way to use gpg,
and it could be that there's an incompatability with how different
versions of gpg do it.
"""]]

55
doc/bugs/Directory_remotes_with_same_mount_point.mdwn
View file

@ -1,55 +0,0 @@
### Please describe the problem.
I have a repository with two directory special remotes that use the same mount point (it's a USB SD card reader using udevil). I use "git-annex-command" with lsblk and grep using the UUID to prevent git-annex from attempting to sync with the remote if it's not mounted. When trying to import from the second remote, I get "file content has changed" for every file, which seems to be related to inode-based caching.
Each of the remotes uses a subdirectory of the mount, though I did change the directory for the first one (it originally used the mount point directly).
### What steps will reproduce the problem?
1. Create a directory special remote on a removable device and import from it
2. Create a second directory special remote on a second removable device at the same mount point and import from it
### What version of git-annex are you using? On what operating system?
8.20201130-g75988a790, Debian Buster
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ git annex import --from=pi1541 master:c64
list pi1541 ok
import pi1541 nav96.d64
file content has changed
ok
import pi1541 fb64
file content has changed
ok
import pi1541 cbmcmd22.d64
file content has changed
ok
import pi1541 cbmcmd22.d80
file content has changed
ok
...
# End of transcript or log.
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I've used it successfully on Linux and Android. My attempts on Windows have been less successful, though I never tried with core.ignoreStat=true. Now I'm trying to use it to manage the SD cards for various devices: my Pi1541, Odroid Go, and Sansa Clip Zip.
> Calling this [[done]] because I'm fairly sure if it was some kind of
> inode related bug, it would be fixed now by changes to the directory
> special remote. Please do followup if you still see the bug with the most
> recent git-annex release, or have other information about how to
> reproduce the alleged behavior. --[[Joey]]

9
doc/bugs/Directory_remotes_with_same_mount_point/comment_1_351063bf8fb7779a9a7ecd21bef8b801._comment
View file

@ -1,9 +0,0 @@
[[!comment format=mdwn
username="seanl@fe5df935169a5440a52bdbfc5fece85cdd002d68"
nickname="seanl"
avatar="http://cdn.libravatar.org/avatar/082ffc523e71e18c45395e6115b3b373"
subject="Works fine after changing mount point"
date="2021-01-17T07:24:38Z"
content="""
I uninstalled udevil in favor of udiskie, which uses a mount point name based on the label instead of the device, updated the directory to point at the new mount point with `git annex enableremote pi1541 <new mount point>/1541`, and the problem went away. But it doesn't seem like having different mount points for removable devices should be a requirement.
"""]]

23
doc/bugs/Directory_remotes_with_same_mount_point/comment_2_90351193c14f4c68f9b92640dae82f13._comment
View file

@ -1,23 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2021-01-18T16:23:42Z"
content="""
I don't think this has anything to do with mount points being in common.
Consider that, a single disk with a git-annex repository, and 2 directory
special remotes all on it has everything under a common mount point. And
I'm pretty sure works. Also there's just nothing in the relevant code that
cares what the mount point is.
It looks to me like the inode was somehow not the same in the "list" stage
of the command as it was in the "import" stage, or the file that was seen
in the first stage was no longer present in the second stage. I don't see
how that could be a git-annex bug. It could, for example, happen if one SD
card was mounted there at the "list" stage, and then a different card got
mounted there at the "import" stage, or even if the SD card just got
unmounted in between the two stages.
I have not been able to replicate the problem either, using 2 removable
devices on the same mount point, 2 special remotes, etc. So you will have
to show me how to reproduce this if it's actually a bug.
"""]]

9
doc/bugs/Directory_remotes_with_same_mount_point/comment_3_6b7727db8e2487394a80fc061d76868e._comment
View file

@ -1,9 +0,0 @@
[[!comment format=mdwn
username="seanl@fe5df935169a5440a52bdbfc5fece85cdd002d68"
nickname="seanl"
avatar="http://cdn.libravatar.org/avatar/082ffc523e71e18c45395e6115b3b373"
subject="Went away with fsck?"
date="2021-01-19T01:15:51Z"
content="""
I tried unmounting the sd card, fscking, and remounting, and then I was able to import without issue. I haven't tried a sync or a second import, though. All of my SD cards use VFAT, and this one has 160k+ files on it. I'm wondering if there are cases where Linux's VFAT implementation will give different inode numbers for the same file during the same session? The only mention of inode numbers in the Linux VFAT documentation is in the NFS section, though. I'll let you know if I run into it again.
"""]]

15
doc/bugs/Directory_remotes_with_same_mount_point/comment_4_5a4b3daa0f8a0d7f330f5a1d922d0f6a._comment
View file

@ -1,15 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2021-01-19T15:40:49Z"
content="""
I don't think linux would change inode numbers while the filesystem was
mounted. But each mount of FAT gets new inode numbers, since FAT doesn't
actually have them and linux makes them up. So the filesystem being
unmounted and remounted for some reason in between the two steps of the
import would explain the behavior.
Note that I've opened
[[todo/import_tree_from_FAT_does_unnecessary_work_due_to_inode_instability]]
after thinking of some other consequences of this.
"""]]

11
doc/bugs/Directory_remotes_with_same_mount_point/comment_5_fb21865b36819a54a23e1ccb7f377fa1._comment
View file

@ -1,11 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2021-01-19T16:53:54Z"
content="""
Well, the changes I made for that todo make changes to inodes due to
remounting in the middle of an import not cause behavior like this. Of
course I don't know that's what caused this behavior, but it does seem
likely those changes would turn out to have fixed this, if we understood
how to reproduce the problem.
"""]]

65
doc/bugs/Disabling_remote_auto-enabling_not_possible.mdwn
View file

@ -1,65 +0,0 @@
### Please describe the problem.
I have a repository with a special remote that is autoenable=true. The special remote is (no longer) accessible to everyone. I want to disable autoenable, but this fails, because the special remote is not accessible.
How can autoenable be disabled, once a remote is no longer (or even temporarily) inaccessible to someone?
### What steps will reproduce the problem?
Here is a small script that demos this for a type-git special remote. However, the problem has been initially observed with different external special remotes. I only reproduced it with a git-annex built-in to see whether an implementation detail in the external special remote is causing this.
```sh
# create an annex-repo (using datalad, but this is the standard init+init dance)
$ datalad create originds
# make a clone with datalad (same as clone+init)
$ datalad clone originds dsclone
# add the autoenabling special remote in the clone
$ git -C dsclone annex initremote myremote type=git "location=$(readlink -f originds)" autoenable=true
# as long as `originds` is accessible everything runs smoothly
$ datalad clone dsclone autoclone1
# make the special remote target inaccessible
$ mv originds originds-moved
# enabling is no longer possible
$ datalad clone dsclone autoclone2
$ git -C autoclone2 annex enableremote myremote
...
Remote myremote cannot currently be accessed.
...
# access by special remote name is no possible
$ git -C dsclone annex enableremote myremote autoenable=false
...
Remote myremote cannot currently be accessed.
...
git-annex: That is a normal git remote; passing these parameters does not make sense: autoenable=false
# determine UUID from
# git -C dsclone annex info | grep myremote
# because `git -C dsclone annex info myremote | grep uuid:` no longer lists it
$ git -C dsclone annex enableremote ba87b7e1-4fec-4826-bd01-302a8e2eda0a autoenable=false
Remote myremote cannot currently be accessed.
...
enableremote ba87b7e1-4fec-4826-bd01-302a8e2eda0a error: remote myremote already exists.
git-annex: git [Param "remote",Param "add",Param "myremote",Param "/tmp/originds"] failed
failed
enableremote: 1 failed
# autoenable is still on
$ git -C dsclone cat-file -p git-annex:remote.log
ba87b7e1-4fec-4826-bd01-302a8e2eda0a autoenable=true location=/tmp/originds name=myremote type=git timestamp=1681453180.645186501s
```
### What version of git-annex are you using? On what operating system?
I am on Debian sid. I tried git-annex versions 10.20221003 and 10.20230126 -- both show the same behavior.
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Loads! This particular issue started to show up in the datalad support channels due to aging infrastructure setups with still popular datasets tracking them using autoenabling special remotes.
> [[fixed|done]] --[[Joey]]

18
doc/bugs/Disabling_remote_auto-enabling_not_possible/comment_1_136c0fa106347bd0474f8298fde227f5._comment
View file

@ -1,18 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2023-04-17T17:42:31Z"
content="""
`git-annex dead` will work.
But it does seem to me that there is perhaps room to have a way
to clear the autoenable setting without killing the remote entirely,
but without needing to be able to enableremote the remote.
And I think, when you say "The special remote is (no longer) accessible to
everyone" you mean some people still have access, right? But you're not
one of those people and want to disable the autoenable.
If so, I ought to add something like `git-annex configremote foo
autoenable=false`.
"""]]

10
doc/bugs/Disabling_remote_auto-enabling_not_possible/comment_2_f6b1d513e664d1a5ee51b90f49731eeb._comment
View file

@ -1,10 +0,0 @@
[[!comment format=mdwn
username="mih"
avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd"
subject="Confirm"
date="2023-04-18T07:54:46Z"
content="""
Yes, in this particular case the situation is as you describe it. The remote itself is fine for a select few, but I (and others) do not have the necessary means to access it. I want to disable autoenable in my \"fork\" to streamline usage for myself and people like me.
I think your proposed solution would work well for this scenario. Thanks!
"""]]

12
doc/bugs/Disabling_remote_auto-enabling_not_possible/comment_3_c2c7144fa74bdc455e1bd36b594902c1._comment
View file

@ -1,12 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2023-04-18T16:28:40Z"
content="""
Hmm, you could dead it in your fork. The issue would be, if they ever merge
from it, their working repo will get marked as dead.
A `git-annex configremote` would likewise cause them problems if they
merged from your fork, if they rely on autoenabling being enabled. That's
less likely to be a problem to them I suppose.
"""]]

14
doc/bugs/Disabling_remote_auto-enabling_not_possible/comment_4_4f3e9f15fcc96cd98c6915ec68cc471f._comment
View file

@ -1,14 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2023-04-18T16:40:57Z"
content="""
A sub-bug here is that enableremote of a normal git remote adding autoenable=false
always fails with
"That is a normal git remote; passing these parameters does not make sense: autoenable=false".
That happens even when the repo is accessible. So there is no way to disable
a normal git remote that has been initremoted with autoenable=true.
Update: Fixed that.
"""]]

33
doc/bugs/Disabling_remote_auto-enabling_not_possible/comment_5_ccb5b6960da43a93ff0b7404190c51de._comment
View file

@ -1,33 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2023-04-18T16:49:45Z"
content="""
I retried your example not using a regular git remote, but a special
remote. When the special remote is made inaccessible, it does not fail in
the same way. Actually, the clone doesn't fail at all:
joey@darkstar:~/tmp/dl>datalad clone dsclone autoclone5
[INFO ] special remote made inacessible for testing
[INFO ] Fetching updates for Dataset(/home/joey/tmp/dl/autoclone5)
update(ok): . (dataset)
update(ok): . (dataset)
configure-sibling(ok): . (sibling)
install(ok): /home/joey/tmp/dl/autoclone5 (dataset)
action summary:
configure-sibling (ok: 1)
install (ok: 1)
update (ok: 1)
Likewise `git-annex init` displays the message from the special remote as a warning
but exits successfully and without autoenabling it.
Compare with datalad's behavior when it is a regular git remote. The failure there
happens because git-annex auto-enables the remote successfully, and then
datalad tries to fetch from it, but the repo is not available.
So other than the special case of a git remote stored by initremote, this bug's
real impact seems to only be that enableremote can't be used to change the
configuration of a remote that's not accessible and `git-annex init` will
warn about such a remote.
"""]]

18
doc/bugs/Disabling_remote_auto-enabling_not_possible/comment_6_5b6359663efb608b9a78ab17c02be9ee._comment
View file

@ -1,18 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 6"""
date="2023-04-18T18:11:18Z"
content="""
Seems that `git-annex configremote` would be very limited in what
configuration it can change. Changing configuration with enableremote
re-runs the `setup` method, which is passed a RemoteConfig with the user's
proposed changes and can either fail if the config is not valid, or return
a RemoteConfig that might not be the same as the input one. Since `setup` also
prepares to use the remote, which is what's failing when the remote is not
accessible, it could not be used by `configremote`.
So, I think autoenable is about the only config setting that it would make
sense for configremote to be able to change. Changing anything else would
risk storing a configuration that `setup` would not allow. Which could
break the remote in arbitrary ways.
"""]]

7
doc/bugs/Disabling_remote_auto-enabling_not_possible/comment_7_995b88efc83f76f14ccb22bc2a6ef4bd._comment
View file

@ -1,7 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 7"""
date="2023-04-18T19:29:28Z"
content="""
Implemented `git-annex configremote`
"""]]

46
doc/bugs/Duplicate_descriptions_cause_confusing_error.mdwn
View file

@ -1,46 +0,0 @@
### Please describe the problem.
Running `git annex info <remote description>` on a remote that shares a description with another remote shows a confusing generic error:
```
fatal: Not a valid object name testname
info testname (not a directory or an annexed file or a treeish or a remote or a uuid) failed
```
It should instead tell the user that there are multiple remotes by that description and unique identification is therefore not possible.
This is also a thing when all remotes except for one are dead. git-annex should prefer the alive remote here and ignore all the dead ones.
### What steps will reproduce the problem?
1. Have a git-annex repo with multiple remotes that share a description
2. Run `git annex info <remotename>`
### What version of git-annex are you using? On what operating system?
```
git-annex version: 10.20220822
build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22.1 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: darwin aarch64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 10
```
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
> [[fixed|done]] --[[Joey]]

67
doc/bugs/Enabling_useConfigOnly_not_honored.mdwn
View file

@ -1,67 +0,0 @@
### Please describe the problem.
Git's `user.useConfigOnly` is not honored by git-annex commands that create commits (e.g., `init` or `sync`).
### What steps will reproduce the problem?
Having not globally set `user.{name,email}`, the following will cause git-annex to ignore said setting and proceed with `$USER` as identity:
$ git config --global user.useConfigOnly true
$ git init annex && cd $_
Initialized empty Git repository in /tmp/annex/.git/
$ git annex init
init Author identity unknown
*** Please tell me who you are.
Run
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
to set your account's default identity.
Omit --global to set the identity only in this repository.
fatal: no email was given and auto-detection is disabled
ok
(recording state in git...)
Looking at the git-annex branch confirms this:
$ git log git-annex
commit 8eb575828cf52a3c150780d89c22672a51291f46
Author: $USER <$USER>
Date: 2023-03-29 13:22:44 +0200
update
commit 0b247e1da51e742bcf7b027aa25fc0d61520270d
Author: $USER <$USER>
Date: 2023-03-29 13:22:44 +0200
branch created
Furthermore, the identity is now even set in the local repository:
$ git config user.name
$USER
Enabling `user.useConfigOnly` should prevent this, making it easier to work with several different identities.
### What version of git-annex are you using? On what operating system?
git-annex version: 10.20230321-gb624394c7
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.24 bloomfilter-2.0.1.0 cryptonite-0.30 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.3 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 10
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Yes, it's great!
> [[fixed|done]] --[[Joey]]

17
doc/bugs/Enabling_useConfigOnly_not_honored/comment_1_4f0e4da06d8c9a9db771e593be1e2608._comment
View file

@ -1,17 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2023-04-04T18:34:32Z"
content="""
Yeah, git-annex goes to considerable trouble to make sure it can convince
git to commit. This is because, when I didn't do that, I was *inundated*
with bug reports from users with poorly configured systems where git cannot
guess a default user.name for various reasons.
I'm willing to support user.useConfigOnly, but only to the extent it
doesn't make me have to deal with all that again.
And I think it's probably unncessary for git-annex to set user.name and
user.email in the git config ever. It should be able to temporarily override
them when it needs to do so.
"""]]

11
doc/bugs/Enabling_useConfigOnly_not_honored/comment_2_5dede8445ba1c20e9750a30fe29de0b0._comment
View file

@ -1,11 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2023-04-04T19:17:56Z"
content="""
I've made user.useConfigOnly be honored when set.
Turns out it was not practical for git-annex init to not set user.name and
user.email when that is necessary to get git commit to work. But it now
avoids setting those when user.useConfigOnly is set.
"""]]

27
doc/bugs/FTBFS_for_the_2nd_day.mdwn
View file

@ -1,27 +0,0 @@
### Please describe the problem.
Got the report in the past days that failed to build across all platforms. Looked today:
```
[274 of 679] Compiling CmdLine.Usage ( CmdLine/Usage.hs, dist/build/git-annex/git-annex-tmp/CmdLine/Usage.o )
CmdLine/Usage.hs:63:1: error:
Duplicate type signatures for `paramRemote'
at CmdLine/Usage.hs:61:1-11
CmdLine/Usage.hs:63:1-11
|
63 | paramRemote :: String
| ^^^^^^^^^^^
CmdLine/Usage.hs:64:1: error:
Multiple declarations of `paramRemote'
Declared at: CmdLine/Usage.hs:62:1
CmdLine/Usage.hs:64:1
|
64 | paramRemote = "REPOSITORY"
| ^^^^^^^^^^^
```
on Debian, didn't check if other systems are anyhow different, will bounce report with pointers just in case.
> [[fixed|done]] --[[Joey]]

29
doc/bugs/FTBFS_for_the_2nd_day/comment_1_b7278de8f98c9fd96219281d61284461._comment
View file

@ -1,29 +0,0 @@
[[!comment format=mdwn
username="jkniiv"
avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d"
subject="comment 1"
date="2022-10-06T06:21:40Z"
content="""
That seems to be a simple typo where the latter/duplicate definition should be named `paramRepository` instead.
The following fixes the build:
[[!format diff \"\"\"
diff --git a/CmdLine/Usage.hs b/CmdLine/Usage.hs
index f3434d709..65879f7c3 100644
--- a/CmdLine/Usage.hs
+++ b/CmdLine/Usage.hs
@@ -60,8 +60,8 @@ paramNumRange :: String
paramNumRange = \"NUM|RANGE\"
paramRemote :: String
paramRemote = \"REMOTE\"
-paramRemote :: String
-paramRemote = \"REPOSITORY\"
+paramRepository :: String
+paramRepository = \"REPOSITORY\"
paramField :: String
paramField = \"FIELD\"
paramGlob :: String
\"\"\"]]
Windows tests are A-OK also.
"""]]

8
doc/bugs/FTBFS_for_the_2nd_day/comment_2_05c93396766cb9ac6a596db679519718._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 2"
date="2022-10-06T12:50:01Z"
content="""
Thank you jkniiv! If Joey doesn't fix it soon, I will add that patch. FTR: [an example of a patch PR for our datalad/git-annex build](https://github.com/datalad/git-annex/pull/77).
"""]]

15
doc/bugs/FTBFS_for_the_2nd_day/comment_3_0037f0e43383f32b22ee1ba2c98d78f8._comment
View file

@ -1,15 +0,0 @@
[[!comment format=mdwn
username="jkniiv"
avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d"
subject="comment 3"
date="2022-10-06T16:04:41Z"
content="""
You're welcome, Yarik! I try to do my part for git-annex although for medical reasons my personal
bandwidth is a bit limited (for instance, I don't pretend to grok Haskell much at all except at
a general functional programming level and I have even trouble with some of the intricacies of git, too).
However, I felt your last sentence was a small hint/suggestion, so with a small leap of faith I made my first
contribution to an outside project in the form of a pull request to datalad/git-annex,
[here](https://github.com/datalad/git-annex/pull/134) . :)
"""]]

28
doc/bugs/FTBFS_since_yesterday_on_linux.mdwn
View file

@ -1,28 +0,0 @@
### Please describe the problem.
thought to check on [another issue test](https://git-annex.branchable.com/todo/standlone__58___growing_number_of_libpcre.so_lookups/) but todays build failed with
```
FWIW, thought to look into current number but that test didn't run since build failed today due to
```
[34 of 35] Compiling Build.LinuxMkLibs ( Build/LinuxMkLibs.hs, Build/LinuxMkLibs.o )
Build/LinuxMkLibs.hs:129:17: error:
parse error (possibly incorrect indentation or mismatched brackets)
|
129 | "arm" ->
| ^
make[3]: *** [Makefile:150: Build/Standalone] Error 1
make[3]: Leaving directory '/home/runner/work/git-annex/git-annex/git-annex-source'
make[2]: *** [Makefile:158: linuxstandalone] Error 2
make[2]: Leaving directory '/home/runner/work/git-annex/git-annex/git-annex-source'
make[1]: *** [debian/rules:24: override_dh_auto_build] Error 2
make[1]: Leaving directory '/home/runner/work/git-annex/git-annex/git-annex-source'
make: *** [debian/rules:17: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
I: FAIL line matches with some context
Error: Process completed with exit code 1.
```
> [[fixed|done]] --[[Joey]]

56
doc/bugs/Failed_adjusted_branch_update_after_error_in_drop.mdwn
View file

@ -1,56 +0,0 @@
### Please describe the problem.
git-annex failed to update the local adjusted branch after `git annex drop` stopped abruptly. In turn, `git annex drop` stopped abruptly because a `git annex copy --from X --to Y` had to be stopped (CTRL-C) because of lack of space.
(Perhaps there are two issues here: `drop` being unable to cope with the unexpected situation, and the adjustment code being unable to cope with the weird stat left by `drop`'s abrupt interruption.)
### What steps will reproduce the problem?
```
$ git copy -J4 --from X --to Y
[notice that git-annex starts saying "not enough free space, need 27.87 MB more"]
[kill process with CTRL-C]
$ git annex drop dirF/
drop dirF/a/a1.txt ok
drop dirF/a/a2.txt ok
[...]
error: Your local changes to the following files would be overwritten by checkout:
dirF/a/a1.txt
dirF/a/a2.txt
[...]
Aborting
Updating adjusted branch failed.
(recording state in git...)
$ git status
HEAD detached at 9d92415fb
nothing to commit, working tree clean
$ git annex status
$ git branch
* (HEAD detached at 9d92415fb)
adjusted/master(unlocked)
adjusted/master(unlockpresent)
git-annex
master
synced/master
```
`dirF/` is the directory whose files were being copied when the process has been stopped with `CTRL-C`.
### What version of git-annex are you using? On what operating system?
```
git-annex version: 10.20230215-gd24914f2a
[...]
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 8
```
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
git-annex is too good. It so rarely causes problems that one does not develop the "git-annex troubleshooting muscle". :)
> [[fixed|done]] --[[Joey]]

11
doc/bugs/Failed_adjusted_branch_update_after_error_in_drop/comment_1_08b3eafdabe5f60ec2206584dff5d230._comment
View file

@ -1,11 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2023-03-23T19:26:12Z"
content="""
I assume this is a --hide-missing adjusted branch?
Update: Oh, I see from a forum post that it's --unlock-present actually.
What is the annex.adjustedbranchrefresh git config set to?
"""]]

42
doc/bugs/Failed_adjusted_branch_update_after_error_in_drop/comment_2_a947f8afde3d7f63fd33b0b7e5998e43._comment
View file

@ -1,42 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 2"""
date="2023-03-23T19:35:33Z"
content="""
I was able to reproduce something that looks similar to this without
needing to interrupt any command:
joey@darkstar:~/tmp/bench>git clone a b
Cloning into 'b'...
done.
joey@darkstar:~/tmp/bench>cd b
joey@darkstar:~/tmp/bench/b>git config annex.adjustedbranchrefresh 1
joey@darkstar:~/tmp/bench/b>git annex adjust --unlock-present
adjust
Switched to branch 'adjusted/master(unlockpresent)'
ok
joey@darkstar:~/tmp/bench/b#master(unlockpresent)>ls
foo@
joey@darkstar:~/tmp/bench/b#master(unlockpresent)>git-annex get
get foo (from origin...)
ok
(recording state in git...)
joey@darkstar:~/tmp/bench/b#master(unlockpresent)>ls
foo
joey@darkstar:~/tmp/bench/b#master(unlockpresent)>git-annex drop
drop foo ok
error: Your local changes to the following files would be overwritten by checkout:
foo
Please commit your changes or stash them before you switch branches.
Aborting
Updating adjusted branch failed.
(recording state in git...)
And it was left in a similar detached head status:
HEAD detached at 2aab85d
nothing to commit, working tree clean
This seems be be a bug with the implementation of annex.adjustedbranchrefresh
"""]]

23
doc/bugs/Failed_adjusted_branch_update_after_error_in_drop/comment_3_c8ce6c9fc35fa6ad5165ecf9a3592c9d._comment
View file

@ -1,23 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2023-03-23T19:50:45Z"
content="""
git-annex is what writes that detached HEAD,
in updateAdjustedBranch. And I've verified that when it fails like this,
git status shows the file is modified. But of course the modification is
only that the annex content has been replaced by an annex pointer. Running
`git add` on the file makes the modification status go away.
And this only happens when annex.adjustedbranchrefresh=1.
At higher values, it calls Annex.Queue.flush, but at 1 it does not, and so
restagePointerFiles does not get called before adjustedBranchRefreshFull.
(Or at least may not, they're both running as cleanup actions and order is
not really defined.)
I wonder if there are other situations where modifications can prevent
checkout of the updated adjusted branch? Eg, what if the user has made some
other modification to an annexed file? It seems wise for git-annex to
defend against it in depth, by making sure no crash can leave it in
detached head state.
"""]]

15
doc/bugs/Failed_adjusted_branch_update_after_error_in_drop/comment_4_2a4de8981ced7894148ca9a68e5ef60b._comment
View file

@ -1,15 +0,0 @@
[[!comment format=mdwn
username="gioele@678b7c03f524f2669b179b603f65352fcc16774e"
nickname="gioele"
avatar="http://cdn.libravatar.org/avatar/366dbda84e78aff8a8a070622aeb63ce"
subject="comment 4"
date="2023-03-24T08:13:00Z"
content="""
Thanks for investigating this!
These are the local settings:
thin = true
adjustedbranchrefresh = 1
"""]]

15
doc/bugs/Failed_adjusted_branch_update_after_error_in_drop/comment_5_7b989f777b211f21ca0e9ca681869bff._comment
View file

@ -1,15 +0,0 @@
[[!comment format=mdwn
username="gioele@678b7c03f524f2669b179b603f65352fcc16774e"
nickname="gioele"
avatar="http://cdn.libravatar.org/avatar/366dbda84e78aff8a8a070622aeb63ce"
subject="comment 5"
date="2023-03-25T08:47:01Z"
content="""
> I wonder if there are other situations where modifications can prevent checkout of the updated adjusted branch?
I clearly remember seeing something similar when a `sync` failed due a non-recoverable network failure. Unfortunately I have no logs from that day.
> It seems wise for git-annex to defend against it in depth, by making sure no crash can leave it in detached head state.
Perhaps also extend `fsck` to automatically recover from such a state? Or at least suggest a couple of possible solutions to the user?
"""]]

13
doc/bugs/Failed_adjusted_branch_update_after_error_in_drop/comment_6_f9ef22fcfabc6445060b964f08080c76._comment
View file

@ -1,13 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 6"""
date="2023-03-28T17:11:53Z"
content="""
Weird, I don't see how a network failure could lead in that direction.
Anyway, [[!commit 038a2600f4cf71294976280c5c29f6710359375f]]
avoids it with a detached head.
git-annex fsck can't do anything about this because the user may have their
own reasons to check out a detached head.
"""]]

19
doc/bugs/Failed_adjusted_branch_update_after_error_in_drop/comment_7_f679af07b798cb4bfae75565dd46e8ed._comment
View file

@ -1,19 +0,0 @@
[[!comment format=mdwn
username="gioele@678b7c03f524f2669b179b603f65352fcc16774e"
nickname="gioele"
avatar="http://cdn.libravatar.org/avatar/366dbda84e78aff8a8a070622aeb63ce"
subject="comment 7"
date="2023-04-01T11:06:25Z"
content="""
> I wonder if there are other situations where modifications can prevent checkout of the updated adjusted branch?
>
>> I clearly remember seeing something similar when a sync failed due a non-recoverable network failure. Unfortunately I have no logs from that day.
>
>>> Weird, I don't see how a network failure could lead in that direction.
All I remember is that the command that was running was `git annex sync --content`.
Perhaps the fact that `annex.alwayscommit = false` was set made a difference.
"""]]

22
doc/bugs/Failed_adjusted_branch_update_after_error_in_drop/comment_8_c7c210961b51a87770932f40a9563622._comment
View file

@ -1,22 +0,0 @@
[[!comment format=mdwn
username="gioele@678b7c03f524f2669b179b603f65352fcc16774e"
nickname="gioele"
avatar="http://cdn.libravatar.org/avatar/366dbda84e78aff8a8a070622aeb63ce"
subject="comment 8"
date="2023-04-01T11:09:49Z"
content="""
<hr/>
> Anyway, 038a2600f4cf71294976280c5c29f6710359375f avoids it with a detached head.
With git-annex 10.20230329-g30d7f9ad7 I got the following message after checking out `adjusted/master(unlockpresent)` and running `git annex copy --from foo --to bar -J2`:
```
git-annex: .git/HEAD: openFile: resource busy (file is locked)\"
```
This message appeared at the very end of the `copy` process.
Regardless of this, everything seems to work just fine.
"""]]

120
doc/bugs/Failure_to_get_small_files_over_P2P_protocol.mdwn
View file

@ -1,120 +0,0 @@
### Please describe the problem.
I'm storing lots of recordings from our satellite receiver into a git annex repo for easy syncing to a backup server that's not always switched on. Now, I'm trying to replace this backup server and pulling all data onto my new node. Most files work fine, but there are a bunch of small files that throw an error. In the debug log the GET command seems to use the file's size as offset, resulting in 0 bytes of data, throwing an error
### What steps will reproduce the problem?
$ git annex get Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.* --debug
[2021-12-19 17:13:10.308043464] process [4214] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
[2021-12-19 17:13:10.309545353] process [4214] done ExitSuccess
[2021-12-19 17:13:10.310048003] process [4215] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
[2021-12-19 17:13:10.312009938] process [4215] done ExitSuccess
[2021-12-19 17:13:10.328167499] process [4216] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--stage","-z","--","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.eit","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.ap","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.cuts","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.sc"]
[2021-12-19 17:13:10.328598306] process [4217] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2021-12-19 17:13:10.328861515] process [4218] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2021-12-19 17:13:10.329676569] process [4219] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
get Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta [2021-12-19 17:13:10.341285273] process [4221] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2021-12-19 17:13:10.342469002] process [4221] done ExitSuccess
[2021-12-19 17:13:10.342787111] process [4222] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2021-12-19 17:13:10.343851119] process [4222] done ExitSuccess
[2021-12-19 17:13:10.34416143] process [4223] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..33675b567939705c283bd955c2c7de13e15ba28a","--pretty=%H","-n1"]
[2021-12-19 17:13:10.345389634] process [4223] done ExitSuccess
[2021-12-19 17:13:10.345949274] process [4224] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2021-12-19 17:13:10.346209136] process [4225] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
(from garage...)
[2021-12-19 17:13:10.349782417] process [4226] read: ssh ["-o","BatchMode=true","-S",".git/annex/ssh/root@10.19.20.147","-o","ControlMaster=auto","-o","ControlPersist=yes","-n","-T","root@10.19.20.147","true"]
[2021-12-19 17:13:11.687712914] process [4226] done ExitSuccess
[2021-12-19 17:13:11.68875019] process [4231] chat: ssh ["root@10.19.20.147","-S",".git/annex/ssh/root@10.19.20.147","-o","ControlMaster=auto","-o","ControlPersist=yes","-T","git-annex-shell 'p2pstdio' '/opt/Sat' '--debug' 'b22315b4-77f0-4f46-aed3-fe376b350ab9' --uuid dd4c0b26-2331-4133-82bf-1888753c3268"]
[2021-12-19 17:13:12.407472022] [ssh connection Just 4231] [ThreadId 4] P2P < AUTH-SUCCESS dd4c0b26-2331-4133-82bf-1888753c3268
[2021-12-19 17:13:12.40779008] [ssh connection Just 4231] [ThreadId 4] P2P > VERSION 1
[2021-12-19 17:13:12.423477213] [ssh connection Just 4231] [ThreadId 4] P2P < VERSION 1
[2021-12-19 17:13:12.423831707] [ssh connection Just 4231] [ThreadId 4] P2P > GET 165 Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta
[2021-12-19 17:13:28.731615325] [ThreadId 4] P2P < GET 165 Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta
[2021-12-19 17:13:28.796144027] [ThreadId 4] P2P > DATA 0
[2021-12-19 17:13:28.796944343] [ThreadId 4] P2P > VALID
[2021-12-19 17:13:12.518895068] [ssh connection Just 4231] [ThreadId 4] P2P < DATA 0
[2021-12-19 17:13:12.521393464] [ssh connection Just 4231] [ThreadId 4] P2P < VALID
[2021-12-19 17:13:12.522008877] [ssh connection Just 4231] [ThreadId 4] P2P > FAILURE
transfer failed
[2021-12-19 17:13:12.524417003] process [4224] done ExitSuccess
[2021-12-19 17:13:12.525545594] process [4225] done ExitSuccess
[2021-12-19 17:13:12.526125714] [ssh connection Just 4231] [ThreadId 4] P2P > GET 165 Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta
[2021-12-19 17:13:28.807577749] [ThreadId 4] P2P < FAILURE
[2021-12-19 17:13:28.812468221] [ThreadId 4] P2P < GET 165 Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta
[2021-12-19 17:13:28.812746153] [ThreadId 4] P2P > DATA 0
[2021-12-19 17:13:28.812854143] [ThreadId 4] P2P > VALID
[2021-12-19 17:13:12.533471116] [ssh connection Just 4231] [ThreadId 4] P2P < DATA 0
[2021-12-19 17:13:12.534230317] [ssh connection Just 4231] [ThreadId 4] P2P < VALID
[2021-12-19 17:13:12.534505767] [ssh connection Just 4231] [ThreadId 4] P2P > FAILURE
transfer failed
Unable to access these remotes: garage
[2021-12-19 17:13:12.537799752] process [4235] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2021-12-19 17:13:12.538818173] process [4236] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2021-12-19 17:13:28.820206504] [ThreadId 4] P2P < FAILURE
No other repository is known to contain the file.
failed
[2021-12-19 17:13:12.55009435] process [4235] done ExitSuccess
[2021-12-19 17:13:12.550952683] process [4236] done ExitSuccess
[2021-12-19 17:13:12.551390026] process [4219] done ExitSuccess
[2021-12-19 17:13:12.551709257] process [4218] done ExitSuccess
[2021-12-19 17:13:12.551975888] process [4217] done ExitSuccess
[2021-12-19 17:13:12.552194866] process [4216] done ExitSuccess
[2021-12-19 17:13:12.561312091] process [4238] read: ssh ["-O","stop","-S","root@10.19.20.147","-o","ControlMaster=auto","-o","ControlPersist=yes","localhost"]
[2021-12-19 17:13:12.569198309] process [4238] done ExitSuccess
git-annex: get: 1 failed
As you can see the offset in the p2p get command is 165, which is exactly the file size of what it's supposed to be getting:
-r-xr-xr-x 1 root root 165 Apr 29 2020 .git/annex/objects/5V/x8/SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta/SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta
### What version of git-annex are you using? On what operating system?
On the old server:
$ git annex version
git-annex version: 8.20200908
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
On the new server:
$ git annex version
git-annex version: 8.20210223
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
Both servers are debian.
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I've been using git annex for years now, some periods more intensively than others. The only data I've lost I can only blame on my own stupidity, even after a terminal disk failure I managed to recover almost everything with a reinject. I'm still using it entirely cli though, no assistants or gui for me.
[[!meta title="corrupted tmp file can prevent getting an object"]]
> [[fixed|done]] (the related issue that I discovered in comment #1
> specifically) --[[Joey]]

34
doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_1_72a8b9b6ed582dad36301cb9beece2f8._comment
View file

@ -1,34 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2021-12-26T16:17:43Z"
content="""
It looks rather like the full content of the file may already have been
downloaded, in `.git/annex/tmp/` which is why it "resumes"
from the end of the file.
But this does not necessarily look like a protocol problem. The protocol
handles no more data being sent. The failure is on the receiving side,
after it's received the data it indicates an overall failure.
One reason this could be happening is if the file in `.git/annex/tmp/` is
corrupted, so verifying the completed download fails.
I tried two experiements. First, I populated `.git/annex/tmp/` with a copy of
the annexed object file from the other repository. Then `git-annex get`
succeeded, and looked this this:
[2021-12-26 11:25:32.364107322] (P2P.IO) [ThreadId 4] P2P < GET 30 foo SHA256E-s30--8559e95c5b77e5e577466959d83097039d0d3f2b6a7e18b2fd5df0188ef648c6
[2021-12-26 11:25:32.365992548] (P2P.IO) [ThreadId 4] P2P > DATA 0
[2021-12-26 11:25:32.366778465] (P2P.IO) [ThreadId 4] P2P > VALID
[2021-12-26 11:25:32.394964731] (P2P.IO) [ThreadId 4] P2P < SUCCESS
Next, I did the same, but corrupted the tmp file. With that I reproduced
your problem exactly.
So, I suggest you try deleting files in `.git/annex/tmp/` and get them again.
I think that git-annex should probably try to recover from this by,
when verification of a downloaded file fails, deleting the tmp file.
But that will need some more thought.
"""]]

15
doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_2_53970994816f1de9816e3496fdf0ef71._comment
View file

@ -1,15 +0,0 @@
[[!comment format=mdwn
username="tim@5431dd39464df207b7d46d3cf1bc74c82123ac68"
nickname="tim"
avatar="http://cdn.libravatar.org/avatar/967c2815f001d24b5f9f33300c03ba6f"
subject="Found the issue"
date="2022-01-03T08:25:05Z"
content="""
Hi Joey,
Thank you for having a look at this, but we can put this one under \"my own stupidity\". The files that give these errors are apparently only status files which get rewritten every time you play a recording, so they continuously get rewritten, get \"git annex add\"ed with a cron job, but don't always get pushed before they get overwritten, so yeah, it's not unexpected that some of these may get lost at some point.
Still no real data lost though, but errors like these worry me enough to file a bug report.
regards,
Tim
"""]]

9
doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_3_4c359c12d5c61351484dfcb2f5547222._comment
View file

@ -1,9 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2022-01-03T17:25:03Z"
content="""
Err, unless you have `annex.thin` set, once you `git-annex add` a file,
it should not be possible for it be overwritten in a way that loses the
annexed copy of it.
"""]]

9
doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_4_1e286c1506fc542df85c6158a8ec3d1c._comment
View file

@ -1,9 +0,0 @@
[[!comment format=mdwn
username="tim@5431dd39464df207b7d46d3cf1bc74c82123ac68"
nickname="tim"
avatar="http://cdn.libravatar.org/avatar/967c2815f001d24b5f9f33300c03ba6f"
subject="well, yes"
date="2022-01-04T11:28:33Z"
content="""
Well, yes, it does, if you start discarding disks after you've done a git annex pull on a temp usb repo and then again to the new server... Hence, my own stupidity.
"""]]

10
doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_5_c667e4d4899925a5484c94256044f6ee._comment
View file

@ -1,10 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2022-01-04T15:22:47Z"
content="""
Ah, I see. :)
I am going to leave this bug open, since the corrupted tmp file problem I
found is a real problem, though not actually your problem.
"""]]

24
doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_6_5ed6591954aeafe6c99ae152f4f4ad67._comment
View file

@ -1,24 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 6"""
date="2022-01-07T16:12:20Z"
content="""
Current thinking on deleting corrupted tmp files: If a download succeeds,
and verification then fails, the whole file content has been downloaded,
and is corrupt. So it would be ok to always delete it then, as far as p2p
transfers goes.
For other remotes, the same is often true. The only exceptions are like
rsync and bittorrent, which can recover from corruption on retry. But,
I don't think either rsync or bittorrent will usually write corrupt data
to a file anyway. They would catch over-the-wire corruption with rolling
checksums etc. So, it seems like a verification should never fail after
a successful rsync or bittorrent download. Unless the disk corrupted the
data in the meantime. Which is an unlikely situation, and not one that it's
really necessary for git-annex to recover from with optimal efficiency.
... Oh interesting.. It already is supposed to do that, in
getViaTmpFromDisk. It seems, what is happening is the transfer fails
when all the file content is present, and so it never gets to the point of
verifying it, let alone deleting it.
"""]]

13
doc/bugs/Failure_to_get_small_files_over_P2P_protocol/comment_7_247db3d1c03b207296e97975a3554756._comment
View file

@ -1,13 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 7"""
date="2022-01-07T16:34:42Z"
content="""
This seems to be a reversion caused by incremental verification. In
P2P.Annex it does incremental verification, and so it notices that the temp
file is corrupted. So, the transfer fails, and the temp file is left
behind.
So I suppose the fix is that, whenever doing incremental verification,
also delete the temp file when it fails.
"""]]

102
doc/bugs/Files_recorded_with_other_file__39__s_checksums.mdwn
View file

@ -1,102 +0,0 @@
### Please describe the problem.
I have a special remote "source" set up as type=directory importtree=yes.
I pulled it into one repo in which none of the files were wanted. So far so good. I cloned that repo to a second, archive, repo. git annex sync worked (but took 2 hours). Then I did `git annex get --auto`, most of the files came through OK. But some said things like this:
```
get Pictures/.dtrash/info/IMG_2979_v1-e9dced7b.dtrashinfo (from source...) (checksum...)
verification of content failed
Unable to access these remotes: source
No other repository is known to contain the file.
failed
```
(Both repos are unlocked via adjust --unlocked)
Upon looking at the file in the repo where it wasn't wanted, I saw this:
```
$ cat IMG_2979_v1-e9dced7b.dtrashinfo
/annex/objects/SHA256E-s144--cec5c7b6a9d97344e374e8395e02b74350678147ff65d6df091f5115cf18bf72
```
Interesting. So, in the source directory:
```
$ sha256sum IMG_2979_v1-e9dced7b.dtrashinfo
aca3ed7243def7a0bd5fcad542c66841b8e7d2a670b4cafe749eb27e032d8975 IMG_2979_v1-e9dced7b.dtrashinfo
```
That's not a match at all. Well, OK then:
```
$ sha256sum * | grep cec5c7
cec5c7b6a9d97344e374e8395e02b74350678147ff65d6df091f5115cf18bf72 IMG_2981_v1-5fc99c7a.dtrashinfo
```
Yikes. So for IMG_2979_v1-e9dced7b.dtrashinfo, git-annex recorded a checksum that belonged to IMG_2981_v1-5fc99c7a.dtrashinfo. Well then, what is this other file recorded as, back in the git-annex repo?
```
$ cat IMG_2981_v1-5fc99c7a.dtrashinfo
/annex/objects/SHA256E-s144--cec5c7b6a9d97344e374e8395e02b74350678147ff65d6df091f5115cf18bf72
```
OK, so two files that were not identical in the source directory got recorded with an identical checksum in git-annex somehow. And, when they were attempted to be imported via `git annex get --auto`, this at least was detected there.
In this .dtrash/info directory, 436 files out of 719 were not loaded by `git annex get`, presumably due to this issue.
In this directory, the source files were ranging in size from 140 to 227 bytes.
In a companion directory, .dtrash/files, 24 out of 719 files exhibited this issue. These files tended to be larger, but one that was 495MB triggered it also.
I have not yet seen it outside .dtrash, but it will be many hours until this get process completes fully, as it needs to copy about 1TB of data.
In case you are wondering if there is a race condition with .dtrash: no. The only application that writes to it isn't running, and the last time a file was modified in there was over a year ago. Also the content of the .info files is just JSON and a corresponding filename embedded in them, so it is very clear that the files on the filesystem are correct and the calculated checksums at issue here were never correct.
### What steps will reproduce the problem?
I have laid that out as best I can above.
### What version of git-annex are you using? On what operating system?
8;.20210223 on Debian
### Please provide any additional information below.
Assistant is not being used.
Setup:
```
REPO=Pictures
cd /acrypt/git-annex/repos
mkdir $REPO
cd $REPO
git init
git config annex.thin true
git annex init 'local hub'
git annex wanted . "include=* and exclude=$REPO/*"
# Now initialize things.
touch mtree
git annex add mtree
git annex sync
git annex adjust --unlock
git annex initremote source type=directory directory=/acrypt/git-annex/bind-ro/$REPO importtree=yes encryption=none
git annex enableremote source directory=/acrypt/git-annex/bind-ro/$REPO
git config remote.source.annex-readonly true
git config remote.source.annex-tracking-branch main:$REPO
git config annex.securehashesonly true
git config annex.genmetadata true
git config annex.diskreserve 100M
```
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
> [[done]] since this has pretty strongly been confirmed to be a bug that
> was already fixed in a newer git-annex version. --[[Joey]]

8
doc/bugs/Files_recorded_with_other_file__39__s_checksums/comment_1_05e3e33e1ca2361546dbe08c6bd476d6._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="jgoerzen"
avatar="http://cdn.libravatar.org/avatar/090740822c9dcdb39ffe506b890981b4"
subject="comment 1"
date="2022-09-05T01:14:53Z"
content="""
Update: This also occurred in other directories, with some video files from 2018. One directory contains 1945 files, and another 21 files. I'm not finding an obvious pattern to the issue.
"""]]

8
doc/bugs/Files_recorded_with_other_file__39__s_checksums/comment_2_0735ed4187e23ddea6bd0aa408451942._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="Lukey"
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
subject="comment 2"
date="2022-09-05T09:35:18Z"
content="""
You really should upgrade to the latest version.
"""]]

12
doc/bugs/Files_recorded_with_other_file__39__s_checksums/comment_3_06b7be9e651e528440a054ef43f7142f._comment
View file

@ -1,12 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 3"""
date="2022-09-05T17:48:49Z"
content="""
Concur that you need to upgrade. In particular, this seems likely to have
been fixed by [[!commit 3e2f1f73cbc5fc10475745b3c3133267bd1850a7]].
If so, the two files in the source directory will turn out to have the
same size and mtime, so were considered to have the same content identifier
when importing from a directory with your version of git-annex.
"""]]

8
doc/bugs/Files_recorded_with_other_file__39__s_checksums/comment_4_bc9cf721f8087cb91954cc2435376bc2._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="jgoerzen"
avatar="http://cdn.libravatar.org/avatar/090740822c9dcdb39ffe506b890981b4"
subject="comment 4"
date="2022-09-05T18:29:31Z"
content="""
Thanks! I noticed some changes in v10 around checksumming, so I got it started with v10 last night. It's still running, but I'll update you when it's done. Appreciate it!
"""]]

10
doc/bugs/Files_recorded_with_other_file__39__s_checksums/comment_5_8f0bbd819b95dfa97e2f4848e08a787a._comment
View file

@ -1,10 +0,0 @@
[[!comment format=mdwn
username="jgoerzen"
avatar="http://cdn.libravatar.org/avatar/090740822c9dcdb39ffe506b890981b4"
subject="comment 5"
date="2022-09-05T18:44:01Z"
content="""
Following up a bit - I had blown away the repos that were in use for this, but based on what I logged here, I can confirm that at least the files I mentioned with conflicts had the same size and mtime.
I may yet want to use ignoreinodes=yes. Eventually I want to be feeding git-annex from zfs clones to get a guaranteed stable source at any given moment, and I'm not really sure what that will do to inodes (it should, at minimum, have a different device number, I would assume).
"""]]

35
doc/bugs/Files_recorded_with_other_file__39__s_checksums/comment_6_17a0864cec6f28a0445998f3c5fc7f10._comment
View file

@ -1,35 +0,0 @@
[[!comment format=mdwn
username="jgoerzen"
avatar="http://cdn.libravatar.org/avatar/090740822c9dcdb39ffe506b890981b4"
subject="comment 6"
date="2022-09-05T22:01:24Z"
content="""
With the latest v10 git, I'm seeing some things like this:
```
get Pictures/0_New/5/20-03-31 21-12-12 0184.mp4 (from source...) file content has changed
Unable to access these remotes: source
No other repository is known to contain the file.
failed
```
Now, the file content really had not chnaged. That file looks like this:
```
$ cat 20-03-31\ 21-12-12\ 0184.mp4
/annex/objects/SHA256E-s55936559--0d418e4bb62cfc7ef5b053f8b622dd72794781a49931abc41bb9499acaf51b09.mp4
```
And on-disk, the file is:
```
$ sha256sum 20-03-31\ 21-12-12\ 0184.mp4
0d418e4bb62cfc7ef5b053f8b622dd72794781a49931abc41bb9499acaf51b09 20-03-31 21-12-12 0184.mp4
```
So, this is something different, but it's producing the same result (files left around not checked out). I obvserve that in .dtrash/info, the directory that had such a problem before, zero files now do. But in .dtrash/files, the problem seems to be triggered by a file that has a duplicate somewhere in the source repository (with the same sha256sum). I was able to find another file with the same sha256sum as that one in the same directory. In the origin repo, in which the given files were not present and it's using unlock-present, all instances of the file contained a broken symlink to the appropriate target. In the repo in question here, which uses plain unlock and is on NTFS (though Linux is the OS accessing it), I'm having this problem.
"""]]

15
doc/bugs/Files_recorded_with_other_file__39__s_checksums/comment_7_d480e85683f0418441f26c7d61c4af02._comment
View file

@ -1,15 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 7"""
date="2022-09-09T18:50:43Z"
content="""
Well, it seems that you have confirmed that the bug you were seeing is the
known and already fixed problem involving files that had the same size and
mtime.
I suggest filing a new bug report for the other problem, since it does not
seem like the same problem. I don't like to have to re-read long threads
about unrelated problems to tease out the "current" problem than a bug
report is talking about. 1 bug report per problem saves everyone time and
confusion.
"""]]

8
doc/bugs/Files_recorded_with_other_file__39__s_checksums/comment_8_80763c1c40acdf0915134182d15e4fd8._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="jgoerzen"
avatar="http://cdn.libravatar.org/avatar/090740822c9dcdb39ffe506b890981b4"
subject="comment 8"
date="2022-09-10T00:37:45Z"
content="""
Sure. Created the new bug at [[/bugs/Files_recorded_with_other_file__39__s_checksums/]]. Thanks Joey! I don't know how to close this one or I would.
"""]]

8
doc/bugs/Files_recorded_with_other_file__39__s_checksums/comment_9_7c3180616e40b49fc5ddf398d23d0201._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="jgoerzen"
avatar="http://cdn.libravatar.org/avatar/090740822c9dcdb39ffe506b890981b4"
subject="comment 9"
date="2022-09-10T00:38:17Z"
content="""
OK, so that link didn't work. I'll try <https://git-annex.branchable.com/bugs/Incorrect___34__file_content_has_changed__34___on_duplicates/>
"""]]

16
doc/bugs/GIN_is_not_recognized_as_a_git-annex_enabled_repos.mdwn
View file

@ -1,16 +0,0 @@
Hi, git-annex Team.
I want to ask you this issue.
GIN is not recognized as a git-annex enabled repository when using the HTTPS protocol.
I cloned a GIN repository with HTTPS protocol and then did `git annex init` on the cloned local repository, but it is not recognized as a git annex-enabled remote.
I checked the .git/config file and it says annex-ignore=true.
![image](https://user-images.githubusercontent.com/91708725/182578704-c04fdb60-5b6a-4faf-8869-04914723c27f.png)<br>
Because of this, git annex copy --to [GIN remote repository] is not possible with HTTPS protocol.
Is git-annex capable of copying content over the HTTPS protocol?
If there are any mistakes in my understanding please let me know.
sincerely yours,
Rino Mizuguchi
> [[notabug|done]] --[[Joey]]

42
doc/bugs/GIN_is_not_recognized_as_a_git-annex_enabled_repos/comment_1_8a27545e11b3d9e4e45aea4c9a5c9af4._comment
View file

@ -1,42 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 1"""
date="2022-08-04T16:22:09Z"
content="""
If you're using GIN, perhaps you're also using Datalad. Have you read
[their documentation](http://handbook.datalad.org/en/0.13_a/basics/101-139-gin.html)
on using it with git-annex?
Part of that documentation that seems likely to be especially
relevant is [here](http://handbook.datalad.org/en/0.13_a/basics/101-139-gin.html#access):
> Important: Take the URL in the browser, not the copy-paste URL
> Please note that you need to use the browser URL of the repository,
> not the copy-paste URL on the upper right hand side of the repository if you want
> to get anonymous HTTPS access! The two URLs differ only by a .git extension:
>
> Brower bar: `https://gin.g-node.org/<user>/<repo>`
>
> Copy-paste “HTTPS clone”: `https://gin.g-node.org/<user>/<repo>.git`
>
> A dataset cloned from `https://gin.g-node.org/<user>/<repoy>.git`, however, can not retrieve annexed files!
The reason is that git-annex needs to construct urls to download
files from the repo, starting with git's `config` file.
Looking at the sample repo <https://gin.g-node.org/adswa/DataLad-101>,
an url based on that works <https://gin.g-node.org/adswa/DataLad-101/config>,
but its web page suggests cloning the url
<https://gin.g-node.org/adswa/DataLad-101.git>, and
<https://gin.g-node.org/adswa/DataLad-101.git/config> does not work.
How does git manage to clone the latter url then? Well,
the http server is speaking the git smart http protocol.
So git makes requests like `https://gin.g-node.org/adswa/DataLad-101.git/info/refs?service=git-upload-pack`
which do work. But that protocol is not useful for git-annex, so it
needs the url where the actual files in the repository can be downloaded,
not the url where the smart http protocol is spoken.
There is not generally a way to discover one url from the other url
(removing ".git" might work in this case, but not in other cases).
So unfortunately you will need to remember to use the right url.
"""]]

75
doc/bugs/GIN_is_not_recognized_as_a_git-annex_enabled_repos/comment_2_3e2f01eaaf717710a08ba092856af193._comment
View file

@ -1,75 +0,0 @@
[[!comment format=mdwn
username="rinomizu5@5ead4c82685c65d7717dbd5591b80425036ae9e3"
nickname="rinomizu5"
avatar="http://cdn.libravatar.org/avatar/62478823018c68821064febcda7e5d4f"
subject="Additional Questions: git annex copy failed"
date="2022-08-08T10:31:04Z"
content="""
Thanks for the advice, joey.
I understand that the URL in the browser bar needs to be set to remote.
Then, after modifying the remote URL, I am trying to sync the contents of the git annex file to GIN using the https protocol.
But git annex copy command failed.
<pre>(base) <font color=\"#8AE234\"><b>mizuguchi@ubuntu</b></font>:<font color=\"#729FCF\"><b>~/https</b></font>$ git annex copy --to=origin test2.txt
copy test2.txt (not found) failed
copy: 1 failed
</pre>
I checked the origin remote has annex-uuid, and it's the same as uuid of [annex] in https://gin.g-node.org/ivis-mizuguchi/https/config.
I would like to know why git annex cannot find test2.txt.
### The version of the tool used in my environment is below.
- git version 2.24.0
- git-annex version: 8.20211123+git12-g02e3756bd-1~ndall+1
### The repository in GIN
[https://gin.g-node.org/ivis-mizuguchi/https](https://gin.g-node.org/ivis-mizuguchi/https)
### I have done the following process and failed.
1. Create a repository named [https](https://gin.g-node.org/ivis-mizuguchi/https) in GIN with initialization. And set the repository to public.
2. git clone https://gin.g-node.org/ivis-mizuguchi/https
3. cd https
4. git annex init
5. echo \"test\" > test.txt
6. git annex add test.txt
7. git commit -m \"test\"
8. git annex sync origin
9. echo \"test\" > test2.txt
10. git annex add 'test2.txt'
11. git commit -m 'test2'
12. git push origin master
13. git annex copy --to=origin test2.txt
<pre>(base) <font color=\"#8AE234\"><b>mizuguchi@ubuntu</b></font>:<font color=\"#729FCF\"><b>~/https</b></font>$ git annex copy --to=origin test2.txt
copy test2.txt (not found) failed
copy: 1 failed
</pre>
14. cat .git/config
<pre>(base) <font color=\"#8AE234\"><b>mizuguchi@ubuntu</b></font>:<font color=\"#729FCF\"><b>~/https</b></font>$ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote &quot;origin&quot;]
url = https://gin.g-node.org/ivis-mizuguchi/https
fetch = +refs/heads/*:refs/remotes/origin/*
annex-uuid = c7a50190-9146-44de-8f1d-288551fa32e3
[branch &quot;master&quot;]
remote = origin
merge = refs/heads/master
[annex]
uuid = 80f0adc3-3144-4bba-a288-09a9ab009cef
version = 8
[filter &quot;annex&quot;]
smudge = git-annex smudge -- %f
clean = git-annex smudge --clean -- %f
</pre>
Any advice would be appreciated.
Sincerely,
Rino Mizuguchi
"""]]

75
doc/bugs/GIN_is_not_recognized_as_a_git-annex_enabled_repos/comment_3_07460def4969896c87e8696955740fa9._comment
View file

@ -1,75 +0,0 @@
[[!comment format=mdwn
username="rinomizu5@5ead4c82685c65d7717dbd5591b80425036ae9e3"
nickname="rinomizu5"
avatar="http://cdn.libravatar.org/avatar/62478823018c68821064febcda7e5d4f"
subject="Additional Questions: git annex copy failed"
date="2022-08-08T10:31:26Z"
content="""
Thanks for the advice, joey.
I understand that the URL in the browser bar needs to be set to remote.
Then, after modifying the remote URL, I am trying to sync the contents of the git annex file to GIN using the https protocol.
But git annex copy command failed.
<pre>(base) <font color=\"#8AE234\"><b>mizuguchi@ubuntu</b></font>:<font color=\"#729FCF\"><b>~/https</b></font>$ git annex copy --to=origin test2.txt
copy test2.txt (not found) failed
copy: 1 failed
</pre>
I checked the origin remote has annex-uuid, and it's the same as uuid of [annex] in https://gin.g-node.org/ivis-mizuguchi/https/config.
I would like to know why git annex cannot find test2.txt.
### The version of the tool used in my environment is below.
- git version 2.24.0
- git-annex version: 8.20211123+git12-g02e3756bd-1~ndall+1
### The repository in GIN
[https://gin.g-node.org/ivis-mizuguchi/https](https://gin.g-node.org/ivis-mizuguchi/https)
### I have done the following process and failed.
1. Create a repository named [https](https://gin.g-node.org/ivis-mizuguchi/https) in GIN with initialization. And set the repository to public.
2. git clone https://gin.g-node.org/ivis-mizuguchi/https
3. cd https
4. git annex init
5. echo \"test\" > test.txt
6. git annex add test.txt
7. git commit -m \"test\"
8. git annex sync origin
9. echo \"test\" > test2.txt
10. git annex add 'test2.txt'
11. git commit -m 'test2'
12. git push origin master
13. git annex copy --to=origin test2.txt
<pre>(base) <font color=\"#8AE234\"><b>mizuguchi@ubuntu</b></font>:<font color=\"#729FCF\"><b>~/https</b></font>$ git annex copy --to=origin test2.txt
copy test2.txt (not found) failed
copy: 1 failed
</pre>
14. cat .git/config
<pre>(base) <font color=\"#8AE234\"><b>mizuguchi@ubuntu</b></font>:<font color=\"#729FCF\"><b>~/https</b></font>$ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote &quot;origin&quot;]
url = https://gin.g-node.org/ivis-mizuguchi/https
fetch = +refs/heads/*:refs/remotes/origin/*
annex-uuid = c7a50190-9146-44de-8f1d-288551fa32e3
[branch &quot;master&quot;]
remote = origin
merge = refs/heads/master
[annex]
uuid = 80f0adc3-3144-4bba-a288-09a9ab009cef
version = 8
[filter &quot;annex&quot;]
smudge = git-annex smudge -- %f
clean = git-annex smudge --clean -- %f
</pre>
Any advice would be appreciated.
Sincerely,
Rino Mizuguchi
"""]]

36
doc/bugs/GIN_is_not_recognized_as_a_git-annex_enabled_repos/comment_4_f97de485f97d8016a5ab4e276799d230._comment
View file

@ -1,36 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 4"""
date="2022-08-08T15:27:21Z"
content="""
Well, that is a different problem than the problem you reported, isn't it?
You seem to have shown that git-annex does in fact recognize your GIN
remote as a git-annex repo.
git-annex does not support uploading files to a git repo on GIN using http.
The datalad documentation says to add a remote using the ssh url of the GIN
repository, which git-annex can store files on. See
<http://handbook.datalad.org/en/0.13_a/basics/101-139-gin.html#publishing-your-dataset-to-gin>
Without using datalad, the equivilant commands would be something like:
git remote add gin git@gin.g-node.org:/ivis-mizuguchi/https.git
git-annex copy --to gin
---
So this leaves the question of why git-annex tries to do a http request
when you ask it to copy the content to the http repo. That http request
failing is what results in the "(not found)" message.
Looks like that is due to `git-annex copy` checking to see if the content
exists on the remote. It does not. So it ought to try to send it,
which would fail with a better "copying to non-ssh repo not supported" error message.
However, intstead it displays the error.
I've fixed this bug, and now it fails with this message:
copy test3.txt (to origin...)
copying to non-ssh repo not supported
"""]]

15
doc/bugs/GIN_is_not_recognized_as_a_git-annex_enabled_repos/comment_5_cf4130cb932d3d884e605d7c6447e61e._comment
View file

@ -1,15 +0,0 @@
[[!comment format=mdwn
username="rinomizu5@5ead4c82685c65d7717dbd5591b80425036ae9e3"
nickname="rinomizu5"
avatar="http://cdn.libravatar.org/avatar/62478823018c68821064febcda7e5d4f"
subject="comment 5"
date="2022-08-09T00:33:55Z"
content="""
Thank you joey for the additional help.
I see that I need to use the SSH protocol to upload the file contents to GIN with git annex.
And I also understand why it showed (not found).
I appreciate your advice.
RIno Mizuguchi
"""]]

109
doc/bugs/Git_repos_corrupt_themselves.mdwn
View file

@ -1,109 +0,0 @@
### Please describe the problem.
This has happened a few times now which is why I'm reporting it. Git-annex seems to do something that causes my git repo to randomly corrupt on their own which is very worrying because I need to trust these repos to keep their content safe.
It has never happened with regular repos and happens on both my Linux machine and my MBP. It hasn't happened on my NAS (SOTERIA) yet though which is exclusively accessed remotely, so I suspect this is an issue with the assistant/daemon.
### What steps will reproduce the problem?
Just have the assistant running, committing and syncing. No idea what other factors migh play into this.
### What version of git-annex are you using? On what operating system?
```
git-annex version: 8.20210428
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.28 DAV-1.3.4 feed-1.3.2.0 ghc-8.10.4 http-client-0.6.4.1 persistent-sqlite-2.11.1.0 torrent-10000.1.1 uuid-1.3.14 yesod-1.6.1.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
```
https://github.com/Atemu/nixpkgs/tree/498831397e77a265c240cf8f8a7d15e738f2f05b
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
[2021-05-09 17:55:13.653645489] Committer: Committing changes to git
(recording state in git...)
[2021-05-09 17:55:13.702843262] Pusher: Syncing with SOTERIA
To ssh://192.168.101.24/~/Annex/Documents.git/
041f52d19..40cc59126 master -> synced/master
git-annex: internal error: evacuate: strange closure type 4325399
(GHC version 8.10.4 for x86_64_unknown_linux)
Please report this as a GHC bug: https://www.haskell.org/ghc/reportabug
(recording state in git...)
(scanning...) ControlSocket .git/annex/ssh/atemu@192.168.101.24 already exists, disabling multiplexing
(started...)
(recording state in git...)
Unpacking all pack files.
remote: error: cannot lock ref 'refs/heads/synced/git-annex': is at 9166c83718695df3021d5bbf4fef2297d3f4cc84 but expected 7f761edbbe3c2a4710c629e9e7fdfb730c0639e7
remote: error: cannot lock ref 'refs/heads/synced/master': is at 42cab466f705c2496fb19249e272487bceb38808 but expected 40cc591261675e872f79b6a9ea966215d3f73581
To ssh://192.168.101.24/~/Annex/Documents.git/
! [remote rejected] git-annex -> synced/git-annex (failed to update ref)
! [remote rejected] master -> synced/master (failed to update ref)
error: failed to push some refs to 'ssh://192.168.101.24/~/Annex/Documents.git/'
To ssh://192.168.101.24/~/Annex/Documents.git/
7f761edbb..9166c8371 git-annex -> synced/git-annex
40cc59126..42cab466f master -> synced/master
fatal: early EOF
git-annex: internal error: evacuate: strange closure type 4325399
(GHC version 8.10.4 for x86_64_unknown_linux)
Please report this as a GHC bug: https://www.haskell.org/ghc/reportabug
(recording state in git...)
ControlSocket .git/annex/ssh/atemu@192.168.101.24 already exists, disabling multiplexing
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
error: Could not read 1f2ab777cdd4eb9add96d42de5022bde0a4a9a8b
error: Could not read 0f74453f84ff23d31b00a9bc9c331c00988f465a
error: Could not read 0f74453f84ff23d31b00a9bc9c331c00988f465a
error: Could not read 1f2ab777cdd4eb9add96d42de5022bde0a4a9a8b
error: Could not read 0f74453f84ff23d31b00a9bc9c331c00988f465a
error: Could not read 1f2ab777cdd4eb9add96d42de5022bde0a4a9a8b
error: Could not read 0f74453f84ff23d31b00a9bc9c331c00988f465a
error: Could not read 1f2ab777cdd4eb9add96d42de5022bde0a4a9a8b
error: Could not read 0f74453f84ff23d31b00a9bc9c331c00988f465a
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /tmp/tmpreporFb6Gq/.git/
Trying to recover missing objects from remote SOTERIA.
fatal: 'SOTERIA' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Trying to recover missing objects from remote SOTERIA.
fatal: 'SOTERIA' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
65 missing objects could not be recovered!
To force a recovery to a usable state, retry with the --force parameter.
# End of transcript or log.
"""]]
I'm not precisely sure when the actual crash happened (could be that it happened after starting up the machine today) as the log is a bit ambiguous. All I know is that it was corrupted today and that I was making commits till yesterday ~18:00.
The objects mentioned in the log are both 6 days old; one is a commit on master and one on git-annex.
Git fsck spat out a huge list of broken blobs and trees.
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
There are a surprising amount of weird bugs and quirks like these but it's such an amazing tool, there's nothing like it.
> [[fixed|done]] although I have opened [[assistant_repair_misfires]] for
> part of the cause of this. --[[Joey]]

601
doc/bugs/Git_repos_corrupt_themselves/comment_10_74e6d17a1b14b2fdc7eddf4ee1895733._comment
View file

@ -1,601 +0,0 @@
[[!comment format=mdwn
username="Atemu"
avatar="http://cdn.libravatar.org/avatar/d1f0f4275931c552403f4c6707bead7a"
subject="comment 10"
date="2021-05-25T11:00:39Z"
content="""
A new crash just happened. I haven't upgraded git-annex to master yet but the strange closure bug didn't occur this time. Annotated daemon.log:
```
(scanning...) (started...)
<lots of this>
ssh: connect to host 192.168.101.68 port 22: Connection timed out
<because the MBP is sleeping>
add file.etherpad ok <note: textconv is set up to pipe .etherpad through `jq .` to pretty-print it>
(recording state in git...)
<git annex fsck all the tree's files. I did not trigger this and automatic consistency checks are off!>
(recording state in git...)
(recording state in git...)
git-annex: fsck: 1 failed
(recovering from race...)
ssh: connect to host 192.168.101.68 port 22: Connection timed out
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
remote: error: cannot lock ref 'refs/heads/master': is at 6e737fee3f50ee203c9db151e3f9d0308c63df9b but expected fc38ba7550f8dd0d29262bc47ae3085403a55784 To ssh://192.168.101.24/~/Annex/Documents.git/ ! [remote rejected] master -> master (failed to update ref)error: failed to push some refs to 'ssh://192.168.101.24/~/Annex/Documents.git/'ssh: connect to host 192.168.101.68 port 22: No route to host
fatal: Could not read from remote repository.Please make sure you have the correct access rightsand the repository exists.ssh: connect to host 192.168.101.68 port 22: Connection timed out
fatal: Could not read from remote repository.Please make sure you have the correct access rightsand the repository exists.ssh: connect to host 192.168.101.68 port 22: Connection timed out
fatal: Could not read from remote repository.Please make sure you have the correct access rightsand the repository exists.ssh: connect to host 192.168.101.68 port 22: Connection timed out
fatal: Could not read from remote repository.Please make sure you have the correct access rightsand the repository exists.ssh: connect to host 192.168.101.68 port 22: No route to host
fatal: Could not read from remote repository.Please make sure you have the correct access rightsand the repository exists.ssh: connect to host 192.168.101.68 port 22: No route to host
fatal: Could not read from remote repository.Please make sure you have the correct access rightsand the repository exists. Transfer failed
Unpacking all pack files.
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /tmp/tmprepoZrGQTy/.git/
Trying to recover missing objects from remote MB-<REDACTED>.
fatal: 'MB-<REDACTED>' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Trying to recover missing objects from remote SOTERIA.
fatal: 'SOTERIA' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Trying to recover missing objects from remote MB-<REDACTED>.
fatal: 'MB-<REDACTED>' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Trying to recover missing objects from remote SOTERIA.
fatal: 'SOTERIA' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
2 missing objects could not be recovered!
To force a recovery to a usable state, retry with the --force parameter.
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
add <.txt render of the etherpad file> fatal: Invalid revision range refs/heads/master..refs/synced/d7d728f7-891a-4035-a758-c7ee80a8017a/master
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
ok
(recording state in git...)
error: refs/heads/master does not point to a valid object!
error: refs/heads/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
CONFLICT (add/add): Merge conflict in <unrelated but frequently edited file>.org
Auto-merging <file>.org
<a handful of conflicts like this>
Automatic merge failed; fix conflicts and then commit the result.
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: Merging is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
100% 3.33 KiB 198 MiB/s 0s
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
ssh: connect to host 192.168.101.68 port 22: Connection timed out
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: Merging is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: Merging is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
ssh: connect to host 192.168.101.68 port 22: No route to host
ssh: connect to host 192.168.101.68 port 22: Connection timed out
fatal: Could not read from remote repository.Please make sure you have the correct access rightsand the repository exists. Transfer failed
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
ssh: connect to host 192.168.101.68 port 22: Connection timed out
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
ssh: connect to host 192.168.101.68 port 22: Connection timed out
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
To ssh://192.168.101.24/~/Annex/Documents.git/
105d6d0..4bd9661 git-annex -> synced/git-annex
! [rejected] master -> synced/master (fetch first)
error: failed to push some refs to 'ssh://192.168.101.24/~/Annex/Documents.git/'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
ssh: connect to host 192.168.101.68 port 22: No route to host
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
error: refs/remotes/SOTERIA/master does not point to a valid object!
error: refs/remotes/SOTERIA/synced/master does not point to a valid object!
From ssh://192.168.101.24/~/Annex/Documents
* [new branch] master -> SOTERIA/master
* [new branch] synced/master -> SOTERIA/synced/master
Auto packing the repository in background for optimum performance.
See \"git help gc\" for manual housekeeping.
error: Merging is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.
error: Merging is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.
error: Merging is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.
error: Merging is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.
To ssh://192.168.101.24/~/Annex/Documents.git/
! [rejected] master -> synced/master (non-fast-forward)
error: failed to push some refs to 'ssh://192.168.101.24/~/Annex/Documents.git/'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. Check out this branch and integrate the remote changes
hint: (e.g. 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
ssh: connect to host 192.168.101.68 port 22: Connection timed out
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
ssh: connect to host 192.168.101.68 port 22: No route to host
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
To ssh://192.168.101.24/~/Annex/Documents.git/
8cd959c..4bd9661 git-annex -> refs/synced/8dc76e63-3dac-4cc3-a752-e73e81d57c1a/git-annex
! [rejected] master -> refs/synced/8dc76e63-3dac-4cc3-a752-e73e81d57c1a/master (non-fast-forward)
error: failed to push some refs to 'ssh://192.168.101.24/~/Annex/Documents.git/'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. Check out this branch and integrate the remote changes
hint: (e.g. 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
ssh: connect to host 192.168.101.68 port 22: Connection timed out
```
Git is now trying to merge refs/synced/d7d728f7-891a-4035-a758-c7ee80a8017a/master which points at a commit made on the MBP 4d ago. This commit has previously been **merged** into the original branch on this machine and contains a change to a txt render of a different .etherpad file. This also means that the commit before it contains a change to the according .etherpad file.
My branch is now a single commit made at 11:49:21 that adds the whole state of the working dir.
However, the diff for the .etherpad file says (without textconv):
```
diff --git a/Uni/LAWR/V10/132c6ef4e421c88547d4a228fef3c39491ba0610.etherpad b/Uni/LAWR/V10/132c6ef4e421c88547d4a228fef3c39491ba0610.etherpad
new file mode 100644
index 0000000..33e0a8f
--- /dev/null
+++ b/Uni/LAWR/V10/132c6ef4e421c88547d4a228fef3c39491ba0610.etherpad
@@ -0,0 +1 @@
+/annex/objects/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2
diff --git a/Uni/LAWR/V10/132c6ef4e421c88547d4a228fef3c39491ba0610.txt b/Uni/LAWR/V10/132c6ef4e421c88547d4a228fef3c39491ba0610.txt
new file mode 100644
index 0000000..f9ce6fe
--- /dev/null
+++ b/Uni/LAWR/V10/132c6ef4e421c88547d4a228fef3c39491ba0610.txt
@@ -0,0 +1 @@
+/annex/objects/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt
```
It added these as a unlocked largefiles despite my largefiles rule excluding them. This has happened before once I think but I manually committed at that time.
The other .etherpad files I had in the tree were not added as largefiles.
I remember having issues with textconv and annex before too and disabling a few rules that I didn't really need.
The git-annex branch seems to be in-tact. However, its commits are a bit weird. Here is `git log synced/git-annex..git-annex --patch`:
```
commit 4bd9661256a12d81186420a16d1bc1fb5f5d261a
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:49:22 2021 +0200
update (recovery from race #1; expected commit parent Ref \"5edabb3782060a334f966387e03252440ef50b10\" but found [Ref \"c518e6a6cac2b966e757a97dc8d26b64743be377\"] )
commit efdaa77dbf258954289366e236f056e50a258e1a
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:49:22 2021 +0200
update
commit c518e6a6cac2b966e757a97dc8d26b64743be377
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:49:22 2021 +0200
update (recovery from race #1; expected commit parent Ref \"5edabb3782060a334f966387e03252440ef50b10\" but found [Ref \"79a6d5f9b7b45ae891529a9bf2ead592f3968faa\"] )
commit fd7d4f7a6c875ca43cfe5e2e34b27c4cd6daab0a
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:49:22 2021 +0200
update
commit 79a6d5f9b7b45ae891529a9bf2ead592f3968faa
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:49:22 2021 +0200
update (recovery from race #1; expected commit parent Ref \"5edabb3782060a334f966387e03252440ef50b10\" but found [Ref \"79190c0161b13aae6570cb24f232ea6ef9288f66\"] )
commit 1276807833ad40c19190049a85ba82b5b171edd2
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:49:22 2021 +0200
update
commit 79190c0161b13aae6570cb24f232ea6ef9288f66
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:49:22 2021 +0200
update
diff --git a/672/674/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt.log b/672/674/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt.log
index 92b1312..0ff0604 100644
--- a/672/674/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt.log
+++ b/672/674/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt.log
@@ -1 +1,2 @@
1621936161.088002573s 1 8dc76e63-3dac-4cc3-a752-e73e81d57c1a
+1621936161.329141017s 1 b0512e28-91b3-4792-a4ef-71357fc4b63a
commit 5edabb3782060a334f966387e03252440ef50b10
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:49:21 2021 +0200
update
diff --git a/672/674/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt.log b/672/674/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt.log
new file mode 100644
index 0000000..92b1312
--- /dev/null
+++ b/672/674/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt.log
@@ -0,0 +1 @@
+1621936161.088002573s 1 8dc76e63-3dac-4cc3-a752-e73e81d57c1a
diff --git a/672/674/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt.log.met b/672/674/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt.log.met
new file mode 100644
index 0000000..9972f75
--- /dev/null
+++ b/672/674/SHA256E-s3406--d5168f8722e96719e2cf32d21760690930264aa8c88d76098477d3438fa247ab.txt.log.met
@@ -0,0 +1 @@
+1621936161.086396922s day +25 month +5 year +2021
commit 6dbb8cc706949f8ef1dbbbd1003dbe728b642f8a
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:45:51 2021 +0200
update (recovery from race #1; expected commit parent Ref \"1ccaa3ce85972818d9afee1ff047183d8836c927\" but found [Ref \"260477393b0e18fbf0d4fc77afa10ab0069f2d40\"] )
commit 1f035559a42966383b424ebb0dd62509f8259543
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:45:51 2021 +0200
update
commit 260477393b0e18fbf0d4fc77afa10ab0069f2d40
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:45:51 2021 +0200
update
diff --git a/activity.log b/activity.log
index e388b9e..890cae5 100644
--- a/activity.log
+++ b/activity.log
@@ -1,37 +1,8 @@
00b14f31-5c38-4eb3-b16f-17140b671470 Fsck timestamp=1610259369.931977s
1968d496-fca5-4b48-ac56-b4de73d6af82 Fsck timestamp=1609853522.83856s
-283a9cfc-cc28-481f-8ab3-28564f61e278 Fsck timestamp=1614085193.126619514s
-283a9cfc-cc28-481f-8ab3-28564f61e278 Fsck timestamp=1614155865.00518998s
283a9cfc-cc28-481f-8ab3-28564f61e278 Fsck timestamp=1614159798.683064383s
4ea64ba8-3d84-439d-88c9-021380c3b097 Fsck timestamp=1610235815.801036s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1614161439.951758036s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1614161875.368919604s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1614503753.902534237s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1618474275.133432744s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1619560801.449019468s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1620081126.285146124s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1620655938.401915298s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621159014.044988654s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621161156.147989006s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621161290.856706788s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621161317.85070715s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621161413.579088689s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621328885.881572492s
-8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621422002.80957147s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621935951.378910574s
a45a1abf-086e-40ea-8db5-f286a4fb69ef Fsck timestamp=1609849766.99771s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1614161148.72413s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1614162182.895703s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1614162687.460662s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1618351280.306775s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1619444550.542679s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1619733602.694413s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1619833893.739576s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1620079818.367555s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1620513005.967486s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1620775822.267693s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1621161025.720264s
-d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1621239795.218301s
d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1621444868.775013s
-f5f5f557-ecf2-4c2e-b704-add87173d58a Fsck timestamp=1612954464.455509s
-f5f5f557-ecf2-4c2e-b704-add87173d58a Fsck timestamp=1614085579.354101s
f5f5f557-ecf2-4c2e-b704-add87173d58a Fsck timestamp=1614160037.595461s
commit 1ccaa3ce85972818d9afee1ff047183d8836c927
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:45:51 2021 +0200
update
diff --git a/af5/b21/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2.log b/af5/b21/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2.log
index 0252166..3150f67 100644
--- a/af5/b21/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2.log
+++ b/af5/b21/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2.log
@@ -1 +1,2 @@
1621935950.772957836s 1 8dc76e63-3dac-4cc3-a752-e73e81d57c1a
+1621935951.025150823s 1 b0512e28-91b3-4792-a4ef-71357fc4b63a
commit 105d6d0ddea616d0f62f5bb78080954545966947
Author: Atemu <atemu.main@gmail.com>
Date: Tue May 25 11:45:50 2021 +0200
update
diff --git a/activity.log b/activity.log
index f925d0a..e388b9e 100644
--- a/activity.log
+++ b/activity.log
@@ -1,8 +1,37 @@
00b14f31-5c38-4eb3-b16f-17140b671470 Fsck timestamp=1610259369.931977s
1968d496-fca5-4b48-ac56-b4de73d6af82 Fsck timestamp=1609853522.83856s
+283a9cfc-cc28-481f-8ab3-28564f61e278 Fsck timestamp=1614085193.126619514s
+283a9cfc-cc28-481f-8ab3-28564f61e278 Fsck timestamp=1614155865.00518998s
283a9cfc-cc28-481f-8ab3-28564f61e278 Fsck timestamp=1614159798.683064383s
4ea64ba8-3d84-439d-88c9-021380c3b097 Fsck timestamp=1610235815.801036s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1614161439.951758036s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1614161875.368919604s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1614503753.902534237s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1618474275.133432744s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1619560801.449019468s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1620081126.285146124s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1620655938.401915298s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621159014.044988654s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621161156.147989006s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621161290.856706788s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621161317.85070715s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621161413.579088689s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621328885.881572492s
8dc76e63-3dac-4cc3-a752-e73e81d57c1a Fsck timestamp=1621422002.80957147s
a45a1abf-086e-40ea-8db5-f286a4fb69ef Fsck timestamp=1609849766.99771s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1614161148.72413s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1614162182.895703s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1614162687.460662s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1618351280.306775s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1619444550.542679s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1619733602.694413s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1619833893.739576s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1620079818.367555s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1620513005.967486s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1620775822.267693s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1621161025.720264s
+d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1621239795.218301s
d7d728f7-891a-4035-a758-c7ee80a8017a Fsck timestamp=1621444868.775013s
+f5f5f557-ecf2-4c2e-b704-add87173d58a Fsck timestamp=1612954464.455509s
+f5f5f557-ecf2-4c2e-b704-add87173d58a Fsck timestamp=1614085579.354101s
f5f5f557-ecf2-4c2e-b704-add87173d58a Fsck timestamp=1614160037.595461s
diff --git a/af5/b21/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2.log b/af5/b21/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2.log
new file mode 100644
index 0000000..0252166
--- /dev/null
+++ b/af5/b21/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2.log
@@ -0,0 +1 @@
+1621935950.772957836s 1 8dc76e63-3dac-4cc3-a752-e73e81d57c1a
diff --git a/af5/b21/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2.log.met b/af5/b21/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2.log.met
new file mode 100644
index 0000000..2aa827d
--- /dev/null
+++ b/af5/b21/SHA256E-s675354--3b69c1b0549173d731e6259d6c817eebabcd5b43a4e5cf6b83c7498603dae6d2.log.met
@@ -0,0 +1 @@
+1621935950.772381243s day +25 month +5 year +2021
diff --git a/efe/f7f/SHA256E-s2496630--0f778a1c32dfddf8098406ff28297d826b28759b4efeb1aa27a1841dd4897248.pdf.log b/efe/f7f/SHA256E-s2496630--0f778a1c32dfddf8098406ff28297d826b28759b4efeb1aa27a1841dd4897248.pdf.log
index 7fd0604..d907850 100644
--- a/efe/f7f/SHA256E-s2496630--0f778a1c32dfddf8098406ff28297d826b28759b4efeb1aa27a1841dd4897248.pdf.log
+++ b/efe/f7f/SHA256E-s2496630--0f778a1c32dfddf8098406ff28297d826b28759b4efeb1aa27a1841dd4897248.pdf.log
@@ -1,3 +1,4 @@
1621350596.596630026s 1 8dc76e63-3dac-4cc3-a752-e73e81d57c1a
+1621350602.99782945s 1 b0512e28-91b3-4792-a4ef-71357fc4b63a
1621350603.027097277s 1 b0512e28-91b3-4792-a4ef-71357fc4b63a
1621445067.293418s 1 d7d728f7-891a-4035-a758-c7ee80a8017a
diff --git a/efe/f7f/SHA256E-s2496630--0f778a1c32dfddf8098406ff28297d826b28759b4efeb1aa27a1841dd4897248.pdf.log.met b/efe/f7f/SHA256E-s2496630--0f778a1c32dfddf8098406ff28297d826b28759b4efeb1aa27a1841dd4897248.pdf.log.met
index b7f2ee1..2e12d4b 100644
--- a/efe/f7f/SHA256E-s2496630--0f778a1c32dfddf8098406ff28297d826b28759b4efeb1aa27a1841dd4897248.pdf.log.met
+++ b/efe/f7f/SHA256E-s2496630--0f778a1c32dfddf8098406ff28297d826b28759b4efeb1aa27a1841dd4897248.pdf.log.met
@@ -1 +1,2 @@
+1621350596.859331937s day +18 month +5 year +2021
1621445071.205197s day -18 +19 month +5 year +2021
diff --git a/schedule.log b/schedule.log
index 88f8251..f29a67c 100644
--- a/schedule.log
+++ b/schedule.log
@@ -1,7 +1,16 @@
00b14f31-5c38-4eb3-b16f-17140b671470 fsck self 1h every day at any time timestamp=1610259355.086542s
1968d496-fca5-4b48-ac56-b4de73d6af82 fsck self 1h every day at any time timestamp=1609853466.30146s
+283a9cfc-cc28-481f-8ab3-28564f61e278 fsck self 1h every day at any time timestamp=1609829311.089081351s
+283a9cfc-cc28-481f-8ab3-28564f61e278 fsck self 1h every day at any time timestamp=1609829331.993124414s
283a9cfc-cc28-481f-8ab3-28564f61e278 fsck self 1h every day at any time timestamp=1609831717.238587482s
+283a9cfc-cc28-481f-8ab3-28564f61e278 fsck self 8d13h every day at any time timestamp=1609829516.974129064s
4ea64ba8-3d84-439d-88c9-021380c3b097 timestamp=1610030955.40944s
+4ea64ba8-3d84-439d-88c9-021380c3b097 fsck self 1h every day at any time timestamp=1609857462.383747s
8dc76e63-3dac-4cc3-a752-e73e81d57c1a timestamp=1621328896.270884603s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a fsck self 1h every day at any time timestamp=1614161325.360455592s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a fsck self 1h every day at any time timestamp=1616330220.055006692s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a fsck self 1h every day at any time timestamp=1620642662.423064041s
+8dc76e63-3dac-4cc3-a752-e73e81d57c1a fsck self 1h every day at any time; fsck self 10h1m every day at any time timestamp=1616329120.414652786s
d7d728f7-891a-4035-a758-c7ee80a8017a timestamp=1621444873.058565s
+d7d728f7-891a-4035-a758-c7ee80a8017a fsck self 1h every day at any time timestamp=1616364223.127229s
f5f5f557-ecf2-4c2e-b704-add87173d58a fsck self 1h every day at any time timestamp=1610259937.483933s
```
synced/master is in the same state as it was 4d ago, the date of the merge commit git is now trying to merge.
There have only been smallfile changes made to master since.
I'm especially confused by the changes to schedule.log.
***
Looks like there are some major incompatibilities with textconv and git-annex that need to be documented and/or fixed.
Smallfiles don't seem to be handled that well either, perhaps my other issues are related to using those almost exclusively in this repo?
Automatic merges have never worked for me for example. (Though I probably wouldn't want them anyways).
Is anyone else having issues with this combination of features? I can't be the only one, right?
"""]]

8
doc/bugs/Git_repos_corrupt_themselves/comment_11_d1f454482feef0350a6fa61bdf7673cc._comment
View file

@ -1,8 +0,0 @@
[[!comment format=mdwn
username="Atemu"
avatar="http://cdn.libravatar.org/avatar/d1f0f4275931c552403f4c6707bead7a"
subject="comment 11"
date="2021-05-31T19:07:31Z"
content="""
The consistency check somehow enabled itself the other day and sure enough, as soon as I started the webapp, the repo corrupted itself on the MBP. Haven't seen a corruption on the Linux machine since I disabled consistency checks.
"""]]

21
doc/bugs/Git_repos_corrupt_themselves/comment_12_5a13945d6bec6fa5fac84555f7ef8e21._comment
View file

@ -1,21 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 12"""
date="2021-06-28T18:04:26Z"
content="""
This seems conclusive that the repair is somehow triggering unncessarily
and also corrupting the repo in this situation.
The comment #3 log shows that the repair is started, and then 1 minute
later a git object is missing.
(It's odd that log shows a second fsck run after the repair was already
triggered. I do not see a way that this would happen unless fscks are
scheduled very close together.)
The automatic repair is supposed to be a non-destructive repair; the
destructive repair only happens after prompting in the UI.
This also reminds me of a persistent issue with a git-annex repo, using the
assistant, on my sister's laptop corrupting itself.
"""]]

20
doc/bugs/Git_repos_corrupt_themselves/comment_13_6715bc3370f115f90627cfc4b06cf0e8._comment
View file

@ -1,20 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 13"""
date="2021-06-28T18:13:40Z"
content="""
The repair process moves all pack files to a temp dir and then unpacks the
loose objects from them. So, there is a time window, when the repair is
running, where git objects that were present before will be missing. And if
the assistant stops before that is complete, it would leave it in that
state. Unpacking pack files can take a long time, so this might
be a sufficient explanation.
But then, something must be causing it to incorrectly think it needs
a repair in the first place. Assuming it is incorrect, of course. Either git
fsck is exiting nonzero for some reason, or git-annex is thinking
it sees git fsck complain about a missing object, that is not really
missing. While there are fsck outputs that it can misinterpret, it
double-checks by trying to cat the object, which should avoid the latter
problem.
"""]]

11
doc/bugs/Git_repos_corrupt_themselves/comment_14_6234b04455acc0b2b2f22a1d8c2ab475._comment
View file

@ -1,11 +0,0 @@
[[!comment format=mdwn
username="joey"
subject="""comment 14"""
date="2021-06-28T18:28:06Z"
content="""
To avoid moving the pack files, repair could set `GIT_OBJECT_DIRECTORY`
to a temp directory, and copy each pack file into it in turn, and unpack.
And after each unpack, move the unpacked objects from the temp directory to
the real object directory, followed by deleting the pack file (in case it's
corrupt).
"""]]

Some files were not shown because too many files have changed in this diff Show more