mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2021-07-15 21:02:29 -04:00
commit a602554ed8
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
3 changed files with 75 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

51
doc/bugs/Assistant_crashes_with_handle_issues_-_never_works.mdwn Normal file
View file

@ -0,0 +1,51 @@
### Please describe the problem.
I have tried to use the assistant as per archival walkthrough. I have
1. Created a client drive locally
2. Created a small archive on an attached drive
3. Created and archive folder on the user drive
4. Moved files into the user drive and then into the archive
I very soon saw the assistant fail with errors all referencing `fd:50: hPutStr: illegal operation (handle is closed)`.
As a result, transactions and syncs were often incomplete and I had to use `git` and `git annex` commands to fix things up. The behaviour I had expected based on the documentation - automated archival of files moved to the `archive` directory - **never** worked.
I upgraded to the PPA that is offering the latest builds but have seen no change.
I ran the built-in self-test and saw 74 failed tests.
```sh
warning
NetWatcherFallback crashed: fd:50: hPutStr: illegal operation (handle is closed)
×
warning
Committer crashed: fd:50: hPutStr: illegal operation (handle is closed)
×
warning
PushRetrier crashed: fd:50: hPutStr: illegal operation (handle is closed)
```
### What steps will reproduce the problem?
I do not have the time to produce a repeatable test suite.
### What version of git-annex are you using? On what operating system?
Latest from PPA on Ubuntu 20.04 LTS current to-the-day.
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
No. But I *was* very enthusiastic about the theory behind it.

25
doc/todo/Incremental_git_annex_sync_--content_--all.mdwn
View file

@ -1,15 +1,18 @@
Hi,<br>
So I have yet another idea to speed up git annex. For now only for the 2nd pass of git annex sync --content --all.
1. Do a normal (full) git annex sync. For every remote that we synced with, record the commit id of the current tip of the git-annex branch.
* Record the commit id only if --content --all was specified
* Record the commit id only if the remote is actually available and every file was sucessfully transfered
2. If any of the remotes doesn't have a commit id recorded, go to 1. Else do a incremental git annex sync: In the 2nd pass of git annex sync --content --all,
only look at keys whose location log changed since the last (full or incremental) sync via `git diff-tree -r --name-only <lowest recorded commit id of all remotes> git-annex`.
Again, update the commit id of remotes that we sucessfully synced with.
3. If one of the following happens, remove all recorded commit ids of all remotes, go to 1. Else go to 2.
* The preferred content expression of us or one of our remotes changed.
* The preferred content expression of a group changed
* The group of any repo (not only remotes) changed. This way remotes containing `copies=<group>:<numcopies>` recheck all keys.
1. Check which remotes are currently available (i.e. online and connected).
2. If any of the (available) remotes doesn't have a commit recorded (See below), do a full sync.
3. If numcopies.log, mincopies.log, trust.log, group.log, preferred-content.log, required-content.log, group-preferred-content.log or transitions.log (on the git-annex branch) changed since the last time we synced, do a full sync (since if any of those logs changed, it may affect the preferred-content expressions and we need to reevaluate every key).
4. If every check passed, do an incremental sync.
This should be pretty reliable, but please double check. It has to be reliable enough to become the default.
##Full sync:
1. Do a normal (full) git annex sync.
2. For every remote that we synced with, record the commit id of the current tip of the git-annex branch.
* Record the commit id only if every file was successfully transferred/dropped.
##Incremental sync:
1. In the 2nd pass of git annex sync --content --all, only look at keys whose location log changed since the last (full or incremental) sync via `git diff-tree -r --name-only <lowest recorded commit id of all remotes> git-annex`.
2. Again, update the commit id of remotes that we successfully synced with.

10
doc/todo/Incremental_git_annex_sync_--content_--all/comment_2_9fff9eb8a7a897c5aabe9878f4d0b23b._comment Normal file
View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="Lukey"
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
subject="comment 2"
date="2021-07-15T18:40:13Z"
content="""
The script needed a few tweaks, you can find the updated version [here](https://gist.github.com/Lukey3332/203ea6f30d48323e7bd1d05c16b5da9c).
I used the script here since half a year now and everything works fine. I even had a problem with failed transfers due to a bad SATA-cable and the script did handle it properly. `git annex fsck --all` confirms that everything is alright.
"""]]