mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2021-01-18 12:18:23 -04:00
commit a9bf2a6cad
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
11 changed files with 291 additions and 18 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

49
doc/bugs/Directory_remotes_with_same_mount_point.mdwn Normal file
View file

@ -0,0 +1,49 @@
### Please describe the problem.
I have a repository with two directory special remotes that use the same mount point (it's a USB SD card reader using udevil). I use "git-annex-command" with lsblk and grep using the UUID to prevent git-annex from attempting to sync with the remote if it's not mounted. When trying to import from the second remote, I get "file content has changed" for every file, which seems to be related to inode-based caching.
Each of the remotes uses a subdirectory of the mount, though I did change the directory for the first one (it originally used the mount point directly).
### What steps will reproduce the problem?
1. Create a directory special remote on a removable device and import from it
2. Create a second directory special remote on a second removable device at the same mount point and import from it
### What version of git-annex are you using? On what operating system?
8.20201130-g75988a790, Debian Buster
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ git annex import --from=pi1541 master:c64
list pi1541 ok
import pi1541 nav96.d64
file content has changed
ok
import pi1541 fb64
file content has changed
ok
import pi1541 cbmcmd22.d64
file content has changed
ok
import pi1541 cbmcmd22.d80
file content has changed
ok
...
# End of transcript or log.
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I've used it successfully on Linux and Android. My attempts on Windows have been less successful, though I never tried with core.ignoreStat=true. Now I'm trying to use it to manage the SD cards for various devices: my Pi1541, Odroid Go, and Sansa Clip Zip.

9
doc/bugs/Directory_remotes_with_same_mount_point/comment_1_351063bf8fb7779a9a7ecd21bef8b801._comment Normal file
View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="seanl@fe5df935169a5440a52bdbfc5fece85cdd002d68"
nickname="seanl"
avatar="http://cdn.libravatar.org/avatar/082ffc523e71e18c45395e6115b3b373"
subject="Works fine after changing mount point"
date="2021-01-17T07:24:38Z"
content="""
I uninstalled udevil in favor of udiskie, which uses a mount point name based on the label instead of the device, updated the directory to point at the new mount point with `git annex enableremote pi1541 <new mount point>/1541`, and the problem went away. But it doesn't seem like having different mount points for removable devices should be a requirement.
"""]]

17
doc/bugs/W__58__prop__95__relPathDirToFileAbs__95__basics_started_to_fail.mdwn Normal file
View file

@ -0,0 +1,17 @@
### Please describe the problem.
Recently, with 8.20201129+git100-g2d84bf992 version of git-annex and whatever states of datalad, we have celebrated a Windows Green Day -- our CI was totally green for Windows.
But then Windows started to fail again. Currently (8.20201129+git125-g7f72314bf) due to
```
2021-01-18T14:42:32.7075079Z prop_relPathDirToFileAbs_basics: FAIL
2021-01-18T14:42:32.7078493Z *** Failed! Falsified (after 54 tests):
2021-01-18T14:42:32.7082732Z TestableFilePath {fromTestableFilePath = "/*LFy?\ETBoj(96"}
2021-01-18T14:42:32.7084569Z Use --quickcheck-replay=725893 to reproduce.
```
full list of recent runs with logs etc: [datalad/git-annex github actions](https://github.com/datalad/git-annex/actions?query=workflow%3A%22Build+git-annex+on+Windows%22). Note that `git annex version` reported is "off": [just filed an issue](https://github.com/datalad/git-annex/issues/44)
[[!meta author=yoh]]
[[!tag projects/datalad]]

142
doc/bugs/__91__PATCH__93___incorrect_behaviour_in_expandTilde.mdwn Normal file
View file

@ -0,0 +1,142 @@
### Please describe the problem.
git-annex has issues when trying to deal with SSH (and possibly other kinds) of URLs which have the form:
```
ssh://user@host/~
```
When git-annex tries to perform tilde-expansion the path part of the URL on the remote side,
it runs into problems because the function responsible for doing this (`expandTilde` in `Git/Construct.hs`)
does not correctly handle the expansion of home directory paths which do not end in a slash,
such as `~` or `/~`. It will correctly handle strings like `/~/` or `~/`, which is why SSH
URLs of the form `ssh://user@host/~/` *will* work.
Examining the definition of `expandTilde` makes it clear why this is true:
```haskell
expandTilde :: FilePath -> IO FilePath
#ifdef mingw32_HOST_OS
expandTilde = return
#else
expandTilde = expandt True
where
expandt _ [] = return ""
expandt _ ('/':cs) = do
v <- expandt True cs
return ('/':v)
expandt True ('~':'/':cs) = do
h <- myHomeDir
return $ h </> cs
expandt True ('~':cs) = do
let (name, rest) = findname "" cs
u <- getUserEntryForName name
return $ homeDirectory u </> rest
expandt _ (c:cs) = do
v <- expandt False cs
return (c:v)
findname n [] = (n, "")
findname n (c:cs)
| c == '/' = (n, cs)
| otherwise = findname (n++[c]) cs
```
The expression `expandTilde "~"` will eventually match the fourth pattern for `expandt`.
Since `cs == ""` in this context, `name` will also evaluate to `""`.
This means that `getUserEntryForName` will be called with the null string as an argument.
Since there is no user on the system with the null string as a username,
`getUserEntryForName` will throw an exception.
This will cause git-annex to spit out an error message:
```
get testfile (from origin...)
git-annex-shell: getUserEntryForName: does not exist (no such user)
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [Receiver=3.1.3]
rsync failed -- run git annex again to resume file transfer
Unable to access these remotes: origin
Try making some of these repositories available:
1f5118ff-a50e-4bf1-a372-960774bce0ab -- user@A:~/ [origin]
failed
git-annex: get: 1 failed
```
Fixing the problem is simple enough.
All that needs to be done is to add an equation for `expandt` to handle the case where `~` appears at the end of a string.
See the following patch:
```
From 680873923197f5eec15365b3e47e3fa05b9573be Mon Sep 17 00:00:00 2001
From: Grond <grond66@riseup.net>
Date: Thu, 14 Jan 2021 18:16:31 -0800
Subject: [PATCH] Fix expandTilde so that it can handle tildes at the end of
it's input
---
Git/Construct.hs | 1 +
1 file changed, 1 insertion(+)
diff --git a/Git/Construct.hs b/Git/Construct.hs
index 8b63ac480..a369bc4a6 100644
--- a/Git/Construct.hs
+++ b/Git/Construct.hs
@@ -187,6 +187,7 @@ expandTilde = expandt True
expandt True ('~':'/':cs) = do
h <- myHomeDir
return $ h </> cs
+ expandt True "~" = myHomeDir
expandt True ('~':cs) = do
let (name, rest) = findname "" cs
u <- getUserEntryForName name
--
2.20.1
```
### What steps will reproduce the problem?
1. Create `testfile` in a git-annex repo of your home directory on host `A`
2. Run `git annex add testfile` in the repo on `A`
3. Run `git commit`
4. Clone your home directory on `A` onto host `B` using `git clone ssh://me@A/~ homedir_A`
5. `cd` into `homedir_A`
6. Run `git annex get testfile`
7. Watch git-annex fail to fetch the file
8. Run `git remote set-url origin ssh://me@A/~/` to set the remote URL to be something git-annex can deal with
9. Run `git annex get testfile` again
10. Watch git-annex suddenly succeed
### What version of git-annex are you using? On what operating system?
I'm running Debian 10.7.
The output of `git annex version` is:
```
git-annex version: 7.20190129
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
operating system: linux x86_64
supported repository versions: 5 7
upgrade supported from repository versions: 0 1 2 3 4 5 6
local repository version: 5
```
### Please provide any additional information below.
[[!format sh """
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Definitely! I'm currently writing some personal file synchronization software that uses git-annex for myself, which is how I noticed this bug.

26
doc/bugs/git-annex_get__58___createDirectory__58___does_not_exist/comment_4_301ac4a37a0f2da6b9c394509a369484._comment Normal file
View file

@ -0,0 +1,26 @@
[[!comment format=mdwn
username="mih"
avatar="http://cdn.libravatar.org/avatar/f881df265a423e4f24eff27c623148fd"
subject="comment 4"
date="2021-01-15T08:32:58Z"
content="""
> Is the relative path actually valid?
As far as I can tell, it is.
```
(venv3.8.6) worker-629-003:datalad_temp_test_basic_scenariocm66i4fm appveyor$ ls -id ../../../../../../../var/folders/5s/g225f6nd6jl4g8tshbh1ltk40000gn/T/datalad_temp_tree_test_basic_scenariodi3ady04/.git/annex /var/folders/5s/g225f6nd6jl4g8tshbh1ltk40000gn/T/datalad_temp_tree_test_basic_scenariodi3ady04/.git/annex
12889649026 ../../../../../../../var/folders/5s/g225f6nd6jl4g8tshbh1ltk40000gn/T/datalad_temp_tree_test_basic_scenariodi3ady04/.git/annex
12889649026 /var/folders/5s/g225f6nd6jl4g8tshbh1ltk40000gn/T/datalad_temp_tree_test_basic_scenariodi3ady04/.git/annex
```
I cannot come up with an explanation, for the consistent behavior of `ls`, but not `mkdir` for a relative path vs. an absolute path.
```
(venv3.8.6) worker-629-003:datalad_temp_test_basic_scenariocm66i4fm appveyor$ mkdir ../../../../../../../var/folders/5s/g225f6nd6jl4g8tshbh1ltk40000gn/T/datalad_temp_tree_test_basic_scenariodi3ady04/.git/annex/testdummy
mkdir: ../../../../../../../var/folders/5s/g225f6nd6jl4g8tshbh1ltk40000gn/T/datalad_temp_tree_test_basic_scenariodi3ady04/.git/annex: No such file or directory
(venv3.8.6) worker-629-003:datalad_temp_test_basic_scenariocm66i4fm appveyor$ mkdir /var/folders/5s/g225f6nd6jl4g8tshbh1ltk40000gn/T/datalad_temp_tree_test_basic_scenariodi3ady04/.git/annex/testdummy
-> exit 0
```
"""]]

15
doc/bugs/git_annex_doesn__39__t_cope_with_a_gcrypt__58____58__rsync__58____47____47__.mdwn Normal file
View file

@ -0,0 +1,15 @@
### Please describe the problem.
After adding a `gcrypt::rsync://` remote, `git-annex-sync` and `git-annex-info` crash with:
git-annex: bad url rsync://192.168.178.241:test
CallStack (from HasCallStack):
error, called at ./Git/Construct.hs:107:15 in main:Git.Construct
### What steps will reproduce the problem?
git init
git annex init test
git remote add test gcrypt::rsync://192.168.178.241:test
git annex sync
### What version of git-annex are you using? On what operating system?
git-annex version: 8.20201127

8
doc/bugs/git_annex_doesn__39__t_cope_with_a_gcrypt__58____58__rsync__58____47____47__/comment_1_f895b93145cc4d86042af14b6f2ab6ac._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Lukey"
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
subject="comment 1"
date="2021-01-18T14:11:44Z"
content="""
I want to use such a remote just for storing the git repo, no need to support it as a (special-)remote for storing annexed files. But full support would be cool too.
"""]]

8
doc/bugs/git_annex_doesn__39__t_cope_with_a_gcrypt__58____58__rsync__58____47____47__/comment_2_5e7bf08ebcb97bcce5589f18898bb1ff._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Lukey"
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
subject="comment 2"
date="2021-01-18T14:31:06Z"
content="""
git-annex crashes with `gcrypt::sftp://` too. And probably with `gcrypt::rclone://`, but I didn't test.
"""]]

9
doc/git-annex-uninit/comment_3_2cf58739c14b8aae6fc12fb1463a303f._comment
View file

@ -1,9 +0,0 @@
[[!comment format=mdwn
username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234"
nickname="eric.w"
avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc"
subject="git-annex uninit --fast"
date="2021-01-11T22:41:04Z"
content="""
are there any caveats to using --fast with this command? I assume it will just skip the hash validation.
"""]]

9
doc/git-annex-uninit/comment_4_c3df10cdd19a15a80d3aa95373f1d071._comment
View file

@ -1,9 +0,0 @@
[[!comment format=mdwn
username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234"
nickname="eric.w"
avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc"
subject="comment 4"
date="2021-01-11T22:42:27Z"
content="""
actually --fast seems to have no affect on the speed. (each file seems to be getting hashed)
"""]]

17
doc/git-annex-uninit/comment_5_53358bdd6093e2fb787df8de7190c5fd._comment Normal file
View file

@ -0,0 +1,17 @@
[[!comment format=mdwn
username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234"
nickname="eric.w"
avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc"
subject="comment 5"
date="2021-01-14T22:15:12Z"
content="""
A faster way of doing uninit is the following:
```cp --no-clobber --dereference --recursive --preserve=all --reflink=auto --verbose ./git_annex_repo/your_symlinks/ ./target_dir/```
This will simply copy (thin COW copy) symlinks (dereferenced) as normal files preserving the mtime, etc. the resulting ./target_dir/ will have your files if they existed in this annex or broken symlinks if the files were not here.
"""]]