Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2022-09-26 12:07:46 -04:00
commit aff5e4376b
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
9 changed files with 445 additions and 22 deletions

View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="asakurareiko@f3d908c71c009580228b264f63f21c7274df7476"
nickname="asakurareiko"
avatar="http://cdn.libravatar.org/avatar/a865743e357add9d15081840179ce082"
subject="comment 14"
date="2022-09-25T19:43:31Z"
content="""
Either of these changes causes the test case in comment 11 to hang indefinitely. Probably no way to fix these issues in general due to [[sqlite write locks aren't respected in WSL|https://github.com/microsoft/WSL/issues/2395]], but as things are currently everything works well enough.
"""]]

View file

@ -0,0 +1,80 @@
### Please describe the problem.
attempting to the use feature which allows me to connect to a friends remote repository results in failure, in which i have tested using multiple methods
### What steps will reproduce the problem?
#### WEBAPP
* open the webapp
* add repository
* choose "share with a friend"
* click "lets get started"
* encounter the error found in the transcript below
#### COMMAND LINE
##### enable tor
* git annex enable tor
* encounter error found in the transcript below
##### p2p --pair
* git annex p2p --pair
* encounter the error found in the transcript
### What version of git-annex are you using? On what operating system?
### Please provide any additional information below.
### external dependency version info
* git annex: using git annex found in nixos repositories
* python version: attempted both python3 and python2.7
* magic-wormhole version: attempted both 0.12.0 under python3, and version 0.10.0 under python2.7
* tor version information: Tor version 0.4.7.10.
Tor is running on Linux with Libevent 2.1.12-stable, OpenSSL 1.1.1q, Zlib 1.2.12, Liblzma 5.2.5, Libzstd 1.5.2 and Glibc 2.34 as libc.
Tor compiled with GCC version 11.3.0
### transcript
[[!format sh """
#### WEBAPP
##### error output after following steps as described above
Failed to enable tor
enable-tor
You may be prompted for a password
git-annex: This can only be run in a git-annex repository.
git-annex: Failed to run as root: /nix/store/c42x83x25df9xd053xiii24mawnrlvrd-git-annex-10.20220504/bin/git-annex enable-tor 1000
failed
enable-tor: 1 failed
#### COMMAND LINE
##### enable tor
git annex enable-tor
Failed to enable tor
enable-tor
You may be prompted for a password
git-annex: This can only be run in a git-annex repository.
git-annex: Failed to run as root: /nix/store/c42x83x25df9xd053xiii24mawnrlvrd-git-annex-10.20220504/bin/git-annex enable-tor 1000
failed
enable-tor: 1 failed
##### git annex p2p --pair
git annex p2p --pair
p2p pair peer1 (using Magic Wormhole)
Failed sending data to pair.
failed
p2p: 1 failed
# End of transcript or log.
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
git annex has been awesome so far in helping track and backup my data, also for allowing me to output podcasts/audiobooks to a small mp3 player, as well as ebooks and pdfs to a kindle paper ewhite, which has made using my ereader no longer an inconvinient endeavour

View file

@ -18,3 +18,5 @@ Through bisection, the problem was found to be introduced in [[!commit 428c91606
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
git-annex has worked wonderfully for managing my files across different machines and cloud storage services.
[[done]]

View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="asakurareiko@f3d908c71c009580228b264f63f21c7274df7476"
nickname="asakurareiko"
avatar="http://cdn.libravatar.org/avatar/a865743e357add9d15081840179ce082"
subject="comment 3"
date="2022-09-24T19:43:47Z"
content="""
This seems to be fixed now, maybe due to the changes made related to [[todo/git_status_smudges_unncessarily_after_unlock]].
"""]]

View file

@ -0,0 +1,18 @@
### Please describe the problem.
v8 repo automatically upgrades to v10 which is contrary to what the changelog states here [[news/version_10.20220822]].
### What steps will reproduce the problem?
```
git init
git annex init --version 8
git config annex.version
git annex info > /dev/null
git config annex.version
```
### What version of git-annex are you using? On what operating system?
10.20220822 Linux

View file

@ -0,0 +1,42 @@
### Please describe the problem.
Started to happen recently (0924):
```shell
(git)smaug:/mnt/datasets/datalad/ci/git-annex/builds/2022/09[master]git
$> git grep -l 'Couldn.t match expected type '
cron-20220924/build-windows.yaml-788-e26581b6-failed/1_build-package (1).txt
cron-20220924/build-windows.yaml-788-e26581b6-failed/build-package/18_Build git-annex.txt
cron-20220925/build-windows.yaml-790-e26581b6-failed/1_build-package (1).txt
cron-20220925/build-windows.yaml-790-e26581b6-failed/build-package/18_Build git-annex.txt
cron-20220926/build-windows.yaml-791-40917e42-failed/1_build-package (1).txt
cron-20220926/build-windows.yaml-791-40917e42-failed/build-package/18_Build git-annex.txt
pr-133/build-windows.yaml-789-06e70ac7-failed/1_build-package (1).txt
pr-133/build-windows.yaml-789-06e70ac7-failed/build-package/18_Build git-annex.txt
```
```
Annex\PidLock.hs:130:48: error:
* Couldn't match expected type `IO a' with actual type `Annex a'
* In the second argument of `($)', namely `a r'
In the expression: liftIO $ a r
In an equation for runsGitAnnexChildProcessViaGit':
runsGitAnnexChildProcessViaGit' r a = liftIO $ a r
* Relevant bindings include
a :: Repo -> Annex a (bound at Annex\PidLock.hs:130:35)
runsGitAnnexChildProcessViaGit' :: Repo
-> (Repo -> Annex a) -> Annex a
(bound at Annex\PidLock.hs:130:1)
|
130 | runsGitAnnexChildProcessViaGit' r a = liftIO $ a r
| ^^^
Warning: Failed to decode module interface:
D:\a\git-annex\git-annex\.stack-work\dist\274b403a\build\git-annex\git-annex-tmp\Annex.hi
Decoding failure: Invalid magic: e49ceb0f
...
```

View file

@ -1,32 +1,221 @@
The following steps are tested on Windows 10 21h1 and are designed to avoid these two bugs:
The following steps are tested on Windows 10 21h1 with Ubuntu 20 and are designed to allow use of the annexed files through both WSL and Windows.
* [[bugs/WSL_adjusted_braches__58___smudge_fails_with_sqlite_thread_crashed_-_locking_protocol]]
* [[bugs/WSL1__58___git-annex-add_fails_in_DrvFs_filesystem]]
** Limitations **
* The repository must be created with `annex.tune.objecthashlower=true`.
* `git annex adjust --unlock` will not work. Unlocked files will work most of the time. Avoid `annex.addunlocked=true` because it is likely to not work.
**Setup**
* Enable Developer mode in Windows settings so that symlinks can be created without elevated privileges.
* Mount the NTFS drive with metadata option. [`/etc/wsl.conf`](https://docs.microsoft.com/en-us/windows/wsl/wsl-config) can be used or a line such as `C: /mnt/c drvfs metadata` can be added in `/etc/fstab`.
* Create an empty directory for the repo.
* With File Explorer go to Properties/Security/Advanced.
* Give "Authenticated Users" permission to [“Write attributes”, “Create files”, “Create folders” and “Delete subfolders and files”](https://devblogs.microsoft.com/commandline/improved-per-directory-case-sensitivity-support-in-wsl/) for this folder and subfolders.
* Enable case sensitivity with `setfattr -n system.wsl_case_sensitive -v 1 <path>`.
* This attribute will be inherited by new subdirectories but will not be automatically recursively applied to existing subdirectories.
* Set this attribute to 0 as appropriate if you do not have files that differ only by case, or if you have non-default git-annex [[tuning]].
* This attribute can be queried with `getfattr -n system.wsl_case_sensitive <path>`.
* Initialize the repo.
* Set `git config annex.crippledfilesystem true` immediately after `git annex init` to avoid triggering the above two bugs.
* Open Properties/Security/Advanced for .git/annex/objects.
* Deny "Authenticated Users" write permissions for files only and select "Replace all child object permission entries with inheritable permission entries from this object".
* If the repository is created by cloning, create local git-annex branch from the remote branch and remove the origin remote before `git annex init`.
* Set `git config annex.crippledfilesystem true` immediately after `git annex init`.
* Add the origin remote back if it was previously removed.
* Safety of locked files will require these settings and scripts and the patch below.
* `git config annex.freezecontent-command 'wsl-freezecontent %path'`
* `git config annex.thawcontent-command 'wsl-thawcontent %path'`
** Setup with repo cloned from SSH **
```
#!/usr/bin/env bash
* If you encounter problems with older versions of git-annex.
* Create a local git-annex branch from the remote branch.
* Remove the origin remote.
* Do `git annex init` and other setup.
* Add back the origin remote.
if [ -f "$1" ]; then
if \[[ "$1" == *.git/annex/objects/* ]]; then
PERM='(DE,WD,AD)'
else
PERM='(WD,AD)'
fi
elif [ -d "$1" ]; then
PERM='(DE,DC,WD,AD)'
else
exit 0
fi
** Using symlinks and locked files **
OUTPUT="$(icacls.exe "$(wslpath -w "$1")" /deny "Authenticated Users:$PERM")"
* Symlinks might be broken and can be fixed by recreating them. You can also delete them and do a `git checkout`.
if [ "$?" -ne 0 ]; then
echo "$OUTPUT"
exit 1
fi
```
```
#!/usr/bin/env bash
if [ -f "$1" ]; then
PERM='(DE,WD,AD)'
elif [ -d "$1" ]; then
PERM='(DE,DC,WD,AD)'
else
exit 0
fi
OUTPUT="$(icacls.exe "$(wslpath -w "$1")" /grant "Authenticated Users:$PERM")"
if [ "$?" -ne 0 ]; then
echo "$OUTPUT"
exit 1
fi
```
** Patches **
This patch allows permissions and freeze/thaw hooks to run on a crippled file system.
```
From 7f3da0dda841bf73645809d3919cff2a37cb21de Mon Sep 17 00:00:00 2001
From: Reiko Asakura <asakurareiko@protonmail.ch>
Date: Sat, 23 Oct 2021 17:14:27 -0400
Subject: [PATCH 2/2] Allow perms on crippled filesystem
---
Annex/Perms.hs | 22 +++++++++-------------
1 file changed, 9 insertions(+), 13 deletions(-)
diff --git a/Annex/Perms.hs b/Annex/Perms.hs
index 6681da7e0..e0c323d05 100644
--- a/Annex/Perms.hs
+++ b/Annex/Perms.hs
@@ -61,7 +61,7 @@ setAnnexPerm :: Bool -> RawFilePath -> Annex ()
setAnnexPerm = setAnnexPerm' Nothing
setAnnexPerm' :: Maybe ([FileMode] -> FileMode -> FileMode) -> Bool -> RawFilePath -> Annex ()
-setAnnexPerm' modef isdir file = unlessM crippledFileSystem $
+setAnnexPerm' modef isdir file =
withShared $ liftIO . go
where
go GroupShared = void $ tryIO $ modifyFileMode file $ modef' $
@@ -89,7 +89,7 @@ resetAnnexFilePerm = resetAnnexPerm False
- usual modes.
-}
resetAnnexPerm :: Bool -> RawFilePath -> Annex ()
-resetAnnexPerm isdir file = unlessM crippledFileSystem $ do
+resetAnnexPerm isdir file = do
defmode <- liftIO defaultFileMode
let modef moremodes _oldmode = addModes moremodes defmode
setAnnexPerm' (Just modef) isdir file
@@ -154,7 +154,7 @@ createWorkTreeDirectory dir = do
- that happens with write permissions.
-}
freezeContent :: RawFilePath -> Annex ()
-freezeContent file = unlessM crippledFileSystem $
+freezeContent file =
withShared $ \sr -> freezeContent' sr file
freezeContent' :: SharedRepository -> RawFilePath -> Annex ()
@@ -199,14 +199,12 @@ freezeContent'' sr file rv = do
- write permissions are ignored.
-}
checkContentWritePerm :: RawFilePath -> Annex (Maybe Bool)
-checkContentWritePerm file = ifM crippledFileSystem
- ( return (Just True)
- , do
+checkContentWritePerm file =
+ do
rv <- getVersion
hasfreezehook <- hasFreezeHook
withShared $ \sr -> liftIO $
checkContentWritePerm' sr file rv hasfreezehook
- )
checkContentWritePerm' :: SharedRepository -> RawFilePath -> Maybe RepoVersion -> Bool -> IO (Maybe Bool)
checkContentWritePerm' sr file rv hasfreezehook
@@ -252,10 +250,8 @@ thawContent' sr file = do
- crippled filesystem, the file may be frozen, so try to thaw its
- permissions. -}
thawPerms :: Annex () -> Annex () -> Annex ()
-thawPerms a hook = ifM crippledFileSystem
- ( void (tryNonAsync a)
- , hook >> a
- )
+thawPerms a hook =
+ hook >> a
{- Blocks writing to the directory an annexed file is in, to prevent the
- file accidentally being deleted. However, if core.sharedRepository
@@ -263,7 +259,7 @@ thawPerms a hook = ifM crippledFileSystem
- file.
-}
freezeContentDir :: RawFilePath -> Annex ()
-freezeContentDir file = unlessM crippledFileSystem $ do
+freezeContentDir file = do
fastDebug "Annex.Perms" ("freezing content directory " ++ fromRawFilePath dir)
withShared go
freezeHook dir
@@ -287,7 +283,7 @@ createContentDir dest = do
unlessM (liftIO $ R.doesPathExist dir) $
createAnnexDirectory dir
-- might have already existed with restricted perms
- unlessM crippledFileSystem $ do
+ do
thawHook dir
liftIO $ allowWrite dir
where
--
2.30.2
```
This patch allows `git annex fix` on a crippled file system.
```
From 65fe6e362dfbf2f54c8da5ca17c59af26de5ff83 Mon Sep 17 00:00:00 2001
From: Reiko Asakura <asakurareiko@protonmail.ch>
Date: Sat, 23 Oct 2021 17:13:50 -0400
Subject: [PATCH 1/2] Allow git-annex fix on crippled filesystem
---
Command/Fix.hs | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Command/Fix.hs b/Command/Fix.hs
index 39853c894..2d66c1461 100644
--- a/Command/Fix.hs
+++ b/Command/Fix.hs
@@ -31,7 +31,7 @@ cmd = noCommit $ withAnnexOptions [annexedMatchingOptions] $
paramPaths (withParams seek)
seek :: CmdParams -> CommandSeek
-seek ps = unlessM crippledFileSystem $
+seek ps =
withFilesInGitAnnex ww seeker =<< workTreeItems ww ps
where
ww = WarnUnmatchLsFiles
--
2.30.2
```
** Usage tips **
* Symlinks are invalid in Windows if created before the target file exists, such as after `git annex add` or `git annex get`. This can be fixed by recreating them with any method, such as delete them and `git checkout`. Below is a sample script.
```
#!/usr/bin/env python3
import pathlib
import os
def do(p):
for c in list(p.iterdir()):
if c.is_symlink() and c.resolve().exists():
target = os.readlink(c) # use readlink here to get the relative link path
c.unlink()
c.symlink_to(target)
elif c.is_dir() and c.name != '.git':
do(c)
do(pathlib.Path('.'))
```
* Sometimes there will SQLite errors using multiple jobs but retrying will work most of the time.
** Related bugs **
* [[bugs/WSL_adjusted_braches__58___smudge_fails_with_sqlite_thread_crashed_-_locking_protocol]]
* [[bugs/WSL1__58___git-annex-add_fails_in_DrvFs_filesystem]]
* [[bugs/problems_with_SSH_and_relative_paths]]

View file

@ -0,0 +1,74 @@
This patch allows hooks to be run in WSL1 for a repo created on an NTFS volume but I think it also applies in general.
```
From d4587806d8fd1ea767a8effc06edc1a4e10f5bca Mon Sep 17 00:00:00 2001
From: Reiko Asakura <asakurareiko@protonmail.ch>
Date: Sun, 25 Sep 2022 15:21:24 -0400
Subject: [PATCH] Run freeze and thaw hooks on crippled filesystems
The user sets these hooks deliberately so they should always be run. For
example this allows hooks to be used to manage file permissions on NTFS
volumes in WSL1.
---
Annex/Perms.hs | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/Annex/Perms.hs b/Annex/Perms.hs
index 6681da7e0..69e344452 100644
--- a/Annex/Perms.hs
+++ b/Annex/Perms.hs
@@ -154,7 +154,7 @@ createWorkTreeDirectory dir = do
- that happens with write permissions.
-}
freezeContent :: RawFilePath -> Annex ()
-freezeContent file = unlessM crippledFileSystem $
+freezeContent file =
withShared $ \sr -> freezeContent' sr file
freezeContent' :: SharedRepository -> RawFilePath -> Annex ()
@@ -163,7 +163,7 @@ freezeContent' sr file = freezeContent'' sr file =<< getVersion
freezeContent'' :: SharedRepository -> RawFilePath -> Maybe RepoVersion -> Annex ()
freezeContent'' sr file rv = do
fastDebug "Annex.Perms" ("freezing content " ++ fromRawFilePath file)
- go sr
+ unlessM crippledFileSystem $ go sr
freezeHook file
where
go GroupShared = if versionNeedsWritableContentFiles rv
@@ -253,7 +253,7 @@ thawContent' sr file = do
- permissions. -}
thawPerms :: Annex () -> Annex () -> Annex ()
thawPerms a hook = ifM crippledFileSystem
- ( void (tryNonAsync a)
+ ( hook >> void (tryNonAsync a)
, hook >> a
)
@@ -263,9 +263,9 @@ thawPerms a hook = ifM crippledFileSystem
- file.
-}
freezeContentDir :: RawFilePath -> Annex ()
-freezeContentDir file = unlessM crippledFileSystem $ do
+freezeContentDir file = do
fastDebug "Annex.Perms" ("freezing content directory " ++ fromRawFilePath dir)
- withShared go
+ unlessM crippledFileSystem $ withShared go
freezeHook dir
where
dir = parentDir file
@@ -287,9 +287,9 @@ createContentDir dest = do
unlessM (liftIO $ R.doesPathExist dir) $
createAnnexDirectory dir
-- might have already existed with restricted perms
- unlessM crippledFileSystem $ do
+ do
thawHook dir
- liftIO $ allowWrite dir
+ unlessM crippledFileSystem $ liftIO $ allowWrite dir
where
dir = parentDir dest
--
2.30.2
```

View file

@ -131,6 +131,6 @@ Seems like this would need Windows 10.
> > > > But here's a bug about sqlite in WSL:
> > > > [[bugs/WSL_adjusted_braches__58___smudge_fails_with_sqlite_thread_crashed_-_locking_protocol]] --[[Joey]]
## update Oct 2021:
## update Sept 2022:
Git-annex has become more reliable on WSL1 recently, see [[tips/Using_git-annex_on_NTFS_with_WSL1]].