Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2023-03-06 12:13:29 -04:00
commit 89373c04bb
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
8 changed files with 356 additions and 2 deletions

View file

@ -110,9 +110,11 @@ This then compiled cleanly and completed the test suite ok.
Git Annex is great. I use it several times a week with my multigigabyte backups, where it gives structure to my image-based backup routines, so you could say I'm a believer. :)
[[!meta author=jkniiv]]
[[!meta title="windows: build fails starting from commit 54ad1b4cf"]]
[[!meta title="windows: build fails starting from commits 54ad1b4cf, 398633c12"]]
> Kind of expected there would be something! The patch is almost right,
> but we're avoiding using System.PosixCompat.Files.getFileStatus
> since it doesn't support long paths, so have to use
> Utility.RawFilePath.getFileStatus instead. [[done]] --[[Joey]]
> Utility.RawFilePath.getFileStatus instead. \[\[done\]\] --[[Joey]]
>> Sorry, I had to reopen the bug as Joey's commit didn't quite compile. See page comments for details. --[[jkniiv]]

View file

@ -0,0 +1,84 @@
[[!comment format=mdwn
username="jkniiv"
avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d"
subject="ok, that didn't quite resolve it"
date="2023-03-04T19:15:31Z"
content="""
@Joey, unfortunately commit [[!commit 398633c12bfb6201ba1fcd9b50e5e8055bfff89e]] wasn't quite enough to
resolve my issue. :) First of all I wondered why you would employ `fromRawFilePath` instead of `toRawFilePath`
considering that the argument of `getstatus` seems to be a FilePath, and indeed Utility.DirWatcher.Win32Notify
didn't compile without replacing the former with the latter. Also you didn't notice that I had changed a qualified
import into a regular one because otherwise `isRegularFile` wouldn't be available.
Here's the tail end of my build log trying to build commit 398633c12:
[[!format sh \"\"\"
jkniiv@AINESIS MINGW64 /c/annx
$ tail -n 35 stack.build.LOG~102
[150 of 676] Compiling Utility.DirWatcher.Win32Notify
Utility\DirWatcher\Win32Notify.hs:55:51: error:
* Variable not in scope: isRegularFile :: FileStatus -> Bool
* Perhaps you meant `System.PosixCompat.Files.isRegularFile' (imported from System.PosixCompat.Files)
|
55 | | isRegularFile s ->
| ^^^^^^^^^^^^^
Utility\DirWatcher\Win32Notify.hs:63:54: error:
* Couldn't match type `[Char]'
with `Data.ByteString.Internal.ByteString'
Expected type: RawFilePath -> RawFilePath
Actual type: RawFilePath -> FilePath
* In the second argument of `(.)', namely `fromRawFilePath'
In the second argument of `(.)', namely
`R.getFileStatus . fromRawFilePath'
In the expression: catchMaybeIO . R.getFileStatus . fromRawFilePath
|
63 | getstatus = catchMaybeIO . R.getFileStatus . fromRawFilePath
| ^^^^^^^^^^^^^^^
Error: [S-7282]
Stack failed to execute the build plan.
While executing the build plan, Stack encountered the following errors:
[S-7011]
While building package git-annex-10.20230227 (scroll up to its section to see the error)
using:
C:\Users\jkniiv\Projektit\git-annex.branchable.com\git-annex--BUILD-230304-398633c12\.stack-work\dist\274b403a\setup\setup --verbose=1 --builddir=.stack-work\dist\274b403a build exe:git-annex --ghc-options \" -fdiagnostics-color=always\"
Process exited with code: ExitFailure 1
\"\"\"]]
So, I ended up with the following simple fix (although this situation left me wondering if `watchDir` ought to
be rewritten to deal with RawFilePaths directly but then that's beyond my abilities):
[[!format diff \"\"\"
diff --git a/Utility/DirWatcher/Win32Notify.hs b/Utility/DirWatcher/Win32Notify.hs
index a6a7e4ce9..0b02053c9 100644
--- a/Utility/DirWatcher/Win32Notify.hs
+++ b/Utility/DirWatcher/Win32Notify.hs
@@ -12,7 +12,7 @@ import Utility.DirWatcher.Types
import qualified Utility.RawFilePath as R
import System.Win32.Notify
-import qualified System.PosixCompat.Files (isRegularFile)
+import System.PosixCompat.Files (isRegularFile)
watchDir :: FilePath -> (FilePath -> Bool) -> Bool -> WatchHooks -> IO WatchManager
watchDir dir ignored scanevents hooks = do
@@ -60,7 +60,7 @@ watchDir dir ignored scanevents hooks = do
where
runhook h s = maybe noop (\a -> a f s) (h hooks)
- getstatus = catchMaybeIO . R.getFileStatus . fromRawFilePath
+ getstatus = catchMaybeIO . R.getFileStatus . toRawFilePath
{- Check each component of the path to see if it's ignored. -}
ignoredPath :: (FilePath -> Bool) -> FilePath -> Bool
\"\"\"]]
"""]]

View file

@ -0,0 +1,19 @@
[[!comment format=mdwn
username="jkniiv"
avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d"
subject="comment 2"
date="2023-03-05T20:18:01Z"
content="""
On a positive note I can confirm that my compiled versions of commits 54ad1b4cf and 398633c12
pass their test suite even in a subdirectory some 64 characters deep (for instance in a directory
named like `c:/Users/jkniiv/annx/annx.10.20230228-g505f1a654--test-test-test`) whereas the commit
just prior to \"Windows: Support long filenames in more (possibly all) of the code\" (54ad1b4cf), or
505f1a654, fails some one third to one half of its tests in said directory. Shortening the test
directory name to say `c:/Users/jkniiv/annx/annx` allows the test suite to pass in older versions
of git-annex all other things considered equal. So far I have been running the built-in tests
always in 'c:/annx' (/c/annx in Git Bash) but here on forward I'll probably start using a deeper
path somewhere in my user folder / home directory just to catch possible future regressions in long
path handling. In any case, @Joey, congratulations 🎉 are in order for your fine achievement from
a Windows user point of view. :)
"""]]

View file

@ -0,0 +1,89 @@
Hi!
I'ce git a Pixel 4a with GrapheneOS running and tried to installiert git-annex, but it fails:
~ $ source git-annex-install
Installing dependencies with termux pkg manager...
Testing the available mirrors:
[*] (10) https://packages-cf.termux.dev/apt/termux-main: ok
[*] (1) https://mirrors.cbrx.io/apt/termux/termux-main: ok
[*] (1) https://packages.nscdn.top/termux-main: bad
[*] (1) https://mirror.nevacloud.com/applications/termux/termux-main: ok
[*] (1) https://mirror.bardia.tech/termux/termux-main: ok
[*] (1) https://mirror.albony.xyz/termux/termux-main: ok
[*] (1) https://linux.domainesia.com/applications/termux/termux-main: ok
[*] (1) https://mirror.textcord.xyz/termux/termux-main: ok
[*] (1) https://mirrors.dgut.edu.cn/termux/apt/termux-main: bad
[*] (1) https://mirrors.cqupt.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirror.iscas.ac.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.aliyun.com/termux/termux-main: ok
[*] (1) https://mirrors.bfsu.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.nju.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.ustc.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.pku.edu.cn/termux/termux-main/: ok
[*] (1) https://mirrors.hit.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.sdu.edu.cn/termux/termux-main: ok
[*] (1) https://mirrors.sau.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.tuna.tsinghua.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirror.nyist.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.njupt.edu.cn/termux/apt/termux-main: bad
[*] (1) https://mirrors.scau.edu.cn/termux/apt/termux-main: bad
[*] (1) https://mirrors.sustech.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirror.mwt.me/termux/main: ok
[*] (4) https://grimler.se/termux/termux-main: ok
[*] (1) https://termux.librehat.com/apt/termux-main: ok
[*] (1) https://termux.sahilister.in/apt/termux-main: bad
[*] (1) https://cdn.lumito.net/termux/termux-main: ok
[*] (1) https://termux.astra.in.ua/apt/termux-main: ok
[*] (1) https://termux.mentality.rip/termux-main: ok
[*] (1) https://termux.3san.dev/termux/termux-main: ok
[*] (1) https://mirror.accum.se/mirror/termux.dev/termux-main: ok
[*] (1) https://termux.cdn.lumito.net/termux-main: ok
[*] (1) https://packages.termux.dev/apt/termux-main: ok
[*] (1) https://mirror.termux.dev/termux-main: ok
[*] (1) https://md.mirrors.hacktegic.com/termux/termux-main: ok
[*] (1) https://mirrors.sahilister.in/termux/termux-main: ok
[*] (4) https://packages.termux.dev/apt/termux-main: ok
[*] (1) https://plug-mirror.rcac.purdue.edu/termux/termux-main: ok
[*] (1) https://mirrors.utermux.dev/termux/termux-main: ok
[*] (1) https://dl.kcubeterm.com/termux-main: ok
[*] (1) https://mirror.vern.cc/termux/termux-main: ok
[*] (1) https://mirror.fcix.net/termux/termux-main: ok
[*] (1) https://mirror.mwt.me/termux/main: ok
[*] (1) https://mirror.csclub.uwaterloo.ca/termux/termux-main: ok
[*] (1) https://mirrors.wale.id.au/termux/termux-main: ok
[*] (1) https://mirror.surf/termux/termux-main/: ok
[*] (1) http://mirror.mephi.ru/termux/termux-main: ok
Picking mirror: (3) /data/data/com.termux/files/usr/etc/termux/mirrors/default
Get:1 https://packages-cf.termux.dev/apt/termux-main stable InRelease [14.0 kB]
Get:2 https://packages-cf.termux.dev/apt/termux-main stable/main aarch64 Packages [490 kB]
Fetched 504 kB in 1s (403 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
git is already the newest version (2.39.2).
wget is already the newest version (1.21.3-5).
tar is already the newest version (1.34-2).
coreutils is already the newest version (9.1-2).
proot is already the newest version (5.1.107-57).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Downloading git-annex...
--2023-03-04 21:36:15-- https://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-arm64.tar.gz
Resolving downloads.kitenet.net... 66.228.36.95
Connecting to downloads.kitenet.net|66.228.36.95|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 66569031 (63M) [application/x-gzip]
Saving to: STDOUT
- 100% 63.48M 6.47MB/s in 12s
2023-03-04 21:36:27 (5.28 MB/s) - written to stdout [66569031/66569031]
Running on Android.. Tuning for optimal behavior.
Bad system call
[Process completed (code 159) - press Enter]

View file

@ -0,0 +1,94 @@
[[!comment format=mdwn
username="derphysiker"
avatar="http://cdn.libravatar.org/avatar/80623354932109c891c2e0ebf523b38f"
subject="comment 1"
date="2023-03-04T20:45:47Z"
content="""
Ok, this time with a correct syntax ^^:
~ $ source git-annex-install
Installing dependencies with termux pkg manager...
Testing the available mirrors:
[*] (10) https://packages-cf.termux.dev/apt/termux-main: ok
[*] (1) https://mirrors.cbrx.io/apt/termux/termux-main: ok
[*] (1) https://packages.nscdn.top/termux-main: bad
[*] (1) https://mirror.nevacloud.com/applications/termux/termux-main: ok
[*] (1) https://mirror.bardia.tech/termux/termux-main: ok
[*] (1) https://mirror.albony.xyz/termux/termux-main: ok
[*] (1) https://linux.domainesia.com/applications/termux/termux-main: ok
[*] (1) https://mirror.textcord.xyz/termux/termux-main: ok
[*] (1) https://mirrors.dgut.edu.cn/termux/apt/termux-main: bad
[*] (1) https://mirrors.cqupt.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirror.iscas.ac.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.aliyun.com/termux/termux-main: ok
[*] (1) https://mirrors.bfsu.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.nju.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.ustc.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.pku.edu.cn/termux/termux-main/: ok
[*] (1) https://mirrors.hit.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.sdu.edu.cn/termux/termux-main: ok
[*] (1) https://mirrors.sau.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.tuna.tsinghua.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirror.nyist.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.njupt.edu.cn/termux/apt/termux-main: bad
[*] (1) https://mirrors.scau.edu.cn/termux/apt/termux-main: bad
[*] (1) https://mirrors.sustech.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirror.mwt.me/termux/main: ok
[*] (4) https://grimler.se/termux/termux-main: ok
[*] (1) https://termux.librehat.com/apt/termux-main: ok
[*] (1) https://termux.sahilister.in/apt/termux-main: bad
[*] (1) https://cdn.lumito.net/termux/termux-main: ok
[*] (1) https://termux.astra.in.ua/apt/termux-main: ok
[*] (1) https://termux.mentality.rip/termux-main: ok
[*] (1) https://termux.3san.dev/termux/termux-main: ok
[*] (1) https://mirror.accum.se/mirror/termux.dev/termux-main: ok
[*] (1) https://termux.cdn.lumito.net/termux-main: ok
[*] (1) https://packages.termux.dev/apt/termux-main: ok
[*] (1) https://mirror.termux.dev/termux-main: ok
[*] (1) https://md.mirrors.hacktegic.com/termux/termux-main: ok
[*] (1) https://mirrors.sahilister.in/termux/termux-main: ok
[*] (4) https://packages.termux.dev/apt/termux-main: ok
[*] (1) https://plug-mirror.rcac.purdue.edu/termux/termux-main: ok
[*] (1) https://mirrors.utermux.dev/termux/termux-main: ok
[*] (1) https://dl.kcubeterm.com/termux-main: ok
[*] (1) https://mirror.vern.cc/termux/termux-main: ok
[*] (1) https://mirror.fcix.net/termux/termux-main: ok
[*] (1) https://mirror.mwt.me/termux/main: ok
[*] (1) https://mirror.csclub.uwaterloo.ca/termux/termux-main: ok
[*] (1) https://mirrors.wale.id.au/termux/termux-main: ok
[*] (1) https://mirror.surf/termux/termux-main/: ok
[*] (1) http://mirror.mephi.ru/termux/termux-main: ok
Picking mirror: (3) /data/data/com.termux/files/usr/etc/termux/mirrors/default
Get:1 https://packages-cf.termux.dev/apt/termux-main stable InRelease [14.0 kB]
Get:2 https://packages-cf.termux.dev/apt/termux-main stable/main aarch64 Packages [490 kB]
Fetched 504 kB in 1s (403 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
git is already the newest version (2.39.2).
wget is already the newest version (1.21.3-5).
tar is already the newest version (1.34-2).
coreutils is already the newest version (9.1-2).
proot is already the newest version (5.1.107-57).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Downloading git-annex...
--2023-03-04 21:36:15-- https://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-arm64.tar.gz
Resolving downloads.kitenet.net... 66.228.36.95
Connecting to downloads.kitenet.net|66.228.36.95|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 66569031 (63M) [application/x-gzip]
Saving to: STDOUT
- 100% 63.48M 6.47MB/s in 12s
2023-03-04 21:36:27 (5.28 MB/s) - written to stdout [66569031/66569031]
Running on Android.. Tuning for optimal behavior.
Bad system call
[Process completed (code 159) - press Enter]
"""]]

View file

@ -0,0 +1,29 @@
Hello,
I am a newbie git annex and english is not my native language. Please bare with me. After evaluation so many options, I have decided to keep track of my family photo collection via git annex. My family photo collection is around 300gb and consisting of 120K items currently and its getting bigger fast.
Previously, I was using my home server and attached usb disks. I had also cold backup disks. But keeping them in sync without making errors had been troublesome.
For this task, I decided to use three external usb disks along with a directory in my home server each keeping a git annex repository. I use my desktop as source annex repository to upload photos to these git annex repositories. I will keep external disks off site and rotate them off and on.
I have implemented this design but I have run into a problem.
Currently, there are five git annex repositories as described above. One of them is on the home serve internal disk, three of them are in the usb disks. One is in my desktop harddrive. Apart from my desktop repository, all other repositories are configured as "standard/backup". My desktop annex repository is configured as standard/manual. Each repository has four remotes of other repositories. They can all sync each other and get the contents. However currently I am receiving git message from all my backup repositories as follows.
````
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
warning: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
````
I have tried `git prune --dry-run` as suggested some internet sites and I have seen that git wants to remove almost all annex blob files.
The annex repositories are mostly on btrfs file system, one is on ext4 file system, I do know if it matters. Git-annex versions are 10.20230126-9 in every machine (arch and manjaro).
`git annex unused` gives nothing. I am not using any encryption or any other git pluggable system other than git-annex.
Could you provide some insight please. Is there something wrong my repositories. It really took days to upload and sync all these repositories. Any help is much appreciated.

View file

@ -0,0 +1,19 @@
[[!comment format=mdwn
username="derphysiker"
avatar="http://cdn.libravatar.org/avatar/80623354932109c891c2e0ebf523b38f"
subject="comment 1"
date="2023-03-04T11:11:26Z"
content="""
Did you have a look at
<https://git-annex.branchable.com/tips/Repositories_with_large_number_of_files/>?
What is your current object count (aka non-packed files)?:
git count-objects
If you have many of them (multiple 10k) then it might be a good idea to repack them:
git repack -ad
git gc
"""]]

View file

@ -0,0 +1,18 @@
[[!comment format=mdwn
username="benibilme"
avatar="http://cdn.libravatar.org/avatar/6c053ef48706932f4bd7ab184de4794a"
subject="Hello,"
date="2023-03-05T20:04:03Z"
content="""
Thank you for the response. Yes, repacking and calling garbage collection solved the problem. I am not familiar with git plumbing commands, even though I have been using git for years, I have never needed using these commands. I had to look it up about what they do. I was afraid to lose files, because running prune with dry run options listed objects as many as my files. After your response, I applied on a backed up repository, it seemed I did not lose anything, and applied all other annex repositories, the problem has been solved.
I read the post you referenced after your pointing out, I would like ask if still `git update-index --index-version 4` is relevant for today. The post is 7 years old, by that time, git must have improved a lot.
I am also not quite sure about losing file tracking information. I like to keep this information. My annex repository which is in source standard group and does not have any actual content files is around 147mb. My full backup repositories are around 320gb and `git annex find` reports around 37500 files which is way less that I thought.
`git status` commands takes about 5 or 6 seconds to complete. Yesterday I have added around 30gb of new photos. From source repository in my desktop to four repositories (home server and attached three usb disks) over ssh took about a day.
Uploading is very slow with ssh. For each file, it copies to each remote in turn. It does not blast out all files to one remote in one connection. How can improve sync speed?
Thank you againt for the directions...
"""]]