mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' of ssh://git-annex.branchable.com

Browse source
This commit is contained in:
Joey Hess 2020-11-07 14:10:19 -04:00
commit c64e95045d
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
7 changed files with 397 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

221
doc/bugs/file_not_correctly_added.mdwn Normal file
View file

@ -0,0 +1,221 @@
### Please describe the problem.
When I run `git annex add .` the file is no longer symlinked into the directory, but the file is keep as it is and simply added to git. This also happens with a minimalistic test repo as described below:
### What steps will reproduce the problem?
- `sudo pacman -S git-annex`
- `mkdir ./test`
- `cd test`
- `git init`
- `git annex init`
- `cp ../myLargeFile.CR2 .`
- `git annex add .`
### What version of git-annex are you using? On what operating system?
```
$ git annex version
git-annex version: 8.20201103-g2dabd4cc2d
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.10.2 http-client-0.7.2.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
```
```
$ uname -a
Linux jcarch 5.4.74-1-lts #1 SMP Sun, 01 Nov 2020 12:58:27 +0000 x86_64 GNU/Linux
```
### Please provide any additional information below.
[[!format sh """
$ git annex add -d test.CR2
[2020-11-05 14:05:18.380112289] process [47259] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2020-11-05 14:05:18.380857707] process [47259] done ExitSuccess
[2020-11-05 14:05:18.381159814] process [47260] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2020-11-05 14:05:18.381788658] process [47260] done ExitSuccess
[2020-11-05 14:05:18.382106752] process [47261] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..ae4cf18c6d72434a514230220967195c0b230adb","--pretty=%H","-n1"]
[2020-11-05 14:05:18.382829403] process [47261] done ExitSuccess
[2020-11-05 14:05:18.38340398] process [47262] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2020-11-05 14:05:18.383651872] process [47263] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2020-11-05 14:05:18.384269396] process [47264] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
[2020-11-05 14:05:18.385141135] process [47264] done ExitSuccess
[2020-11-05 14:05:18.385420658] process [47265] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
[2020-11-05 14:05:18.386064486] process [47265] done ExitFailure 1
[2020-11-05 14:05:18.386480384] process [47266] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","-z","--others","--exclude-standard","--","test.CR2"]
[2020-11-05 14:05:18.387250532] process [47267] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
add test.CR2 [2020-11-05 14:05:18.389436209] process [47268] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","diff.external=","-c","filter.annex.smudge=","-c","filter.annex.clean=","diff","--cached","--raw","-z","--no-abbrev","-G^/annex/objects/","--diff-filter=AMUT","--no-renames","--ignore-submodules=all","--no-ext-diff"]
[2020-11-05 14:05:18.391345028] process [47268] done ExitSuccess
ok
[2020-11-05 14:05:18.391581576] process [47266] done ExitSuccess
[2020-11-05 14:05:18.39193215] process [47270] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","-z","--modified","--","test.CR2"]
[2020-11-05 14:05:18.392531099] process [47270] done ExitSuccess
[2020-11-05 14:05:18.392820948] process [47271] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff","--name-only","--diff-filter=T","-z","--cached","--","test.CR2"]
[2020-11-05 14:05:18.393490388] process [47271] done ExitSuccess
(recording state in git...)
[2020-11-05 14:05:18.39392609] process [47272] feed: xargs ["-0","git","--git-dir=.git","--work-tree=.","--literal-pathspecs","add","--"]
[2020-11-05 14:05:18.679370104] process [47272] done ExitSuccess
[2020-11-05 14:05:18.680381167] process [47262] done ExitSuccess
[2020-11-05 14:05:18.680659505] process [47263] done ExitSuccess
[2020-11-05 14:05:18.680881174] process [47267] done ExitSuccess
"""]]
The directory structure of the repo after running `git annex add .`:
[[!format sh """
$ tree -a
.
├── .git
│   ├── annex
│   │   ├── gitqueue.lck
│   │   ├── index
│   │   ├── index.lck
│   │   ├── journal
│   │   ├── journal.lck
│   │   ├── keysdb
│   │   │   └── db
│   │   ├── keysdb.cache
│   │   ├── keysdb.lck
│   │   ├── othertmp.lck
│   │   ├── sentinal
│   │   └── sentinal.cache
│   ├── branches
│   ├── config
│   ├── description
│   ├── HEAD
│   ├── hooks
│   │   ├── applypatch-msg.sample
│   │   ├── commit-msg.sample
│   │   ├── fsmonitor-watchman.sample
│   │   ├── post-checkout
│   │   ├── post-merge
│   │   ├── post-receive
│   │   ├── post-update.sample
│   │   ├── pre-applypatch.sample
│   │   ├── pre-commit
│   │   ├── pre-commit.sample
│   │   ├── pre-merge-commit.sample
│   │   ├── prepare-commit-msg.sample
│   │   ├── pre-push.sample
│   │   ├── pre-rebase.sample
│   │   ├── pre-receive.sample
│   │   └── update.sample
│   ├── index
│   ├── info
│   │   ├── attributes
│   │   └── exclude
│   ├── logs
│   │   └── refs
│   │   └── heads
│   │   └── git-annex
│   ├── objects
│   │   ├── 0d
│   │   │   └── b47ff2773fdd41f7be54839f6a3450c25da595
│   │   ├── 4b
│   │   │   └── 825dc642cb6eb9a060e54bf8d69288fbee4904
│   │   ├── 53
│   │   │   └── 40ddb9312ba2c09f08cb0308f4f8679b70735e
│   │   ├── 7f
│   │   │   └── 92a6e546f860b696d1ee94694296975761b8f4
│   │   ├── 9d
│   │   │   └── c023b630e35c131d96c10641e57de7529c3c64
│   │   ├── ae
│   │   │   └── 4cf18c6d72434a514230220967195c0b230adb
│   │   ├── info
│   │   └── pack
│   └── refs
│   ├── heads
│   │   └── git-annex
│   └── tags
└── test.CR2
"""]]
The repo after running `git annex unlock test.CR2 && git annex lock test.CR2`. Now, I do get a symlink, but it is completely broken:
[[!format sh """
$ tree -a
.
├── .git
│   ├── annex
│   │   ├── gitqueue.lck
│   │   ├── index
│   │   ├── index.lck
│   │   ├── journal
│   │   ├── journal.lck
│   │   ├── keysdb
│   │   │   └── db
│   │   ├── keysdb.cache
│   │   ├── keysdb.lck
│   │   ├── objects
│   │   │   └── Z8
│   │   │   └── ff
│   │   │   └── SHA256E-s29334740--1ddc219a95f1d1fa37e94667dae44c8a5ec42fff0c5dfd5dc46ac9e6ad43e7fe.CR2
│   │   ├── othertmp.lck
│   │   ├── sentinal
│   │   └── sentinal.cache
│   ├── branches
│   ├── config
│   ├── description
│   ├── HEAD
│   ├── hooks
│   │   ├── applypatch-msg.sample
│   │   ├── commit-msg.sample
│   │   ├── fsmonitor-watchman.sample
│   │   ├── post-checkout
│   │   ├── post-merge
│   │   ├── post-receive
│   │   ├── post-update.sample
│   │   ├── pre-applypatch.sample
│   │   ├── pre-commit
│   │   ├── pre-commit.sample
│   │   ├── pre-merge-commit.sample
│   │   ├── prepare-commit-msg.sample
│   │   ├── pre-push.sample
│   │   ├── pre-rebase.sample
│   │   ├── pre-receive.sample
│   │   └── update.sample
│   ├── index
│   ├── info
│   │   ├── attributes
│   │   └── exclude
│   ├── logs
│   │   └── refs
│   │   └── heads
│   │   └── git-annex
│   ├── objects
│   │   ├── 0d
│   │   │   └── b47ff2773fdd41f7be54839f6a3450c25da595
│   │   ├── 3a
│   │   │   └── 23d9e0574e7b37778991638c07005f6c27418f
│   │   ├── 45
│   │   │   └── 36e1509db6efcf5b3f213e94e139dfffd909b7
│   │   ├── 4b
│   │   │   └── 825dc642cb6eb9a060e54bf8d69288fbee4904
│   │   ├── 53
│   │   │   └── 40ddb9312ba2c09f08cb0308f4f8679b70735e
│   │   ├── 74
│   │   │   └── 4ae89ab07cb3c592c4e815db44adb1f2b0f679
│   │   ├── 7f
│   │   │   └── 92a6e546f860b696d1ee94694296975761b8f4
│   │   ├── 85
│   │   │   └── f161fd698805ff508cf17712f7297751ea4d51
│   │   ├── 9c
│   │   │   └── 1989036a57661474b3442062b4fcc3c285681b
│   │   ├── 9d
│   │   │   └── c023b630e35c131d96c10641e57de7529c3c64
│   │   ├── ae
│   │   │   └── 4cf18c6d72434a514230220967195c0b230adb
│   │   ├── f8
│   │   │   └── c3a1f4e637bb0cffd25ae8fa61ce38d3b9d2f2
│   │   ├── info
│   │   └── pack
│   └── refs
│   ├── heads
│   │   └── git-annex
│   └── tags
└── test.CR2 -> .git/annex/objects/Z8/ff/SHA256E-s29334740--1ddc219a95f1d1fa37e94667dae44c8a5ec42fff0c5dfd5dc46ac9e6ad43e7fe.CR2/SHA256E-s29334740--1ddc219a95f1d1fa37e94667dae44c8a5ec42fff0c5dfd5dc46ac9e6ad43e7fe.CR2
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
It used to work before. It seem to be broken since a system update. I have tried reinstalling `git-annex` as well as all dependencies, bit without any luck.

8
doc/bugs/file_not_correctly_added/comment_1_3da19ed6975dc562c194175ce956f05a._comment Normal file
View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Lukey"
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
subject="comment 1"
date="2020-11-06T14:14:18Z"
content="""
I can't reproduce this with the latest [[standalone build|https://git-annex.branchable.com/install/Linux_standalone/]] (8.20201008). What does `git status` say after `git annex add`? What filesystem is this on? Can you try this again with the standalone build?
"""]]

30
doc/bugs/file_not_correctly_added/comment_2_aadc2c2faa8455614efd6e8427965ece._comment Normal file
View file

@ -0,0 +1,30 @@
[[!comment format=mdwn
username="jcjgraf"
avatar="http://cdn.libravatar.org/avatar/9dda752f83ac44906fefbadb35e8a6ac"
subject="comment 2"
date="2020-11-06T14:57:52Z"
content="""
Thanks for the reply!
- `git status` after running `git annex add` simply reports that there is a new file:
```
$ git annex add
add test.CR2 ok
(recording state in git...)
$ git status
On branch master
No commits yet
Changes to be committed:
(use \"git rm --cached <file>...\" to unstage)
new file: test.CR2
```
- The file system is just plain ext4.
- The behaviour of the standalone build does not differ.
"""]]

103
doc/bugs/make_debianstandalone_-_FTBFS.mdwn Normal file
View file

@ -0,0 +1,103 @@
### Please describe the problem.
```
$> make debianstandalone
...
[644 of 645] Compiling CmdLine.GitAnnex ( CmdLine/GitAnnex.hs, dist/build/git-annex/git-annex-tmp/CmdLine/GitAnnex.o )
[645 of 645] Compiling Main ( git-annex.hs, dist/build/git-annex/git-annex-tmp/Main.o )
Linking dist/build/git-annex/git-annex ...
ghc --make Build/Standalone -Wall -fno-warn-tabs
[18 of 30] Compiling BuildInfo ( BuildInfo.hs, BuildInfo.o )
[19 of 30] Compiling Build.BundledPrograms ( Build/BundledPrograms.hs, Build/BundledPrograms.o )
[20 of 30] Compiling Utility.DebugLocks ( Utility/DebugLocks.hs, Utility/DebugLocks.o )
[21 of 30] Compiling Utility.Env ( Utility/Env.hs, Utility/Env.o )
[22 of 30] Compiling Utility.FileSize ( Utility/FileSize.hs, Utility/FileSize.o )
[23 of 30] Compiling Utility.PartialPrelude ( Utility/PartialPrelude.hs, Utility/PartialPrelude.o )
[24 of 30] Compiling Utility.Network ( Utility/Network.hs, Utility/Network.o )
[25 of 30] Compiling Utility.Path.AbsRel ( Utility/Path/AbsRel.hs, Utility/Path/AbsRel.o )
[26 of 30] Compiling Utility.LinuxMkLibs ( Utility/LinuxMkLibs.hs, Utility/LinuxMkLibs.o )
[27 of 30] Compiling Common ( Common.hs, Common.o )
[28 of 30] Compiling Utility.CopyFile ( Utility/CopyFile.hs, Utility/CopyFile.o )
[29 of 30] Compiling Build.LinuxMkLibs ( Build/LinuxMkLibs.hs, Build/LinuxMkLibs.o )
Build/LinuxMkLibs.hs:45:41: error:
• Couldn't match type ‘Data.ByteString.Internal.ByteString’
with ‘[Char]’
Expected type: String
Actual type: System.Posix.ByteString.FilePath.RawFilePath
• In the second argument of ‘writeFile’, namely
‘(parentDir $ head gconvlibs)’
In a stmt of a 'do' block:
writeFile (top </> "gconvdir") (parentDir $ head gconvlibs)
In the expression:
do fs <- dirContentsRecursive top
exes <- filterM checkExe fs
libs <- parseLdd <$> readProcess "ldd" exes
glibclibs <- glibcLibs
....
|
45 | writeFile (top </> "gconvdir") (parentDir $ Prelude.head gconvlibs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Build/LinuxMkLibs.hs:45:53: error:
• Couldn't match type ‘[Char]’
with ‘Data.ByteString.Internal.ByteString’
Expected type: System.Posix.ByteString.FilePath.RawFilePath
Actual type: FilePath
• In the second argument of ‘($)’, namely ‘head gconvlibs’
In the second argument of ‘writeFile’, namely
‘(parentDir $ head gconvlibs)’
In a stmt of a 'do' block:
writeFile (top </> "gconvdir") (parentDir $ head gconvlibs)
|
45 | writeFile (top </> "gconvdir") (parentDir $ Prelude.head gconvlibs)
| ^^^^^^^^^^^^^^^^^^^^^^
Build/LinuxMkLibs.hs:110:17: error:
• Variable not in scope:
relPathDirToFile :: FilePath -> [Char] -> IO FilePath
• Perhaps you meant ‘relPathDirToFileAbs’ (imported from Utility.Path)
|
110 | link <- relPathDirToFile (top </> exedir) (top ++ linker)
| ^^^^^^^^^^^^^^^^
Build/LinuxMkLibs.hs:132:31: error:
• Couldn't match type ‘Data.ByteString.Internal.ByteString’
with ‘[Char]’
Expected type: FilePath
Actual type: System.Posix.ByteString.FilePath.RawFilePath
• In the second argument of ‘($)’, namely ‘parentDir f’
In the expression: inTop top $ parentDir f
In an equation for ‘destdir’: destdir = inTop top $ parentDir f
|
132 | destdir = inTop top $ parentDir f
| ^^^^^^^^^^^
Build/LinuxMkLibs.hs:132:41: error:
• Couldn't match type ‘[Char]’
with ‘Data.ByteString.Internal.ByteString’
Expected type: System.Posix.ByteString.FilePath.RawFilePath
Actual type: FilePath
• In the first argument of ‘parentDir’, namely ‘f’
In the second argument of ‘($)’, namely ‘parentDir f’
In the expression: inTop top $ parentDir f
|
132 | destdir = inTop top $ parentDir f
| ^
make[4]: *** [Makefile:165: Build/Standalone] Error 1
make[4]: Leaving directory '/home/yoh/proj/git-annex'
make[3]: *** [Makefile:173: linuxstandalone] Error 2
make[3]: Leaving directory '/home/yoh/proj/git-annex'
make[2]: *** [debian/rules:24: override_dh_auto_build] Error 2
make[2]: Leaving directory '/home/yoh/proj/git-annex'
make[1]: *** [debian/rules:17: build] Error 2
make[1]: Leaving directory '/home/yoh/proj/git-annex'
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
make: *** [Makefile:205: dpkg-buildpackage-F] Error 2
make debianstandalone 289.29s user 9.60s system 99% cpu 5:00.49 total
$> git describe
8.20201103-34-gd8e8d145e
```
```

11
doc/forum/How_are_people_handling_the_v8_repo_upgrade__63__/comment_3_b05a8c63d2594290769aeeaabc91f491._comment Normal file
View file

@ -0,0 +1,11 @@
[[!comment format=mdwn
username="interfect@b151490178830f44348aa57b77ad58c7d18e8fe7"
nickname="interfect"
avatar="http://cdn.libravatar.org/avatar/3ba541a3204f4ea4b274049a152489c5"
subject="comment 3"
date="2020-11-05T17:55:31Z"
content="""
Oh, that must be the problem; the remote that's giving me trouble is on NTFS, so I think everything has been automatically unlocked there.
I'll just wait for Raspian to upgrade, and see about switching to a better filesystem in the meantime.
"""]]

9
doc/forum/Moving_S3_special_remote/comment_1_b030a7afcbf1a329bb35260cf338ff96._comment Normal file
View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="a@92214b2eae95307f511e06845d2cb5b5931b00a3"
nickname="a"
avatar="http://cdn.libravatar.org/avatar/d76a578b33d7d7bb570e176909fa8fb1"
subject="Re: Moving S3 special remote"
date="2020-11-05T03:44:52Z"
content="""
And the answer is that running `git annex enableremote bucket=bucketbar fileprefix=annex/` works just fine.
"""]]

15
doc/todo/Incremental_git_annex_sync_--content_--all.mdwn Normal file
View file

@ -0,0 +1,15 @@
Hi,<br>
So I have yet another idea to speed up git annex. For now only for the 2nd pass of git annex sync --content --all.
1. Do a normal (full) git annex sync. For every remote that we synced with, record the commit id of the current tip of the git-annex branch.
* Record the commit id only if --content --all was specified
* Record the commit id only if the remote is actually available and every file was sucessfully transfered
2. If any of the remotes doesn't have a commit id recorded, go to 1. Else do a incremental git annex sync: In the 2nd pass of git annex sync --content --all,
only look at keys whose location log changed since the last (full or incremental) sync via `git diff-tree -r --name-only <lowest recorded commit id of all remotes> git-annex`.
Again, update the commit id of remotes that we sucessfully synced with.
3. If one of the following happens, remove all recorded commit ids of all remotes, go to 1. Else go to 2.
* The preferred content expression of us or one of our remotes changed.
* The preferred content expression of a group changed
* The group of any repo (not only remotes) changed. This way remotes containing `copies=<group>:<numcopies>` recheck all keys.
This should be pretty reliable, but please double check. It has to be reliable enough to become the default.