mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Merge branch 'master' into rawfilepath

Browse source
This commit is contained in:
Joey Hess 2020-11-02 14:35:46 -04:00
commit c41be0d3bd
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
13 changed files with 300 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
Annex/ExternalAddonProcess.hs
View file

@ -55,14 +55,20 @@ startExternalAddonProcess basecmd pid = do
started cmd errrelayer pall@(Just hin, Just hout, Just herr, ph) = do
stderrelay <- async $ errrelayer herr
let shutdown forcestop = do
cancel stderrelay
-- Close the process's stdin, to let it know there
-- are no more requests, so it will exit.
hClose hout
-- Close the procces's stdout as we're not going to
-- process any more output from it.
hClose hin
if forcestop
then cleanupProcess pall
else flip onException (cleanupProcess pall) $ do
hClose herr
hClose hin
hClose hout
void $ waitForProcess ph
else void (waitForProcess ph)
`onException` cleanupProcess pall
-- This thread will exit after consuming any
-- remaining stderr from the process.
wait stderrelay
hClose herr
return $ ExternalAddonProcess
{ externalSend = hin
, externalReceive = hout

4
CHANGELOG
View file

@ -23,6 +23,10 @@ git-annex (8.20201008) UNRELEASED; urgency=medium
on this.
* view: Avoid using ':' from metadata when generating a view, because
it's a special character on Windows ("c:")
* Improve shutdown process for external special remotes and external
backends. Make sure to relay any remaining stderr from the process,
and potentially avoid the process getting a SIGPIPE if it writes to
stderr too late.
-- Joey Hess <id@joeyh.name> Thu, 08 Oct 2020 10:48:17 -0400

30
doc/bugs/Error_cloning_repository_on_Windows/comment_5_6ff246a8cda6bc964fc30e1ccee5ab6f._comment Normal file
View file

@ -0,0 +1,30 @@
[[!comment format=mdwn
username="joey"
subject="""comment 5"""
date="2020-11-02T17:38:34Z"
content="""
I think windows has a limit of around 256 characters on the total length of
a path. (vs 4096 on linux)
So, even if ikiwiki had a way to limit the length of a page name,
which could perhaps be done with `wiki_file_regexp`, that would not help,
because several short page names can be nested into a path that ends up too
long for windows. As is the case here; neither of the bug report filenames
is too long, only the comments on them end up too long. Allowing posting a
bug report but not commenting on it would not be good.
Since I am flatly NOT going to decouple the bug tracking repo from the
source code repo, this seems to put me in the position of going around
cleaning up things that happen to be too long for windows manually. Which I
frankly, also do not want to do. I guess I'll take patches if people want
to send them, but is this really how we want to spend our time?
(Also, renaming someone's bug report out from under where it was risks them
not seeing a request for more information.)
Note that, windows provides ways to avoid these length limits, by prefixing
paths with a sequence that avoids using a compatability layer that imposes
them, and instead uses a more modern layer that has more reasonable limits.
I think git-annex actually uses that itself, thanks to some improvements to
ghc. git for windows could be made to also, I suppose.
"""]]

49
doc/bugs/__34__no_such_file_or_directory__34___when_using_bundle.mdwn Normal file
View file

@ -0,0 +1,49 @@
### Please describe the problem.
The bundle provided at https://git-annex.branchable.com/install/Linux_standalone/ doesn't work
### What steps will reproduce the problem?
1. I downloaded this file: https://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz
2. I untared it in /usr/local and renamed the resulting directory to '/usr/local/git-annex'
3. I added '/usr/local/git-annex' to my PATH
4. I ran 'git-annex' and got:
mv: cannot move '/home/lyderic/.cache/git-annex/locales.tmp/ec977e22ef909b450a3a84bd55783865.1075139' to '/home/lyderic/.cache/git-annex/locales/ec977e22ef909b450a3a84bd55783865': No such file or directory
5. I fixed the problem by doing this:
$ mkdir ~/.cache/git-annex/locales
[[done]]
### What version of git-annex are you using? On what operating system?
git-annex version: 8.20200909-g2d5036e44
OS: Ubuntu 20.04.1
### Please provide any additional information below.
This is a transcript of the commands I ran to show and fix the bug:
[[!format sh """
~$ sudo tar -xzf Downloads/git-annex-standalone-amd64.tar.gz -C /usr/local
~$ sudo mv /usr/local/git-annex.linux /usr/local/git-annex
~$ sudo chown -R 0.0 /usr/local/git-annex
~$ export PATH=$PATH:/usr/local/git-annex
~$ git-annex version
mv: cannot move '/home/lyderic/.cache/git-annex/locales.tmp/ec977e22ef909b450a3a84bd55783865.1075139' to '/home/lyderic/.cache/git-annex/locales/ec977e22ef909b450a3a84bd55783865': No such file or directory
~$ mkdir ~/.cache/git-annex/locales
~$ git-annex version | head -1
git-annex version: 8.20200909-g2d5036e44
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I have never used it. I am learning it / evaluating it right now. I am very impressed so far!
> [[fixed|done]], the bundle will be updated in the upcoming release.
> --[[Joey]]

9
doc/bugs/__34__no_such_file_or_directory__34___when_using_bundle/comment_1_9c2410c55154bc8d9f733377cde58eb9._comment Normal file
View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="lyderic"
avatar="http://cdn.libravatar.org/avatar/d03e900b5e94b54891ee9596460f76c3"
subject="This bug has alreay been raised"
date="2020-10-29T10:13:50Z"
content="""
I found out that this bug has already been raised and addressed...
https://git-annex.branchable.com/bugs/git_annex_init_fails/
"""]]

43
doc/bugs/git-annex_looses_itself_on_Windows__63__.mdwn Normal file
View file

@ -0,0 +1,43 @@
Well, I know that my setup is "experimental" - I am trying to establish consistent `conda` environment, ATM everything (`git` and all the `posix` tools) come from conda (conda-forge channel at large) and then git-annex is installed on top - pretty much simply extracted into conda environment from our git-annex-installer.exe built with magic.
Running (in a windows shell, so not in bash)
```
(datalad-2) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_hufbz7i1>git add --verbose "fi le2.dat"
git-annex smudge --clean -- 'fi le2.dat': git-annex: command not found
error: external filter 'git-annex smudge --clean -- %f' failed 127
error: external filter 'git-annex smudge --clean -- %f' failed
add 'fi le2.dat'
```
suggests that `git` does see `git-annex`, but then `git-annex smudge` doesn't see `git-annex` in the PATH?
```
(datalad-2) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_hufbz7i1>echo %PATH%
C:\Users\DataLad\miniconda3\envs\datalad-2;C:\Users\DataLad\miniconda3\envs\datalad-2\Library\mingw-w64\bin;C:\Users\DataLad\miniconda3\envs\datalad-2\Library\usr\bin;C:\Users\DataLad\miniconda3\envs\datalad-2\Library\bin;C:\Users\DataLad\miniconda3\envs\datalad-2\Scripts;C:\Users\DataLad\miniconda3\envs\datalad-2\bin;C:\Users\DataLad\miniconda3\condabin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\Git\cmd;C:\Users\DataLad\AppData\Local\Microsoft\WindowsApps;.
```
and there is `C:\Users\DataLad\miniconda3\envs\datalad-2\Library\bin\git-annex.exe` .
<details>
<summary>git annex version</summary>
```shell
(datalad-2) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_hufbz7i1>git annex version
git-annex version: 8.20201008-g3be4731
build flags: Assistant Webapp Pairing TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external
operating system: mingw32 x86_64
supported repository versions: 8
upgrade supported from repository versions: 2 3 4 5 6 7
local repository version: 8
```
</details>
[[!meta author=yoh]]
[[!tag projects/datalad]]
> [[notabug|done]] --[[Joey]]

24
doc/bugs/git-annex_looses_itself_on_Windows__63__/comment_1_e809445ff0109d8aa98093b024b3b35c._comment Normal file
View file

@ -0,0 +1,24 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 1"
date="2020-10-27T19:44:36Z"
content="""
probably nevermind and feel free to close -- something is finicky with the `bash` which I have
```
(datalad-2) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_hufbz7i1>bash -c \"git-annex version\"
bash: git-annex: command not found
```
need to figure that out
```
(datalad-2) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_hufbz7i1>echo %PATH%
C:\Users\DataLad\miniconda3\envs\datalad-2;C:\Users\DataLad\miniconda3\envs\datalad-2\Library\mingw-w64\bin;C:\Users\DataLad\miniconda3\envs\datalad-2\Library\usr\bin;C:\Users\DataLad\miniconda3\envs\datalad-2\Library\bin;C:\Users\DataLad\miniconda3\envs\datalad-2\Scripts;C:\Users\DataLad\miniconda3\envs\datalad-2\bin;C:\Users\DataLad\miniconda3\condabin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\Git\cmd;C:\Users\DataLad\AppData\Local\Microsoft\WindowsApps;.
(datalad-2) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_hufbz7i1>bash -c \"echo $PATH\"
/c/Users/DataLad/miniconda3/envs/datalad-2:/mingw-w64/bin:/usr/bin:/bin:/c/Users/DataLad/miniconda3/envs/datalad-2/Scripts:/c/Users/DataLad/miniconda3/envs/datalad-2/bin:/c/Users/DataLad/miniconda3/condabin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/Program Files/Git/cmd:/c/Users/DataLad/AppData/Local/Microsoft/WindowsApps:.
```
so it seems to discard `C:\Users\DataLad\miniconda3\envs\datalad-2\Library\bin`, may be I should not \"install\" there ;) (but it also looses `conda` itself in that `bash` \"shell\")
"""]]

26
doc/bugs/old__40__er__41___cp_from_coreutils_lacks_--reflink/comment_9_552e36f88bc39bc68b65ccba06002082._comment Normal file
View file

@ -0,0 +1,26 @@
[[!comment format=mdwn
username="yarikoptic"
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
subject="comment 9"
date="2020-10-27T19:30:24Z"
content="""
Working in the same environment as in [the other issue](https://git-annex.branchable.com/bugs/git-annex_looses_itself_on_Windows__63__/?updated). It does come with more recent `cp` (coreutils) 8.25 BUT that one blows with (sorry -- this is debug output of datalad, not direct output from git/annex invocation):
<details>
<summary></summary>
```shell
datalad.cmd: DEBUG: Async run ['git', '-c', 'annex.largefiles=nothing', 'add', '--dry-run', '-N', '--ignore-missing', '--verbose', '--', 'fi le.dat']
datalad.cmd: DEBUG: Launching process ['git', '-c', 'annex.largefiles=nothing', 'add', '--dry-run', '-N', '--ignore-missing', '--verbose', '--', 'fi le.dat']
datalad.cmd: DEBUG: Process 2932 started
datalad.cmd: DEBUG: Waiting for process 2932 to complete
datalad.cmd: DEBUG: Process 2932 exited with return code 0
datalad.cmd: DEBUG: Process file list chunk 0 (length 1)
datalad.annex: DEBUG: Running add resulted in stderr output: cp: preserving permissions for ‘.git\\annex\\objects\\6f0\\fbd\\SHA256E-s7--ed7002b439e9ac845f22357d822bac1444730fbdb6016d3ec9432297b9ec9f73.dat\\SHA256E-s7--ed7002b439e9ac845f22357d822bac1444730fbdb6016d3ec9432297b9ec9f73.dat’: Not supported
git-annex: add: 1 failed
```
</details>
so it echoes your \"-a/-p/--preserve-timestamps are more important and also probed at build time.\" and is yet another gotcha of using different build and run environments. I only wonder how is that our github windows workflow environment supports `cp` with preserving permissions while conda's doesn't
"""]]

67
doc/bugs/view.mdwn Normal file
View file

@ -0,0 +1,67 @@
### Please describe the problem.
git-annex-view fails with an error message that `viewindex.lock` exists and another git process is running in the repository.
The problem appears even with newly initialized repositories.
### What steps will reproduce the problem?
The steps I followed to reproduce the problem:
[[!format sh """
cd /tmp
mkdir testgit
cd testgit
git init
git annex init
echo 123 > test1
echo 456 > test2
git annex add test1 test2
git commit -m 'initial commit'
git annex metadata -s test=test1 test1
git annex metadata -s test=test2 test2
git annex view test=test1
"""]]
Note: The problem does not appear with git-annex version 7.20190819+git2-g908476a9b-1~ndall+1 but it appears with 8.20201007-g903b2f1.
### What version of git-annex are you using? On what operating system?
I could reproduce the problem at two different machines, both with Ubuntu 18.04 LTS.
`git-annex` was installed via conda using the command `conda install -c conda-forge git-annex`
```
git-annex version: 8.20201007-g903b2f1
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
```
### Please provide any additional information below.
Here is the complete error message:
[[!format sh """
view (searching...) fatal: Unable to create '/tmp/mytmp/.git/annex/viewindex.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
git-annex: failed to read sha from git write-tree
CallStack (from HasCallStack):
error, called at ./Git/Sha.hs:23:15 in main:Git.Sha
failed
git-annex: view: 1 failed
"""]]
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
git-annex is an awesome tool! Really appreciate your work and everything else works great!

7
doc/forum/git_annex_view__58___viewindex.lock__44___file_exists___40__it_doesn__39__t__41__.mdwn Normal file
View file

@ -0,0 +1,7 @@
Hi!
I tagged some files with `tag=telegram`, but upon trying to try out views with `git annex view tag=telegram` I get:
https://del.dog/nifumairra.txt
Any ideas? :( The file viewindex.lock doesn't exist..

2
doc/install/OSX/Homebrew.mdwn
View file

@ -1,4 +1,4 @@
[Homebrew](http://brew.sh/) has [a formula](https://github.com/Homebrew/homebrew/commits/master/Library/Formula/git-annex.rb) for git-annex.
[Homebrew](http://brew.sh/) has [a formula](https://formulae.brew.sh/formula/git-annex) for git-annex.
Homebrew users can simply run `brew install git-annex` to install git-annex.

13
doc/install/Windows/comment_14_7b5f0c6b31d9c409ea0989123788bacf._comment Normal file
View file

@ -0,0 +1,13 @@
[[!comment format=mdwn
username="nix.zahlen@1211ac6c964ba2d68b70655f747bef1383032e77"
nickname="nix.zahlen"
avatar="http://cdn.libravatar.org/avatar/66cc45a05749fe8d4ca36d8c6071da51"
subject="where and how to submit Windows autobuild related issues?"
date="2020-10-31T17:44:18Z"
content="""
see <br>
- http://git-annex.branchable.com/news/version_8.20201007/#comment-cc425c38dbd1e3e6ee308bf2d9de1fd1 <br>
- http://git-annex.branchable.com/bugs/windows_dosn__39__t_build_again__63____33__/#comment-90263bc0689187fb5df4d44ce668c08c (bug moved to \"done\"?!? but there is still no updated Windows installer available...) <br>
"""]]

15
doc/install/Windows/comment_15_19226be8cae0c29b0e9a54cebcf1f17b._comment Normal file
View file

@ -0,0 +1,15 @@
[[!comment format=mdwn
username="nix.zahlen@1211ac6c964ba2d68b70655f747bef1383032e77"
nickname="nix.zahlen"
avatar="http://cdn.libravatar.org/avatar/66cc45a05749fe8d4ca36d8c6071da51"
subject="build a CI/CD chain for git-annex"
date="2020-10-31T17:47:15Z"
content="""
would this be: <br>
* possible? <br>
* desired? <br>
* appreciated? <br>
If yes, how and where? Ideas?
"""]]