Merge branch 'master' of ssh://git-annex.branchable.com
This commit is contained in:
Joey Hess
commit
2b3525fbbe
15 changed files with 371 additions and 0 deletions
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="torpidus"
|
||||
avatar="http://cdn.libravatar.org/avatar/997fb77747b008a26383426ae6561368"
|
||||
subject="comment 5"
|
||||
date="2018-09-16T18:40:43Z"
|
||||
content="""
|
||||
This affects me also. git-annex in version of August 8th, 2018.
|
||||
"""]]
|
||||
193
doc/bugs/Remote_does_not_work_with_UTF-8_filenames.mdwn
Normal file
193
doc/bugs/Remote_does_not_work_with_UTF-8_filenames.mdwn
Normal file
|
|
@ -0,0 +1,193 @@
|
|||
### Please describe the problem.
|
||||
|
||||
I'm using git-annex-remote-googledrive to sync a large music library. Some files contain special characters like:
|
||||
chillgressive/\[Astropilot\ Music\]/Unusual\ Cosmic\ Process\ -\ Spacetime/Unusual\ Cosmic\ Process\ -\ Spacetime\ -\ 03\ Souvenirs\ Éphémère.flac
|
||||
|
||||
When running:
|
||||
"git annex sync google --content -v -d"
|
||||
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
The problem seems to be, that git annex sends non utf-8 data to the remote script which crashes due to decoding errors. This seems to be git annex bug as the remote should not have to guess which encoding is used but should be able to use utf-8 always.
|
||||
|
||||
environment:
|
||||
LANG=en_US.UTF-8
|
||||
LANGUAGE=en_US.UTF-8
|
||||
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
nixos
|
||||
|
||||
git-annex version: 6.20180807
|
||||
build flags: Assistant Pairing WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.2 feed-1.0.0.0 ghc-8.4.3 http-client-0.5.13.1 persistent-sqlite-2.8.1.2 torrent-10000.1.1 uuid-1.3.13
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
local repository version: 5
|
||||
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
# If you can, paste a complete transcript of the problem occurring here.
|
||||
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
|
||||
git annex sync google --content -v -d 1.8m Tue 18 Sep 2018 12:29:32 PM CEST
|
||||
[2018-09-18 12:29:41.854290811] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
|
||||
[2018-09-18 12:29:41.856338835] process done ExitSuccess
|
||||
[2018-09-18 12:29:41.85643436] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2018-09-18 12:29:41.858259966] process done ExitSuccess
|
||||
[2018-09-18 12:29:41.859926397] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..97d762166d735b12c5393a21d5d2eeb27b33eb8d","--pretty=%H","-n1"]
|
||||
[2018-09-18 12:29:41.861562154] process done ExitSuccess
|
||||
[2018-09-18 12:29:41.864078093] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
|
||||
[2018-09-18 12:29:41.864393271] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
|
||||
[2018-09-18 12:29:41.866339921] read: git ["config","--null","--list"]
|
||||
[2018-09-18 12:29:41.867406295] process done ExitSuccess
|
||||
[2018-09-18 12:29:41.867536122] read: git ["config","--null","--list"]
|
||||
[2018-09-18 12:29:41.86768986] read: git ["config","--null","--list"]
|
||||
[2018-09-18 12:29:41.86875252] chat: /home/poelzi/.nix-profile/bin/git-annex-remote-googledrive []
|
||||
[2018-09-18 12:29:42.10211189] git-annex-remote-googledrive[1] --> VERSION 1
|
||||
[2018-09-18 12:29:42.102224585] git-annex-remote-googledrive[1] <-- EXTENSIONS INFO
|
||||
[2018-09-18 12:29:42.10234806] git-annex-remote-googledrive[1] --> EXTENSIONS
|
||||
[2018-09-18 12:29:42.10240219] git-annex-remote-googledrive[1] <-- EXPORTSUPPORTED
|
||||
[2018-09-18 12:29:42.102579476] git-annex-remote-googledrive[1] --> EXPORTSUPPORTED-SUCCESS
|
||||
commit
|
||||
[2018-09-18 12:29:42.106079317] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit","-a","-m","git-annex in poelzi@galaxy:/run/media/poelzi/backup-1/Music"]
|
||||
On branch master
|
||||
Your branch is ahead of 'origin/master' by 3 commits.
|
||||
(use "git push" to publish your local commits)
|
||||
|
||||
nothing to commit, working tree clean
|
||||
[2018-09-18 12:29:42.325422783] process done ExitFailure 1
|
||||
ok
|
||||
[2018-09-18 12:29:42.325517478] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
|
||||
[2018-09-18 12:29:42.326719154] process done ExitSuccess
|
||||
[2018-09-18 12:29:42.326780286] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
|
||||
[2018-09-18 12:29:42.328496909] process done ExitSuccess
|
||||
[2018-09-18 12:29:42.328549028] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/heads/synced/master"]
|
||||
[2018-09-18 12:29:42.329769099] process done ExitSuccess
|
||||
[2018-09-18 12:29:42.329841385] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/heads/synced/master","--pretty=%H","-n1"]
|
||||
[2018-09-18 12:29:42.33163463] process done ExitSuccess
|
||||
[2018-09-18 12:29:42.331711233] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
|
||||
[2018-09-18 12:29:42.333577663] process done ExitSuccess
|
||||
[2018-09-18 12:29:42.333638691] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2018-09-18 12:29:42.335399941] process done ExitSuccess
|
||||
[2018-09-18 12:29:42.337137236] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..97d762166d735b12c5393a21d5d2eeb27b33eb8d","--pretty=%H","-n1"]
|
||||
[2018-09-18 12:29:42.338812954] process done ExitSuccess
|
||||
[2018-09-18 12:29:42.338954942] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--"]
|
||||
[2018-09-18 12:29:42.349381649] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
|
||||
[2018-09-18 12:29:42.349788044] read: git ["--version"]
|
||||
[2018-09-18 12:29:42.351035475] process done ExitSuccess
|
||||
[2018-09-18 12:30:46.421010356] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","rev-parse","master:"]
|
||||
[2018-09-18 12:30:46.423208056] process done ExitSuccess
|
||||
[2018-09-18 12:30:46.426223794] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2018-09-18 12:30:46.428877128] process done ExitSuccess
|
||||
[2018-09-18 12:30:46.428974324] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-tree","--full-tree","-z","-r","-t","--","97d762166d735b12c5393a21d5d2eeb27b33eb8d"]
|
||||
[2018-09-18 12:30:50.326843882] process done ExitSuccess
|
||||
[2018-09-18 12:30:50.326943526] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","mktree","--batch","-z"]
|
||||
[2018-09-18 12:30:50.357438031] process done ExitSuccess
|
||||
[2018-09-18 12:30:50.357526145] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit-tree","ee08aadd5604249699ce157df47426c70f62312b","--no-gpg-sign","-p","97d762166d735b12c5393a21d5d2eeb27b33eb8d"]
|
||||
[2018-09-18 12:30:50.359994305] process done ExitSuccess
|
||||
[2018-09-18 12:30:50.360058172] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","mktree","--batch","-z"]
|
||||
[2018-09-18 12:30:50.383728957] process done ExitSuccess
|
||||
[2018-09-18 12:30:50.383984605] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit-tree","acf111be942ccf1cfda391d7ecffc4a9c6a27d6f","--no-gpg-sign","-p","46bf4e6b14c01c587d5eb1be2db75c66479b6e15"]
|
||||
[2018-09-18 12:30:50.388013862] process done ExitSuccess
|
||||
[2018-09-18 12:30:50.388101083] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","update-ref","refs/heads/git-annex","125e9dfe8129591025117cc9dc0c752265be7e70"]
|
||||
[2018-09-18 12:30:50.391058496] process done ExitSuccess
|
||||
[2018-09-18 12:30:50.391473142] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff-tree","-z","--raw","--no-renames","-l0","-r","9557efae9721f725d73f2b2776a2070aad7b0a39","9557efae9721f725d73f2b2776a2070aad7b0a39","--"]
|
||||
[2018-09-18 12:30:50.402405642] process done ExitSuccess
|
||||
[2018-09-18 12:30:50.402521039] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff-tree","-z","--raw","--no-renames","-l0","-r","33bf55cdf1f58109ee31097308693f833c4cca97","9557efae9721f725d73f2b2776a2070aad7b0a39","--"]
|
||||
[2018-09-18 12:30:50.415691771] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
|
||||
[2018-09-18 12:30:50.41669191] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
|
||||
[2018-09-18 12:30:50.4222699] process done ExitSuccess
|
||||
rename google chillgressive/[Astropilot Music]/Unusual Cosmic Process - Spacetime/Unusual Cosmic Process - Spacetime - 03 Souvenirs <20>ph<70>m<EFBFBD>re.flac -> .git-annex-tmp-content-SHA256E-s97469487--14f5f5a819093a154f8d660ae1d986332108eb556a9c27dfd36d8f8dd90d30ae.flac [2018-09-18 12:30:50.422844753] git-annex-remote-googledrive[1] <-- PREPARE
|
||||
[2018-09-18 12:30:50.423065052] git-annex-remote-googledrive[1] --> DEBUG Running .git-annex-remote-googledrive-wrapped version 0.11.1
|
||||
[2018-09-18 12:30:50.423111151] Running .git-annex-remote-googledrive-wrapped version 0.11.1
|
||||
[2018-09-18 12:30:50.423145446] git-annex-remote-googledrive[1] --> DEBUG Using AnnexRemote version 1.2.0
|
||||
[2018-09-18 12:30:50.423239295] Using AnnexRemote version 1.2.0
|
||||
[2018-09-18 12:30:50.423278034] git-annex-remote-googledrive[1] --> GETCONFIG prefix
|
||||
[2018-09-18 12:30:50.423319342] git-annex-remote-googledrive[1] <-- VALUE Music
|
||||
[2018-09-18 12:30:50.423424438] git-annex-remote-googledrive[1] --> GETCONFIG root_id
|
||||
[2018-09-18 12:30:50.423486989] git-annex-remote-googledrive[1] <-- VALUE 1vnnTRDDMJWqTjm5hc0BGzxhCW4S7-j7b
|
||||
[2018-09-18 12:30:50.42358159] git-annex-remote-googledrive[1] --> GETCREDS credentials
|
||||
[2018-09-18 12:30:50.423727488] git-annex-remote-googledrive[1] <-- CREDS {"access_token":"ya...8Q","client_id":"...","client_secret":"...","refresh_token":"..."}
|
||||
[2018-09-18 12:30:51.007447341] git-annex-remote-googledrive[1] --> SETCREDS credentials {"access_token":...}
|
||||
[2018-09-18 12:30:51.348855667] git-annex-remote-googledrive[1] --> PREPARE-SUCCESS
|
||||
[2018-09-18 12:30:51.348957671] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Spacetime/Unusual Cosmic Process - Spacetime - 03 Souvenirs <20>ph<70>m<EFBFBD>re.flac
|
||||
[2018-09-18 12:30:51.34902213] git-annex-remote-googledrive[1] <-- RENAMEEXPORT SHA256E-s97469487--14f5f5a819093a154f8d660ae1d986332108eb556a9c27dfd36d8f8dd90d30ae.flac .git-annex-tmp-content-SHA256E-s97469487--14f5f5a819093a154f8d660ae1d986332108eb556a9c27dfd36d8f8dd90d30ae.flac
|
||||
Traceback (most recent call last):
|
||||
File "/nix/store/478kqwy2yfrx61swlsnhjajna7sl8lhh-git-annex-remote-googledrive-0.11.1/bin/.git-annex-remote-googledrive-wrapped", line 584, in <module>
|
||||
main()
|
||||
File "/nix/store/478kqwy2yfrx61swlsnhjajna7sl8lhh-git-annex-remote-googledrive-0.11.1/bin/.git-annex-remote-googledrive-wrapped", line 580, in main
|
||||
master.Listen()
|
||||
File "/nix/store/5wncs360nzq0fgqmv30p8qn0cga8zi0d-python3.6-annexremote-1.2.0/lib/python3.6/site-packages/annexremote/annexremote.py", line 394, in Listen
|
||||
for line in self.input:
|
||||
File "/nix/store/hy65mn4wjswqih75gfr6g4q3xgqdm325-python3-3.6.6/lib/python3.6/codecs.py", line 321, in decode
|
||||
(result, consumed) = self._buffer_decode(data, self.errors, final)
|
||||
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 125: invalid continuation byte
|
||||
git-annex: external special remote protocol error, unexpectedly received "" (unable to parse command)
|
||||
|
||||
rename failed; deleting instead
|
||||
[2018-09-18 12:30:51.384823525] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Spacetime/Unusual Cosmic Process - Spacetime - 03 Souvenirs <20>ph<70>m<EFBFBD>re.flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
failed
|
||||
rename google chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 04 Souvenirs <20>ph<70>m<EFBFBD>re (Trance Version).flac -> .git-annex-tmp-content-SHA256E-s90158047--578d406e3db07835d3c4045096c7f8333c5adbbfe1003d37121d5b0656f45688.flac [2018-09-18 12:30:51.385017728] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 04 Souvenirs <20>ph<70>m<EFBFBD>re (Trance Version).flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
|
||||
rename failed; deleting instead
|
||||
[2018-09-18 12:30:51.385128278] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 04 Souvenirs <20>ph<70>m<EFBFBD>re (Trance Version).flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
failed
|
||||
rename google chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 01 Artem<65>sia.flac -> .git-annex-tmp-content-SHA256E-s41053932--7bb9b80766e1d445bb39442e39e3e727ba7329d00186cccb5b2b1e2b28b493b1.flac [2018-09-18 12:30:51.38521185] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 01 Artem<65>sia.flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
|
||||
rename failed; deleting instead
|
||||
[2018-09-18 12:30:51.385281886] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 01 Artem<65>sia.flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
failed
|
||||
rename google psychill/[Badger Records]/Badgers Records - VA - Inner Holographic Reality/Badgers Records - VA - Inner Holographic Reality - 01 Cord & Frankie L<>t<EFBFBD> - Lullaby.flac -> .git-annex-tmp-content-SHA256E-s43901186--9d51034d81247e3e0890d81c1e1c20c06149aac3bf4451d880523dcc89c3de82.flac [2018-09-18 12:30:51.385360183] git-annex-remote-googledrive[1] <-- EXPORT psychill/[Badger Records]/Badgers Records - VA - Inner Holographic Reality/Badgers Records - VA - Inner Holographic Reality - 01 Cord & Frankie L<>t<EFBFBD> - Lullaby.flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
|
||||
rename failed; deleting instead
|
||||
[2018-09-18 12:30:51.385429424] git-annex-remote-googledrive[1] <-- EXPORT psychill/[Badger Records]/Badgers Records - VA - Inner Holographic Reality/Badgers Records - VA - Inner Holographic Reality - 01 Cord & Frankie L<>t<EFBFBD> - Lullaby.flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
failed
|
||||
rename google chillgressive/[Astropilot Music]/Skytechnic - Astral Insıgnias/Skytechnic - Astral Ins?gnias - 08 Vac<61>o.flac -> .git-annex-tmp-content-SHA256E-s89899720--afbc07b561f058db6da5849f728d665d14cc6ea42e2768fecf6396d0009e2861.flac [2018-09-18 12:30:51.385513597] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Skytechnic - Astral Insıgnias/Skytechnic - Astral Ins?gnias - 08 Vac<61>o.flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
|
||||
rename failed; deleting instead
|
||||
[2018-09-18 12:30:51.385594622] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Skytechnic - Astral Insıgnias/Skytechnic - Astral Ins?gnias - 08 Vac<61>o.flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
failed
|
||||
rename google chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 03 Souvenirs <20>ph<70>m<EFBFBD>re (Ambient Version).flac -> .git-annex-tmp-content-SHA256E-s81128227--c4222d79979c8562c44847869c20d172f51a8d4670192d860aa2dae1ba3d54c8.flac [2018-09-18 12:30:51.385768001] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 03 Souvenirs <20>ph<70>m<EFBFBD>re (Ambient Version).flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
|
||||
rename failed; deleting instead
|
||||
[2018-09-18 12:30:51.385830411] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Unusual Cosmic Process - Elysium/Unusual Cosmic Process - Elysium - 03 Souvenirs <20>ph<70>m<EFBFBD>re (Ambient Version).flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
failed
|
||||
[2018-09-18 12:30:51.38631869] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-tree","--full-tree","-z","-r","--","9557efae9721f725d73f2b2776a2070aad7b0a39"]
|
||||
export google .gitignore (not available) failed
|
||||
export google bin/bandcamp-rip-album.variant-5bf5 (not available) failed
|
||||
export google bin/flac2opus.variant-1201.py (not available) failed
|
||||
export google chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 01 Artemísia.flac
|
||||
[2018-09-18 12:30:59.661174242] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 01 Artemísia.flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
failed
|
||||
export google chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 02 Marwell.flac
|
||||
[2018-09-18 12:30:59.663165427] git-annex-remote-googledrive[1] <-- EXPORT chillgressive/[Astropilot Music]/Dream Twice - Enta/Dream Twice - Enta - 02 Marwell.flac
|
||||
git-annex: fd:16: hFlush: resource vanished (Broken pipe)
|
||||
failed
|
||||
|
||||
|
||||
# End of transcript or log.
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
I love git annex, using it very day :)
|
||||
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
[[!comment format=mdwn
|
||||
username="duncan_bayne"
|
||||
avatar="http://cdn.libravatar.org/avatar/2fc23e2009234ad965f9d5d796400417"
|
||||
subject="Bug raised"
|
||||
date="2018-09-16T22:15:23Z"
|
||||
content="""
|
||||
Hi Joey,
|
||||
|
||||
I've raised a bug ( https://git-annex.branchable.com/bugs/Can__39__t_compile_on_FreeBSD__58____Issues_with_System.Posix.Files/ ) as requested. I also happen to have a machine sitting around at home that I can configure for SSH access, if you or anyone would like a current (FreeBSD 11.1) dev environment to use. Let me know.
|
||||
|
||||
Yours,
|
||||
Duncan
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
### Please describe the problem.
|
||||
|
||||
I quickly searched but could not find exactly the same issue, so filing it new, but feel welcome to refer/close if it is already filed.
|
||||
|
||||
Upon some transfers between local hosts, I am getting some files failing to be transferred upon initial run, and I need to rerun `get` to get them. Could may be `annex get` internally retry a few times (I think the same is done for http already) upon transfer failure?
|
||||
|
||||
[[!format sh """
|
||||
|
||||
(git)smaug:/mnt/btrfs/datasets/datalad/crawl/dbic/QA[master]git
|
||||
$> git annex get -J4 --from=origin --all --json 2>&1 | tee /tmp/qa-get.log
|
||||
{"command":"get","note":"from origin...\nchecksum...","success":true,"key":"SHA256E-s9037046--92b3f24faf39b0c3710311fa57a614bdc24f998e5a4f7c8b33063d4ef01a0aaf.nii.gz","file":null}
|
||||
...
|
||||
git-annex: get: 1 failed
|
||||
git annex get -J4 --from=origin --all --json 2>&1 87.01s user 22.58s system 73% cpu 2:29.82 total
|
||||
tee /tmp/qa-get.log 0.00s user 0.02s system 0% cpu 2:29.82 total
|
||||
|
||||
$> grep -v -e 'success":true' /tmp/qa-get.log
|
||||
{"command":"get","note":"from origin...\nrsync failed -- run git annex again to resume file transfer","success":false,"key":"SHA256E-s44080906--a534bdaa5b4143fb64135aa83d86cf4d4f29d2c23b7f2e7e1d72a811835f2bb8.nii.gz","file":null}
|
||||
git-annex: get: 1 failed
|
||||
|
||||
$> git annex get --key SHA256E-s44080906--a534bdaa5b4143fb64135aa83d86cf4d4f29d2c23b7f2e7e1d72a811835f2bb8.nii.gz --from origin
|
||||
get SHA256E-s44080906--a534bdaa5b4143fb64135aa83d86cf4d4f29d2c23b7f2e7e1d72a811835f2bb8.nii.gz (from origin...)
|
||||
(checksum...) ok
|
||||
(recording state in git...)
|
||||
|
||||
$> git annex version
|
||||
git-annex version: 6.20180807+git291-g549114118-1~ndall+1
|
||||
|
||||
"""]]
|
||||
|
||||
|
||||
[[!meta author=yoh]]
|
||||
|
||||
14
doc/forum/Please_help__44___half_my_files_disappeared.mdwn
Normal file
14
doc/forum/Please_help__44___half_my_files_disappeared.mdwn
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
Hello, I was writing a message to ask for help but I found a solution for my problem so I deleted it but now I think it could be useful to someone else so hereafter is my initial message and my workaround.
|
||||
|
||||
|
||||
> Hello, this time I am very very frightened by the odds but I am still looking for your help. Long story short: windows update.
|
||||
|
||||
> Yesterday, everything was fine with my local git-annex using WSL. This morning, windows had updated itself to version 1803 automatically, (by the way, thanks to Microsoft for their forced updates), and half my locked files are gone. I tried git annex fix ., nothing, I tried to find the associated keys with find .git/annex/objects -name 'sha...', nothing. git annex whereis <file> says the file is here whereas git annex fsck <file> fails telling the file's content is missing.
|
||||
|
||||
> Now, it seems like the only remaining files have links pointing to annex/objects directories that are exclusively composed of uppercase letters or numbers. Looking more closely to the objects folders, it seems like they are all strangely duplicated depending on the spelling: for example the variations zZ, Zz, ZZ, zz which all have the same subdirectories and files. Is this the way those directories should be named or did Windows break everything?
|
||||
|
||||
> The last thing that keeps me hoping is that the hard drive usage seems to be the same as before but with the apparent duplication of object directories, my hopes are fading away pretty fast... If anyone can help me, or even finish to turn my hopes down, I am listening.
|
||||
|
||||
I solved this by using the shiny feature that lets a Windows user revert the OS to previous version once yet another a big flaw has been found in the 1803 version.
|
||||
Strangely, after the revert, all my files were back to their normal state, as I left them before the upgrade.
|
||||
I hope none of you will need this ;)
|
||||
11
doc/forum/Some_files_missing_extension_in_annex.mdwn
Normal file
11
doc/forum/Some_files_missing_extension_in_annex.mdwn
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
I hadn't noticed this before, but I added some files in the past and, finally looking at them today, the file extension is missing in the annex. This confuses OS X something awful, so it opens the file in TextEdit instead of VLC. If I open the file directly using VLC, it works.
|
||||
|
||||
I just dropped and re-added the file that has an extension. It lacked it before:
|
||||
|
||||
ls -l 10*
|
||||
lrwxr-xr-x 1 jasonb staff 215 Sep 16 15:23 10 Minute Ab Chisel.mkv -> ../.git/annex/objects/zw/mG/WORM-s103155193-m1516830000--Masters,32Hammer,32and,32Chisel-a67ed6d2a3dfd5fe29be62146fbd0c04/WORM-s103155193-m1516830000--Masters,32Hammer,32and,32Chisel-a67ed6d2a3dfd5fe29be62146fbd0c04
|
||||
lrwxr-xr-x 1 jasonb staff 209 Sep 16 15:04 10 Minute Ab Hammer.mkv -> ../.git/annex/objects/jx/5p/WORM-s117692368-m1537124682--Masters,32Hammer,32and,32Chisel%10,32Minute,32Ab,32Hammer.mkv/WORM-s117692368-m1537124682--Masters,32Hammer,32and,32Chisel%10,32Minute,32Ab,32Hammer.mkv
|
||||
|
||||
Is there something I can do to recover my file extensions besides manually dropping and re-adding each file?
|
||||
|
||||
Thanks.
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="jasonb@ab4484d9961a46440958fa1a528e0fc435599057"
|
||||
nickname="jasonb"
|
||||
avatar="http://cdn.libravatar.org/avatar/c7330f4da122c671b935fc1d58bb02b1"
|
||||
subject="comment 1"
|
||||
date="2018-09-16T20:25:39Z"
|
||||
content="""
|
||||
Doing a git annex unannex and then a git annex add fixes it. Now sure how this happened originally.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="jasonb@ab4484d9961a46440958fa1a528e0fc435599057"
|
||||
nickname="jasonb"
|
||||
avatar="http://cdn.libravatar.org/avatar/c7330f4da122c671b935fc1d58bb02b1"
|
||||
subject="comment 2"
|
||||
date="2018-09-16T20:32:08Z"
|
||||
content="""
|
||||
Unfortunately, my remotes don't know anything about the change, so when I push it, the symlinks point to files that don't exist, even though the data is still in the annex with the original mysterious filenames.
|
||||
|
||||
How to get git-annex to recover the existing data?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="torpidus"
|
||||
avatar="http://cdn.libravatar.org/avatar/997fb77747b008a26383426ae6561368"
|
||||
subject="comment 3"
|
||||
date="2018-09-16T18:45:15Z"
|
||||
content="""
|
||||
Thanks Joey,
|
||||
after your comment, I decided to migrate the repo's contents to a new one in v6 format. However, I run into several other issues. I wished there was a live support that I can pay with coffee or beer... :-) Instead, I try to figure around the issues by myself, hopefully not spamming too much here into this forum. :-(
|
||||
|
||||
BTW, thank you so much for the effort you have put into git-annex!
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
I can't find out why, but under certain circumstances the symlinks don't work in my environment. On all hosts I have latest Ubuntu Linux and latest git-annex. V6 repos. Issue is, that instead of real symlinks, the filenames remain the original ones and the content is the relative link to the corresponding annex object.
|
||||
|
||||
Strange behavior is, I can drop and get the files but it doesn't seem to affect the situation after all.
|
||||
|
||||
Comparing the size of .git/*, it is very likely that the contents are present and so does "git annex whereis bla.jpg". Leading me to the conclusion that everything works except the symlinking. Is this a known issue? What steps can I perform to find out the root cause?
|
||||
|
||||
Thank you.
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="mario"
|
||||
avatar="http://cdn.libravatar.org/avatar/4c63b0935789d29210d0bd8cad8d7ac7"
|
||||
subject="3rd party "mover" / speed up git annex move"
|
||||
date="2018-09-17T12:16:08Z"
|
||||
content="""
|
||||
git annex move seems to be quite slow if many small files have to be transferred. I think the main issue is that for each file a new TCP connection is opened. In this case reusing the TCP connection could significantly speed up the transfers. Two questions about this. a) Am I just doing something wrong? (I don't want to use -J because the remote is cluster and -J connects to different servers.) b) Is there an efficient method to write my own mover for the files in question? Basically \"git annex find [...] --json\" gives me all the files in question, so I know what to move. But after copying them to the remote, how do I tell git annex that they are there? Does it make sense to use git annex reinject?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
[A Debian user reports](https://bugs.debian.org/909023) that 6.20180913 cannot be used with a server running the version of git-annex in Debian stretch. The error is:
|
||||
|
||||
fd:23: hClose: resource vanished (Broken pipe)
|
||||
|
||||
Presumably this sudden lack of backwards incompatibility is unintentional.
|
||||
|
||||
--spwhitton
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="comment 3"
|
||||
date="2018-09-18T18:41:01Z"
|
||||
content="""
|
||||
The main reason for wanting git-annex to follow symlinks, is that the semantics of its commands (get/add/copy) operating on directories would be much more intuitive.
|
||||
I want to know that after 'git annex get subdir' all annexed files accessible under subdir/ are available; that after 'git annex add subdir && git annex move subdir --to my-remote' all
|
||||
large files accessible under subdir/ are at my-remote; etc. I want to be able to treat a subdirectory as a self-contained unit. But this isn't possible if relative symlinks stored
|
||||
undir subdir/ but pointing outside subdir/ might still be broken after 'git annex get subdir', etc.
|
||||
|
||||
E.g. I have
|
||||
|
||||
proj1/
|
||||
big_file.dat: symlink to ../.git/annex/objects/....
|
||||
proj2/
|
||||
big_file.dat: symlink to ../proj1/big_file.dat
|
||||
then if I do 'git annex get proj1', I can safely work on proj1 knowing all its files have been fetched; but if I do 'git annex get proj2', I can't safely work on it because
|
||||
the symlink proj2/big_file.dat is still broken. I can of course make proje2/big_file.dat a direct link to the annex, but that loses the relationship between proj1 and proj2.
|
||||
It's pretty common to want to create a \"variant\" of a project by making most files in proj2/ symlinks to corresponding files in proj1/ , except for a few files that differ.
|
||||
|
||||
With 'git annex add', it'd help a lot to know that 'git annex add /my/dir' will definitely store _everything_ under /my/dir. In fact, if a symlink under /my/dir points outside the
|
||||
repository, git-annex could still store the target file in the annex and check in a symlink to that (or perhaps warn the user about the situation). Then I can 'git annex add' my local
|
||||
setup, even if it points to some absolute paths, and check it out on another machine, with all links working.
|
||||
|
||||
\"Supporting this would require that git-annex stat every file that non-git-annex symlinks point to\" -- only ones that point outside the subdir being worked on. If the target of
|
||||
such a symlink is a directory, you'd need to process that directory too. But if this all is off by default, and turned on by a flag, then the normal operation won't be affected.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="comment 4"
|
||||
date="2018-09-18T18:48:06Z"
|
||||
content="""
|
||||
p.s. you're right that symlinks to outside the repository would be a security risk. They should not be added to the annex without explicit options. But it's important to at least warn about them, so e.g. the user knows that 'git annex add' did not add everything, or that 'git annex get subdir/' did not get everything under subdir/ . It _would_ be useful to handle relative symlinks into submodules though.
|
||||
Symlink loops would of course be errors.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://christian.amsuess.com/chrysn"
|
||||
nickname="chrysn"
|
||||
avatar="http://christian.amsuess.com/avatar/c6c0d57d63ac88f3541522c4b21198c3c7169a665a2f2d733b4f78670322ffdc"
|
||||
subject="symlinks into git-annex"
|
||||
date="2018-09-18T19:15:11Z"
|
||||
content="""
|
||||
I often work with scenarios where I want to `git annex get` files where I'm not actually in a git annex repository but in a symlink farm that fans out (often via directory-level indirections) into git annices (is there a canonical plural?). There, what I use is a short script that `readlink`s the to-be-fetched files until it contains a `.git/annex/objects` part and then fetches that in its original annex. If there were a way to do that with a standard `git annex get`, possibly (if after some deliberation there remain security concerns) behind a --follow-symlinks switch, I'd appreciate that.
|
||||
"""]]
|
||||
Loading…
Reference in a new issue