Merge branch 'master' into no-xmpp
This commit is contained in:
Joey Hess
commit
ab66bbfeb6
377 changed files with 7442 additions and 875 deletions
|
|
@ -1,3 +1,5 @@
|
|||
Having a windows build of Git-Annex in an archived format would be very usefull for automation, and deploy.
|
||||
Could it be possible to add this to the buildserver of gitannex?
|
||||
|
||||
[[!tag moreinfo]]
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,19 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-11-16T18:41:25Z"
|
||||
content="""
|
||||
It would be helpful to have more details, such as an example of software
|
||||
distributed for windows that way, or documentation of how such an archive
|
||||
is used on windows.
|
||||
|
||||
The git-annex Windows installer is a exe file that uses the NullSoft
|
||||
installation system. As far as I know that's pretty common in the Windows
|
||||
world.
|
||||
|
||||
I don't see any point in zipping up the single exe. It would be possible to
|
||||
make a zip containing all the files that instlling the exe installs. But,
|
||||
the installation process has to integrate git-annex with git, it installs
|
||||
menu items, etc. A zip file would not be able to handle that integration.
|
||||
So its use seems limited to me.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
[[!comment format=mdwn
|
||||
username="luckcolorsgoo@ab4f3c1c44a7dbcbcb9d9a29315b59ad524ceaaa"
|
||||
nickname="luckcolorsgoo"
|
||||
avatar="http://cdn.libravatar.org/avatar/ddff84cd2a97252a05cccb4bc5010448"
|
||||
subject="comment 2"
|
||||
date="2016-11-16T22:56:46Z"
|
||||
content="""
|
||||
In my case i was going to make a script for automatically downloading and updating an git portbale / git annex instance, by first fetching git portbale and then downloading the gitannex exe.
|
||||
|
||||
So yeah it's more reliable to extract an archive rather than trying to extract the setup without executing it.
|
||||
That's why i'm asking for this feature. :)
|
||||
|
||||
|
||||
|
||||
"""]]
|
||||
|
|
@ -59,5 +59,3 @@ SHA256E-s41311329--69c3b054a3fe9676133605730d85b7fcef8696f6782d402a524e41b836253
|
|||
|
||||
|
||||
[[!meta title="Detect stalled transfer and retry or abort it"]]
|
||||
|
||||
> [[done]] --[[Joey]]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,16 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 4"""
|
||||
date="2016-12-13T16:05:42Z"
|
||||
content="""
|
||||
Could the original bug reporter please show what your ~/.ssh/config
|
||||
contains? As far as I can tell, ssh's TCPKeepAlive option, which is
|
||||
supposed to be enabled by default, unless you have disabled it, should
|
||||
avoid such problems.
|
||||
|
||||
It may also help to set ServerAliveInterval.
|
||||
|
||||
Unfortunately, my attempt to make git-annex set ServerAliveInterval
|
||||
when running ssh broke too many systems with old sshed, and I have had to
|
||||
revert it.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
[[!comment format=mdwn
|
||||
username="boh"
|
||||
avatar="http://cdn.libravatar.org/avatar/e7fa2d1c5d95e323fe48887f7f827b1f"
|
||||
subject="comment 9"
|
||||
date="2016-11-27T12:23:20Z"
|
||||
content="""
|
||||
Seems as if the problem still exists in 6.20161118 (Debian).
|
||||
|
||||
I have three repositories (among others), `jolla`, `sts-3xx`, and `here`. `jolla` and `here` are in group `manual`, `sts-3xx` is `backup`; `here` and `sts-3xx` have assistants running, `jolla` not. `jolla` and `sts-3xx` have slightly older versions of git-annex installed.
|
||||
|
||||
Now, when I copy a file from `here` to `jolla` like this
|
||||
|
||||
git annex copy real_programmers.png -t jolla
|
||||
|
||||
the file is subsequently dropped by the assistant:
|
||||
|
||||
```
|
||||
drop real_programmers.png (locking jolla...) [2016-11-27 13:00:02.667376556] chat: ssh [\"-S\",\".git/annex/ssh/jolla\",\"-o\",\"ControlMaster
|
||||
=auto\",\"-o\",\"ControlPersist=yes\",\"-F\",\".git/annex/ssh.config\",\"-T\",\"jolla\",\"git-annex-shell 'lockcontent' '/~/Music/media/' '--debug' '
|
||||
SHA256E-s84499--ff98a733cc0122858fb11433c720e2d038fec190a3d36380d0e7e8dab468f883.png' --uuid 5298e3ce-1106-4d5e-b052-0aee4b27a344\"]
|
||||
(locking sts-3xx...) [2016-11-27 13:00:03.252473676] chat: ssh [..., \"git-annex-shell 'lockcontent' '/backups/exot/media/' '--debug' 'SHA256E-s84499--ff98a733cc0122858fb11433c720e2d038fec190a3d 36380d0e7e8dab468f883.png' --uuid 1fec6253-171d-4f86-885b-e233be2d65ec\"]
|
||||
(lockcontent failed) [2016-11-27 13:00:03.486158016] process done ExitFailure 1
|
||||
(checking sts-3xx...) [2016-11-27 13:00:03.487047149] call: ssh [..., \"git-annex-shell 'inannex' '/backups/exot/media/' '--debug' 'SHA256E-s84499--ff98a733cc0122858fb11433c720e2d038fec190a3d363 80d0e7e8dab468f883.png' --uuid 1fec6253-171d-4f86-885b-e233be2d65ec\"]
|
||||
[2016-11-27 13:00:03.76435136] process done ExitSuccess
|
||||
[2016-11-27 13:00:03.764705754] Dropping from here proof: Just (SafeDropProof (NumCopies 2) [RecentlyVerifiedCopy UUID \"1fec6253-171d-4 f86-885b-e233be2d65ec\",LockedCopy UUID \"5298e3ce-1106-4d5e-b052-0aee4b27a344\"] (Just (ContentRemovalLock (Key {keyName = \"ff98a733cc012 2858fb11433c720e2d038fec190a3d36380d0e7e8dab468f883.png\", keyBackendName = \"SHA256E\", keySize = Just 84499, keyMtime = Nothing, keyChun kSize = Nothing, keyChunkNum = Nothing}))))
|
||||
[2016-11-27 13:00:04.24333081] process done ExitFailure 1
|
||||
ok
|
||||
[2016-11-27 13:00:04.251232455] dropped real_programmers.png (from here) (copies now 4) : drop wanted after Upload UUID \"5298e3ce-1106- 4d5e-b052-0aee4b27a344\" real_programmers.png Just 84499
|
||||
```
|
||||
|
||||
However, I failed to reproduce the problem by replicating my setup with fresh repositories …
|
||||
|
||||
Please let me know if you need more information, and *so* many thanks for git-annex!
|
||||
"""]]
|
||||
49
doc/bugs/Build_with_aws_head_fails.mdwn
Normal file
49
doc/bugs/Build_with_aws_head_fails.mdwn
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
### Please describe the problem.
|
||||
https://github.com/aristidb/aws/issues/206 was recently resolved in https://github.com/aristidb/aws/pull/213.
|
||||
|
||||
A newer version will be tagged imminently according to https://github.com/aristidb/aws/issues/206#issuecomment-260214736.
|
||||
|
||||
With the http-conduit (<2.2.0) constraint removed from git-annex.cabal, and the aws dependency set to use aws head (currently c8806dc), the git-annex build fails.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
Remove the http-conduit (<2.2.0) constraint and attempt to build git-annex with aws head.
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
macOS 10.11, git-annex 6.20161118.
|
||||
|
||||
### Please provide any additional information below.
|
||||
Full build log: https://gist.github.com/ilovezfs/15bcd8f1086b3d825beff58140e04eec
|
||||
[[!format sh """
|
||||
[ 90 of 542] Compiling Types.Crypto ( Types/Crypto.hs, dist/dist-sandbox-6b15e8f0/build/git-annex/git-annex-tmp/Types/Crypto.o )
|
||||
[ 91 of 542] Compiling Utility.Metered ( Utility/Metered.hs, dist/dist-sandbox-6b15e8f0/build/git-annex/git-annex-tmp/Utility/Metered.o )
|
||||
[ 92 of 542] Compiling Messages.JSON ( Messages/JSON.hs, dist/dist-sandbox-6b15e8f0/build/git-annex/git-annex-tmp/Messages/JSON.o )
|
||||
[ 93 of 542] Compiling Utility.Url ( Utility/Url.hs, dist/dist-sandbox-6b15e8f0/build/git-annex/git-annex-tmp/Utility/Url.o )
|
||||
|
||||
Utility/Url.hs:354:34: error:
|
||||
• The constructor ‘StatusCodeException’ should have 2 arguments, but has been given 3
|
||||
• In the pattern: StatusCodeException s _ _
|
||||
In an equation for ‘matchStatusCodeException’:
|
||||
matchStatusCodeException want e@(StatusCodeException s _ _)
|
||||
| want s = Just e
|
||||
| otherwise = Nothing
|
||||
|
||||
Utility/Url.hs:354:34: error:
|
||||
• Couldn't match expected type ‘HttpException’
|
||||
with actual type ‘HttpExceptionContent’
|
||||
• In the pattern: StatusCodeException s _ _
|
||||
In an equation for ‘matchStatusCodeException’:
|
||||
matchStatusCodeException want e@(StatusCodeException s _ _)
|
||||
| want s = Just e
|
||||
| otherwise = Nothing
|
||||
cabal: Leaving directory '.'
|
||||
cabal: Error: some packages failed to install:
|
||||
git-annex-6.20161118 failed during the building phase. The exception was:
|
||||
ExitFailure 1
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
Yes :)
|
||||
|
||||
> [[done]] via the nice patch! --[[Joey]]
|
||||
|
|
@ -0,0 +1,149 @@
|
|||
[[!comment format=mdwn
|
||||
username="alpernebbi"
|
||||
avatar="http://cdn.libravatar.org/avatar/daf2abb14f39e28ad75d5f9a03fcd106"
|
||||
subject="Patch to fix aws head build issue"
|
||||
date="2016-12-10T13:08:58Z"
|
||||
content="""
|
||||
I think I fixed this. I'm attaching the output of `git format-patch origin/master`.
|
||||
|
||||
In an Arch Linux chroot with their [PKGBUILD](https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/git-annex) (with a small modification to apply the patch), `haskell-http-client 0.5.3.3-1` and `http-conduit 2.2.3-5` both the build and the tests are successful.
|
||||
It's also successful in a Debian Sid chroot, where `sudo apt build-dep git-annex` gives me `libghc-http-client-dev 0.4.31.1-3+b2`, `libghc-http-conduit-dev 2.1.11-3+b2`.
|
||||
|
||||
### Patch
|
||||
|
||||
[[!format patch \"\"\"
|
||||
From 2ce09420aa8f3d916c6562abea4ed8911a186902 Mon Sep 17 00:00:00 2001
|
||||
From: Alper Nebi Yasak <alpernebiyasak@gmail.com>
|
||||
Date: Sat, 10 Dec 2016 15:24:27 +0300
|
||||
Subject: [PATCH] Remove http-conduit (<2.2.0) constraint
|
||||
|
||||
Since https://github.com/aristidb/aws/issues/206 is resolved, this
|
||||
constraint is no longer necessary. However, http-conduit (>=2.2.0)
|
||||
requires http-client (>=0.5.0) which introduces some breaking changes.
|
||||
This commit also implements those changes depending on the version.
|
||||
Fixes: https://git-annex.branchable.com/bugs/Build_with_aws_head_fails/
|
||||
|
||||
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
|
||||
---
|
||||
Remote/S3.hs | 8 +++++++-
|
||||
Remote/WebDAV.hs | 17 +++++++++++++++++
|
||||
Utility/Url.hs | 8 ++++++++
|
||||
git-annex.cabal | 3 +--
|
||||
4 files changed, 33 insertions(+), 3 deletions(-)
|
||||
|
||||
diff --git a/Remote/S3.hs b/Remote/S3.hs
|
||||
index 4c1bd57..9563b5a 100644
|
||||
--- a/Remote/S3.hs
|
||||
+++ b/Remote/S3.hs
|
||||
@@ -49,6 +49,12 @@ import Annex.Content
|
||||
import Annex.Url (withUrlOptions)
|
||||
import Utility.Url (checkBoth, managerSettings, closeManager)
|
||||
|
||||
+#if MIN_VERSION_http_client(0,5,0)
|
||||
+import Network.HTTP.Client (responseTimeoutNone)
|
||||
+#else
|
||||
+responseTimeoutNone = Nothing
|
||||
+#endif
|
||||
+
|
||||
type BucketName = String
|
||||
|
||||
remote :: RemoteType
|
||||
@@ -430,7 +436,7 @@ withS3HandleMaybe c gc u a = do
|
||||
where
|
||||
s3cfg = s3Configuration c
|
||||
httpcfg = managerSettings
|
||||
- { managerResponseTimeout = Nothing }
|
||||
+ { managerResponseTimeout = responseTimeoutNone }
|
||||
|
||||
s3Configuration :: RemoteConfig -> S3.S3Configuration AWS.NormalQuery
|
||||
s3Configuration c = cfg
|
||||
diff --git a/Remote/WebDAV.hs b/Remote/WebDAV.hs
|
||||
index 19dbaa8..14947f1 100644
|
||||
--- a/Remote/WebDAV.hs
|
||||
+++ b/Remote/WebDAV.hs
|
||||
@@ -5,6 +5,7 @@
|
||||
- Licensed under the GNU GPL version 3 or higher.
|
||||
-}
|
||||
|
||||
+{-# LANGUAGE CPP #-}
|
||||
{-# LANGUAGE ScopedTypeVariables #-}
|
||||
|
||||
module Remote.WebDAV (remote, davCreds, configUrl) where
|
||||
@@ -34,6 +35,10 @@ import Utility.Url (URLString, matchStatusCodeException)
|
||||
import Annex.UUID
|
||||
import Remote.WebDAV.DavLocation
|
||||
|
||||
+#if MIN_VERSION_http_client(0,5,0)
|
||||
+import Network.HTTP.Client (HttpExceptionContent(..), responseStatus)
|
||||
+#endif
|
||||
+
|
||||
remote :: RemoteType
|
||||
remote = RemoteType {
|
||||
typename = \"webdav\",
|
||||
@@ -302,6 +307,17 @@ goDAV (DavHandle ctx user pass _) a = choke $ run $ prettifyExceptions $ do
|
||||
{- Catch StatusCodeException and trim it to only the statusMessage part,
|
||||
- eliminating a lot of noise, which can include the whole request that
|
||||
- failed. The rethrown exception is no longer a StatusCodeException. -}
|
||||
+#if MIN_VERSION_http_client(0,5,0)
|
||||
+prettifyExceptions :: DAVT IO a -> DAVT IO a
|
||||
+prettifyExceptions a = catchJust (matchStatusCodeException (const True)) a go
|
||||
+ where
|
||||
+ go (HttpExceptionRequest _ (StatusCodeException response message)) = error $ unwords
|
||||
+ [ \"DAV failure:\"
|
||||
+ , show (responseStatus response)
|
||||
+ , show (message)
|
||||
+ ]
|
||||
+ go e = throwM e
|
||||
+#else
|
||||
prettifyExceptions :: DAVT IO a -> DAVT IO a
|
||||
prettifyExceptions a = catchJust (matchStatusCodeException (const True)) a go
|
||||
where
|
||||
@@ -311,6 +327,7 @@ prettifyExceptions a = catchJust (matchStatusCodeException (const True)) a go
|
||||
, show (statusMessage status)
|
||||
]
|
||||
go e = throwM e
|
||||
+#endif
|
||||
|
||||
prepDAV :: DavUser -> DavPass -> DAVT IO ()
|
||||
prepDAV user pass = do
|
||||
diff --git a/Utility/Url.hs b/Utility/Url.hs
|
||||
index 9b68871..d0e1b37 100644
|
||||
--- a/Utility/Url.hs
|
||||
+++ b/Utility/Url.hs
|
||||
@@ -350,8 +350,16 @@ hUserAgent = \"User-Agent\"
|
||||
-
|
||||
- > catchJust (matchStatusCodeException (== notFound404))
|
||||
-}
|
||||
+#if MIN_VERSION_http_client(0,5,0)
|
||||
+matchStatusCodeException :: (Status -> Bool) -> HttpException -> Maybe HttpException
|
||||
+matchStatusCodeException want e@(HttpExceptionRequest _ (StatusCodeException r _))
|
||||
+ | want (responseStatus r) = Just e
|
||||
+ | otherwise = Nothing
|
||||
+matchStatusCodeException _ _ = Nothing
|
||||
+#else
|
||||
matchStatusCodeException :: (Status -> Bool) -> HttpException -> Maybe HttpException
|
||||
matchStatusCodeException want e@(StatusCodeException s _ _)
|
||||
| want s = Just e
|
||||
| otherwise = Nothing
|
||||
matchStatusCodeException _ _ = Nothing
|
||||
+#endif
|
||||
diff --git a/git-annex.cabal b/git-annex.cabal
|
||||
index ec54a14..83d45a1 100644
|
||||
--- a/git-annex.cabal
|
||||
+++ b/git-annex.cabal
|
||||
@@ -357,8 +357,7 @@ Executable git-annex
|
||||
resourcet,
|
||||
http-client,
|
||||
http-types,
|
||||
- -- Old version needed due to https://github.com/aristidb/aws/issues/206
|
||||
- http-conduit (<2.2.0),
|
||||
+ http-conduit,
|
||||
time,
|
||||
old-locale,
|
||||
esqueleto,
|
||||
--
|
||||
2.7.4
|
||||
|
||||
\"\"\"]]
|
||||
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
### Please describe the problem.
|
||||
|
||||
I have files that match annex.largefiles and therefore should be added to git but not to annex, they seem to be getting corrupted after cloning the repo.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
I couldn't immediately find the exact steps to reproduce the issue but I have multiple git repositories showing this.
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
The problem has occurred a while ago but I have just noticed it. This is on macOS if it helps. I also tend to use the latest released version of git-annex (installed via Homebrew)
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
# If you can, paste a complete transcript of the problem occurring here.
|
||||
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
|
||||
|
||||
$ cd Documents
|
||||
$ cat .gitattributes
|
||||
* annex.largefiles=((not(mimetype=text/*))or(largerthan=100kb))
|
||||
|
||||
*.png binary
|
||||
*.jpg binary
|
||||
*.jpeg binary
|
||||
*.gif binary
|
||||
*.ico binary
|
||||
|
||||
*.mp3 binary
|
||||
*.fla binary
|
||||
|
||||
*.mov binary
|
||||
*.mp4 binary
|
||||
*.flv binary
|
||||
*.swf binary
|
||||
*.avi binary
|
||||
*.mkv binary
|
||||
*.mpg binary
|
||||
*.mpeg binary
|
||||
|
||||
*.gz binary
|
||||
*.zip binary
|
||||
*.7z binary
|
||||
*.rar binary
|
||||
*.bz2 binary
|
||||
|
||||
*.ttf binary
|
||||
|
||||
*.pdf binary
|
||||
|
||||
$ ls -la Docs/2016-XXX/XXX/
|
||||
total 696
|
||||
drwx------@ 4 denis staff 136 Jul 11 15:05 ./
|
||||
drwxr-xr-x@ 9 denis staff 306 Dec 12 19:42 ../
|
||||
-rwxr-xr-x@ 1 denis staff 265898 Jul 11 13:03 XXX.pdf*
|
||||
-rwxr-xr-x@ 1 denis staff 89586 Jul 11 13:03 Summary.pdf*
|
||||
$ file --mime-type Docs/2016-XXX/XXX/XXX.pdf
|
||||
Docs/2016-XXX/XXX/XXX.pdf: application/pdf
|
||||
$ git show 60a76858a57a73967131b929af45a99703f67335
|
||||
commit 60a76858a57a73967131b929af45a99703f67335
|
||||
Author: Denis Dzyubenko <denis@ddenis.info>
|
||||
Date: Mon Jul 11 15:05:37 2016 +0200
|
||||
|
||||
XXX
|
||||
|
||||
diff --git a/Docs/2016-XXX/XXX/XXX.pdf b/Docs/2016-XXX/XXX/XXX.pdf
|
||||
new file mode 100755
|
||||
index 00000000..112f68d0
|
||||
Binary files /dev/null and b/Docs/2016-XXX/XXX/XXX.pdf differ
|
||||
diff --git a/Docs/2016-XXX/XXX/Summary.pdf b/Docs/2016-XXX/XXX/Summary.pdf
|
||||
new file mode 100755
|
||||
index 00000000..3828383e
|
||||
Binary files /dev/null and b/Docs/2016-XXX/XXX/Summary.pdf differ
|
||||
diff --git a/Docs/2016-XXX/XXX.pdf b/Docs/2016-XXX/XXX.pdf
|
||||
deleted file mode 120000
|
||||
index 6d347a22..00000000
|
||||
--- a/Docs/2016-XXX/XXX.pdf
|
||||
+++ /dev/null
|
||||
@@ -1 +0,0 @@
|
||||
-../../.git/annex/objects/zJ/X1/SHA256E-s190749--ee0c8329c88f9c1656cc75cf37d4df64060a022e73d199164c5e5222ba1739d1.pdf/SHA256E-s190749--ee0c8329c88f9c1656cc
|
||||
\ No newline at end of file
|
||||
|
||||
|
||||
|
||||
$ git clone Documents Documents.tmp
|
||||
Cloning into 'Documents.tmp'...
|
||||
done.
|
||||
$ cd ./Documents.tmp/
|
||||
$ ls -la Docs/2016-XXX/XXX/
|
||||
total 184
|
||||
drwxr-xr-x 4 denis staff 136 Dec 19 00:09 ./
|
||||
drwxr-xr-x 8 denis staff 272 Dec 19 00:09 ../
|
||||
-rwxr-xr-x 1 denis staff 101 Dec 19 00:09 XXX.pdf*
|
||||
-rwxr-xr-x 1 denis staff 89586 Dec 19 00:09 Summary.pdf*
|
||||
$ cat Docs/2016-XXX/XXX/XXX.pdf
|
||||
/annex/objects/SHA256E-s265898--9c750c01dce9689ac3880224d2e95da6287b0cc89759c0c882e7a9a0fe48d664.pdf
|
||||
|
||||
# End of transcript or log.
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
### Please describe the problem.
|
||||
Issue uploading to S3 remote (Dreamhost)
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
git-annex copy massart/f16_Web1/screencaptures/IMG_5159.MOV --to=cloud
|
||||
on my repo
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
6.20161031-g0a58e94
|
||||
OS-X 10.11.6
|
||||
|
||||
### Please provide any additional information below.
|
||||
I am using a different WIFI I haven't used before. Maybe it is blocking something…
|
||||
|
||||
[[!format sh """
|
||||
git-annex copy massart/f16_Web1/screencaptures/IMG_5159.MOV --to=cloud
|
||||
copy massart/f16_Web1/screencaptures/IMG_5159.MOV (checking cloud...) (to cloud...)
|
||||
17% 0.0 B/s 0sgpg: error running `/Users/joey/homebrew/opt/gpg-agent/bin/gpg-agent': probably not installed
|
||||
gpg: DBG: running `/Users/joey/homebrew/opt/gpg-agent/bin/gpg-agent' for testing failed: Configuration error
|
||||
gpg: can't connect to the agent: IPC connect call failed
|
||||
gpg: problem with the agent: No agent running
|
||||
35% 1021.8KB/s 30s
|
||||
user error (gpg ["--quiet","--trust-model","always","--batch","--passphrase-fd","26","--symmetric","--force-mdc","--no-textmode"] exited 2)
|
||||
failed
|
||||
git-annex: copy: 1 failed
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
Yes.
|
||||
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
[[!comment format=mdwn
|
||||
username="andrew"
|
||||
avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435"
|
||||
subject="RESOLVED"
|
||||
date="2016-11-17T14:59:15Z"
|
||||
content="""
|
||||
Ooops. I am on OS-X. I use brew for my gnupg installation. It appears I had removed gpg from the path when installing something. I just needed to run to fix:
|
||||
|
||||
brew link gnupg2
|
||||
|
||||
Thanks,
|
||||
|
||||
Andrew
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
### Please describe the problem.
|
||||
|
||||
I'm seeing some inconsistent results between runs of `git annex fsck` and `git annex whereis` that I'm not able to explain. When I run `git annex fsck`, it reports a few keys that only have 1 copy, and advises me to make more copies. If I run `git annex whereis --key <key>`, git annex confirms that it only knows about 1 copy of this key. If I then use `git log --stat -S'<key>'` to find the actual file that it refers to, and run `git annex whereis <file>`, git annex report 9 copies of this file. Checking on remotes shows that these files do exist on the remote, so why does `git annex fsck` and `git annex whereis` mis-report the number of copies when querying for the key - but not for the actual filename? Additionally, `git annex find --lackingcopies 1` doesn't return any results, but should if there are actually files with not enough copies?
|
||||
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
5.20151208-1build1 on Ubuntu Xenial, one remote running 5.20141024~bpo70+1 on Debian Wheezy
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
# If you can, paste a complete transcript of the problem occurring here.
|
||||
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
|
||||
|
||||
[william@hactar ~/Pictures/Photo Library]$ git annex whereis SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9
|
||||
git-annex: SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9 not found
|
||||
git-annex: whereis: 1 failed
|
||||
[william@hactar ~/Pictures/Photo Library]$ git annex whereis --key SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9
|
||||
whereis SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9 (1 copy)
|
||||
7691934f-2542-4103-9122-2db4e6cfc887 -- hactar [here]
|
||||
ok
|
||||
[william@hactar ~/Pictures/Photo Library]$ git annex fsck --key SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9
|
||||
fsck SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9
|
||||
Only 1 of 3 trustworthy copies exist of SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9
|
||||
Back it up with git-annex copy.
|
||||
failed
|
||||
(recording state in git...)
|
||||
git-annex: fsck: 1 failed
|
||||
[william@hactar ~/Pictures/Photo Library]$ git log --stat -S'SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9'
|
||||
[william@hactar ~/Pictures/Photo Library]$ git annex whereis 2009/05/05/P1040890.JPG
|
||||
whereis 2009/05/05/P1040890.JPG (9 copies)
|
||||
0e825a69-1927-4f62-b731-6f3e98bba998 -- william@marvin:/media/backup/annex/photos [marvin]
|
||||
1b728ab5-1e32-45a6-bc11-2a4bfdc9d6ab -- backup1
|
||||
5c0caa42-b489-467b-a612-9590fa9d5a94 -- backup2
|
||||
7691934f-2542-4103-9122-2db4e6cfc887 -- hactar [here]
|
||||
894b2216-72e0-40e1-8765-1386e1e9e4b4 -- backup3
|
||||
96f19fa8-d385-4e8b-b000-61ee15993a70 -- backup3
|
||||
a862b121-d794-4af4-bb56-21adfe8962f2 -- S3
|
||||
b083f8ae-42fb-41f0-a2a3-4e7c9f93aadb -- [guide]
|
||||
bf021ce9-465b-4419-86e7-bddfd208fca4 -- git@newzaphod:~/repositories/annex/photos.git [zaphod]
|
||||
ok
|
||||
|
||||
|
||||
# End of transcript or log.
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
I trust Git Annex to keep hundreds of GB of data safe, and it has never failed me - despite my best efforts
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-12-13T16:42:08Z"
|
||||
content="""
|
||||
The obvious reason for this would be if the file no longer points to that
|
||||
same key. Perhaps the file got modified and the key is the old version of
|
||||
the file.
|
||||
|
||||
That would explain everything you showed, so currently I don't see any
|
||||
bug..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
### Please describe the problem.
|
||||
I had also commented this on [[another bug|bugs/git-annex_fromkey_barfs_on_utf-8_input]], but the original issue there is fixed now.
|
||||
I tested `fromkey`, `calckey --batch`, `lookupkey --batch` (in standalone) after your fix, they work nicely.
|
||||
|
||||
However, `git-annex metadata --batch --json` using the [[linux standalone|install/Linux_standalone]] (autobuild) still fails when it encounters UTF-8 characters (e.g. ü, ç, ä).
|
||||
Also, `git-annex metadata --json` gives `"file":"<22><>.txt"` for `ü.txt`.
|
||||
|
||||
This happens only in the standalone builds.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
[[!format sh """
|
||||
$ .../git-annex.linux/runshell
|
||||
$ touch u.txt ü.txt
|
||||
$ git-annex add .
|
||||
|
||||
$ git-annex metadata --batch --json
|
||||
{"file":"ü.txt"}
|
||||
git-annex: Batch input parse failure: Error in $: Failed reading: Cannot decode byte '\xb3': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream
|
||||
|
||||
$ git-annex metadata --batch --json
|
||||
{"file":"u.txt","fields":{"ç":["b"]}}
|
||||
git-annex: Batch input parse failure: Error in $: Failed reading: Cannot decode byte '\xb3': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream
|
||||
|
||||
$ git-annex metadata --batch --json
|
||||
{"file":"u.txt","fields":{"b":["ä"]}}
|
||||
git-annex: Batch input parse failure: Error in $: Failed reading: Cannot decode byte '\xb3': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream
|
||||
|
||||
$ git-annex metadata --json
|
||||
{"command":"metadata","note":"","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"u.txt","fields":{}}
|
||||
{"command":"metadata","note":"","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"<22><>.txt","fields":{}}
|
||||
# success, but the second line should have "file":"ü.txt"
|
||||
"""]]
|
||||
|
||||
It's the same even if I call `.../git-annex.linux/git-annex` directly (without `runshell`)
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
Using the Linux standalone: [git-annex-standalone-amd64.tar.gz](https://downloads.kitenet.net/git-annex/autobuild/amd64/git-annex-standalone-amd64.tar.gz) on Xubuntu 16.04
|
||||
|
||||
[[!format sh """
|
||||
$ git-annex version
|
||||
git-annex version: 6.20161213-g55a34b493
|
||||
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
local repository version: 5
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
operating system: linux x86_64
|
||||
"""]]
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
None of the characters I used have `\xb3` in it, but all the errors happen due to it:
|
||||
[[!format sh """
|
||||
$ .../git-annex.linux/runshell
|
||||
$ echo -n ü | xxd
|
||||
00000000: c3bc ..
|
||||
$ echo -n ç | xxd
|
||||
00000000: c3a7 ..
|
||||
$ echo -n ä | xxd
|
||||
00000000: c3a4 ..
|
||||
"""]]
|
||||
|
||||
In `runshell`, `ls` can't show UTF-8, but `git-annex status` can:
|
||||
[[!format sh """
|
||||
$ .../git-annex.linux/runshell
|
||||
$ ls
|
||||
u.txt ??.txt
|
||||
$ git-annex status
|
||||
A u.txt
|
||||
A ü.txt
|
||||
"""]]
|
||||
|
||||
`man` complains about locale in `runshell` as well:
|
||||
[[!format sh """
|
||||
$ .../git-annex.linux/runshell
|
||||
$ man
|
||||
man: can\'t set the locale; make sure $LC_* and $LANG are correct
|
||||
What manual page do you want?
|
||||
# I escaped that \', formatting was messy otherwise
|
||||
$ set | grep LANG
|
||||
GDM_LANG='en_GB'
|
||||
LANG='en_GB.UTF-8'
|
||||
LANGUAGE='en_GB:en'
|
||||
$ set | grep LC
|
||||
# nothing
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-12-19T20:37:56Z"
|
||||
content="""
|
||||
runshell was recently changed to bypass using the system locales, it
|
||||
includes its own locale data and attempts to generate a locale definition
|
||||
file for the locale. The code that did that was failing to notice that
|
||||
en_GB.UTF-8 was a UTF-8 locale (en_GB.utf8 would work though), which
|
||||
explains why the locale is not set inside runshell
|
||||
(git-annex.linux/git-annex is a script that uses runshell). I've corrected
|
||||
that problem, and verified it fixes the problem you reported.
|
||||
|
||||
----
|
||||
|
||||
However.. The same thing happens when using LANG=C with git-annex
|
||||
installed by any method and --json --batch. So the deeper problem is that
|
||||
it's forcing the batch input to be decoded as utf8 via the current locale.
|
||||
This happens in Command/MetaData.hs parseJSONInput which uses
|
||||
`BU.fromString`.
|
||||
|
||||
I tried swapping in `encodeBS` for `BU.fromString`. That prevented the
|
||||
decoding error, but made git-annex complain that the file was not annexed,
|
||||
due to a Mojibake problem:
|
||||
|
||||
With `encodeBS`, the input `{"file":"ü.txt"}` is encoded as
|
||||
`"{\"file\":\"\195\188.txt\"}"`. Aeson parses that input to this:
|
||||
|
||||
JSONActionItem {itemCommand = Nothing, itemKey = Nothing, itemFile = Just "\252.txt", itemAdded = Nothing}
|
||||
|
||||
Note that the first two bytes have been
|
||||
parsed by Aeson as unicode (since JSON is unicode encoded),
|
||||
yielding character 252 (ü).
|
||||
|
||||
In a unicode locale, this works ok, because the encoding layer is able to
|
||||
convert that unicode character back to two bytes 195 188
|
||||
and finds the file on disk. But in a non-unicode locale, it doesn't know
|
||||
what to do with the unicode character, and in fact it gets discarded
|
||||
and so it looks for a file named ".txt".
|
||||
|
||||
So, to make --batch --json input work in non-unicode locales, it would
|
||||
need, after parsing the json, to re-encode filenames (and perhaps other
|
||||
data), from utf8 to the filesystem encoding. I have not yet worked out how
|
||||
to do that.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,85 @@
|
|||
### Please describe the problem.
|
||||
|
||||
While running `git-annex metadata --batch --json`, repeatedly assigning a field to the same value in the same run (with different values in between the assignments of the same value) causes a value to get stuck.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
$ touch test.txt
|
||||
$ git annex add
|
||||
$ git-annex metadata --batch --json
|
||||
{"file":"test.txt","fields":{"f":["a"]}}
|
||||
# prints { ... "f":["a"] ... }
|
||||
{"file":"test.txt","fields":{"f":["b"]}}
|
||||
# prints { ... "f":["b"] ... }
|
||||
{"file":"test.txt","fields":{"f":["c"]}}
|
||||
# prints { ... "f":["c"] ... }
|
||||
{"file":"test.txt","fields":{"f":["a"]}}
|
||||
# prints { ... "f":["a", "c"] ... }
|
||||
{"file":"test.txt","fields":{"f":["b"]}}
|
||||
# prints { ... "f":["c"] ... }
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
git-annex version: 6.20161122+gitg9f179ae-1~ndall+1
|
||||
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
local repository version: 5
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
operating system: linux x86_64
|
||||
|
||||
I'm using Xubuntu 16.04, with the `git-annex-standalone` package from NeuroDebian repository.
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
If you keep reassigning the same values, things get very weird. Full inputs/outputs from a sample run:
|
||||
|
||||
{"file":"test.txt","fields":{"f":["a"]}}
|
||||
{"command":"metadata","note":"f=a\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields": {"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["a"]}}
|
||||
{"file":"test.txt","fields":{"f":["b"]}}
|
||||
{"command":"metadata","note":"f=b\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["b"]}}
|
||||
{"file":"test.txt","fields":{"f":["c"]}}
|
||||
{"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}}
|
||||
{"file":"test.txt","fields":{"f":["a"]}}
|
||||
{"command":"metadata","note":"f=a\nf=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["a","c"]}}
|
||||
{"file":"test.txt","fields":{"f":["b"]}}
|
||||
{"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}}
|
||||
{"file":"test.txt","fields":{"f":[]}}
|
||||
{"command":"metadata","note":"lastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"lastchanged":["2016-12-05@21-17-39"]}}
|
||||
{"file":"test.txt","fields":{"f":["b"]}}
|
||||
{"command":"metadata","note":"lastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"lastchanged":["2016-12-05@21-17-39"]}}
|
||||
{"file":"test.txt","fields":{"f":["c"]}}
|
||||
{"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}}
|
||||
{"file":"test.txt","fields":{"f":["a"]}}
|
||||
{"command":"metadata","note":"f=a\nf=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["a","c"]}}
|
||||
{"file":"test.txt","fields":{"f":["b"]}}
|
||||
{"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}}
|
||||
{"file":"test.txt","fields":{"f":["c"]}}
|
||||
{"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}}
|
||||
{"file":"test.txt","fields":{"f":["a"]}}
|
||||
{"command":"metadata","note":"f=a\nf=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["a","c"]}}
|
||||
{"file":"test.txt","fields":{"f":["b"]}}
|
||||
{"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}}
|
||||
{"file":"test.txt","fields":{"f":["b"]}}
|
||||
{"command":"metadata","note":"f=b\nf=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["b","c"]}}
|
||||
{"file":"test.txt","fields":{"f":["a"]}}
|
||||
{"command":"metadata","note":"f=a\nf=b\nf=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["a","b","c"]}}
|
||||
{"file":"test.txt","fields":{"f":["d"]}}
|
||||
{"command":"metadata","note":"f=b\nf=c\nf=d\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["b","c","d"]}}
|
||||
{"file":"test.txt","fields":{"f":["a"]}}
|
||||
{"command":"metadata","note":"f=b\nf=c\nf=d\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["b","c","d"]}}
|
||||
{"file":"test.txt","fields":{"f":["a"]}}
|
||||
{"command":"metadata","note":"f=b\nf=c\nf=d\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["b","c","d"]}}
|
||||
{"file":"test.txt","fields":{"f":[]}}
|
||||
{"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}}
|
||||
|
||||
Restarting the process solves the issue.
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
I love the metadata functionality so much that I wrote [[tips/a_gui_for_metadata_operations]] and discovered this bug.
|
||||
Metadata driven views are awesome (but I don't like the entire folder hierarchy being appended to the filename).
|
||||
I haven't used the other commands much since I have not yet organized most of my stuff (and their naively copy-pasted backups), but I am glad I discovered git-annex before I began organizing.
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-12-13T14:40:02Z"
|
||||
content="""
|
||||
I thought this would involve the journal, but it seems not; same
|
||||
behavior occurs if the journal is committed after each metadata change.
|
||||
|
||||
Looking at the new metadata value in the case where a and c both get set,
|
||||
it is:
|
||||
|
||||
MetaData (fromList [(MetaField "f",fromList [MetaValue (CurrentlySet True) "a",MetaValue (CurrentlySet False) "c"])])
|
||||
|
||||
That is supposed to unset c, with the CurrentlySet False, but instead c
|
||||
remains set somehow.
|
||||
|
||||
Aha, the use of `addMetaData'` causes the bug. That reuses the same
|
||||
timestamp, and indeed the same timestamp is used for all the batch
|
||||
changes. With the same timestamp for the log line that sets c as the line
|
||||
that removes it, it's indeterminite which line will be acted on first, and
|
||||
so the removal can be processed before the addition, leaving c "stuck".
|
||||
|
||||
Fixing..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
[[!comment format=mdwn
|
||||
username="justin.lebar@7a36fcafc322d9a381e89f08ab6289033c6dde91"
|
||||
nickname="justin.lebar"
|
||||
avatar="http://cdn.libravatar.org/avatar/9fca4b61a1ab555f231851e7543f9a3e"
|
||||
subject="comment 2"
|
||||
date="2016-11-20T03:47:23Z"
|
||||
content="""
|
||||
Thanks for your reply, Joey. Sorry for the delay getting back to this -- I didn't realize I hadn't enabled notifications on the thread.
|
||||
|
||||
The GCS docs suggest that 400 errors should be accompanied by an explanation in the reply body.
|
||||
|
||||
> Error responses usually include a JSON document in the response body, which contains information about the error.
|
||||
|
||||
https://cloud.google.com/storage/docs/json_api/v1/status-codes
|
||||
|
||||
Do you think we're not getting an http response body here, or that it's not being printed out?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="justin.lebar@7a36fcafc322d9a381e89f08ab6289033c6dde91"
|
||||
nickname="justin.lebar"
|
||||
avatar="http://cdn.libravatar.org/avatar/9fca4b61a1ab555f231851e7543f9a3e"
|
||||
subject="comment 3"
|
||||
date="2016-12-04T04:30:38Z"
|
||||
content="""
|
||||
For a while things were working, but now it's not working again, same problem as before.
|
||||
|
||||
Do you think maybe it's a timestamp bug in the signature or something? That could explain this \"mysteriously works then stops working\" behavior.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
### Please describe the problem.
|
||||
|
||||
If a filename has a single space (and only one space), `git annex add` will fail out with the following message:
|
||||
|
||||
add one two git-annex: unknown response from git cat-file ("HEAD:./one two missing",Ref "HEAD:./one two")
|
||||
CallStack (from HasCallStack):
|
||||
error, called at ./Git/CatFile.hs:102:28 in main:Git.CatFile
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
Run the following:
|
||||
|
||||
git init .
|
||||
git annex init
|
||||
touch "one two"
|
||||
# this will cause error
|
||||
git annex add "one two"
|
||||
touch "one two three"
|
||||
# this is fine
|
||||
git annex add "one two three"
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
Output of `git annex version`
|
||||
|
||||
git-annex version: 6.20161027
|
||||
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
local repository version: 5
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
operating system: linux x86_64
|
||||
|
||||
Operating System: Linux (NixOS 16.09.909.238c7e0 (Flounder))
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
Maybe related to [https://git-annex.branchable.com/forum/unknown_response_from_git_cat-file/](https://git-annex.branchable.com/forum/unknown_response_from_git_cat-file/) or [https://git-annex.branchable.com/bugs/git_annex_import_fails_on_filenames_with_newlines_in_them/](https://git-annex.branchable.com/bugs/git_annex_import_fails_on_filenames_with_newlines_in_them/)?
|
||||
|
||||
EDIT: Somewhat surprisingly, if I build from source using `cabal`, everything works fine.
|
||||
|
||||
.cabal-sandbox/bin/git-annex version
|
||||
git-annex version: 6.20161113-g1e88c12
|
||||
build flags: Assistant Webapp Pairing Testsuite WebDAV Inotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
|
||||
Not sure whether this means that this bug is actually fixed or whether it's an artifact of how things are built in Nix.
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Just starting out with it as a way of archiving some of my media seems to work great apart from this. Thanks a bunch!
|
||||
|
||||
> This bug was already fixed in git-annex 6.20161031. I told the Debian
|
||||
> maintainer about the bug fix at the time; package has not been updated
|
||||
> yet. [[done]] on git-annex side. --[[Joey]]
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
[[!comment format=mdwn
|
||||
username="gfa@1e86118cd41fbfea50004af221471ad97b55af18"
|
||||
nickname="gfa"
|
||||
avatar="http://cdn.libravatar.org/avatar/4678da4da55c67fa668e31ea0a76b201"
|
||||
subject="same on Debian"
|
||||
date="2016-11-14T09:13:19Z"
|
||||
content="""
|
||||
I face the same issue on Debian testing
|
||||
|
||||
git-annex 6.20161012-1
|
||||
git 1:2.10.2-1
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://launchpad.net/~stephane-gourichon-lpad"
|
||||
nickname="stephane-gourichon-lpad"
|
||||
avatar="http://cdn.libravatar.org/avatar/02d4a0af59175f9123720b4481d55a769ba954e20f6dd9b2792217d9fa0c6089"
|
||||
subject="Known bug, fixed."
|
||||
date="2016-11-23T18:04:27Z"
|
||||
content="""
|
||||
This is a known bug introduced in 6.20161012 and fixed in 6.20161031.
|
||||
|
||||
Solution is: just update your copy of git-annex. At this time most recent is 6.20161119 .
|
||||
|
||||
For more details, see changelog at https://github.com/joeyh/git-annex/blob/master/CHANGELOG#L53
|
||||
|
||||
|
||||
|
||||
"""]]
|
||||
|
|
@ -30,4 +30,5 @@ operating system: darwin x86_64
|
|||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
|
||||
> [[done]], my fix worked! Don't know entirely why it was needed..
|
||||
> --[[Joey]]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,21 @@
|
|||
[[!comment format=mdwn
|
||||
username="christopher@5845ecd3cef9edadd4dc084df00e1fa60ce311eb"
|
||||
nickname="christopher"
|
||||
avatar="http://cdn.libravatar.org/avatar/4b722efb21f38d9944730c93727bc602"
|
||||
subject="comment 3"
|
||||
date="2016-11-15T12:15:37Z"
|
||||
content="""
|
||||
Hi Joey,
|
||||
|
||||
I installed git-annex using the homebrew recipe from https://github.com/Homebrew/homebrew-core/blob/master/Formula/git-annex.rb, OS X 10.11.6 (15G31)
|
||||
|
||||
These are the dependencies reported by homebrew:
|
||||
|
||||
Build: ghc ✔, cabal-install ✔, pkg-config ✔
|
||||
Required: gsasl ✔, libidn ✔, libmagic ✔, gnutls ✔, quvi ✔
|
||||
|
||||
I've re-installed using \"brew install git-annex --HEAD\" to pull in your latest commit and I can confirm that everything works as expected and the /static/ resources load correctly.
|
||||
|
||||
Thanks,
|
||||
Chris
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
### Please describe the problem.
|
||||
|
||||
```
|
||||
> git annex get Narnia/
|
||||
get Narnia/Course of a Generation/01 Sail Around the World.mp3 (from Seagate...)
|
||||
SHA256E-s8395599--2fea961006a279f0765c45755b35a06f0a4fc6bfbab6118182ebc693d7b47a91.mp3
|
||||
8,395,599 100% 29.65MB/s 0:00:00 (xfr#1, to-chk=0/1)
|
||||
(checksum...) ^C⏎
|
||||
```
|
||||
|
||||
```
|
||||
> mpv ~/Music/sorted/Narnia/Course\ of\ a\ Generation/
|
||||
Playing: /home/philip/Music/sorted/Narnia/Course of a Generation/
|
||||
[file] This is a directory - adding to playlist.
|
||||
|
||||
Playing: /home/philip/Music/sorted/Narnia/Course of a Generation/01 Sail Around the World.mp3
|
||||
Failed to recognize file format.
|
||||
|
||||
Playing: /home/philip/Music/sorted/Narnia/Course of a Generation/02 When the Stars Are Falling.mp3
|
||||
```
|
||||
|
||||
```
|
||||
> git annex version
|
||||
git-annex version: 6.20161012
|
||||
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
local repository version: 6
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
operating system: linux x86_64
|
||||
```
|
||||
|
||||
Any consecutive `git annex get` commands don’t notice that the file is not completely transferred and leave it in a broken state.
|
||||
`git annex get --failed` does not correct the problem.
|
||||
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Yes, it (kind of) works for keeping my music library in sync.
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-11-16T18:36:34Z"
|
||||
content="""
|
||||
Thing is, git-annex get does not update the file in place. Only once the
|
||||
entire file is downloaded, and its content is verified correct is it moved
|
||||
into a place where you can access it.
|
||||
|
||||
So, it seems much more likely to me that the content of the file, as
|
||||
originally added to git-annex, was bad, and the it had just finished
|
||||
verifying the content and moving it into place when you interruped the
|
||||
command.
|
||||
|
||||
Please check with `git annex fsck` on the file and see if it determines
|
||||
it has the content git-annex expects it to have.
|
||||
|
||||
However, I notice you're using a v6 repository. Is the file an unlocked
|
||||
file? It's possible that in that specific case there could be a bug.
|
||||
I've interrupted `git annex get` on a nearly daily basis for years, but
|
||||
v6 is still experimental and not as well tested.
|
||||
"""]]
|
||||
74
doc/bugs/YouTube_-_error_in_importfeed.mdwn
Normal file
74
doc/bugs/YouTube_-_error_in_importfeed.mdwn
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
### Please describe the problem.
|
||||
When adding a YouTube channel via importfeed I get the error:
|
||||
|
||||
```
|
||||
warning: bad feed content; no enclosures to download
|
||||
```
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
1. `cd $(mktemp -d)`
|
||||
2. `git init && git annex init`
|
||||
3. `git annex importfeed https://www.youtube.com/feeds/videos.xml\?playlist_id\=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet`
|
||||
4. Get sad. :-(
|
||||
|
||||
(URL [https://www.youtube.com/feeds/videos.xml?playlist_id=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet](https://www.youtube.com/feeds/videos.xml?playlist_id=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet) looks like a feed to Firefox)
|
||||
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
OSX (MacOS?) - installed via homebrew
|
||||
|
||||
git-annex version: 6.20161210
|
||||
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
|
||||
Debian Jessie - installed via apt-get (ASIDE: why is the apt-get version sooooo old?)
|
||||
|
||||
git-annex version: 5.20141125
|
||||
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
|
||||
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
|
||||
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
|
||||
|
||||
|
||||
### Additional information
|
||||
|
||||
Running with `--debug` (see below) seems to indicate that the feed downloads correctly, but it is the parsing that is failing. I don't know what command is being run to parse the feed though.
|
||||
|
||||
|
||||
``` shell
|
||||
git annex importfeed --debug https://www.youtube.com/feeds/videos.xml\?playlist_id\=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet
|
||||
```
|
||||
results in:
|
||||
|
||||
``` shell
|
||||
(checking known urls...) [2016-12-19 12:39:36.387714] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
|
||||
[2016-12-19 12:39:36.392367] process done ExitSuccess
|
||||
[2016-12-19 12:39:36.392496] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2016-12-19 12:39:36.396484] process done ExitSuccess
|
||||
[2016-12-19 12:39:36.406716] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--stage","-z","--","."]
|
||||
[2016-12-19 12:39:36.412674] process done ExitSuccess
|
||||
importfeed https://www.youtube.com/feeds/videos.xml?playlist_id=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet
|
||||
[2016-12-19 12:39:36.413555] call: wget ["--clobber","-c","-O","/var/folders/l0/l60294_970b9fh46062znm0r0000gn/T/feed16807282475249","https://www.youtube.com/feeds/videos.xml?playlist_id=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet","--user-agent","git-annex/6.20161210"]
|
||||
--2016-12-19 12:39:36-- https://www.youtube.com/feeds/videos.xml?playlist_id=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet
|
||||
Resolving www.youtube.com... 216.58.199.78, 2404:6800:4006:806::200e
|
||||
Connecting to www.youtube.com|216.58.199.78|:443... connected.
|
||||
HTTP request sent, awaiting response... 200 OK
|
||||
Length: unspecified [text/xml]
|
||||
Saving to: ‘/var/folders/l0/l60294_970b9fh46062znm0r0000gn/T/feed16807282475249’
|
||||
|
||||
/var/folders/l0/l60294_970b9fh46062znm0r0000gn/T/f [ <=> ] 23.81K --.-KB/s in 0.02s
|
||||
|
||||
2016-12-19 12:39:37 (1.22 MB/s) - ‘/var/folders/l0/l60294_970b9fh46062znm0r0000gn/T/feed16807282475249’ saved [24386]
|
||||
|
||||
[2016-12-19 12:39:37.595869] process done ExitSuccess
|
||||
|
||||
warning: bad feed content; no enclosures to download
|
||||
ok
|
||||
```
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Yes, for years. I donated to fund the dev and proudly display my git-annex stickers!
|
||||
|
||||
> This is now fixed in feed's git repository, and will be in the next
|
||||
> release of feed after the current 0.3.11.1 release. [[done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-12-19T19:55:23Z"
|
||||
content="""
|
||||
It's somewhat misleading that it complains there are no enclosures in the
|
||||
feed. While importfeed mostly downloads only enclosures in podcast feeds,
|
||||
it also checks link tags, which this feed contains, to see if quvi supports
|
||||
downloading content from them. Quvi does support the links in this feed,
|
||||
so it should work despite there being no enclosures.
|
||||
|
||||
I've reproduced it not working, and it seems that the problem is this is
|
||||
not quite a valid Atom feed, and the feed parsing library is failing to
|
||||
parse it. Perhaps that can be improved; I filed a bug here
|
||||
<https://github.com/bergmark/feed/issues/18>
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="m8r-achx62@7323980ed426b7f78c85dfefe7358672bce44e98"
|
||||
nickname="m8r-achx62"
|
||||
avatar="http://cdn.libravatar.org/avatar/adaf4c4277529e10e32c467fa4ed4b9a"
|
||||
subject="comment 2"
|
||||
date="2016-12-19T22:33:13Z"
|
||||
content="""
|
||||
Thanks for following this up Joey!
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
### Please describe the problem.
|
||||
|
||||
In my unlocked adjusted branch, I get a lot of errors during "git annex sync". It appears to work fine otherwise (the files actually get synced). Below is what I see on the terminal. The repository is otherwise clean (no local or remote changes).
|
||||
This has started to happen around a month ago, though I cannot pinpoint the exact version. This is in the same repo you used to debug the disappearing files in direct mode recently (thanks a lot btw!).
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
[[!format sh """
|
||||
$ git annex version
|
||||
git-annex version: 6.20161110-gd48f4ca
|
||||
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
$ lsb_release -a
|
||||
No LSB modules are available.
|
||||
Distributor ID: Debian
|
||||
Description: Debian GNU/Linux 8.6 (jessie)
|
||||
Release: 8.6
|
||||
Codename: jessie
|
||||
"""]]
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
$ git annex sync --content
|
||||
commit
|
||||
On branch adjusted/master(unlocked)
|
||||
nothing to commit, working tree clean
|
||||
ok
|
||||
pull origin
|
||||
remote: Counting objects: 113, done.
|
||||
remote: Compressing objects: 100% (113/113), done.
|
||||
remote: Total 113 (delta 112), reused 0 (delta 0)
|
||||
Receiving objects: 100% (113/113), 7.16 KiB | 0 bytes/s, done.
|
||||
Resolving deltas: 100% (112/112), completed with 112 local objects.
|
||||
From /srv/annex/bilder
|
||||
97a4806..78cb4ef git-annex -> origin/git-annex
|
||||
ok
|
||||
(merging origin/git-annex into git-annex...)
|
||||
|
||||
git-annex: fd:25: commitBuffer: invalid argument (invalid character)
|
||||
failed
|
||||
|
||||
git-annex: fd:25: commitBuffer: invalid argument (invalid character)
|
||||
failed
|
||||
|
||||
[...]
|
||||
|
||||
git-annex: fd:25: commitBuffer: invalid argument (invalid character)
|
||||
failed
|
||||
git-annex: sync: 2653 failed
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-11-16T18:49:09Z"
|
||||
content="""
|
||||
I'm not able to reproduce the problem with your test case and git-annex
|
||||
version 6.20161012.
|
||||
|
||||
Can you still reproduce it after upgrading?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
[[!comment format=mdwn
|
||||
username="t.z.mates"
|
||||
avatar="http://cdn.libravatar.org/avatar/90f15fad216078fd08d62cc676487925"
|
||||
subject="comment 2"
|
||||
date="2016-11-19T04:42:25Z"
|
||||
content="""
|
||||
Thanks for looking into it; I just checked again, and even on the newest version (6.20161118 binary), I'm still experiencing the behavior. However, I checked on an older OpenSuse box I have, and there it works (6.20161031 from OpenSuse repo).
|
||||
|
||||
Since my two machines experiencing the problem are both running arch, it seems it's somehow related to that distro. I've checked both installing via the binary (from kitenet) and from the arch community repo, but both produce the same behavior. Further, the OpenSuse install has the same build flags as the binaries, so that doesn't seem to be it. Are there any other diagnostics I can run?
|
||||
|
||||
This particular problem isn't very troublesome (it doesn't seem to have any material impact aside from error messages); however, I also occasionally experience a more serious bug. Namely, when certain (seemingly random) files are added to the repo locked, their content disappears and the symlink is broken (this is the other problem I alluded to in the description). I suspect that problem is related to this one though, since it also only affects my arch machines. I haven't yet submitted a report for that bug yet, though, since I can't reliably replicate it.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2016-12-13T16:54:11Z"
|
||||
content="""
|
||||
Perhaps it's caused by a particular/old version of git?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
[[!comment format=mdwn
|
||||
username="t.z.mates"
|
||||
avatar="http://cdn.libravatar.org/avatar/90f15fad216078fd08d62cc676487925"
|
||||
subject="comment 4"
|
||||
date="2016-12-20T23:08:44Z"
|
||||
content="""
|
||||
Hmm, I don't think an old version of git is the cause. I'm currently running the most recent build of git (2.11.0), but have used a number of versions over the past year.
|
||||
|
||||
I'm not sure if this is relevant, but this other bug reports similar behavior: [sync --content, fatal is outside repository errors](https://git-annex.branchable.com/forum/sync_--content__44___fatal_is_outside_repository_errors/). Specifically, it notes that there is an odd use of relative paths:
|
||||
> The relative path ../Users is curious
|
||||
|
||||
My error also appends an extra period. In particular, the path should be \"./1/2/3/4/foo\" but prints \"../1/2/3/4/foo\".
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
### Please describe the problem.
|
||||
|
||||
When addurl'ing a big file with .gitattributes configured to add only some files directly into git (and 'git annex add' operating correctly), addurl adds large files straight into git.
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
git-annex version: 6.20161018+gitgf3c366a-1~ndall+1
|
||||
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
$> cat .gitattributes
|
||||
* annex.backend=MD5E
|
||||
* annex.largefiles=(largerthan=100kb)
|
||||
*.json annex.largefiles=nothing
|
||||
*.txt annex.largefiles=nothing
|
||||
*.tsv annex.largefiles=nothing
|
||||
*.nii.gz annex.largefiles=(largerthan=0kb)
|
||||
*.tgz annex.largefiles=(largerthan=0kb)
|
||||
*.tar.gz annex.largefiles=(largerthan=0kb)
|
||||
*.gz annex.largefiles=(largerthan=0kb)
|
||||
|
||||
$> git annex addurl http://fcp-indi.s3.amazonaws.com/data/Projects/HBNSSI/RawDataTars/sub-0031121_baseline.tar.gz\?versionId\=7FvexHgyazWF.dUo238FA7XRiK0FWQDw.
|
||||
addurl fcp_indi.s3.amazonaws.com_data_Projects_HBNSSI_RawDataTars_sub_0031121_baseline.tar.gz_versionId_7FvexHgyazWF.dUo238FA7XRiK0FWQDw. (downloading http://fcp-indi.s3.amazonaws.com/data/Projects/HBNSSI/RawDataTars/sub-0031121_baseline.tar.gz?versionId=7FvexHgyazWF.dUo238FA7XRiK0FWQDw. ...)
|
||||
/mnt/btrfs/datasets/datalad/crawl-misc/indi/ 100%[==============================================================================================>] 195.44M 21.2MB/s in 12s
|
||||
(non-large file; adding content to git repository) ok
|
||||
(recording state in git...)
|
||||
cached/staged changes:
|
||||
\u2026r.gz_versionId_7FvexHgyazWF.dUo238FA7XRiK0FWQDw. | Bin 0 -> 204937338 bytes
|
||||
|
||||
$> ls -l fcp_indi.s3.amazonaws.com_data_Projects_HBNSSI_RawDataTars_sub_0031121_baseline.tar.gz_versionId_7FvexHgyazWF.dUo238FA7XRiK0FWQDw.
|
||||
-rw------- 1 yoh datalad 204937338 Oct 25 17:30 fcp_indi.s3.amazonaws.com_data_Projects_HBNSSI_RawDataTars_sub_0031121_baseline.tar.gz_versionId_7FvexHgyazWF.dUo238FA7XRiK0FWQDw.
|
||||
cached/staged changes:
|
||||
\u2026r.gz_versionId_7FvexHgyazWF.dUo238FA7XRiK0FWQDw. | Bin 0 -> 204937338 bytes
|
||||
|
||||
"""]]
|
||||
|
||||
[[!meta author=yoh]]
|
||||
|
||||
> [[fixed|done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-11-21T15:12:54Z"
|
||||
content="""
|
||||
It's sufficient to have "* annex.largefiles=(largerthan=100kb)"
|
||||
in .gitattributes.
|
||||
|
||||
Even "* annex.largefiles=(largerthan=0kb)" will reproduce it.
|
||||
|
||||
Ok, I see why.. It's running the largefile matcher on the destination file
|
||||
before it renames the temp file to it!
|
||||
|
||||
Seems to have been broken this way ever since addurl got largefiles
|
||||
support. Testing didn't catch it because it only affects largefiles
|
||||
expressions that need to examine the file.
|
||||
|
||||
Fixed in git. Audited other checkFileMatcher calls for this problem;
|
||||
the rest are ok.
|
||||
"""]]
|
||||
|
|
@ -11,3 +11,7 @@ This isn't how it behaves. It would be more accurate as (emphasis on changes):
|
|||
> For example, adding the url http://www.example.com/dir/subdir/bigfile with --pathdepth=1 will use "**dir_subdir_bigfile**", while --pathdepth=3 will use "bigfile".
|
||||
|
||||
For what I am doing (adding a directory tree with addurl and file:// URLs), I'd actually like the behaviour described (to recreate the tree), but I'm not sure which one was the *intended* behaviour..
|
||||
|
||||
> [[done]]; bug report didn't show what was wrong; I can see nothing wrong;
|
||||
> bug reporter cannot seem to remember what was wrong. Probably user error.
|
||||
> --[[Joey]]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,21 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2016-11-16T19:04:37Z"
|
||||
content="""
|
||||
Seems to work as described here:
|
||||
|
||||
joey@darkstar:~/tmp/r>rm localhost__joey_blog_index.html
|
||||
joey@darkstar:~/tmp/r>git annex addurl --pathdepth 2 http://localhost/~joey/blog/index.html
|
||||
addurl blog/index.html (downloading http://localhost/~joey/blog/index.html ...)
|
||||
/home/joey/tmp/r/ 100%[===========>] 40.70K --.-KB/s in 0s
|
||||
ok
|
||||
(recording state in git...)
|
||||
joey@darkstar:~/tmp/r>ls
|
||||
blog/
|
||||
joey@darkstar:~/tmp/r>ls blog
|
||||
index.html
|
||||
|
||||
It would probably help if you can provide a test case where it does not work
|
||||
as described.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="CandyAngel"
|
||||
avatar="http://cdn.libravatar.org/avatar/15c0aade8bec5bf004f939dd73cf9ed8"
|
||||
subject="comment 4"
|
||||
date="2016-11-25T20:27:07Z"
|
||||
content="""
|
||||
I really don't know what to say. I can't even figure out which computer I updated git-annex on to test if it was still happening.. let alone reproduce it anymore. It does work fine.
|
||||
|
||||
I'm so sorry to bother you with this, I've done something stupid! This is exactly why you ask for a transcript of bugs occurring. (Feel free to use this as an example for why you ask for them, so some good can come of it at least..).
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="moc514@eb7af2cd9147722b29f32b6606feb2b8563dfac8"
|
||||
nickname="moc514"
|
||||
avatar="http://cdn.libravatar.org/avatar/c8c98fc66ef014e61c163375ca9e7422"
|
||||
subject="Nexus 6p"
|
||||
date="2016-12-16T02:08:21Z"
|
||||
content="""
|
||||
I also have the same issue with the Nexus 6p with 7.1.1
|
||||
"""]]
|
||||
78
doc/bugs/annex_add_ignores_.-prefixed_directories.mdwn
Normal file
78
doc/bugs/annex_add_ignores_.-prefixed_directories.mdwn
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
### Please describe the problem.
|
||||
|
||||
annex add seems to ignore content under directories having . prefix.
|
||||
|
||||
We thought to unify (across direct/indirect/v6) adding files to annex repository by using 'git annex add' with corresponding setting for largefiles for any addition, but it seems to ignore content under .-prefixed directories, unlike git
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
6.20161122+gitg9f179ae-1~ndall+1
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
hopa:/tmp/datalad_temp_test_annex_add_no_dotfilesqMXck8
|
||||
$> git status
|
||||
On branch master
|
||||
|
||||
Initial commit
|
||||
|
||||
nothing to commit (create/copy files and use "git add" to track)
|
||||
|
||||
$> mkdir .dir dir; echo 123 > .dir/123; echo 124 > dir/124
|
||||
|
||||
$> git status
|
||||
On branch master
|
||||
|
||||
Initial commit
|
||||
|
||||
Untracked files:
|
||||
(use "git add <file>..." to include in what will be committed)
|
||||
|
||||
.dir/
|
||||
dir/
|
||||
|
||||
nothing added to commit but untracked files present (use "git add" to track)
|
||||
|
||||
$> git annex add -c 'annex.largefiles=nothing' .
|
||||
add dir/124 (non-large file; adding content to git repository) ok
|
||||
(recording state in git...)
|
||||
|
||||
$> git status
|
||||
On branch master
|
||||
|
||||
Initial commit
|
||||
|
||||
Changes to be committed:
|
||||
(use "git rm --cached <file>..." to unstage)
|
||||
|
||||
new file: dir/124
|
||||
|
||||
Untracked files:
|
||||
(use "git add <file>..." to include in what will be committed)
|
||||
|
||||
.dir/
|
||||
|
||||
|
||||
# and with regular git
|
||||
$> git -c 'annex.largefiles=nothing' add .
|
||||
|
||||
$> git status
|
||||
On branch master
|
||||
|
||||
Initial commit
|
||||
|
||||
Changes to be committed:
|
||||
(use "git rm --cached <file>..." to unstage)
|
||||
|
||||
new file: .dir/123
|
||||
new file: dir/124
|
||||
|
||||
|
||||
"""]]
|
||||
|
||||
Ref: https://github.com/datalad/datalad/issues/1027
|
||||
|
||||
[[!meta author=yoh]]
|
||||
|
||||
[[done]]; oh -- it is RTFM: --include-dotfiles --[[yoh]]
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://launchpad.net/~felixonmars"
|
||||
nickname="felixonmars"
|
||||
avatar="http://cdn.libravatar.org/avatar/17284a3bb2e4ad9d3be8fab31f49865be9c1dc22143c728de731fe800a335d38"
|
||||
subject="comment 1"
|
||||
date="2016-11-28T04:17:12Z"
|
||||
content="""
|
||||
aws has merged a PR to support http-conduit 2.2, but git-annex itself doesn't build with the new component yet:
|
||||
|
||||
```
|
||||
[ 95 of 544] Compiling Utility.Url ( Utility/Url.hs, dist/build/git-annex/git-annex-tmp/Utility/Url.o )
|
||||
|
||||
Utility/Url.hs:354:34: error:
|
||||
* The constructor `StatusCodeException' should have 2 arguments, but has been given 3
|
||||
* In the pattern: StatusCodeException s _ _
|
||||
In an equation for `matchStatusCodeException':
|
||||
matchStatusCodeException want e@(StatusCodeException s _ _)
|
||||
| want s = Just e
|
||||
| otherwise = Nothing
|
||||
|
||||
Utility/Url.hs:354:34: error:
|
||||
* Couldn't match expected type `HttpException'
|
||||
with actual type `HttpExceptionContent'
|
||||
* In the pattern: StatusCodeException s _ _
|
||||
In an equation for `matchStatusCodeException':
|
||||
matchStatusCodeException want e@(StatusCodeException s _ _)
|
||||
| want s = Just e
|
||||
| otherwise = Nothing
|
||||
```
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2016-12-13T15:58:25Z"
|
||||
content="""
|
||||
This got fixed in the meantime. Note that posting comments to a bug that
|
||||
has already been closed is a good way to get new problems not to be
|
||||
noticed..
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
[[!comment format=mdwn
|
||||
username="http://svario.it/gioele"
|
||||
nickname="Gioele"
|
||||
avatar="http://cdn.libravatar.org/avatar/af2f2ba0dafe4650011d20f2168d43cff773aba97f55ae5b252bb873f391c1e2"
|
||||
subject="compiled with GHC 8, but LOCPATH is still set"
|
||||
date="2016-12-21T21:51:09Z"
|
||||
content="""
|
||||
This bug does not want to die.
|
||||
|
||||
The current standalone build (`6.20161211-gc3ab3c668`) has been compiled with GHC 8 but when I launch `runshell`, I still see that `LOCPATH` is set and the character encoding is messed up.
|
||||
|
||||
I deduced the version of GHC used to compile git-annex with `strings ./shimmed/git-annex/git-annex | grep 'GHC [0-9]'`.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 6"""
|
||||
date="2016-12-24T16:57:45Z"
|
||||
content="""
|
||||
This is an old closed bug report. The recent comments are about a
|
||||
completely unrelated issue, which I suspect was fixed by
|
||||
[[!commit 95c8b37544c383983712c5c368dd803c51bf8eeb]].
|
||||
|
||||
Please file new bug reports if you have an issue, if the old bug report was
|
||||
closed years ago.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
### Please describe the problem.
|
||||
directory 1.3.0.0 causes a conflict for "getFileSize"
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
Build git-annex with directory 1.3.0.0 (first need to bump max directory version on concurrent-output (and aws if building with s3))
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
6.20161210 on macOS 10.11 El Capitan
|
||||
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
[23 of 34] Compiling Common ( Common.hs, dist/setup/Common.o )
|
||||
|
||||
Common.hs:3:16: error:
|
||||
Conflicting exports for ‘getFileSize’:
|
||||
‘module X’ exports ‘X.getFileSize’
|
||||
imported from ‘Utility.Directory’ at Common.hs:28:1-29
|
||||
(and originally defined in ‘System.Directory’)
|
||||
‘module X’ exports ‘X.getFileSize’
|
||||
imported from ‘Utility.FileSize’ at Common.hs:34:1-28
|
||||
(and originally defined at Utility/FileSize.hs:26:1-11)
|
||||
"""]]
|
||||
|
||||
A fix, though possibly not best, is to make this change in Common.hs:
|
||||
[[!format sh """
|
||||
import Utility.Directory as X hiding (getFileSize)
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
Yes :)
|
||||
|
||||
> [[fixed|done]]; thanks for reporting!
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
### Please describe the problem.
|
||||
I tried to use `git-annex-fsck --all --from remote` to check files on a special remote, but git-annex did a scan of the local repo instead. If I don't use the `--all` flag, it correctly checks the files on the remote (but just the files in the current checked out branch).
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
mkdir repo
|
||||
mkdir special
|
||||
cd repo
|
||||
git init
|
||||
git annex init
|
||||
git annex initremote special type=directory directory=../special encryption=none
|
||||
touch testfile
|
||||
git annex add testfile
|
||||
git annex copy testfile --to special
|
||||
chmod -R +w ../special/*
|
||||
rm -r ../special/*
|
||||
git annex fsck --all --from special # should check special remote but checks local repo instead
|
||||
git diff git-annex^ git-annex # activity log shows that it checked special remote
|
||||
git annex fsck --from special # correctly checks special remote, identifies missing file
|
||||
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
6.20161012 on Ubuntu 16.10
|
||||
|
||||
### Have you had any luck using git-annex before?
|
||||
Yes, it's been very helpful for managing large files between laptops, desktops, external storage, and remote storage.
|
||||
|
||||
> Thanks for an excellent test case and a clear bug report. I've fixed this
|
||||
> bug. [[done]] --[[Joey]]
|
||||
38
doc/bugs/git-annex_fromkey_barfs_on_utf-8_input.mdwn
Normal file
38
doc/bugs/git-annex_fromkey_barfs_on_utf-8_input.mdwn
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
### Please describe the problem.
|
||||
|
||||
I'm sending a stream of keys and filenames to git-annex fromkey on stdin, and it errors out with "git-annex: <stdin>: hGetContents: invalid argument (invalid byte sequence)". On the other hand yipdw tried to reproduce this and it worked fine for him, so I must be doing something wrong.
|
||||
|
||||
I have LANG=en_US.UTF-8 set in my environment, if that matters.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
[[!format sh """
|
||||
echo "MD5-s3263532--0b4d070eff7baa8ef314ca330aecb71f é" | git-annex fromkey
|
||||
"""]]
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
[[!format sh """
|
||||
git-annex version: 6.20161118-g0a34f08
|
||||
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
local repository version: 5
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
operating system: linux x86_64
|
||||
"""]]
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
Note that this is indeed valid utf-8:
|
||||
|
||||
[[!format sh """
|
||||
db48x ~ projects IA.BAK-server echo "é" | hexdump -C
|
||||
00000000 c3 a9 0a |...|
|
||||
00000003
|
||||
"""]]
|
||||
|
||||
> Despite my strange inability to reproduce these, there's really only one
|
||||
> thing that can fix it, namely using fileEncoding. Now done for all batch
|
||||
> and stdin reading stuff. [[fixed|done]] I suppose. --[[Joey]]
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
[[!comment format=mdwn
|
||||
username="alpernebbi"
|
||||
avatar="http://cdn.libravatar.org/avatar/daf2abb14f39e28ad75d5f9a03fcd106"
|
||||
subject="UTF-8 problems in some other commands"
|
||||
date="2016-12-05T20:46:07Z"
|
||||
content="""
|
||||
Running the command above gives me the same error on Xubuntu 16.04, using `git-annex-standalone` package from NeuroDebian repositories.
|
||||
|
||||
git-annex version: 6.20161122+gitg9f179ae-1~ndall+1
|
||||
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
local repository version: 5
|
||||
supported repository versions: 3 5 6
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5
|
||||
operating system: linux x86_64
|
||||
|
||||
I encountered other commands that fail as well:
|
||||
|
||||
$ touch u.txt ü.txt
|
||||
$ git annex add
|
||||
|
||||
$ git-annex calckey ü.txt
|
||||
# prints key
|
||||
|
||||
$ git-annex calckey --batch
|
||||
ü.txt
|
||||
# dies
|
||||
|
||||
$ git-annex lookupkey ü.txt
|
||||
# prints key
|
||||
|
||||
$ git-annex lookupkey --batch
|
||||
ü.txt
|
||||
# dies
|
||||
|
||||
$ git-annex metadata --batch --json
|
||||
{\"file\":\"ü.txt\"}
|
||||
# dies
|
||||
|
||||
$ git-annex metadata --batch --json
|
||||
{\"file\":\"u.txt\",\"fields\":{\"ü\":[\"b\"]}}
|
||||
# dies
|
||||
|
||||
$ git-annex metadata --batch --json
|
||||
{\"file\":\"u.txt\",\"fields\":{\"a\":[\"ü\"]}}
|
||||
# dies
|
||||
|
||||
All those die without output, all $? are 0.
|
||||
No values were recorded to metadata.
|
||||
Also:
|
||||
|
||||
$ git-annex-metadata --json
|
||||
# entry for \"ü.txt\" has \"file\":\"<22><>.txt\"
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2016-12-08T21:03:14Z"
|
||||
content="""
|
||||
I'm not able to reproduce the original reported problem; it works even when
|
||||
using the C locale. However, it may be that this patch would fix it:
|
||||
|
||||
<pre>
|
||||
diff --git a/Command/FromKey.hs b/Command/FromKey.hs
|
||||
index dca63aa..6a81c1f 100644
|
||||
--- a/Command/FromKey.hs
|
||||
+++ b/Command/FromKey.hs
|
||||
@@ -45,7 +45,9 @@ startMass = do
|
||||
next massAdd
|
||||
|
||||
massAdd :: CommandPerform
|
||||
-massAdd = go True =<< map (separate (== ' ')) . lines <$> liftIO getContents
|
||||
+massAdd = do
|
||||
+ liftIO $ fileEncoding stdin
|
||||
+ go True =<< map (separate (== ' ')) . lines <$> liftIO getContents
|
||||
where
|
||||
go status [] = next $ return status
|
||||
go status ((keyname,f):rest) | not (null keyname) && not (null f) = do
|
||||
</pre>
|
||||
|
||||
(The NeuroDebian git-annex-standalone may well have had its locale
|
||||
installation broken by [[!commit c07981122672f6cc87ca08efb57d8a7b1e2f5725]],
|
||||
which assumes that the git-annex.linux is writable by the user.
|
||||
I doubt that is related to the original bug report.)
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="alpernebbi"
|
||||
avatar="http://cdn.libravatar.org/avatar/daf2abb14f39e28ad75d5f9a03fcd106"
|
||||
subject="comment 3"
|
||||
date="2016-12-10T07:36:04Z"
|
||||
content="""
|
||||
I experienced all these with the [linux standalone](https://git-annex.branchable.com/install/Linux_standalone/) from this site as well.
|
||||
|
||||
However, I couldn't reproduce them in a Debian unstable chroot where I installed the `git-annex` package from their repos.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 8"""
|
||||
date="2016-11-16T21:48:49Z"
|
||||
content="""
|
||||
The arm daily build now uses a 32kb page size. So try
|
||||
<https://downloads.kitenet.net/git-annex/autobuild/armel/git-annex-standalone-armel.tar.gz>
|
||||
|
||||
That has been verified to fix the problem on a Drobo 5N.
|
||||
|
||||
This may still not be enough for some of the affected NAS devices, which
|
||||
use a 64kb page size. Unfortunately, gold fails to link with a 64kb page
|
||||
size: <http://bugs.debian.org/844467>
|
||||
"""]]
|
||||
|
|
@ -65,3 +65,5 @@ I would just like to be sure nothing else is hidden.
|
|||
> .git/config to remove that in order to recover from the problem, so might
|
||||
> as well remove `.git/annex/ssh_config` too.
|
||||
> --[[Joey]]
|
||||
|
||||
>> Fixed more by stopping using `.git/annex/ssh_config` at all. --[[Joey]]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,17 @@
|
|||
[[!comment format=mdwn
|
||||
username="scottgorlin@a32946b2aad278883c1690a0753241583a9855b9"
|
||||
nickname="scottgorlin"
|
||||
avatar="http://cdn.libravatar.org/avatar/2dd1fc8add62bbf4ffefac081b322563"
|
||||
subject="IgnoreUnknown Include considered harmful?"
|
||||
date="2016-11-23T20:07:45Z"
|
||||
content="""
|
||||
As noted, include appears to not work on a mac at the moment. This means git-annex silently ignores the included configs, which may be required to ssh to the remotes of interest. This is happening to me.
|
||||
|
||||
My understanding is that ssh aliases are the recommended way of juggling multiple private keys amongst multiple hosts, so it is a required part of many git workflows. In this particular case, I have set up git annex on a NAS which does not allow multiple ssh users (QNAP) and the authentication is done only via key identity, not username. Thus, host aliases are necessary.
|
||||
|
||||
If one config can't include another, I would prefer an early failure indicating a problem with the config file, or better, a solution where git-annex doesn't require a config. In this scenario, git fetch remote_name and git annex copy --to remotename do not resolve to the same alias definitions (the latter is missing because of the ignored config!).
|
||||
|
||||
I got my setup to work only by finding and manually editing <repo>/.git/annex/ssh_config, which to my knowledge is undocumented (ie when is it written? do any commands change it?); manual mucking around inside .git to me is not a good practice, and for now I have two different alias's defined (in repo and in ~/.ssh/config)
|
||||
|
||||
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
[[!comment format=mdwn
|
||||
username="palday@91f366b5178879146d2b6e1e53bfa21389ee89a8"
|
||||
nickname="palday"
|
||||
avatar="http://cdn.libravatar.org/avatar/077a63af75ddba159980fbf88690f401"
|
||||
subject="Temporary workaround until the brew formula is updated"
|
||||
date="2016-11-29T02:17:52Z"
|
||||
content="""
|
||||
The homebrew formula doesn't yet this fix, but you can get around the problem in the meantime by getting a newer SSH via homebrew:
|
||||
|
||||
```
|
||||
brew install homebrew/dupes/openssh
|
||||
```
|
||||
|
||||
You can then choose to keep that or get rid of it when the formula for git annex is later updated.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
### Please describe the problem.
|
||||
|
||||
The OpenSSH client parses configuration in a "first match wins" manner, and this also applies to `IgnoreUnknown`. This means that when git-annex's `Include ~/.ssh/config` is processed, any user-specified `IgnoreUnknown` setting in the global configuration will be ignored because it has already been set. As a result, every time git-annex runs ssh, it immediately exits with an error:
|
||||
|
||||
[[!format text """
|
||||
drop vol3 somefile.mkv (locking vol5...) (lockcontent failed) (checking vol5...)
|
||||
/home/grawity/.ssh/config: line 217: Bad configuration option: gssapikeyexchange
|
||||
/home/grawity/.ssh/config: terminating, 1 bad configuration options
|
||||
failed
|
||||
"""]]
|
||||
|
||||
To be fair, this might be an OpenSSH bug (IgnoreUnknown ought to be merged), but it seems git-annex is triggering it unnecessarily.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
1. In `~/.ssh/config`, have some unrecognized options (e.g. `GSSAPIKeyExchange`) and a corresponding `IgnoreUnknown`.
|
||||
|
||||
2. Try to use a git-annex feature which directly invokes ssh, e.g. get or drop.
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
6.20161210 on Arch, but I think this was introduced in a 201611* release.
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Yes, it's been taking care of my archives for nearly a year.
|
||||
|
||||
> How annoying, ssh seems to make it impossible to set this in a way that
|
||||
> doesn't break some configurations. Oh well, gave up on setting it
|
||||
> and removed the code to do so. [[done]] --[[Joey]]
|
||||
10
doc/bugs/ssh___34__Include__34___feature_broke_Android.mdwn
Normal file
10
doc/bugs/ssh___34__Include__34___feature_broke_Android.mdwn
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
### Please describe the problem.
|
||||
https://git-annex.branchable.com/bugs/git_annex_init_failed_due_to_unsupported_ssh_option/ deal with Include not being supported by pre 7.3 by using the 6.4+ IgnoreUnknown directive.
|
||||
|
||||
Unfortunately, the Android apk (which I got from https://downloads.kitenet.net/git-annex/android/current/5.0/git-annex.apk) has (according to ssh -v) OpenSSH_6.0p1.
|
||||
|
||||
I had to edit .git/annex/ssh.config to comment out the three offending lines and then append the contents of ~/.ssh/config to get git-annex working again.
|
||||
|
||||
(This is on a Nexus 10 running stock, though I doubt it matters)
|
||||
|
||||
> Reverted use of this feature for now.[[done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-12-08T21:11:31Z"
|
||||
content="""
|
||||
Indeed, the ssh bundled in the apk is shockingly old by now, and needs to
|
||||
get updated.
|
||||
"""]]
|
||||
46
doc/bugs/unRAID_shares_treated_as_a_crippled_filesystem.mdwn
Normal file
46
doc/bugs/unRAID_shares_treated_as_a_crippled_filesystem.mdwn
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
### Please describe the problem.
|
||||
|
||||
Running `git annex init` on an [unRAID server](https://lime-technology.com/what-is-unraid/) results in an annex created with `crippledfilesystem = true` and `direct = true`. I understand from reading [this](https://git-annex.branchable.com/design/assistant/blog/day_188__crippled_filesystem_support/) that it occurs when `git annex init` performs a probe to determine if all of the following are supported:
|
||||
|
||||
1. symlinks
|
||||
2. hard links
|
||||
3. unix permissions
|
||||
|
||||
Although unRAID disks are formatted with xfs, and therefore support all three of the above, I'm assuming that unRAID's method of combining multiple disks into one "share" is the cause of the problem (hardlinks still work on a single disk, but not on shares that span multiple disks). Symlinks and unix permissions work normally in the unRAID-created shares.
|
||||
|
||||
Is there any way to allow the use of 'indirect' mode with multi-disk shares? As I mentioned, symlinks and unix permissions work normally--it's only the hardlinks that won't work across the multi-disk shares.
|
||||
|
||||
I can create a 'normal' annex as long as I `cd` to a single disk drive first--what would happen if the annex was later moved onto a multi-disk share? Would it still work? Would it fail gracefully? Would it cause data loss?
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
cd /mnt/user/NameOfShare
|
||||
git init
|
||||
git annex init
|
||||
|
||||
The following will result in the creation of a 'normal' indirect share:
|
||||
|
||||
cd /mnt/disk1
|
||||
git init
|
||||
git annex init
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
git-annex version: 6.20161211-gc3ab3c668
|
||||
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
# If you can, paste a complete transcript of the problem occurring here.
|
||||
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
|
||||
|
||||
|
||||
# End of transcript or log.
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Has been working great, so far, except for the above.
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
### Please describe the problem.
|
||||
Recent builds of git-annex spew out many lines such as:
|
||||
|
||||
git-annex: unable to decommit memory: Invalid argument
|
||||
git-annex: unable to decommit memory: Invalid argument
|
||||
git-annex: unable to decommit memory: Invalid argument
|
||||
git-annex: unable to decommit memory: Invalid argument
|
||||
git-annex: unable to decommit memory: Invalid argument
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
This happens to me syncing any large repository now.
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
git-annex version: 6.20161118-g0a34f08
|
||||
|
||||
uname -r: 4.4.14-11.pvops.qubes.x86_64
|
||||
|
||||
/etc/system-release: Fedora release 23 (Twenty Three)
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
I found this: https://ghc.haskell.org/trac/ghc/ticket/12495
|
||||
|
||||
It looks like this is a problem that occurs only on kernels < 4.5, when ghc is built with a newer glibc, I think.
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
Git annex rocks!
|
||||
|
||||
> [[fixed|done]]; fix is confirmed, all linux standalone builds are updated
|
||||
> (and I pinged Yoh to update the neurodebian standalone build). --[[Joey]]
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 10"""
|
||||
date="2016-12-19T19:53:11Z"
|
||||
content="""
|
||||
The only way the files could be in lost+found is if the system crashed or
|
||||
there was a disk error etc. Can't be due to this bug. So, it may be that
|
||||
this bug did not actually cause any data loss. The screwed up symlinks could
|
||||
have been caused by a disk error.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
[[!comment format=mdwn
|
||||
username="0xloem@0bd8a79a57e4f0dcade8fc81d162c37eae4d6730"
|
||||
nickname="0xloem"
|
||||
avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b"
|
||||
subject="Corrupt Links Produced, Significant Data Loss"
|
||||
date="2016-12-10T12:31:31Z"
|
||||
content="""
|
||||
Additionally, I added a mess of files with this release of git-annex, and of the 200 files added, three of the links produced were corrupt. I'm still searching for where these files have gone to recover the data.
|
||||
|
||||
The files look like this in `ls -l`, they were bup files:
|
||||
|
||||
lrwxrwxrwx 1 user user 338 Jun 17 22:36 bup.git/objects/pack/pack-47b493a3bbbd22200d2b390c277e49ce713243cc.pack -> *??:?;J?????????
|
||||
lrwxrwxrwx 1 user user 336 Jun 17 21:41 bup.git/objects/pack/pack-4d202b3929b187d4acaf1602526e7344eef1edc8.pack -> ?p???GWj??????ܥ??{b?#???>C??%??????~à???/hjT;?p??d?8??oyE?K?)6?uL+??h??&???SB}?'s??֫{?>^i?&?f??^{ш??aD??t4?C?sBTk>d6H???5h3?ڋ6fAa??=?r????j?????a8K??????????B?~????I͕?T7?Y??=???b?7C???鋤??8???\"?????#???M?????}z?A??9?C>?-?GD??7?ј;'P?H??ɑ??Zr?/U???W?G??3@\"??Ȧ?z?h???U??Ԇ???R??u??I????62??>@??@?a??x???}?????)d?G;(???m_?^3?????T
|
||||
lrwxrwxrwx 1 user user 332 Jul 20 07:32 bup.git/objects/pack/pack-5328381f3b023c1356581c22d1e74d4eda0b46a3.idx -> c??'w??????????m?q#?ٱCO??o????ʃ?Ʃڌ??[???Ѐ??*?;.?c?N?0?????D$ o?r????8BGn?96gY?B?Z1?=???{??z?71????!aG?>?u)???i\?G[???:?Kk??%??.mu???n???K??ǚ????q&Z-?E???]??/?6???}
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 2"""
|
||||
date="2016-12-10T15:13:44Z"
|
||||
content="""
|
||||
I think that the "x86-32, for ancient kernels" build should avoid this
|
||||
problem. <http://git-annex.branchable.com/install/Linux_standalone/>
|
||||
|
||||
It's very surprising if this lead to symlinks being created that apparently
|
||||
contain garbage in their link targets. Perhaps glibc is failing in a way
|
||||
with the old kernel that leads to memory corruption? I have asked the GHC
|
||||
developers if that could be the case in
|
||||
<https://ghc.haskell.org/trac/ghc/ticket/12865>
|
||||
|
||||
I hope that the content of your files is in fact somewhere under
|
||||
`.git/annex/objects/` -- look around, and with some luck, you may find it.
|
||||
Unfortunately, the information about, which object file goes with which
|
||||
working tree has apparently been lost. (Also, you might check if these
|
||||
symlinks have been staged in git; it's possible though unlikely that the
|
||||
correct link target got staged in git.)
|
||||
|
||||
I have filed a bug on Debian's ghc to get them to fast-track getting the
|
||||
patch into ghc. <https://bugs.debian.org/847677>
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="0xloem@0bd8a79a57e4f0dcade8fc81d162c37eae4d6730"
|
||||
nickname="0xloem"
|
||||
avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b"
|
||||
subject="comment 3"
|
||||
date="2016-12-11T00:26:41Z"
|
||||
content="""
|
||||
Thank you so much for the prompt response. My system wouldn't shut down cleanly after this, either, so there may have been something else screwy going on. Still, I'll be using the build for ancient kernels exclusively for the near future. Thank you.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2016-12-11T19:35:21Z"
|
||||
content="""
|
||||
All Linux standalone builds have been updated with a version of ghc that
|
||||
has that bug fixed. Can you please upgrade and verify it's fixed?
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="0xloem@0bd8a79a57e4f0dcade8fc81d162c37eae4d6730"
|
||||
nickname="0xloem"
|
||||
avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b"
|
||||
subject="Verification"
|
||||
date="2016-12-11T00:59:50Z"
|
||||
content="""
|
||||
I saw the new comment on the download page and tried running `git annex test`. I can confirm that `git annex test` eventually segfaults using the normal build on my system, whereas it passes successfully using the 'ancient kernels' build. The version strings output for the two are identical.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="0xloem@0bd8a79a57e4f0dcade8fc81d162c37eae4d6730"
|
||||
nickname="0xloem"
|
||||
avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b"
|
||||
subject="Nope a Fluke"
|
||||
date="2016-12-11T13:26:29Z"
|
||||
content="""
|
||||
Apologies. I can't reproduce the segfault running the tests again. The corruption and crashing seems to have been some horrifying fluke.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 7"""
|
||||
date="2016-12-12T17:30:03Z"
|
||||
content="""
|
||||
Can you please check if the current builds still have the "unable to
|
||||
decommit memory" problem or not?
|
||||
|
||||
(What it does after that error is probably nondeterministic, fixing that
|
||||
error is the crucial thing.)
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="0xloem@0bd8a79a57e4f0dcade8fc81d162c37eae4d6730"
|
||||
nickname="0xloem"
|
||||
avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b"
|
||||
subject="comment 8"
|
||||
date="2016-12-13T15:52:34Z"
|
||||
content="""
|
||||
It looks like the errors are gone. Thank you so much.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="xloem"
|
||||
avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b"
|
||||
subject="Coda"
|
||||
date="2016-12-18T14:56:17Z"
|
||||
content="""
|
||||
The missing files turned up in 'lost+found' the next time I ran fsck =)
|
||||
"""]]
|
||||
25
doc/bugs/wget_always_shows_progress_bar.mdwn
Normal file
25
doc/bugs/wget_always_shows_progress_bar.mdwn
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
### Please describe the problem.
|
||||
|
||||
git annex addurl or importfeed operations were quiet on git-annex versions older than 5.20141219, if
|
||||
annex.web-options was set to "--quiet". But now the --show-progress option is always passed to wget.
|
||||
|
||||
In some use cases this might be nice, but I'm using GNU Parallel to update multiple podcast feeds
|
||||
concurrently, and it causes wget to output the ugly "dot" indicator for every feed, which is totally
|
||||
useless since parallel buffers and groups the output. Adding "--no-show-progress" to web-options
|
||||
does not help (it does not override --show-progress), nor does redirecting stdout to /dev/null.
|
||||
Redirecting stderr would hide possible errors.
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
|
||||
parallel git annex importfeed --relaxed --quiet ::: http://feeds.feedburner.com/freakonomicsradio
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
|
||||
5.20151208-1~bpo8+1 on Debian.
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
I love git annex and use it daily.
|
||||
|
||||
|
||||
> [[done]] --[[Joey]]
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-12-13T15:49:20Z"
|
||||
content="""
|
||||
Since 2014, git-annex has used wget -q --show-progress to get a
|
||||
progress bar without the several other lines of output it would normally
|
||||
display. Whether a given git-annex build does this depends on what
|
||||
version of wget it saw at configure time.
|
||||
|
||||
Running git-annex with --quiet will disable the wget progress bar (and
|
||||
other git-annex output). This seems like the thing to do if you're running
|
||||
git-annex concurrently. (Of course, git-annex also has its own built-in
|
||||
concurrency with -J which can display multiple download progress bars in a
|
||||
nice way.)
|
||||
|
||||
Still, might as well make the web-options come after the default options so
|
||||
they can be overridden. Doing so.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
The suggestion to make remotes available isn't really applicable, since the error was local.
|
||||
|
||||
This is with git annex 6.20161110-gd48f4ca.
|
||||
|
||||
[[!format sh """
|
||||
../git-annex.linux/git-annex get archiveteam-fire/metro.co.uk-urls-2007-04-12-20150627/metro.co.uk-urls-2007-04-12-20150627_meta.xml
|
||||
get archiveteam-fire/metro.co.uk-urls-2007-04-12-20150627/metro.co.uk-urls-2007-04-12-20150627_meta.xml
|
||||
not enough free space, need 98.82 GB more (use --force to override this check or adjust annex.diskreserve)
|
||||
|
||||
Unable to access these remotes: web
|
||||
|
||||
Try making some of these repositories available:
|
||||
00000000-0000-0000-0000-000000000001 -- web
|
||||
9f8218c0-763f-463d-9152-ecdc56d4452c -- iabak@redwyne.jwintjen.de:~/IA.BAK/shard12
|
||||
failed
|
||||
git-annex: get: 1 failed
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
|
||||
mixed success
|
||||
|
|
@ -49,7 +49,7 @@
|
|||
<iframe width=1024 scrolling=no frameborder=0 marginheight=0 marginwidth=0 src="https://downloads.kitenet.net/git-annex/autobuildtest/x86_64-apple-yosemite/testresult/status">
|
||||
</iframe>
|
||||
<h2><a href="https://qa.nest-initiative.org/view/msysGit/job/msysgit-git-annex-assistant-test/">Windows</a></h2>
|
||||
<a href="https://qa.nest-initiative.org/view/msysGit/job/msysgit-git-annex-assistant-test/">here</a>
|
||||
<a href="http://vps.zaytsev.net:8080/">here</a> (firewalled from most locations; kite can access it)
|
||||
<h2><a href="https://buildd.debian.org/status/package.php?p=git-annex&suite=sid">Debian</a></h2>
|
||||
<iframe width=1024 scrolling=no height=500px frameborder=0 marginheight=0 marginwidth=0 src="https://buildd.debian.org/status/package.php?p=git-annex&suite=sid">
|
||||
</iframe>
|
||||
|
|
|
|||
|
|
@ -46,13 +46,7 @@ or [cjdns](https://github.com/cjdelisle/cjdns) or tor or i2p or [magic wormhole]
|
|||
|
||||
* Awesome.
|
||||
* Easy to install, use; very well known.
|
||||
* May need root to set up a hidden service.
|
||||
* There's been some [haskell packages developed recently](http://www.leonmergen.com/haskell/privacy/2015/05/30/on-anonymous-networking-in-haskell-announcing-tor-and-i2p-for-haskell.html)
|
||||
to communicate with tor and set up onion addresses for a service.
|
||||
Could be used to make git-annex run as a hidden service.
|
||||
However, that relies on tor being configured with a ControlPort,
|
||||
without authentication. The normal tor configuration does not enable a
|
||||
ControlPort.
|
||||
* Supported in git-annex now!
|
||||
|
||||
## i2p status
|
||||
|
||||
|
|
@ -66,26 +60,25 @@ or [cjdns](https://github.com/cjdelisle/cjdns) or tor or i2p or [magic wormhole]
|
|||
|
||||
## general design
|
||||
|
||||
* Make address.log that contains (uuid, transport, address, Maybe authtoken)
|
||||
* The authtoken is an additional guard, to protect against transports
|
||||
where the address might be able to be guessed, or observed by the rest of
|
||||
the network.
|
||||
* Some addresses can be used with only the provided authtoken
|
||||
from the address.log. Remotes can be auto-enabled for these.
|
||||
* Other addresses have Nothing povided for the authtoken, and one
|
||||
has to instead be provided during manual enabling of the remote.
|
||||
* There is a generic P2P protocol, which should be usable with any P2P
|
||||
system that can send messages between peers.
|
||||
* A p2p remote has an url like tor-annex::fijdksajdksjfkj, which connects
|
||||
to a specific peer. The peer's address may be kept private, but
|
||||
the design allows the address to be public without giving access to
|
||||
the peer.
|
||||
* An authtoken also needs to be presented when connecting with a peer.
|
||||
This is stored in local creds storage and must be kept private.
|
||||
* The remotedaemon runs, and/or communicates with the program implementing
|
||||
the network transport. For example for tor, the remotedaemon runs
|
||||
the hidden service, and also connects to the tor hidden services of
|
||||
other nodes.
|
||||
the P2P network. For example for tor, the remotedaemon runs the
|
||||
hidden service.
|
||||
* The remotedaemon handles both sides of git push over the transport.
|
||||
* The remotedaemon may also support sending objects over the transport,
|
||||
depending on the transport.
|
||||
|
||||
## address discovery
|
||||
## address exchange
|
||||
|
||||
The address is a public key, and the authtoken is some large chunk of data,
|
||||
so won't want to type that in. Need discovery.
|
||||
so won't want to type that in. Need discovery or exchange for peering.
|
||||
|
||||
* Easy way is any set of repos that are already connected can communicate
|
||||
them via address.log.
|
||||
|
|
@ -96,32 +89,26 @@ so won't want to type that in. Need discovery.
|
|||
it can be read over the phone.
|
||||
* Users may not have a way to communicate with perfect forward secrecy.
|
||||
So it would be good to have a address+authtoken that can only be used
|
||||
one time during pairing:
|
||||
one time during pairing.
|
||||
* Check out [PAKE](https://en.wikipedia.org/wiki/Password-authenticated_key_agreement)
|
||||
for MITM resistance.
|
||||
* Possibly use magic wormhole to exchange the address, which avoids
|
||||
the users needing to exchange so much data. The magic wormhole code
|
||||
is just 3 words, and it uses PAKE.
|
||||
|
||||
I tried it, and opened a couple of bug reports that would be useful in
|
||||
integrating it with git-annex:
|
||||
|
||||
1. Alice uses the webapp to generate a one-time address+authtoken,
|
||||
and sends it into a message to Bob.
|
||||
2. Bob enters it into his webapp.
|
||||
3. Bob's assistant contacts Alice's over the transport, presents the
|
||||
one-time authtoken. (Alice's assistant accepts it, and marks it as
|
||||
used so it cannot be used again.)
|
||||
4. Alice's webapp shows that it's ready to finish pairing; so does Bob's.
|
||||
Both wait for their users to confirm before proceeding.
|
||||
5. Alice's assistant generates a new, permanant use authtoken, sends it
|
||||
to Bob's assistant, which stores it and enables a remote using it.
|
||||
6. Bob's assistant generates a new, permanant use authtoken, sends it to
|
||||
Alice's assistant, which stores it and enables a remote using it.
|
||||
7. Alice and Bob's assistants are now paired.
|
||||
- [option to receive to a specific file](https://github.com/warner/magic-wormhole/issues/101)
|
||||
- [machine readable wormhole code ](https://github.com/warner/magic-wormhole/issues/104])
|
||||
|
||||
Note that this exchange can be actively MITMed. If Eve can intercept
|
||||
Alice's message to Bob, then Eve can pair with Alice. Or, if Eve can
|
||||
forge a message from Alice to Bob, Eve can trick Bob into pairing with
|
||||
her.
|
||||
## local lan detection
|
||||
|
||||
If they make a phone call, it's much harder for Eve to MITM it.
|
||||
Eve would need to listen to Alice reading the authtoken and enter it
|
||||
before Bob does, so pairing with Alice. But as long as Alice waits
|
||||
for Bob to confirm he's ready to finish pairing, this will fail,
|
||||
because Bob won't get to that point if the authtoken is intercepted.
|
||||
At connection time, after authentication, the remote can send
|
||||
(ip address, ssh host key). Try sshing to the ip address to check if
|
||||
the host key matches. If so, can enable a ssh remote, which will
|
||||
be cheaper than using the transport. Send the ssh public key back to the
|
||||
remote to get it authorized.
|
||||
|
||||
## remotedaemon
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,6 @@
|
|||
* [[design/caching_database]] for metadata views
|
||||
* [[assistant/deltas]]
|
||||
* [[assistant/gpgkeys]] management for the assistant
|
||||
* [[assistant/telehash]] or similar
|
||||
* [[design/requests_routing]]
|
||||
* [[design/new_repo_versions]]
|
||||
|
||||
|
|
|
|||
23
doc/devblog/day_425__tor.mdwn
Normal file
23
doc/devblog/day_425__tor.mdwn
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
Have waited too long for some next-generation encrypted P2P network, like
|
||||
telehash to emerge. Time to stop waiting; tor hidden services are not as
|
||||
cutting edge, but should work. Updated the [[design|design/assistant/telehash]]
|
||||
and started implementation in the `tor` branch.
|
||||
|
||||
Unfortunately, Tor's default configuration does not enable the ControlPort.
|
||||
And, changing that in the configuration could be problimatic. This
|
||||
makes it harder than it ought to be to register a tor hidden service.
|
||||
So, I implemented a `git annex enable-tor` command, which can be run as root
|
||||
to set it up. The webapp will probably use `su-to-root` or `gksu` to run it.
|
||||
There's some Linux-specific parts in there, and it uses a socket for
|
||||
communication between tor and the hidden service, which may cause problems
|
||||
for Windows porting later.
|
||||
|
||||
Next step will be to get `git annex remotedaemon` to run as a tor hidden
|
||||
service.
|
||||
|
||||
Also made a `no-xmpp` branch which removes xmpp support from the assistant.
|
||||
That will remove 3000 lines of code when it's merged. Will probably wait
|
||||
until after tor hidden services are working.
|
||||
|
||||
Today's work was sponsored by Jake Vosloo on
|
||||
[Patreon](https://www.patreon.com/joeyh/).
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="grawity@2ea26be48562f66fcb9b66307da72b1e2e37453f"
|
||||
nickname="grawity"
|
||||
avatar="http://cdn.libravatar.org/avatar/7003e967f47003bae82966aa373de8ef"
|
||||
subject="comment 1"
|
||||
date="2016-11-15T18:01:18Z"
|
||||
content="""
|
||||
…or `pkexec`, which is present on many systems and generally integrates better with whatever DE/non-DE the user may be running.
|
||||
|
||||
(OTOH, pkexec does not set up X11 access – then again, root helpers shouldn't need it.)
|
||||
"""]]
|
||||
63
doc/devblog/day_426__grab_bag.mdwn
Normal file
63
doc/devblog/day_426__grab_bag.mdwn
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
Fixed one howler of a bug today. Turns out that
|
||||
`git annex fsck --all --from remote` didn't actually check the content of
|
||||
the remote, but checked the local repository. Only `--all` was buggy;
|
||||
`git annex fsck --from remote` was ok. Don't think this is crash priority
|
||||
enough to make a release for, since only `--all` is affected.
|
||||
|
||||
Somewhat uncomfortably made `git annex sync` pass
|
||||
`--allow-unrelated-histories` to git merge. While I do think that git's
|
||||
recent refusal to merge unrelated histories is good in general, the
|
||||
problem is that initializing a direct mode repository involves making an
|
||||
empty commit. So merging from a remote into such a direct mode repository
|
||||
means merging unrelated histories, while an indirect mode repository doesn't.
|
||||
Seems best to avoid such inconsistencies, and the only way I could see to
|
||||
do it is to always use `--allow-unrelated-histories`. May revisit this once
|
||||
direct mode is finally removed.
|
||||
|
||||
Using the git-annex arm standalone bundle on some WD NAS boxes used to
|
||||
work, and then it seems they changed their kernel to use a nonstandard page
|
||||
size, and broke it. This actually seems to be a
|
||||
[bug in the gold linker](http://bugs.debian.org/844467), which defaults to an
|
||||
unncessarily small page size on arm. The git-annex arm bundle is being
|
||||
adjusted to try to deal with this.
|
||||
|
||||
ghc 8 made `error` include some backtrace information. While it's really
|
||||
nice to have backtraces for unexpected exceptions in Haskell, it turns
|
||||
out that git-annex used `error` a lot with the intent of showing an error
|
||||
message to the user, and a backtrace clutters up such messages. So,
|
||||
bit the bullet and checked through every `error` in git-annex and made such
|
||||
ones not include a backtrace.
|
||||
|
||||
Also, I've been considering what protocol to use between git-annex nodes
|
||||
when communicating over tor. One way would be to make it very similar to
|
||||
`git-annex-shell`, using rsync etc, and possibly reusing code from
|
||||
git-annex-shell. However, it can take a while to make a connection across
|
||||
the tor network, and that method seems to need a new connection for each
|
||||
file transfered etc. Also thought about using a http based protocol. The
|
||||
servant library is great for that, you get both http client and server
|
||||
implementations almost for free. Resuming interrupted transfers might
|
||||
complicate it, and the hidden service side would need to listen on a unix
|
||||
socket, instead of the regular http port. It might be worth it to use http
|
||||
for tor, if it could be reused for git-annex http servers not on the tor
|
||||
network. But, then I'd have to make the http server support git pull and
|
||||
push over http in a way that's compatable with how git uses http, including
|
||||
authentication. Which is a whole nother ball of complexity. So, I'm leaning
|
||||
instead to using a simple custom protocol something like:
|
||||
|
||||
> AUTH $localuuid $token
|
||||
< AUTH-SUCCESS $remoteuuid
|
||||
> SENDPACK $length
|
||||
> $gitdata
|
||||
< RECVPACK $length
|
||||
< $gitdata
|
||||
> GET $pos $key
|
||||
< DATA $length
|
||||
< $bytes
|
||||
> SUCCESS
|
||||
> PUT $key
|
||||
< PUT-FROM $pos
|
||||
> DATA $length
|
||||
> $bytes
|
||||
< SUCCESS
|
||||
|
||||
Today's work was sponsored by Riku Voipio.
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://anarc.at/openid/"
|
||||
nickname="anarcat"
|
||||
avatar="http://cdn.libravatar.org/avatar/b36dcf65657dd36128161355d8920a99503def9461c1bb212410980fe6f07125"
|
||||
subject="how about reusing the special remote protocol?"
|
||||
date="2016-11-16T21:58:08Z"
|
||||
content="""
|
||||
git-annex already has a custom protocol detailed in [[design/external_special_remote_protocol]]. it could be quite useful to have that protocol extended to support direct object transfer instead of having to mess around with temporary files like may remotes do, for example...
|
||||
|
||||
maybe that makes no sense at all, i don't know. :) --[[anarcat]]
|
||||
"""]]
|
||||
51
doc/devblog/day_427__free_p2p.mdwn
Normal file
51
doc/devblog/day_427__free_p2p.mdwn
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
For a Haskell programmer, and day where a big thing is implemented
|
||||
without the least scrap of code that touches the IO monad is a good day.
|
||||
And this was a good day for me!
|
||||
|
||||
Implemented the p2p protocol for tor hidden services. Its needs are somewhat
|
||||
similar to the external special remote protocol, but the two protocols are
|
||||
not fully overlapping with one-another. Rather than try to unify them, and
|
||||
so complicate both cases, I prefer to reuse as much code as possible between
|
||||
separate protocol implementations. The generating and parsing of messages
|
||||
is largely shared between them. I let the new p2p protocol otherwise
|
||||
develop in its own direction.
|
||||
|
||||
But, I *do* want to make this p2p protocol reusable for other types of p2p
|
||||
networks than tor hidden services. This was an opportunity to use the Free
|
||||
monad, which I'd never used before. It worked out great, letting me write
|
||||
monadic code to handle requests and responses in the protocol, that reads
|
||||
the content of files and resumes transfers and so on, all independent
|
||||
of any concrete implementation.
|
||||
|
||||
The whole implementation of the protocol only needed 74 lines of monadic code.
|
||||
It helped that I was able to factor out functions like this one, that is used
|
||||
both for handling a download, and by the remote when an upload is sent to it:
|
||||
|
||||
receiveContent :: Key -> Offset -> Len -> Proto Bool
|
||||
receiveContent key offset len = do
|
||||
content <- receiveBytes len
|
||||
ok <- writeKeyFile key offset content
|
||||
sendMessage $ if ok then SUCCESS else FAILURE
|
||||
return ok
|
||||
|
||||
To get transcripts of the protocol in action, the Free monad can be evaluated
|
||||
purely, providing the other side of the conversation:
|
||||
|
||||
ghci> putStrLn $ protoDump $ runPure (put (fromJust $ file2key "WORM--foo")) [PUT_FROM (Offset 10), SUCCESS]
|
||||
> PUT WORM--foo
|
||||
< PUT-FROM 10
|
||||
> DATA 90
|
||||
> bytes
|
||||
< SUCCESS
|
||||
result: True
|
||||
|
||||
ghci> putStrLn $ protoDump $ runPure (serve (toUUID "myuuid")) [GET (Offset 0) (fromJust $ file2key "WORM--foo")]
|
||||
< GET 0 WORM--foo
|
||||
> PROTO-ERROR must AUTH first
|
||||
result: ()
|
||||
|
||||
Am very happy with all this pure code and that I'm finally using Free monads.
|
||||
Next I need to get down the the dirty business of wiring this up to
|
||||
actual IO actions, and an actual network connection.
|
||||
|
||||
Today's work was sponsored by Jake Vosloo on Patreon.
|
||||
31
doc/devblog/day_428-429__git_push_to_hiddden_service.mdwn
Normal file
31
doc/devblog/day_428-429__git_push_to_hiddden_service.mdwn
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
The `tor` branch is coming along nicely.
|
||||
|
||||
This weekend, I continued working on the P2P protocol, implementing
|
||||
it for network sockets, and extending it to support connecting up
|
||||
git-send-pack/git-receive-pack.
|
||||
|
||||
There was a bit of a detour when I split the Free monad into two separate
|
||||
ones, one for Net operations and the other for Local filesystem operations.
|
||||
|
||||
This weekend's work was sponsored by Thomas Hochstein on Patreon.
|
||||
|
||||
----
|
||||
|
||||
Today, implemented a `git-remote-tor-annex` command that git will
|
||||
use for tor-annex:: urls, and made `git annex remotedaemon`
|
||||
serve the tor hidden service.
|
||||
|
||||
Now I have git push/pull working to the hidden service, for example:
|
||||
|
||||
git pull tor-annex::eeaytkuhaupbarfi.onion:47651
|
||||
|
||||
That works very well, but does not yet check that the user is authorized
|
||||
to use the repo, beyond knowing the onion address. And currently
|
||||
it only works in git-annex repos; with some tweaks it should
|
||||
also work in plain git repos.
|
||||
|
||||
Next, I need to teach git-annex how to access tor-annex remotes.
|
||||
And after that, an interface in the webapp for setting them up and
|
||||
connecting them together.
|
||||
|
||||
Today's work was sponsored by Josh Taylor on Patreon.
|
||||
13
doc/devblog/day_430__tor_socket_problem.mdwn
Normal file
13
doc/devblog/day_430__tor_socket_problem.mdwn
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
Debian's tor daemon is very locked down in the directories it can read
|
||||
from, and so I've had a hard time finding a place to put the unix socket
|
||||
file for git-annex's tor hidden service. Painful details in
|
||||
<http://bugs.debian.org/846275>. At least for now, I'm putting it under
|
||||
/etc/tor/, which is probably a FHS violation, but seems to be the only
|
||||
option that doesn't involve a lot of added complexity.
|
||||
|
||||
---
|
||||
|
||||
The Windows autobuilder is moving, since
|
||||
[NEST](http://nest-initiative.org/) is shutting down the server it has been
|
||||
using. Yury Zaytsev has set up a new Windows autobuilder, hosted at
|
||||
Dartmouth College this time.
|
||||
27
doc/devblog/day_431__p2p_linking.mdwn
Normal file
27
doc/devblog/day_431__p2p_linking.mdwn
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
Today I finished the second-to-last big missing peice for tor hidden service
|
||||
remotes. Networks of these remotes are P2P networks, and there needs to be
|
||||
a way for peers to find one-another, and to authenticate with one-another.
|
||||
The `git annex p2p` command sets up links between peers in such a network.
|
||||
|
||||
So far it has only a basic interface that sets up a one way link between
|
||||
two peers. In the first repository, run `git annex p2p --gen-address`.
|
||||
That outputs a long address. In the second repository, run
|
||||
`git annex p2p --link peer1`, and paste the address into it. That sets up a
|
||||
git remote named "peer1" that connects back to the first repository over tor.
|
||||
|
||||
That is a one-directional link, while a bi-directional link would be
|
||||
much more convenient to have between peers. Worse, the address can be reused by
|
||||
anyone who sees it, to link into the repository. And, the address is far
|
||||
too long to communicate in any way except for pasting it.
|
||||
|
||||
So I want to improve that later. What I'd really like to have is an
|
||||
interface that displays a one-time-use phrase of five to ten words, that
|
||||
can be read over the phone or across the room. Exchange phrases with a
|
||||
friend, and get your repositories securely linked together with tor.
|
||||
|
||||
But, `git annex p2p` is good enough for now. I can move on to the final
|
||||
keystone of the tor support, which is file transfer over tor.
|
||||
That should, fingers crossed, be relatively easy, and the `tor` branch is
|
||||
close to mergeable now.
|
||||
|
||||
Today's work was sponsored by Riku Voipio.
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
[[!comment format=mdwn
|
||||
username="https://anarc.at/openid/"
|
||||
nickname="anarcat"
|
||||
avatar="http://cdn.libravatar.org/avatar/b36dcf65657dd36128161355d8920a99503def9461c1bb212410980fe6f07125"
|
||||
subject="magic wormhole"
|
||||
date="2016-11-30T22:16:19Z"
|
||||
content="""
|
||||
> What I'd really like to have is an interface that displays a
|
||||
> one-time-use phrase of five to ten words, that can be read over the
|
||||
> phone or across the room. Exchange phrases with a friend, and get
|
||||
> your repositories securely linked together with tor.
|
||||
|
||||
I already mentionned the project in [[design/assistant/telehash/]],
|
||||
but [magic-wormhole](https://github.com/warner/magic-wormhole) does
|
||||
exactly that:
|
||||
|
||||
% wormhole send README.md
|
||||
Sending 7924 byte file named 'README.md'
|
||||
On the other computer, please run: wormhole receive
|
||||
Wormhole code is: 7-crossover-clockwork
|
||||
|
||||
Sending (<-10.0.1.43:58988)..
|
||||
100%|=========================| 7.92K/7.92K [00:00<00:00, 6.02MB/s]
|
||||
File sent.. waiting for confirmation
|
||||
Confirmation received. Transfer complete.
|
||||
|
||||
Receiver:
|
||||
|
||||
% wormhole receive
|
||||
Enter receive wormhole code: 7-crossover-clockwork
|
||||
Receiving file (7924 bytes) into: README.md
|
||||
ok? (y/n): y
|
||||
Receiving (->tcp:10.0.1.43:58986)..
|
||||
100%|===========================| 7.92K/7.92K [00:00<00:00, 120KB/s]
|
||||
Received file written to README.md
|
||||
|
||||
While that example shows a file transfer, arbitrary data can be
|
||||
transfered this way. There's a documented protocol, and it's not
|
||||
completely peer-to-peer: there are relay servers to deal with NAT'd
|
||||
machines. But the [PAKE
|
||||
protocol](https://en.wikipedia.org/wiki/Password-authenticated_key_agreement)
|
||||
(basically SPAKE2) could be a good inspiration here.
|
||||
|
||||
Otherwise, I must say that, as a user, I don't mind copy-pasting a
|
||||
hidden service string (if that's what it's about): i can do that over
|
||||
a secure medium (email + OpenPGP or IM + OTR) easily... But I
|
||||
understand it can be difficult to do for new users.
|
||||
|
||||
"""]]
|
||||
13
doc/devblog/day_432-433__almost_there.mdwn
Normal file
13
doc/devblog/day_432-433__almost_there.mdwn
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
Friday and today were spent implementing both sides of the P2P protocol for
|
||||
git-annex content transfers.
|
||||
|
||||
There were some tricky cases to deal with. For example, when a file is being
|
||||
sent from a direct mode repository, or v6 annex.thin repository, the
|
||||
content of the file can change as it's being transferred. Including being
|
||||
appended to or truncated. Had to find a way to deal with that, to avoid
|
||||
breaking the protocol by not sending the indicated number of bytes of data.
|
||||
|
||||
It all seems to be done now, but it's not been tested at all, and there are
|
||||
probably some bugs to find. (And progress info is not wired up yet.)
|
||||
|
||||
Today's work was sponsored by Trenton Cronholm on Patreon.
|
||||
27
doc/devblog/day_434__it_works.mdwn
Normal file
27
doc/devblog/day_434__it_works.mdwn
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
Git annex transfers over Tor worked correctly the first time I tried them
|
||||
today. I had been expecting protocol implementation bugs, so this was a
|
||||
nice surprise!
|
||||
|
||||
Of course there were some bugs to fix. I had forgotten to add UUID
|
||||
discovery to `git annex p2p --link`. And, resuming interrupted transfers
|
||||
was buggy.
|
||||
|
||||
Spent some time adding progress updates to the Tor remote. I was curious to
|
||||
see what speed transfers would run. Speed will of course vary depending on
|
||||
the Tor relays being used, but this example with a 100 mb file is not bad:
|
||||
|
||||
copy big4 (to peer1...)
|
||||
62% 1.5MB/s 24s
|
||||
|
||||
There are still a couple of [[known bugs|todo/tor]],
|
||||
but I've merged the `tor` branch into `master` already.
|
||||
|
||||
----
|
||||
|
||||
Alpernebbi has built a GUI for editing git-annex metadata.
|
||||
Something I always wanted!
|
||||
[[Read about it here|tips/a_gui_for_metadata_operations]]
|
||||
|
||||
----
|
||||
|
||||
Today's work was sponsored by Ethan Aubin.
|
||||
20
doc/devblog/day_435-436_post_tor_merge.mdwn
Normal file
20
doc/devblog/day_435-436_post_tor_merge.mdwn
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
More improvements to tor support. Yesterday, debugged a reversion that
|
||||
broke push/pull over tor, and made actual useful error messages be
|
||||
displayed when there were problems. Also fixed a memory leak, although I
|
||||
fixed it by reorganizing code and could not figure out quite why it happened,
|
||||
other than that the ghc runtime was not managing to be as lazy as I would
|
||||
expect.
|
||||
|
||||
Today, added git ref change notification to the
|
||||
P2P protocol, and made the remotedaemon automatically fetch changes from
|
||||
tor remotes. So, it should work to use the assistant to keep
|
||||
repositories in sync over tor. I have not tried it yet, and linking over tor
|
||||
still needs to be done at the command line, so it's not really ready for
|
||||
webapp users yet.
|
||||
|
||||
Also fixed a denial of service attack in git-annex-shell and git-annex when
|
||||
talking to a remote git-annex-shell. It was possible to feed either a large
|
||||
amount of data when they tried to read a line of data, and summon the OOM
|
||||
killer. Next release will be expedited some because of that.
|
||||
|
||||
Today's work was sponsored by Thomas Hochstein on Patreon.
|
||||
20
doc/devblog/day_437__catching_up.mdwn
Normal file
20
doc/devblog/day_437__catching_up.mdwn
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
Quite a backlog developed in the couple of weeks I was concentrating on tor
|
||||
support. I've taken a first pass through it and fixed the most pressing
|
||||
issues now.
|
||||
|
||||
Most important was an ugly memory corruption problem in the GHC runtime
|
||||
system that may have led to data corruption when using git-annex with Linux
|
||||
kernels older than 4.5. All the Linux standalone builds of git-annex have
|
||||
been updated to fix that issue.
|
||||
|
||||
Today dealt with several more things, including fixing a buggy timestamp
|
||||
issue with `metadata --batch`, reverting the ssh ServerAliveInterval
|
||||
setting (broke on too many systems with old ssh or complicated ssh
|
||||
configurations), making batch input not be rejected when it can't be decoded
|
||||
as UTF-8, and more.
|
||||
|
||||
Also, spent some time learning a little bit about Magic Wormhole and SPAKE,
|
||||
as a way to exchange tor remote addresses. Using Magic Wormhole for that
|
||||
seems like a reasonable plan. I did file a couple bugs on it which will
|
||||
need to get fixed, and then using it is mostly a question of whether it's
|
||||
easy enough to install that git-annex can rely on it.
|
||||
6
doc/devblog/day_438__bi-directional_p2p_links.mdwn
Normal file
6
doc/devblog/day_438__bi-directional_p2p_links.mdwn
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
Improved `git annex p2p --link` to create a bi-directional link
|
||||
automatically. Bi-directional links are desirable more often than not, so
|
||||
it's the default behavior.
|
||||
|
||||
Also continued thinking about using magic wormhole for communicating
|
||||
p2p addresses for pairing. And filed some more bugs on magic wormhole.
|
||||
51
doc/devblog/day_439__wormhole_pairing.mdwn
Normal file
51
doc/devblog/day_439__wormhole_pairing.mdwn
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
`git annex p2p --pair` implemented, using Magic Wormhole codes
|
||||
that have to be exchanged between the repositories being paired.
|
||||
|
||||
It looks like this, with the same thing being done at the same time
|
||||
in the other repository.
|
||||
|
||||
joey@elephant:~/tmp/bench3/a>git annex p2p --pair
|
||||
p2p pair peer1 (using Magic Wormhole)
|
||||
|
||||
This repository's pairing code is: 1-select-bluebird
|
||||
|
||||
Enter the other repository's pairing code: (here I entered 8-fascinate-sawdust)
|
||||
Exchanging pairing data...
|
||||
Successfully exchanged pairing data. Connecting to peer1...
|
||||
ok
|
||||
|
||||
And just that simply, the two repositories find one another,
|
||||
Tor onion addresses and authentication data is exchanged, and a git remote
|
||||
is set up connecting via Tor.
|
||||
|
||||
joey@elephant:~/tmp/bench3/a>git annex sync peer1
|
||||
commit
|
||||
ok
|
||||
pull peer1
|
||||
warning: no common commits
|
||||
remote: Counting objects: 5, done.
|
||||
remote: Compressing objects: 100% (3/3), done.
|
||||
remote: Total 5 (delta 0), reused 0 (delta 0)
|
||||
Unpacking objects: 100% (5/5), done.
|
||||
From tor-annex::5vkpoyz723otbmzo.onion:61900
|
||||
* [new branch] git-annex -> peer1/git-annex
|
||||
|
||||
Very pleased with this, and also the whole thing worked on the very first
|
||||
try!
|
||||
|
||||
It might be slightly annoying to have to exchange two codes during pairing.
|
||||
It would be possible to make this work with only one code. I decided to go
|
||||
with two codes, even though it's only marginally more secure than one,
|
||||
mostly for UI reasons. The pairing interface and
|
||||
[[instructions for using it|tips/peer_to_peer_network_with_tor]] is simplfied
|
||||
by being symmetric.
|
||||
|
||||
(I also decided to revert the work I did on Friday to make `p2p --link`
|
||||
set up a bidirectional link. Better to keep `--link` the simplest possible
|
||||
primitive, and pairing makes bidirectional links more easily.)
|
||||
|
||||
Next: Some more testing of this and the Tor hidden services, a webapp UI
|
||||
for P2P peering, and then finally removing XMPP support. I hope to finish
|
||||
that by New Years.
|
||||
|
||||
Today's work was sponsored by Jake Vosloo on Patreon.
|
||||
|
|
@ -20,11 +20,11 @@ git-annex will continue to work.
|
|||
For the really tricky memory leaks, here's how to make a profiling build of
|
||||
git-annex.
|
||||
|
||||
1. `cabal configure` with only the flags you really need
|
||||
2. `cabal build --ghc-options="-prof -auto-all -caf-all"`
|
||||
1. `cabal configure --enable-profiling`
|
||||
This will probably fail due to some missing profiling libraries.
|
||||
You have to get the profiling versions of all needed haskell libraries
|
||||
installed somehow.
|
||||
2. `cabal build`
|
||||
3. Run git-annex with the special flags `+RTS -hc -p`
|
||||
4. Reproduce the memory leak problem.
|
||||
5. If the assistant was run, stop it.
|
||||
|
|
|
|||
9
doc/forum/Extra___38___missing_folders_on_remote.mdwn
Normal file
9
doc/forum/Extra___38___missing_folders_on_remote.mdwn
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
I created a local annex directory that's an adjusted branch used with the assistant. On another machine, I initialized an annex directory, then made this into a full-backup ssh remote for my local.
|
||||
|
||||
After the assistant pushes to the remote, and the remote runs `git annex sync`, the remote is missing some directories and has some extra directories. For example, it has the extra directory `Documents/programs/Documents/programs/`, which has different contents than `Documents/programs/`. Both directories are missing the subdirectory `graphing_experiments/`.
|
||||
|
||||
From my local, `git annex whereis Documents/programs/graphing_experiments` says the directory exists on the remote. But it's not there.
|
||||
|
||||
I recreated the remote from scratch and the problem persists.
|
||||
|
||||
The assistant says the remote is caught up, and is keeping up with new content changes. What could cause this?
|
||||
3
doc/forum/GIT__95__SSH.mdwn
Normal file
3
doc/forum/GIT__95__SSH.mdwn
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
Is there any way to get git-annex to respect the GIT_SSH environment variable just like git? This would be super helpful for specifying ssh options (keys in my application)
|
||||
|
||||
Hacks like a ~/.ssh/config with a fake hostname to specify the key are not appropriate in my case, because keys are generated on the fly by a server-side application that may be distributed across multiple servers. So I really need to be able to specify a key file at run time and then use the normal remote URI. I can do this with git and GIT_SSH, but have not found a solution for git-annex yet.
|
||||
41
doc/forum/Git-annex_link_to_different_file_names.mdwn
Normal file
41
doc/forum/Git-annex_link_to_different_file_names.mdwn
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
This is a recreation of a stackexchange question, in case the community here is more knowledgeable.
|
||||
|
||||
Link to stackexchange question : http://unix.stackexchange.com/questions/325753/git-annex-link-to-different-file-names
|
||||
|
||||
Content :
|
||||
"Maybe this is just a crazy use case that doesn't work, but I was wondering if there's a way to build a file's history from files with different file names. I'm exploring this idea because I'd like to have a git-annex system but I can't force my coworkers to adapt.
|
||||
|
||||
Here's what I have in mind :
|
||||
|
||||
Folder 1, managed by coworkers (On a shared disk) :
|
||||
|
||||
drawing_shop_12_nov_2015.pdf
|
||||
drawing_shop_13_nov_2015.pdf
|
||||
drawing_asbuilt_14_nov_2015.pdf
|
||||
drawing_asbuilt_rev1_15_nov_2015.pdf
|
||||
|
||||
And
|
||||
|
||||
Git-annex, managed by me :
|
||||
|
||||
drawing.pdf
|
||||
|
||||
(with a shop branch and a asbuilt branch)
|
||||
|
||||
The git-annex's drawing.pdf would have an history like this :
|
||||
|
||||
[shop]
|
||||
|
|
||||
Commit A "Initial shop drawing"
|
||||
|
|
||||
Commit B "Add corrections from Wizzbasket"
|
||||
\
|
||||
|
|
||||
[asbuilt]
|
||||
Commit C "Reflect as built"
|
||||
|
|
||||
Commit D "Change dweezelbox block for simplicity"
|
||||
|
||||
But somehow the "managed by coworkers" repo would be a direct mode repo with Commit A pointing to drawing_shop_12_nov_2015.pdf, Commit B to drawing_shop_13_nov_2015.pdf etc.
|
||||
|
||||
Can this be done?"
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-12-20T19:27:31Z"
|
||||
content="""
|
||||
Yes, you can do this. Effectively, you have two branches. In the master
|
||||
branch, you have drawing.pdf with a single name and changes commited to it.
|
||||
In the coworkers branch, you have the multiple different versions. Git has
|
||||
no difficulty representing this, but it's up to you to maintain the
|
||||
different branches.
|
||||
|
||||
For example:
|
||||
|
||||
joey@darkstar:~/tmp/shop>git commit -m 'updated drawing some more'
|
||||
[master 1403dd4] updated drawing some more
|
||||
1 file changed, 1 insertion(+), 1 deletion(-)
|
||||
joey@darkstar:~/tmp/shop>git checkout coworkers
|
||||
Switched to branch 'coworkers'
|
||||
joey@darkstar:~/tmp/shop#coworkers>git show master
|
||||
commit 1403dd49b2c378e78b8b8ec82d73e295c486697b
|
||||
Author: Joey Hess <joeyh@joeyh.name>
|
||||
Date: Tue Dec 20 15:31:17 2016 -0400
|
||||
|
||||
updated drawing some more
|
||||
|
||||
diff --git a/drawing.pdf b/drawing.pdf
|
||||
index b59371e..c05ed95 120000
|
||||
--- a/drawing.pdf
|
||||
+++ b/drawing.pdf
|
||||
@@ -1 +1 @@
|
||||
-.git/annex/objects/55/MZ/SHA256E-s13--c5f6529491f9e6d40e893d2ffc008bc297bcc56a680040c124e4019fb5c1a94d.pdf/SHA256E-s13--c5f6529491f9e6d40e893d2ffc008bc297bcc56a680040c124e4019fb5c1a94d.pdf
|
||||
\ No newline at end of file
|
||||
+.git/annex/objects/xj/XF/SHA256E-s17--7786e857a89634ff9242d899245cbcc5e009736af6b0553cb7283b2daef77d16.pdf/SHA256E-s17--7786e857a89634ff9242d899245cbcc5e009736af6b0553cb7283b2daef77d16.pdf
|
||||
\ No newline at end of file
|
||||
joey@darkstar:~/tmp/shop#coworkers>ln -s .git/annex/objects/xj/XF/SHA256E-s17--7786e857a89634ff9242d899245cbcc5e009736af6b0553cb7283b2daef77d16.pdf/SHA256E-s17--7786e857a89634ff9242d899245cbcc5e009736af6b0553cb7283b2daef77d16.pdf drawing_rev2.pdf
|
||||
joey@darkstar:~/tmp/shop#coworkers>git add drawing_rev2.pdf
|
||||
joey@darkstar:~/tmp/shop#coworkers>ls
|
||||
drawing.pdf@ drawing_rev2.pdf@
|
||||
joey@darkstar:~/tmp/shop#coworkers>git commit -m 'added rev2 of drawing'
|
||||
[coworkers cf27781] added rev2 of drawing
|
||||
1 file changed, 1 insertion(+)
|
||||
create mode 120000 drawing_rev2.pdf
|
||||
|
||||
In the example, I looked at what was committed to master, and copied and
|
||||
pasted the git-annex symlink into a new drawing_rev2.pdf file.
|
||||
|
||||
That's the basic idea. There might be a better way to do that. Another way,
|
||||
for example, would be to have 2 clones of the repo, one with master checked
|
||||
out and one with coworkers checked out. You could then run, in the
|
||||
coworkers checkout:
|
||||
|
||||
cp -a ../master/drawing.pdf drawing_rev2.pdf
|
||||
git add drawing_rev2.pdf
|
||||
git commit -m 'added rev2 of drawing'
|
||||
|
||||
That results in the same commit as the method I showed.
|
||||
|
||||
With some scripting, you should be able to automate keeping the two
|
||||
branches in sync.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2016-12-13T16:55:21Z"
|
||||
content="""
|
||||
Creating a separate special remote pointing to the same content is not a
|
||||
good idea. This will confuse git-annex's location tracking, and in some
|
||||
cases can lead to data loss, since git-annex will assume it can safely
|
||||
delete a file from one of the "two" repositories, since it thinks the
|
||||
"other" one will still have the content of the file.
|
||||
|
||||
Instead, you need a way to configure a regular git remote,
|
||||
pointing at the repository, that is read-only, or is accessed over rsync,
|
||||
or whatever your requirements are.
|
||||
"""]]
|
||||
27
doc/forum/Odd_Hybrid_Symlinks_To_Content.mdwn
Normal file
27
doc/forum/Odd_Hybrid_Symlinks_To_Content.mdwn
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
I've somehow managed to get my indirect repository to symlink to literal content instead of object files.
|
||||
|
||||
By this I mean literally the symlink is pointing at the contents of the file as the filename.
|
||||
|
||||
So if I have a blah.txt file with this content:
|
||||
|
||||
* First line
|
||||
* second line
|
||||
|
||||
And I ls -al to view the symlink pointer, it shows up as this:
|
||||
|
||||
* blah.txt -> First line?second line
|
||||
|
||||
It literally has the contents of the file as the destination filename.
|
||||
|
||||
I've tried a couple things I could think of to re-symlink the files, but they don't seem to do anything as they think everything is fine:
|
||||
|
||||
* git annex indirect //returns nothing
|
||||
* git annex lock blah.txt //returns nothing
|
||||
* git annex fix blah.txt //returns nothing
|
||||
* git annex fsck //returns nothing
|
||||
|
||||
I'm actually able to find several of these files hanging around by searching for all symlinks that don't point to something in the .git directory.
|
||||
|
||||
Is there a way for me to replace the symlinks with correct symlinks to the objects in .git/annex? Can it even figure out which ones it was supposed to point to if the symlinks are messed up (are filenames -> content hashes stored anywhere else)?
|
||||
|
||||
Else I might have to go do some manual rebasing and history editing to try to undo the bad commits manually. I've synced this repo to another direct repo so I'll need to figure out how to manually fix that repo too (using proxy). From what I can tell the annex/direct/master seems to be same as master and synced/master branches? Is there an [[internals]] page for direct branches besides [[direct_mode]] so I know what should be fixed where?
|
||||
47
doc/forum/Preserving_Directories_in_Metadata_Views.mdwn
Normal file
47
doc/forum/Preserving_Directories_in_Metadata_Views.mdwn
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
I want to use metadata views to sort files into top-level directories based on a tag, but then preserve the directory structure underneath that. I'm having trouble with this.
|
||||
|
||||
Say I have an annex at `~/annex` with a structure like this:
|
||||
|
||||
$ tree
|
||||
.
|
||||
├── foo
|
||||
│ └── bar
|
||||
│ ├── one.txt
|
||||
│ ├── three.txt
|
||||
│ └── two.txt
|
||||
└── waldo
|
||||
└── fred
|
||||
├── a.txt
|
||||
├── b.txt
|
||||
└── c.txt
|
||||
|
||||
I tag some of the files with `blah`:
|
||||
|
||||
$ git annex metadata -t blah foo/bar/*
|
||||
|
||||
Now I want to change my view to only see those files with a certain tag, but I want to maintain their directory structure, ie I want to end up with something like this:
|
||||
|
||||
$ tree
|
||||
.
|
||||
├── blah
|
||||
│ └── foo
|
||||
│ └── bar
|
||||
│ ├── one.txt
|
||||
│ ├── three.txt
|
||||
│ └── two.txt
|
||||
|
||||
If I do `git annex view blah` I see the files `one.txt`, `two.txt` and `three.txt` but they are in the top level of `~/annex`. The `foo` and `bar` directories are not present.
|
||||
|
||||
If I do `git annex view blah "/=*"` then the files I present under the `foo` directory, but the `bar` subdirectory is not there.
|
||||
|
||||
It would also be fine if I could just hide the files that did not have the `blah` tag, so that I ended up with this:
|
||||
|
||||
$ tree
|
||||
.
|
||||
├── foo
|
||||
│ └── bar
|
||||
│ ├── one.txt
|
||||
│ ├── three.txt
|
||||
│ └── two.txt
|
||||
|
||||
Is something like this possible?
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="neocryptek@659edac901ffbc8e541a974f8f18987eeafc63bd"
|
||||
nickname="neocryptek"
|
||||
avatar="http://cdn.libravatar.org/avatar/d9bfdefa9b503f1ac4844a686618374e"
|
||||
subject="comment 3"
|
||||
date="2016-11-13T22:39:44Z"
|
||||
content="""
|
||||
Thanks, that makes sense.
|
||||
|
||||
All git annex repositories using the same branch will have the same (symlink) working directory right (assuming entire network has been synced eventually)?
|
||||
"""]]
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue