Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2021-06-21 11:32:48 -04:00
commit 17170cddcc
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
13 changed files with 377 additions and 0 deletions

View file

@ -0,0 +1,144 @@
[[!comment format=mdwn
username="falsifian"
avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae"
subject="comment 2"
date="2021-06-19T23:41:24Z"
content="""
I tried again with 8.20210428 and saw a similar error. `git annex fsck` thinks the file is okay. It doesn't exist on the remote (`elbow`) in this case (confirmed that the symlink is broken).
copy&paste showing `version`, `copy --debug` and `fsck`:
james copter annex-neu $ git annex version [61/5067]
git-annex version: 8.20210428
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser M
agicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.28 DAV-1.3.4 feed
-1.3.2.0 ghc-8.10.4 http-client-0.6.4.1 persistent-sqlite-2.11.1.0 torrent-10000
.1.1 uuid-1.3.15 yesod-1.6.1.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEI
N256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512
E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM UR
L X* remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb ta
hoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
james copter annex-neu $ git remote show -n elbow
* remote elbow
Fetch URL: /d/elbow-ext2/falsifian/w/annex-neu
Push URL: /d/elbow-ext2/falsifian/w/annex-neu
HEAD branch: (not queried)
Remote branches: (status not queried)
git-annex
master
synced/git-annex
synced/master
Local ref configured for 'git push' (status not queried):
(matching) pushes to (matching)
james copter annex-neu $ git annex fsck software_releases/openbsd/68/amd64/insta
ll68.img
fsck software_releases/openbsd/68/amd64/install68.img (checksum...) ok
(recording state in git...)
james copter annex-neu $ git annex copy --debug --to=elbow software_releases/ope
nbsd/68/amd64/install68.img
[2021-06-19 23:37:59.820462299] (Utility.Process) process [28235] read: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"sho
w-ref\",\"git-annex\"]
[2021-06-19 23:37:59.824734704] (Utility.Process) process [28235] done ExitSucce
ss
[2021-06-19 23:37:59.825397492] (Utility.Process) process [28236] read: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"sho
w-ref\",\"--hash\",\"refs/heads/git-annex\"]
[2021-06-19 23:37:59.829279502] (Utility.Process) process [28236] done ExitSucce
ss
[2021-06-19 23:37:59.830626358] (Utility.Process) process [28237] read: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"log
\",\"refs/heads/git-annex..ddbef20f72b72bf2c36bc07574cf211c954bbc41\",\"--pretty=%H\"
,\"-n1\"]
[2021-06-19 23:37:59.834246281] (Utility.Process) process [28237] done ExitSucce
ss
[2021-06-19 23:37:59.834758464] (Utility.Process) process [28238] read: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"log
\",\"refs/heads/git-annex..061e90a61d21283803bd3a16628337e3d098452d\",\"--pretty=%H\"
,\"-n1\"]
[2021-06-19 23:37:59.838545279] (Utility.Process) process [28238] done ExitSucce
ss
[2021-06-19 23:37:59.838996643] (Utility.Process) process [28239] read: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"log
\",\"refs/heads/git-annex..5c367f9a01548348bffa89e8d441c64b18e0dd51\",\"--pretty=%H\"
,\"-n1\"]
[2021-06-19 23:37:59.842728492] (Utility.Process) process [28239] done ExitSucce
ss
[2021-06-19 23:37:59.84317283] (Utility.Process) process [28240] read: git [\"--g
it-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"log\"
,\"refs/heads/git-annex..194c38c1020fe2f5fb62a1358c08589feb845e06\",\"--pretty=%H\",
\"-n1\"]
[2021-06-19 23:37:59.847513494] (Utility.Process) process [28240] done ExitSucce
ss
[2021-06-19 23:37:59.847961228] (Utility.Process) process [28241] read: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"log
\",\"refs/heads/git-annex..9dbdfd430c2d74c58de025c9f30d8a218cf137ca\",\"--pretty=%H\"
,\"-n1\"]
[2021-06-19 23:37:59.851669136] (Utility.Process) process [28241] done ExitSucce
ss
[2021-06-19 23:37:59.859466727] (Utility.Process) process [28242] chat: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"cat
-file\",\"--batch\"]
[2021-06-19 23:37:59.860034811] (Utility.Process) process [28243] chat: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"cat
-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"]
[2021-06-19 23:37:59.863651703] (Utility.Process) process [28244] read: git [\"co
nfig\",\"--null\",\"--list\"]
[2021-06-19 23:37:59.86580683] (Utility.Process) process [28244] done ExitSucces
s
[2021-06-19 23:37:59.867051153] (Utility.Process) process [28246] read: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"sym
bolic-ref\",\"-q\",\"HEAD\"]
[2021-06-19 23:37:59.86930561] (Utility.Process) process [28246] done ExitSucces
s
[2021-06-19 23:37:59.870034557] (Utility.Process) process [28247] read: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"sho
w-ref\",\"refs/heads/master\"]
[2021-06-19 23:37:59.873057419] (Utility.Process) process [28247] done ExitSucce
ss
[2021-06-19 23:37:59.873564996] (Utility.Process) process [28248] read: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"ls-
files\",\"--stage\",\"-z\",\"--\",\"software_releases/openbsd/68/amd64/install68.img\"]
[2021-06-19 23:37:59.874034406] (Utility.Process) process [28249] chat: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"cat
-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\",\"--buffer\"]
[2021-06-19 23:37:59.874572424] (Utility.Process) process [28250] chat: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"cat
-file\",\"--batch=%(objectname) %(objecttype) %(objectsize)\",\"--buffer\"]
[2021-06-19 23:37:59.875594055] (Utility.Process) process [28251] chat: git [\"--
git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"annex.debug=true\",\"cat
-file\",\"--batch=%(objectname) %(objecttype) %(objectsize)\",\"--buffer\"]
copy software_releases/openbsd/68/amd64/install68.img (to elbow...)
[2021-06-19 23:37:59.994828229] (Utility.Process) process [28253] read: cp [\"--r
eflink=always\",\"--preserve=timestamps\",\".git/annex/objects/MF/F8/SHA256-s6967459
84--14ea602583030b33e91ee8fde8dd76113984e9fac6598f9f609f408137c4cff2/SHA256-s696
745984--14ea602583030b33e91ee8fde8dd76113984e9fac6598f9f609f408137c4cff2\",\"/d/el
bow-ext2/falsifian/w/annex-neu/.git/annex/tmp/SHA256-s696745984--14ea602583030b3
3e91ee8fde8dd76113984e9fac6598f9f609f408137c4cff2\"]
[2021-06-19 23:37:59.996355598] (Utility.Process) process [28253] done ExitFailu
re 1
failed to send content to remote
failed to send content to remote
failed
[2021-06-19 23:38:16.192591254] (Utility.Process) process [28251] done ExitSucce
ss
[2021-06-19 23:38:16.192851278] (Utility.Process) process [28250] done ExitSucce
ss
[2021-06-19 23:38:16.193064177] (Utility.Process) process [28249] done ExitSucce
ss
[2021-06-19 23:38:16.193167533] (Utility.Process) process [28248] done ExitSucce
ss
git-annex: copy: 1 failed
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="falsifian"
avatar="http://cdn.libravatar.org/avatar/59c3c23c500d20d83ecb9d1f149be9ae"
subject="comment 3"
date="2021-06-19T23:48:20Z"
content="""
If it helps: I see a progress indicator go from 0% to 100% four times when I run the copy command. The \"failed to send content to remote\" messages appear after the second and fourth times.
Also, I was able to copy that reddit dataset file (from my original comment) to this remote (`elbow`) today. So I guess it is not a property of the particular file on the source repository?
"""]]

View file

@ -0,0 +1,7 @@
I'm trying to make git-annex work with the ipfs special remote here: (https://git-annex.branchable.com/special_remotes/ipfs/). Publishing to IPFS works, but I cannot retrieve the files onto another machine through git-annex
Running `git-annex copy --to ipfs` does copy files onto ipfs, I can access them through the CIDs (through an IPFS gateway like ipfs.io/ipfs/<CID>), and I can view the annexed files on running `git annex whereis`
But, `git annex whereis` on the other computer gives an empty output. (even after running `git annex sync` on both systems). `git annex copy --from ipfs` also gives no output on the other system.
Is there something obvious that I'm missing?

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Lukey"
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
subject="comment 1"
date="2021-06-18T14:50:44Z"
content="""
You have to setup a way so the two remotes can push/pull the git data itself too. You could add each other as git remotes or setup a git server somewhere that both repos talk to.
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="kinshukkashyap.me@35751535d5ebba69df2f2ead4f868251d1aadacf"
nickname="kinshukkashyap.me"
avatar="http://cdn.libravatar.org/avatar/3b7e38fd42309b3ecebf85e0619fc3d8"
subject="comment 2"
date="2021-06-18T15:11:23Z"
content="""
Yes I have a repository on Github. Starting with an empty repo, I clone it on both machines, then run all the git-annex commands (first enabling the remote, then adding the file, then copy to remote, then sync).
Is this what you meant?
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="setting up sync"
date="2021-06-18T18:54:59Z"
content="""
Are you running `git-annex-sync` with the `--content` flag?
"""]]

View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="kinshukkashyap.me@35751535d5ebba69df2f2ead4f868251d1aadacf"
nickname="kinshukkashyap.me"
avatar="http://cdn.libravatar.org/avatar/3b7e38fd42309b3ecebf85e0619fc3d8"
subject="comment 4"
date="2021-06-19T10:22:58Z"
content="""
No, I'm not using the `--content` flag. I thought that was for syncing content as well, rather than just the locations of files?
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="Lukey"
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
subject="comment 5"
date="2021-06-19T12:14:33Z"
content="""
Hmm, it should work as you described it. Did you run `git annex sync` on the 2nd repo _after_ running `git annex sync` on the 1st repo (where you copied the files to the ipfs remote)? Can you post the output of the following actions, in that order:
repo1# git annex copy somefile --to=ipfs
repo1# git annex sync
repo2# git annex sync
"""]]

View file

@ -0,0 +1,10 @@
[[!comment format=mdwn
username="Ilya_Shlyakhter"
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
subject="comment 6"
date="2021-06-19T18:31:59Z"
content="""
>No, I'm not using the --content flag. I thought that was for syncing content as well, rather than just the locations of files?
You're right, sorry, I misread your post. For syncing just the location info `--content` is not needed.
"""]]

View file

@ -0,0 +1,128 @@
[[!comment format=mdwn
username="kinshukkashyap.me@35751535d5ebba69df2f2ead4f868251d1aadacf"
nickname="kinshukkashyap.me"
avatar="http://cdn.libravatar.org/avatar/3b7e38fd42309b3ecebf85e0619fc3d8"
subject="comment 7"
date="2021-06-20T06:54:37Z"
content="""
Here's a detailed sequence of what I'm doing:
I made an empty repository on github. Then, on machine #1:
$ git init
Reinitialized existing Git repository in /home/kinshuk/code/annex-again/.git/
$ git annex init origin
init origin (scanning for unlocked files...)
Remote origin not usable by git-annex; setting annex-ignore
ok
(recording state in git...)
$ git annex initremote ipfs type=external externaltype=ipfs encryption=none
initremote ipfs ok
(recording state in git...)
$ vim test1.txt
$ git annex add test1.txt
add test1.txt
ok
(recording state in git...)
$ git annex copy --to ipfs
copy test1.txt (to ipfs...)
ok
(recording state in git...)
$ git annex whereis
whereis test1.txt (2 copies)
27123293-bec2-49a6-ac67-8c87fe72ffd6 -- origin [here]
e7582073-415a-48d2-9dec-29c58311fd21 -- [ipfs]
ipfs: ipfs:QmUk36hewmeivRKf1LKYvn2sG1ALqnt2NtAxaJAHJrrSH8
ok
$ git annex sync
commit
[master (root-commit) d2860cd] git-annex in origin
1 file changed, 1 insertion(+)
create mode 120000 test1.txt
ok
pull origin
ok
push origin
Enumerating objects: 26, done.
Counting objects: 100% (26/26), done.
Delta compression using up to 4 threads
Compressing objects: 100% (20/20), done.
Writing objects: 100% (26/26), 2.33 KiB | 1.16 MiB/s, done.
Total 26 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), done.
To https://github.com/plsrgb/annex-again
* [new branch] git-annex -> synced/git-annex
* [new branch] master -> synced/master
ok
$ git annex log
+ Sun, 20 Jun 2021 12:02:21 IST test1.txt | e7582073-415a-48d2-9dec-29c58311fd21 -- [ipfs]
+ Sun, 20 Jun 2021 12:02:08 IST test1.txt | 27123293-bec2-49a6-ac67-8c87fe72ffd6 -- origin
Then I clone the repository from github on machine #2:
$ git annex sync
commit
On branch synced/git-annex
Your branch is up to date with 'origin/synced/git-annex'.
nothing to commit, working tree clean
ok
pull origin
ok
push origin
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 12 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 750 bytes | 750.00 KiB/s, done.
Total 6 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/plsrgb/annex-again
c95a7b3..d9cc5d1 git-annex -> synced/git-annex
* [new branch] synced/git-annex -> synced/synced/git-annex
ok
To https://github.com/plsrgb/annex-again ! [rejected] synced/git-annex -> synced/git-annex (non-fast-forward)error: failed to push some refs to 'https://github.com/plsrgb/annex-again'hint: Updates were rejected because the tip of your current branch is behindhint: its remote counterpart. Integrate the remote changes (e.g.hint: 'git pull ...') before pushing again.hint: See the 'Note about fast-forwards' in 'git push --help' for details.
To resolve this, I run git pull, then run sync again:
$ git pull
Updating c95a7b3..d9cc5d1
Fast-forward
remote.log | 2 +-
uuid.log | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
$ git annex sync
commit
On branch synced/git-annex
Your branch is up to date with 'origin/synced/git-annex'.
nothing to commit, working tree clean
ok
pull origin
ok
push origin
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/plsrgb/annex-again
c95a7b3..d9cc5d1 synced/git-annex -> synced/synced/git-annex
ok
"""]]

View file

@ -0,0 +1,8 @@
[[!comment format=mdwn
username="Lukey"
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
subject="comment 8"
date="2021-06-20T07:15:04Z"
content="""
Ah that explains it. You have to switch to the `master` branch on the 2nd repository with `git checkout master`.
"""]]

View file

@ -0,0 +1,11 @@
[[!comment format=mdwn
username="kinshukkashyap.me@35751535d5ebba69df2f2ead4f868251d1aadacf"
nickname="kinshukkashyap.me"
avatar="http://cdn.libravatar.org/avatar/3b7e38fd42309b3ecebf85e0619fc3d8"
subject="comment 9"
date="2021-06-20T07:44:34Z"
content="""
That works! That completely slipped my mind.
Thanks a lot!
"""]]

View file

@ -0,0 +1,12 @@
[[!comment format=mdwn
username="james@06209b7878fcf3b5c46b8028dacb3cec6609369c"
nickname="james"
avatar="http://cdn.libravatar.org/avatar/a11da294e0e89c859bd11a3b1dbb0dda"
subject="comment 2"
date="2021-06-17T18:14:45Z"
content="""
Thats great, thanks! I decided to allow some more files to be added as `smallfiles` when I saw the outcome of adding my files.
Going to see if I can wrangle some GNU-Fu and get this into a one liner.
"""]]