mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Browse source
This commit is contained in:
leavingchicago@c04c893e78d1c4c76cb3e32b5c227cf42bbf7682 2017-05-15 04:40:17 +00:00 • committed by admin
parent 9a772ee987
commit 7ee0e9d816
1 changed files with 215 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

215
doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking.mdwn Normal file
View file

@ -0,0 +1,215 @@
### Please describe the problem.
Sync / Assistant between a `source` and `transfer` repo results in zero copies for whereis, but data is available.
### What steps will reproduce the problem?
See a script below...
``` sh
#!/bin/sh
TMP=$(mktemp -d)
##########################################
# Set up the repos
##########################################
###
# Create working directories
cd ${TMP}
mkdir source transfer
###
# Set up the source repo
cd ${TMP}/source
git init
git annex init source
git-annex group here source
git annex wanted here standard
###
# Set up the transfer repo
# N.B. It can see the source repo
cd ${TMP}/transfer
git init
git remote add source ${TMP}/source
git annex init transfer
git-annex group here transfer
git annex wanted here \
'not (inallgroup=client and copies=client:1) and ((include=* and ((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1)))) or approxlackingcopies=1)'
##########################################
# Add some files and run the assistant
##########################################
###
# Make sure it works by adding some files to the source...
cd ${TMP}/source
date > file.1
date > file.2
date > file.3
# Run the assistant in the source...
(cd ${TMP}/source && \
git annex assistant && \
sleep 30 && \
git annex assistant --stop && \
git annex whereis file.*)
# PASS: All files are in source.
# Now run it in the transfer group
(cd ${TMP}/transfer && \
git annex assistant && \
sleep 30 && \
git annex assistant --stop && \
git annex whereis file.*)
# PASS: All files are in transfer
# Check the source
(cd ${TMP}/source && \
git annex whereis file.*)
# FAIL: Only file.1 was updated to be in source.
# whereis file.1 (1 copy)
# 64b193e0-86f0-4349-b073-70af919ce628 -- transfer
# ok
# whereis file.2 (0 copies) failed
# whereis file.3 (0 copies) failed
# git-annex: whereis: 2 failed
##########################################
# Start to scratch head and ask 'Why?'
##########################################
##########
# What's happening???
cd ${TMP}/source && git annex sync
# Source thinks it is updated - working tree clean
cd ${TMP}/transfer && git annex sync
# Transfer pulls from source, so run again
git annex sync
# OK... working tree clean
# Check again
cd ${TMP}/source && git annex whereis file.*
# whereis file.1 (1 copy)
# 64b193e0-86f0-4349-b073-70af919ce628 -- transfer
# ok
# whereis file.2 (0 copies) failed
# whereis file.3 (0 copies) failed
# git-annex: whereis: 2 failed
cd ${TMP}/transfer && git annex whereis file.*
# whereis file.1 (1 copy)
# 64b193e0-86f0-4349-b073-70af919ce628 -- transfer [here]
# ok
# whereis file.2 (0 copies) failed
# whereis file.3 (0 copies) failed
# git-annex: whereis: 2 failed
# HOWEVER... Files are definitely in the transfer repo
cd ${TMP}/source && cat file.*
# cat: file.1: No such file or directory
# cat: file.2: No such file or directory
# cat: file.3: No such file or directory
cd ${TMP}/transfer && cat file.*
# Sun May 14 21:22:02 PDT 2017
# Sun May 14 21:22:03 PDT 2017
# Sun May 14 21:22:06 PDT 2017
```
### What version of git-annex are you using? On what operating system?
#### Version:
```
git-annex version: 6.20170510
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents ConcurrentOutput TorrentParser MagicMime Feeds Quvi
dependency versions: aws-0.16 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.6.1 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
local repository version: 5
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: darwin x86_64
```
#### OS:
MacOS 10.12.4, but seems to apply to Debian Jessie too.
### Please provide any additional information below.
Annex assistant daemon.log for `source` repo:
``` sh
[2017-05-15 14:11:24.150202] main: starting assistant version 6.20170510
[2017-05-15 14:11:24.196597] Cronner: You should enable consistency checking to protect your data.
(scanning...) [2017-05-15 14:11:24.257123] Watcher: Performing startup scan
(started...)
[2017-05-15 14:11:45.505855] Committer: Adding file.1 file.2 file.3
add file.1 ok
add file.2 ok
add file.3 ok
[2017-05-15 14:11:45.544277] Committer: Committing changes to git
(recording state in git...)
(recording state in git...)
```
Annex assistant daemon.log for `transfer` repo:
``` sh
[2017-05-15 14:12:09.127406] main: starting assistant version 6.20170510
[2017-05-15 14:12:09.157611] Cronner: You should enable consistency checking to protect your data.
[2017-05-15 14:12:09.177452] TransferScanner: Syncing with source
(scanning...) [2017-05-15 14:12:09.210605] Watcher: Performing startup scan
(started...)
warning: no common commits
From /Users/olaf/tmp/git-annex/play/bug-play/source
* [new branch] git-annex -> source/git-annex
* [new branch] master -> source/master
* [new branch] synced/master -> source/synced/master
(merging source/git-annex into git-annex...)
(recording state in git...)
Already up-to-date.
[2017-05-15 14:12:10.256195] Committer: Committing changes to git
(recording state in git...)
(checksum...) [2017-05-15 14:12:10.425494] Transferrer: Downloaded file.1
[2017-05-15 14:12:11.17664] Pusher: Syncing with source
(recording state in git...)
[2017-05-15 14:12:11.356447] Committer: Committing changes to git
(recording state in git...)
To /Users/olaf/tmp/git-annex/play/bug-play/source
* [new branch] git-annex -> synced/git-annex
(checksum...) [2017-05-15 14:12:11.428676] Transferrer: Downloaded file.2
(checksum...) [2017-05-15 14:12:11.451457] Transferrer: Downloaded file.3
drop source file.1 ok
drop source file.2 ok
drop source file.3 ok
[2017-05-15 14:12:13.477636] Pusher: Syncing with source
(recording state in git...)
To /Users/olaf/tmp/git-annex/play/bug-play/source
825d3dd..60c96cb git-annex -> synced/git-annex
```
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Yes. Love it. Donated. Have been using it for years. Recommend it and get(/force) my collaborators to use it. ;-)