Merge branch 'master' into importtree
This commit is contained in:
Joey Hess
commit
760f26ebc6
8 changed files with 223 additions and 3 deletions
|
|
@ -3,6 +3,8 @@ git-annex (7.20190220) UNRELEASED; urgency=medium
|
|||
* Fix storage of metadata values containing newlines.
|
||||
(Reversion introduced in version 7.20190122.)
|
||||
* Sped up git-annex export in repositories with lots of keys.
|
||||
* S3: Support enabling bucket versioning when built with aws-0.21.1.
|
||||
* stack.yaml: Build with aws-0.21.1
|
||||
* Fix cleanup of git-annex:export.log after git-annex forget --drop-dead.
|
||||
* export: Deprecated the --tracking option.
|
||||
Instead, users can configure remote.<name>.annex-tracking-branch
|
||||
|
|
|
|||
|
|
@ -861,7 +861,7 @@ getS3VersionIDPublicUrls mk info u k =
|
|||
-- setting versioning in a bucket that git-annex has already exported
|
||||
-- files to risks losing the content of those un-versioned files.
|
||||
enableBucketVersioning :: SetupStage -> RemoteConfig -> RemoteGitConfig -> UUID -> Annex ()
|
||||
#if MIN_VERSION_aws(0,22,0)
|
||||
#if MIN_VERSION_aws(0,21,1)
|
||||
enableBucketVersioning ss c gc u = do
|
||||
#else
|
||||
enableBucketVersioning ss c _ _ = do
|
||||
|
|
@ -876,7 +876,7 @@ enableBucketVersioning ss c _ _ = do
|
|||
giveup "Cannot change versioning= of existing S3 remote."
|
||||
where
|
||||
enableversioning b = do
|
||||
#if MIN_VERSION_aws(0,22,0)
|
||||
#if MIN_VERSION_aws(0,21,1)
|
||||
showAction "enabling bucket versioning"
|
||||
hdl <- mkS3HandleVar c gc u
|
||||
withS3HandleOrFail u hdl $ \h ->
|
||||
|
|
|
|||
|
|
@ -0,0 +1,112 @@
|
|||
### Please describe the problem.
|
||||
background: I'm trying to use git-annex to manage tracking/off-site replication of a borgbackup repository. This requires direct access to files, and it's not feasible for me to operate without annex.thin set. I'm trying to workaround some of borg's inconvenient behavior--sometimes, a borg transaction will do something that is effectively just a file rename, but internally, it truncates+deletes the original file and creates a new file with the old content. This leads to annex adding the "new" file as a reference to known content, but that content was already lost. (maybe) MWE below
|
||||
|
||||
### What steps will reproduce the problem?
|
||||
set -x && \
|
||||
mkdir tmp && cd tmp && \
|
||||
git init && \
|
||||
git-annex init --version 7 base && \
|
||||
git-annex config --set annex.thin true && \
|
||||
\
|
||||
echo "bar">foo && cat foo && git-annex add foo && ls -l && git-annex unlock foo && git-annex sync && \
|
||||
truncate -s 0 foo && rm foo && git-annex sync && \
|
||||
\
|
||||
echo "bar">ffoo && cat ffoo && git-annex add ffoo && ls -l && git-annex unlock ffoo && git-annex sync && \
|
||||
cat ffoo && git-annex whereis && git-annex fsck
|
||||
|
||||
### What version of git-annex are you using? On what operating system?
|
||||
# git-annex version
|
||||
git-annex version: 7.20190129+git78-g3fa6be1fe-1~ndall+1
|
||||
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
|
||||
dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
|
||||
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
|
||||
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
|
||||
operating system: linux x86_64
|
||||
supported repository versions: 5 7
|
||||
upgrade supported from repository versions: 0 1 2 3 4 5 6
|
||||
|
||||
# lsb_release -a
|
||||
LSB Version: core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
|
||||
Distributor ID: LinuxMint
|
||||
Description: Linux Mint 19 Tara
|
||||
Release: 19
|
||||
Codename: tara
|
||||
|
||||
|
||||
### Please provide any additional information below.
|
||||
|
||||
[[!format sh """
|
||||
# If you can, paste a complete transcript of the problem occurring here.
|
||||
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
|
||||
+ mkdir tmp
|
||||
+ cd tmp
|
||||
+ git init
|
||||
Initialized empty Git repository in /home/user/tmp/.git/
|
||||
+ git-annex init --version 7 base
|
||||
init base ok
|
||||
(recording state in git...)
|
||||
+ git-annex config --set annex.thin true
|
||||
annex.thin true ok
|
||||
(recording state in git...)
|
||||
+ echo bar
|
||||
+ cat foo
|
||||
bar
|
||||
+ git-annex add foo
|
||||
add foo ok
|
||||
(recording state in git...)
|
||||
+ ls -l
|
||||
total 4
|
||||
lrwxrwxrwx 1 root root 178 Feb 23 13:34 foo -> .git/annex/objects/g7/9v/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
|
||||
+ git-annex unlock foo
|
||||
unlock foo ok
|
||||
(recording state in git...)
|
||||
+ git-annex sync
|
||||
commit
|
||||
[master (root-commit) eebbbb5] git-annex in base
|
||||
1 file changed, 1 insertion(+)
|
||||
create mode 100644 foo
|
||||
ok
|
||||
+ truncate -s 0 foo
|
||||
+ rm foo
|
||||
+ git-annex sync
|
||||
commit
|
||||
[master d043022] git-annex in base
|
||||
1 file changed, 1 deletion(-)
|
||||
delete mode 100644 foo
|
||||
ok
|
||||
+ echo bar
|
||||
+ cat ffoo
|
||||
bar
|
||||
+ git-annex add ffoo #content is discarded here
|
||||
add ffoo ok
|
||||
(recording state in git...)
|
||||
+ cat ffoo
|
||||
+ ls -l
|
||||
total 4
|
||||
lrwxrwxrwx 1 root root 178 Feb 23 13:34 ffoo -> .git/annex/objects/g7/9v/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
|
||||
+ git-annex unlock ffoo
|
||||
unlock ffoo ok
|
||||
(recording state in git...)
|
||||
+ git-annex sync
|
||||
commit
|
||||
[master fbc08bc] git-annex in base
|
||||
1 file changed, 1 insertion(+)
|
||||
create mode 100644 ffoo
|
||||
ok
|
||||
+ cat ffoo
|
||||
/annex/objects/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
|
||||
+ git-annex whereis
|
||||
whereis ffoo (1 copy)
|
||||
e470821b-a101-4acc-91ac-a0ae5d3aa40d -- base [here]
|
||||
ok
|
||||
+ git-annex fsck
|
||||
fsck ffoo ok
|
||||
(recording state in git...)
|
||||
|
||||
|
||||
# End of transcript or log.
|
||||
"""]]
|
||||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
I'm hopeful! But just getting started :(
|
||||
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
[[!comment format=mdwn
|
||||
username="david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8"
|
||||
nickname="david.j.buckmaster"
|
||||
avatar="http://cdn.libravatar.org/avatar/1650cdf23a0999fd0e03650e20c90ee7"
|
||||
subject="comment 1"
|
||||
date="2019-02-24T01:20:14Z"
|
||||
content="""
|
||||
I guess the issue here isn't specifically the truncation, but rather that annexed content isn't confirmed when committing an rm.
|
||||
|
||||
set -x && \
|
||||
mkdir tmp && cd tmp && \
|
||||
git init && \
|
||||
git-annex init --version 7 base && \
|
||||
git-annex config --set annex.thin true && \
|
||||
\
|
||||
echo \"bar\">foo && cat foo && git-annex add foo && ls -l && git-annex sync && git-annex unlock foo && git-annex sync && \
|
||||
echo \"barbar\">foo && cat foo && rm foo && git-annex sync && \
|
||||
\
|
||||
echo \"bar\">ffoo && cat ffoo && git-annex add ffoo && git-annex sync && cat ffoo
|
||||
|
||||
This results in ffoo containing the text \"barbar\" rather than \"bar\" (symlink to foo's last committed content), and unlocking ffoo after this leads to a pointer file instead of hardlink (and git-annex info ffoo shows present: false).
|
||||
|
||||
I'm new to this, so apologies if I'm doing something stupid.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
[[!comment format=mdwn
|
||||
username="david.j.buckmaster@984ff2704feacab53415ac5647206517d18f88f8"
|
||||
nickname="david.j.buckmaster"
|
||||
avatar="http://cdn.libravatar.org/avatar/1650cdf23a0999fd0e03650e20c90ee7"
|
||||
subject="comment 2"
|
||||
date="2019-02-24T07:01:35Z"
|
||||
content="""
|
||||
In case this is NOT a bug but rather an incompatibility between borg/git-annex, the bash snippet below seems to sidestep the issue if run AFTER a borg operation but BEFORE any other annex commit/sync. Unsure if I'm the only person in the world that wants a not-permanently-append-only borg repository in an unlocked+thin git-annex...but...after a borg operation that truncates+deletes files, the function uses the status printout to identify all missing, borg-controlled files, and for each
|
||||
|
||||
1. Reverts the deletion (git checkout -- ..)
|
||||
2. Checks if the restored file is a (dead) file pointer (file contents start with /annex/objects). If no, redo the delete and continue loop; if yes, truncate (but do not delete) the file and stage a commit.
|
||||
|
||||
Truncations are then committed to force annex to see that contents have changed, and all deletions are repeated. After that, git annex add-ing a new file with old content restores the annex content that borg truncated, rather than ending up with a pointer file and missing contents.
|
||||
|
||||
Test case:
|
||||
|
||||
set -x && \
|
||||
mkdir tmp && cd tmp && \
|
||||
git init && \
|
||||
git-annex init --version 7 base && \
|
||||
git-annex config --set annex.thin true && \
|
||||
\
|
||||
echo \"bar\">foo && cat foo && git-annex add foo && ls -l && git-annex sync && git-annex unlock foo && git-annex sync && \
|
||||
echo \"barbar\">foo && cat foo && rm foo && \
|
||||
\
|
||||
git checkout -- foo && truncate -s 0 foo && git add foo && git-annex sync && \
|
||||
rm foo && git-annex sync && \
|
||||
\
|
||||
echo \"bar\">ffoo && cat ffoo && git-annex add ffoo && git-annex sync && cat ffoo
|
||||
|
||||
Bash function:
|
||||
|
||||
handleborgdeletes() {
|
||||
echo \"(handleborgdeletes\"
|
||||
|
||||
local PTRFILES=()
|
||||
local HAVETRUNCS=
|
||||
local f
|
||||
|
||||
for f in $(IFS=$'\n' $GITANNEX status | grep ^D | grep -E '(index|hints|integrity|data/)'); do
|
||||
f=${f:2}
|
||||
|
||||
$GIT checkout -- \"$f\" #restore
|
||||
if [ -n \"$(grep -F --text '/annex/objects' $f)\" ]; then #is broken annex pointer file
|
||||
echo \"$f is a pointer file\"
|
||||
HAVETRUNCS=1
|
||||
truncate -s 0 \"$f\" #truncate
|
||||
PTRFILES+=(\"$f\") #track
|
||||
$GIT add \"$f\" #stage content change
|
||||
else
|
||||
echo \"$f is not a pointer file\"
|
||||
rm \"$f\"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -n \"$HAVETRUNCS\" ]; then
|
||||
$GIT commit -m \"truncating borg deletions\" #commit content changes
|
||||
|
||||
for f in ${PTRFILES[@]}; do
|
||||
rm \"$f\" #repeat delete to be committed later
|
||||
done
|
||||
fi
|
||||
echo \"handleborgdeletes)\"
|
||||
}
|
||||
"""]]
|
||||
9
doc/devblog/day_575__core_of_import_tree_done.mdwn
Normal file
9
doc/devblog/day_575__core_of_import_tree_done.mdwn
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
Yesterday, got the git tree generation done and working. The main import
|
||||
tree code is also implemented, though it may need some fine tuning.
|
||||
|
||||
Today I've been working on firming up user interface design and documentation.
|
||||
Turns out that import tree is going to lead to some changes to export tree.
|
||||
A remote tracking branch will be updated by both export tree and import
|
||||
tree, since those operations are similar to git push and git fetch.
|
||||
And `git annex export --tracking` will be deprecated in favor
|
||||
of a git config setting that configures both import and export.
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="sameerds"
|
||||
avatar="http://cdn.libravatar.org/avatar/b6d36a03115d3ececaafe16e327e9e27"
|
||||
subject="+1"
|
||||
date="2019-02-24T17:06:30Z"
|
||||
content="""
|
||||
I am facing the same problem. It's a v6 repo on an NTFS partition mounted on an Ubuntu system. Many (but not all) of my files look like ordinary text files with a path to the key. Double-clicking them in the GUI obviously doesn't work because although the extention is .jpg, the file contents are plain text. As far as I can see, it should have been a symlink. Commands like \"get\" and \"find\" work as if those files are present in the repo.
|
||||
"""]]
|
||||
|
|
@ -16,7 +16,7 @@ packages:
|
|||
- '.'
|
||||
extra-deps:
|
||||
- IfElse-0.85
|
||||
- aws-0.20
|
||||
- aws-0.21.1
|
||||
- bloomfilter-2.0.1.0
|
||||
- tasty-1.1.0.4
|
||||
- tasty-rerun-1.1.13
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue