Merge branch 'master' into rawfilepath
This commit is contained in:
Joey Hess
commit
f9fc26f05a
29 changed files with 299 additions and 59 deletions
|
|
@ -67,7 +67,7 @@ startExternalAddonProcess basecmd pid = do
|
|||
`onException` cleanupProcess pall
|
||||
-- This thread will exit after consuming any
|
||||
-- remaining stderr from the process.
|
||||
wait stderrelay
|
||||
() <- wait stderrelay
|
||||
hClose herr
|
||||
return $ ExternalAddonProcess
|
||||
{ externalSend = hin
|
||||
|
|
|
|||
|
|
@ -360,7 +360,7 @@ applyView' mkviewedfile getfilemetadata view = do
|
|||
topf <- inRepo (toTopFilePath f)
|
||||
go uh topf sha (toTreeItemType mode) =<< lookupKey f
|
||||
liftIO $ void clean
|
||||
genViewBranch view
|
||||
genViewBranch view
|
||||
where
|
||||
genviewedfiles = viewedFiles view mkviewedfile -- enables memoization
|
||||
|
||||
|
|
|
|||
27
CHANGELOG
27
CHANGELOG
|
|
@ -1,11 +1,5 @@
|
|||
git-annex (8.20201008) UNRELEASED; urgency=medium
|
||||
git-annex (8.20201103) upstream; urgency=medium
|
||||
|
||||
* Fix bug that prevented linux standalone bundle from working on a fresh
|
||||
install.
|
||||
* Fix build on Windows with network-3.
|
||||
* Fix a memory leak introduced in the last release.
|
||||
* add, import: Fix a reversion in 7.20191009 that broke handling
|
||||
of --largerthan and --smallerthan.
|
||||
* move: Improve resuming a move that was interrupted after the object
|
||||
was transferred, in cases where numcopies checks prevented the resumed
|
||||
move from dropping the object from the source repository.
|
||||
|
|
@ -14,7 +8,12 @@ git-annex (8.20201008) UNRELEASED; urgency=medium
|
|||
chunks using the configured chunk size.
|
||||
* Fixed some problems that prevented this command from working:
|
||||
git submodule foreach git annex init
|
||||
* testremote: Display exceptions when tests fail, to aid debugging.
|
||||
* Improve shutdown process for external special remotes and external
|
||||
backends. Make sure to relay any remaining stderr from the process,
|
||||
and potentially avoid the process getting a SIGPIPE if it writes to
|
||||
stderr too late.
|
||||
* Fix a bug that prevented linux standalone bundle from working on a fresh
|
||||
install.
|
||||
* Windows build changed to one done by the datalad-extensions project
|
||||
using Github actions.
|
||||
* Windows build now includes libmagic, so mimetype and mimeencoding
|
||||
|
|
@ -23,12 +22,14 @@ git-annex (8.20201008) UNRELEASED; urgency=medium
|
|||
on this.
|
||||
* view: Avoid using ':' from metadata when generating a view, because
|
||||
it's a special character on Windows ("c:")
|
||||
* Improve shutdown process for external special remotes and external
|
||||
backends. Make sure to relay any remaining stderr from the process,
|
||||
and potentially avoid the process getting a SIGPIPE if it writes to
|
||||
stderr too late.
|
||||
* Fix a memory leak introduced in the last release.
|
||||
* add, import: Fix a reversion in 7.20191009 that broke handling
|
||||
of --largerthan and --smallerthan.
|
||||
* view: Fix a reversion in 8.20200522 that broke entering or changing views.
|
||||
* Fix build on Windows with network-3.
|
||||
* testremote: Display exceptions when tests fail, to aid debugging.
|
||||
|
||||
-- Joey Hess <id@joeyh.name> Thu, 08 Oct 2020 10:48:17 -0400
|
||||
-- Joey Hess <id@joeyh.name> Tue, 03 Nov 2020 11:40:56 -0400
|
||||
|
||||
git-annex (8.20201007) upstream; urgency=medium
|
||||
|
||||
|
|
|
|||
9
Test.hs
9
Test.hs
|
|
@ -299,6 +299,7 @@ unitTests note = testGroup ("Unit Tests " ++ note)
|
|||
, testCase "export_import_subdir" test_export_import_subdir
|
||||
, testCase "shared clone" test_shared_clone
|
||||
, testCase "log" test_log
|
||||
, testCase "view" test_view
|
||||
, testCase "magic" test_magic
|
||||
, testCase "import" test_import
|
||||
, testCase "reinject" test_reinject
|
||||
|
|
@ -448,6 +449,14 @@ test_log :: Assertion
|
|||
test_log = intmpclonerepo $ do
|
||||
git_annex "log" [annexedfile] @? "log failed"
|
||||
|
||||
test_view :: Assertion
|
||||
test_view = intmpclonerepo $ do
|
||||
git_annex "metadata" ["-s", "test=test1", annexedfile] @? "metadata failed"
|
||||
git_annex "metadata" ["-s", "test=test2", sha1annexedfile] @? "metadata failed"
|
||||
git_annex "view" ["test=test1"] @? "entering view failed"
|
||||
checkexists annexedfile
|
||||
checkdoesnotexist sha1annexedfile
|
||||
|
||||
test_magic :: Assertion
|
||||
test_magic = intmpclonerepo $ do
|
||||
#ifdef WITH_MAGICMIME
|
||||
|
|
|
|||
|
|
@ -0,0 +1,12 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 6"""
|
||||
date="2020-11-02T19:17:01Z"
|
||||
content="""
|
||||
Hmm, the form has size=40, but people are pasting entire long error
|
||||
messages into it. Although special characters also being escaped increases
|
||||
the length, if it were really limited to around that length, it would still
|
||||
be short enough.
|
||||
|
||||
Ok, added maxlength=50 to it.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
### Please describe the problem.
|
||||
|
||||
Happens as a part of our test [for datalad-archives external special remote](https://github.com/datalad/datalad/blob/master/datalad/customremotes/tests/test_archives.py#L171) we drop/get the file we just added to git-annex from a tarball.
|
||||
|
||||
It works fine on Linux, and from todays fix [8.20201007-144-g7245a9ed5](https://git.kitenet.net/index.cgi/git-annex.git/commit/?id=7245a9ed533c87abb70b2b9d06a5d3d5ea2f23b4) we no longer stall, and proceed to that `drop`. But drop fails claiming that the content is locked. I have checked processes - there is no some other git annex hanging around and .lck even doesn't exist before we call `drop`.
|
||||
|
||||
(datalad-3) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\a\d>dir ..\..\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\
|
||||
Volume in drive C has no label.
|
||||
Volume Serial Number is AC6C-BABE
|
||||
|
||||
Directory of C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc
|
||||
|
||||
11/02/2020 05:17 PM <DIR> .
|
||||
11/02/2020 05:17 PM <DIR> ..
|
||||
11/02/2020 05:17 PM 3 SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc 1 File(s) 3 bytes
|
||||
2 Dir(s) 23,168,851,968 bytes free
|
||||
|
||||
|
||||
but then it is created by `drop`:
|
||||
|
||||
(datalad-3) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\a\d>git annex drop --debug -- a.datc
|
||||
[2020-11-02 17:21:26.8245443] process [5128] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","symbolic-ref","-q","HEAD"]
|
||||
[2020-11-02 17:21:26.8393936] process [5128] done ExitSuccess
|
||||
[2020-11-02 17:21:26.8393936] process [1452] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","show-ref","refs/heads/adjusted/dl-test-branch(unlocked)"]
|
||||
[2020-11-02 17:21:26.8860693] process [1452] done ExitSuccess
|
||||
[2020-11-02 17:21:26.8860693] process [2004] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","ls-files","--stage","-z","--","a.datc"]
|
||||
[2020-11-02 17:21:26.9331701] process [3452] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
|
||||
[2020-11-02 17:21:26.9501808] process [4632] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
|
||||
[2020-11-02 17:21:26.9501808] process [2920] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","show-ref","git-annex"]
|
||||
[2020-11-02 17:21:27.0113712] process [2920] done ExitSuccess
|
||||
[2020-11-02 17:21:27.0113712] process [976] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
|
||||
[2020-11-02 17:21:27.0580403] process [976] done ExitSuccess
|
||||
[2020-11-02 17:21:27.0580403] process [8] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","log","refs/heads/git-annex..6217b1e92de175374dcb552d2bddee0b785073ce","--pretty=%H","-n1"]
|
||||
[2020-11-02 17:21:27.1363275] process [8] done ExitSuccess
|
||||
[2020-11-02 17:21:27.1363275] process [6948] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","hash-object","-w","--stdin-paths","--no-filters"]
|
||||
[2020-11-02 17:21:27.1675675] process [1728] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch"]
|
||||
[2020-11-02 17:21:27.1675675] process [4840] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
|
||||
[2020-11-02 17:21:27.1834634] process [7832] feed: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","update-index","-z","--index-info"]
|
||||
[2020-11-02 17:21:27.2146236] process [6244] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","diff-index","--raw","-z","-r","--no-renames","-l0","--cached","refs/heads/git-annex","--"]
|
||||
[2020-11-02 17:21:27.2769931] process [6244] done ExitSuccess
|
||||
[2020-11-02 17:21:27.3111859] process [7832] done ExitSuccess
|
||||
[2020-11-02 17:21:27.3238724] process [5528] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
|
||||
[2020-11-02 17:21:27.3864742] process [8556] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
|
||||
[2020-11-02 17:21:27.4331328] process [1100] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch"]
|
||||
[2020-11-02 17:21:27.4331328] process [3516] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
|
||||
drop a.datc
|
||||
git-annex: content is locked
|
||||
failed
|
||||
[2020-11-02 17:21:27.5114868] process [1100] done ExitSuccess
|
||||
[2020-11-02 17:21:27.5114868] process [3516] done ExitSuccess
|
||||
[2020-11-02 17:21:27.5114868] process [8556] done ExitSuccess
|
||||
[2020-11-02 17:21:27.5275735] process [6948] done ExitSuccess
|
||||
[2020-11-02 17:21:27.5422938] process [5528] done ExitSuccess
|
||||
[2020-11-02 17:21:27.5422938] process done ExitSuccess
|
||||
[2020-11-02 17:21:27.5422938] process done ExitSuccess
|
||||
[2020-11-02 17:21:27.5422938] process [4632] done ExitSuccess
|
||||
[2020-11-02 17:21:27.5422938] process [3452] done ExitSuccess
|
||||
[2020-11-02 17:21:27.5422938] process [2004] done ExitSuccess
|
||||
[2020-11-02 17:21:27.5422938] process done ExitSuccess
|
||||
git-annex: drop: 1 failed
|
||||
|
||||
|
||||
(datalad-3) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\a\d>dir ..\..\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\
|
||||
Volume in drive C has no label.
|
||||
Volume Serial Number is AC6C-BABE
|
||||
|
||||
Directory of C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc
|
||||
|
||||
11/02/2020 05:21 PM <DIR> .
|
||||
11/02/2020 05:21 PM <DIR> ..
|
||||
11/02/2020 05:17 PM 3 SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc
|
||||
11/02/2020 05:21 PM 0 SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck
|
||||
2 File(s) 3 bytes
|
||||
2 Dir(s) 23,168,679,936 bytes free
|
||||
|
||||
|
||||
so it seems that `drop` somehow steps on its own feet and gets freaked out.
|
||||
|
||||
Here is also some dump from procmon captured during `drop`, and filtered on this lck file name
|
||||
|
||||
```
|
||||
5:07:49.2392479 PM git-annex.exe 2144 CreateFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS Desired Access: Generic Read/Write, Disposition: OpenIf, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, Delete, AllocationSize: 0, OpenResult: Opened File System 8228 8664
|
||||
5:07:49.2393198 PM git-annex.exe 2144 QueryInformationVolume C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS VolumeCreationTime: 10/22/2020 2:55:22 PM, VolumeSerialNumber: AC6C-BABE, SupportsObjects: True, VolumeLabel: File System 8228 8664
|
||||
5:07:49.2393295 PM git-annex.exe 2144 QueryAllInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck BUFFER OVERFLOW CreationTime: 11/2/2020 5:02:41 PM, LastAccessTime: 11/2/2020 5:04:56 PM, LastWriteTime: 11/2/2020 5:04:56 PM, ChangeTime: 11/2/2020 5:04:56 PM, FileAttributes: A, AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x2e000000015258, EaSize: 0, Access: Generic Read/Write, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Word File System 8228 8664
|
||||
5:07:49.2393613 PM git-annex.exe 2144 QueryInformationVolume C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS VolumeCreationTime: 10/22/2020 2:55:22 PM, VolumeSerialNumber: AC6C-BABE, SupportsObjects: True, VolumeLabel: File System 8228 8664
|
||||
5:07:49.2393740 PM git-annex.exe 2144 QueryAllInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck BUFFER OVERFLOW CreationTime: 11/2/2020 5:02:41 PM, LastAccessTime: 11/2/2020 5:04:56 PM, LastWriteTime: 11/2/2020 5:04:56 PM, ChangeTime: 11/2/2020 5:04:56 PM, FileAttributes: A, AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x2e000000015258, EaSize: 0, Access: Generic Read/Write, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Word File System 8228 8664
|
||||
5:07:49.2394246 PM git-annex.exe 2144 QueryStandardInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: False File System 8228 8664
|
||||
5:07:49.2394500 PM git-annex.exe 2144 SetEndOfFileInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS EndOfFile: 0 File System 8228 8664
|
||||
5:07:49.2395339 PM git-annex.exe 2144 SetAllocationInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS AllocationSize: 0 File System 8228 8664
|
||||
5:07:49.2395801 PM git-annex.exe 2144 CloseFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS File System 8228 8664
|
||||
```
|
||||
|
||||
so it seems to happily create that file.
|
||||
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/datalad]]
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 1"
|
||||
date="2020-11-03T02:12:16Z"
|
||||
content="""
|
||||
hehehey -- I think I got the reason at least!
|
||||
|
||||
```
|
||||
In [4]: len(r'C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4
|
||||
...: fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck')
|
||||
Out[4]: 257
|
||||
|
||||
```
|
||||
so we exceeded the volume of magical dust windows could swallow without it sticking out somewhere... If I change backend to MD5E we do not observe that issue.
|
||||
"""]]
|
||||
|
|
@ -65,3 +65,7 @@ git-annex: view: 1 failed
|
|||
|
||||
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
|
||||
git-annex is an awesome tool! Really appreciate your work and everything else works great!
|
||||
|
||||
> Thank you for reporting, I've fixed this reversion and am also going to
|
||||
> add something like that to the test suite, which lacked any testing of
|
||||
> views. [[done]] --[[Joey]]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 16"
|
||||
date="2020-11-02T21:04:35Z"
|
||||
content="""
|
||||
FWIW -- there is now CI for git-annex on linux (debian-based), osx, and Windows on https://github.com/datalad/datalad-extensions/ which builds git-annex installer, tests git-annex, and then tests datalad against it in a few scenarios.
|
||||
ATM it runs on cron, so not for each git-annex commit. We will see if we need/want more frequent builds. Artifacts (re)distribution will eventually be established on http://datasets.datalad.org/datalad/packages/ but we aren't there yet.
|
||||
"""]]
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
git-annex 8.20200720.1 released with [[!toggle text="these changes"]]
|
||||
[[!toggleable text="""
|
||||
* Fix a bug in find --batch in the previous version."""]]
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
git-annex 8.20200720 released with [[!toggle text="these changes"]]
|
||||
[[!toggleable text="""
|
||||
* import: Added --no-content option, which avoids downloading files
|
||||
from a special remote. Currently only supported by the directory
|
||||
special remote.
|
||||
* Honor annex.largefiles when importing a tree from a special remote.
|
||||
(Except for when --no-content is used.)
|
||||
* Fix a deadlock that could occur after git-annex got an unlocked
|
||||
file, causing the command to hang indefinitely. Known to happen on
|
||||
vfat filesystems, possibly others.
|
||||
* Build with the http-client-restricted and git-lfs libraries when
|
||||
available, otherwise use the vendored copy as before.
|
||||
* testremote: Fix over-allocation of resources and bad caching,
|
||||
including starting up a large number of external special remote processes.
|
||||
(Regression introduced in version 8.20200501)
|
||||
* test: Fix some test cases that assumed git's default branch name.
|
||||
* importfeed: Added some additional --template variables:
|
||||
itempubyear, itempubmonth, itempubday, itempubhour,
|
||||
itempubminute, itempubsecond.
|
||||
* Made several special remotes support locking content on them,
|
||||
which allows dropping from other special remotes in some situations
|
||||
where it was not possible before. Supported special remotes:
|
||||
S3 (with versioning=yes), git-lfs, tahoe
|
||||
* Fix reversion that broke passing annex.* and remote.*.annex-*
|
||||
git configs with -c. (Since version 8.20200330.)
|
||||
* Bring back git-annex branch read cache. This speeds up some operations,
|
||||
eg git-annex sync --content --all gets 20% faster.
|
||||
* Fix a recently introduced bug that could cause a "fork: resource exhausted"
|
||||
after getting several thousand files.
|
||||
* Sped up the --all option by 2x to 16x by using git cat-file --buffer.
|
||||
Thanks to Lukey for finding this optimisation.
|
||||
* Sped up seeking for annexed files to operate on by a factor of nearly 2x.
|
||||
* Sped up sync --content by 2x and other commands like fsck --fast and
|
||||
whereis by around 50%, by using git cat-file --buffer.
|
||||
* importfeed: Made checking known urls step around 15% faster.
|
||||
* fsck: Detect if WORM keys contain a carriage return, and recommend
|
||||
upgrading the key. (git-annex could have maybe created such keys back
|
||||
in 2013).
|
||||
* When on an adjust --hide-missing branch, fix handling of files that
|
||||
have been deleted but the deletion is not yet staged."""]]
|
||||
33
doc/news/version_8.20201103.mdwn
Normal file
33
doc/news/version_8.20201103.mdwn
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
git-annex 8.20201103 released with [[!toggle text="these changes"]]
|
||||
[[!toggleable text="""
|
||||
git-annex (8.20201103) upstream; urgency=medium
|
||||
.
|
||||
* move: Improve resuming a move that was interrupted after the object
|
||||
was transferred, in cases where numcopies checks prevented the resumed
|
||||
move from dropping the object from the source repository.
|
||||
* When a special remote has chunking enabled, but no chunk sizes are
|
||||
recorded (or the recorded ones are not found), speculatively try
|
||||
chunks using the configured chunk size.
|
||||
* Fixed some problems that prevented this command from working:
|
||||
git submodule foreach git annex init
|
||||
* Improve shutdown process for external special remotes and external
|
||||
backends. Make sure to relay any remaining stderr from the process,
|
||||
and potentially avoid the process getting a SIGPIPE if it writes to
|
||||
stderr too late.
|
||||
* Fix a bug that prevented linux standalone bundle from working on a fresh
|
||||
install.
|
||||
* Windows build changed to one done by the datalad-extensions project
|
||||
using Github actions.
|
||||
* Windows build now includes libmagic, so mimetype and mimeencoding
|
||||
will work.
|
||||
Thanks to John Thorvald Wodder II and Yaroslav Halchenko for their work
|
||||
on this.
|
||||
* view: Avoid using ':' from metadata when generating a view, because
|
||||
it's a special character on Windows ("c:")
|
||||
* Fix a memory leak introduced in the last release.
|
||||
* add, import: Fix a reversion in 7.20191009 that broke handling
|
||||
of --largerthan and --smallerthan.
|
||||
* view: Fix a reversion in 8.20200522 that broke entering or changing views.
|
||||
* Fix build on Windows with network-3.
|
||||
* testremote: Display exceptions when tests fail, to aid debugging.
|
||||
"""]]
|
||||
16
doc/templates/blogpost.tmpl
vendored
Normal file
16
doc/templates/blogpost.tmpl
vendored
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
<form <TMPL_IF POSTFORMID>id="<TMPL_VAR POSTFORMID ESCAPE=HTML>"</TMPL_IF> action="<TMPL_VAR CGIURL>" method="get">
|
||||
<div class="blogform">
|
||||
<TMPL_IF RSSURL>
|
||||
<a class="feedbutton" type="application/rss+xml" rel="alternate" title="<TMPL_VAR RSSDESC>" href="<TMPL_VAR RSSURL>">RSS</a>
|
||||
</TMPL_IF>
|
||||
<TMPL_IF ATOMURL>
|
||||
<a class="feedbutton" type="application/atom+xml" rel="alternate" title="<TMPL_VAR ATOMDESC>" href="<TMPL_VAR ATOMURL>">Atom</a>
|
||||
</TMPL_IF>
|
||||
<input type="hidden" name="do" value="blog" />
|
||||
<input type="hidden" name="from" value="<TMPL_VAR ROOTPAGE>" />
|
||||
<input type="hidden" name="subpage" value="1" />
|
||||
<TMPL_VAR POSTFORMTEXT>
|
||||
<input name="title" size="40" maxlength="50" />
|
||||
<input type="submit" value="Edit" />
|
||||
</div>
|
||||
</form>
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
In the light of the recent debugging of git-annex+datalad special remote tandem, I would have appreciated if there was a way to see reflection of git-annex actions beyond current `--debug`: what is annex intending to do internally (not just reporting which commands to be ran etc). Most likely such higher level of debug information would generally be not needed, so probably not worth "enriching" default `--debug` output, so I wonder if there could be an option or config setting which would set `git-annex` on the path to provide more debug output if requested (e.g. like those on closing stdin/stdout/stderr/stderr-handler), may be it could be a config setting which would allow for `DEFAULT|HEAVY|VERYHEAVY` or alike values.
|
||||
I hope that I would never need to use that, but it might end up being "days-savior"
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/datalad]]
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 1"""
|
||||
date="2020-11-02T18:48:44Z"
|
||||
content="""
|
||||
Every call to debugM does add overhead both in time to run it
|
||||
(checking if debugging is enabled is nonzero overhead and this will add up
|
||||
if it's done a few million times or whatever) and in time to write
|
||||
a useful explanation.
|
||||
|
||||
It seems to me that debugging is best added when trying to debug something,
|
||||
or when there's a readily available value that can be output (such as the
|
||||
safety proof that is debugged when dropping, or external special remote
|
||||
messages).
|
||||
|
||||
Trying to add enough useful debugging information to 60,000 lines of code
|
||||
that any possible thing it might do will have a useful debug trace seems
|
||||
like a recipe to end up with 100,000 lines of code. It would be easier to
|
||||
run the code in a debugger and step through it at that point.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
[[!comment format=mdwn
|
||||
username="yarikoptic"
|
||||
avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
|
||||
subject="comment 2"
|
||||
date="2020-11-02T20:37:47Z"
|
||||
content="""
|
||||
> Every call to debugM does add overhead both in time to run it (checking if debugging is enabled is nonzero overhead and this will add up if it's done a few million times or whatever) and in time to write a useful explanation.
|
||||
|
||||
debug-vs-not could be decided at start up once, and debug function could be set to noop if no debugging output (given the level requested) was desired. I really doubt that any reasonable number of noop function calls would add any notable burden on top of operation/interaction with external git commands and remotes.
|
||||
|
||||
> It seems to me that debugging is best added when trying to debug something, or when there's a readily available value that can be output (such as the safety proof that is debugged when dropping, or external special remote messages).
|
||||
|
||||
yes, but then first you need to identify the code where to add it and know how to add it... users (myself included) would not be be able to do so that easily.
|
||||
|
||||
> Trying to add enough useful debugging information to 60,000 lines of code that any possible thing it might do will have a useful debug trace seems like a recipe to end up with 100,000 lines of code
|
||||
|
||||
sure thing should not be added to every code line! But there would be value from knowing some critical \"interface\" points where e.g. hanging or incorrect operation could happen, e.g. if I saw earlier the output of what `trace.patch` you provided today I might have saved myself quite a lot of time, so sounds like a location of value to add some `heavydebug()` logging.
|
||||
|
||||
> It would be easier to run the code in a debugger and step through it at that point.
|
||||
|
||||
might not be that easy especially when relevant git-annex call is sandwiched deep in the hierarchy of datalad/git/git-annex/datalad/git-annex/datalad/git-annex calls through all the special remotes etc.
|
||||
"""]]
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
[[!comment format=mdwn
|
||||
username="joey"
|
||||
subject="""comment 3"""
|
||||
date="2020-11-03T01:47:26Z"
|
||||
content="""
|
||||
It would be possible to split out of AnnexState a value that is
|
||||
immutable, and have that part not gated behind a MVar, and so avoid the
|
||||
MVar lookup overhead, since it would just be part of the Annex Reader
|
||||
monad. And the debug action in that could just be a noop when it's disabled,
|
||||
and so be essentially zero overhead. One little problem is that option
|
||||
parsing (eg for --debug) currently runs in the Annex monad, so too late to
|
||||
change that immutable value.
|
||||
|
||||
Obviously having debug logging in a place where you just needed debug
|
||||
logging would have been useful. It does not logically follow that we can
|
||||
determine such places prior to your needing debug logging, or that we will
|
||||
need debug logging there later. After all, it seems to have been debugged
|
||||
now.
|
||||
|
||||
I suppose we can say that anything involving canceling a thread, or closing
|
||||
a file descriptor connected to a pipe, is fairly delicate and so it makes
|
||||
sense to wrap and debug log those. (Which were the things needing debug
|
||||
logging in your case.) But the only information available to display is
|
||||
"thread N" or "FD N", so it would also need to debug log each thread spawn
|
||||
and FD open point in order to work out which are interesting. (Or wrap the
|
||||
values with more information, but that rapidly gets ugly.)
|
||||
|
||||
Anyway, I rarely find myself adding debugM calls when debugging git-annex,
|
||||
generally just playing with the test case rapidly lets a theory be formed
|
||||
about what's happening, and then do something to test the theory, and I'm
|
||||
probably most of the way to a fix or at least understanding the problem.
|
||||
|
||||
The main way the current debug output is useful to me is it often hints at
|
||||
the proximate location of the problem, because of whatever git command
|
||||
git-annex ran just before it. It would certianly be useful to have
|
||||
additional debugM wherever there's a lot of code that does not involve
|
||||
anything that currently logs.. But I don't know a good way to identify
|
||||
such stretches of code.
|
||||
"""]]
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
Name: git-annex
|
||||
Version: 8.20201007
|
||||
Version: 8.20201103
|
||||
Cabal-Version: >= 1.10
|
||||
License: AGPL-3
|
||||
Maintainer: Joey Hess <id@joeyh.name>
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue