mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

export: Let concurrent transfers be done with -J or annex.jobs

Browse source 
Tested working, although I did find this bug in my testing, which also
afflicts sync -J to an export remote.
This commit is contained in:
Joey Hess 2020-05-26 11:43:58 -04:00
parent f7fe71602c
commit 0bcecb67f5
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
4 changed files with 23 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

1
CHANGELOG
View file

@ -4,6 +4,7 @@ git-annex (8.20200523) UNRELEASED; urgency=medium
* import: Added --json-progress. * import: Added --json-progress.
* addurl: Make --preserve-filename also apply when eg a torrent contains * addurl: Make --preserve-filename also apply when eg a torrent contains
multiple files. multiple files.
* export: Let concurrent transfers be done with -J or annex.jobs.
-- Joey Hess <id@joeyh.name> Tue, 26 May 2020 10:20:52 -0400 -- Joey Hess <id@joeyh.name> Tue, 26 May 2020 10:20:52 -0400

4
Command/Export.hs
View file

@ -44,7 +44,7 @@ import qualified Data.Map as M
import Control.Concurrent import Control.Concurrent
cmd :: Command cmd :: Command
cmd = withGlobalOptions [jsonOptions, jsonProgressOption] $ cmd = withGlobalOptions [jobsOption, jsonOptions, jsonProgressOption] $
command "export" SectionCommon command "export" SectionCommon
"export content to a remote" "export content to a remote"
paramTreeish (seek <$$> optParser) paramTreeish (seek <$$> optParser)
@ -77,7 +77,7 @@ exportTempName ek = mkExportLocation $ toRawFilePath $
".git-annex-tmp-content-" ++ serializeKey (asKey (ek)) ".git-annex-tmp-content-" ++ serializeKey (asKey (ek))
seek :: ExportOptions -> CommandSeek seek :: ExportOptions -> CommandSeek
seek o = do seek o = startConcurrency commandStages $ do
r <- getParsed (exportRemote o) r <- getParsed (exportRemote o)
unlessM (isExportSupported r) $ unlessM (isExportSupported r) $
giveup "That remote does not support exports." giveup "That remote does not support exports."

13
doc/bugs/sync_to_export_with-J_STM_error.mdwn Normal file
View file

@ -0,0 +1,13 @@
git annex sync exportremote -J2 --content
git-annex: thread blocked indefinitely in an MVar operation
failed
git-annex: thread blocked indefinitely in an STM transaction
Also, git-annex export -J2 crashes the same way. I discovered this bug
when adding -J to export, but then found sync had the same bug.
To reproduce this, there may need there to be a tree of several annexed
files whose content is not locally available. In my case,
there were 338 of them. It seems to export all, or all but 1
before crashing. --[[Joey]]

7
doc/git-annex-export.mdwn
View file

@ -86,6 +86,13 @@ so the overwritten modification is not lost.)
the remote. You can later run `git annex sync --content` to upload the remote. You can later run `git annex sync --content` to upload
the files to the export. the files to the export.
* `--jobs=N` `-JN`
Exports multiple files in parallel. This may be faster.
For example: `-J4`
Setting this to "cpus" will run one job per CPU core.
* `--json` * `--json`
Enable JSON output. This is intended to be parsed by programs that use Enable JSON output. This is intended to be parsed by programs that use