Merge branch 'master' of ssh://git-annex.branchable.com
This commit is contained in:
commit
a2b566be29
3 changed files with 34 additions and 0 deletions
|
@ -0,0 +1,9 @@
|
|||
[[!comment format=mdwn
|
||||
username="linnearight02@915958f850452a19de84ec14a765402d1f7ecdb0"
|
||||
nickname="linnearight02"
|
||||
avatar="http://cdn.libravatar.org/avatar/9c146ceff6ab204aa75ec5a686bd6cfb"
|
||||
subject="Online Coursework Service"
|
||||
date="2019-11-26T11:11:07Z"
|
||||
content="""
|
||||
Get the best [online coursework service](https://www.allassignmenthelp.com/online-coursework-service.html) by the top Aussie writers at cheap rates. We at [AllAssignmentHelp](https://www.allassignmenthelp.com/) known to provide custom coursework services and unlimited support to the Australian students when they place an order with us. All of our writers are well-qualified and trained professional writers, thus no need to be worried about the quality of the delivered work.
|
||||
"""]]
|
|
@ -0,0 +1,14 @@
|
|||
In neurophysiology we encounter HUGE files (HDF5 .nwb files).
|
||||
Sizes reach hundreds of GBs per file (thus exceeding any possible file system memory cache size). While operating in the cloud or on a fast connection it is possible to fetch the files with speeds up to 100 MBps.
|
||||
Upon successful download such files are then loaded back by git-annex for the checksum validation, and often at slower speeds (eg <60MBps on EC2 SSD drive).
|
||||
So, ironically, it does not just double, but rather nearly triples overall time to obtain a file.
|
||||
|
||||
I think ideally,
|
||||
|
||||
- (at minimum) for built-in special remotes (such as web), it would be great if git-annex was check-summing incrementally as data comes in;
|
||||
- made it possible to for external special remotes to provide desired checksum on obtained content. First git-annex should of cause inform them on type (backend) of the checksum it is interested in, and may be have some information reported by external remotes on what checksums they support.
|
||||
|
||||
If needed example, here is http://datasets.datalad.org/allen-brain-observatory/visual-coding-2p/.git with >50GB files such as ophys_movies/ophys_experiment_576261945.h5 .
|
||||
|
||||
[[!meta author=yoh]]
|
||||
[[!tag projects/dandi]]
|
|
@ -0,0 +1,11 @@
|
|||
[[!comment format=mdwn
|
||||
username="Ilya_Shlyakhter"
|
||||
avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0"
|
||||
subject="use named pipes?"
|
||||
date="2019-11-25T16:45:26Z"
|
||||
content="""
|
||||
For external remotes can pass to the `TRANSFER` request, as the `FILE` parameter, a named pipe, and use `tee` to create a separate stream for checksumming.
|
||||
|
||||
An external remote could also do its own checksum checking and then set remote.<name>.annex-verify=false.
|
||||
Could also make a “wrapper” external remote that delegates all requests to a given external remote but does checksum-checking in parallel with downloading (by creating a named pipe and passing that to the wrapped remote).
|
||||
"""]]
|
Loading…
Add table
Reference in a new issue