Merge branch 'master' of ssh://git-annex.branchable.com
This commit is contained in:
commit
ba0adefe4c
4 changed files with 202 additions and 0 deletions
20
doc/todo/idea__58___git-annex_cat.mdwn
Normal file
20
doc/todo/idea__58___git-annex_cat.mdwn
Normal file
|
@ -0,0 +1,20 @@
|
|||
I wanted to share some thoughts for an idea I had.
|
||||
|
||||
There are times when I want to stream data from a remote -- I want to start processing it immediately, and do not want to keep it in my annex when I am done with it.
|
||||
|
||||
I can give some examples:
|
||||
|
||||
* I have several projects which have a large number of similar text files, and they compress really well with borg or bup. For example, I have a repo with many [ncdu](https://dev.yorhel.nl/ncdu) json index files. They total 60G, but in a bup special remote, they are ~3G. In another repo, I have large highly differential tsv files.
|
||||
* I have an annex with 5-10G video files that are stored in a variety of network special remotes. Most of them are in my Google Drive. I would like to be able to immediately start playing them with VLC rather than downloading and verifying them in their entirety.
|
||||
|
||||
It would look like this:
|
||||
|
||||
```
|
||||
git annex cat "someindex.ncdu" | ncdu -f -
|
||||
|
||||
diff <(git annex cat "huge-data-dump1.tsv" -f mybupremote ) <(git annex cat "huge-data-dump2.tsv" -f mybupremote )
|
||||
|
||||
git annex cat "myvideo.mp4" -f googledrive | vlc -
|
||||
```
|
||||
|
||||
I imagine that there might be issues with verification. But I really am ok with not verifying a video file I am streaming.
|
|
@ -0,0 +1,8 @@
|
|||
[[!comment format=mdwn
|
||||
username="Lukey"
|
||||
avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b"
|
||||
subject="comment 10"
|
||||
date="2020-07-06T21:20:58Z"
|
||||
content="""
|
||||
I'm seeing larger improvements in my repo: ~40% speedup with -J2 and even ~200% speedup without jobs. Good work!
|
||||
"""]]
|
Loading…
Add table
Add a link
Reference in a new issue