Merge branch 'master' of ssh://git-annex.branchable.com

This commit is contained in:
Joey Hess 2018-03-29 13:32:21 -04:00
commit fd53b80624
No known key found for this signature in database
GPG key ID: DB12DB0FF05F8F38
6 changed files with 151 additions and 0 deletions

View file

@ -0,0 +1,42 @@
### Please describe the problem.
Not really related to datalad -- thought to addurl youtube playlist
youtube-dl seems might be capable of doing it
### Please provide any additional information below.
[[!format sh """
$> git annex addurl --debug 'https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ' [2018-03-28 13:09:06.337738339] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2018-03-28 13:09:06.34708705] process done ExitSuccess
[2018-03-28 13:09:06.347202003] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2018-03-28 13:09:06.35303272] process done ExitSuccess
[2018-03-28 13:09:06.353205536] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..734b368eab4f39d8494671657977952b02a35d9a","--pretty=%H","-n1"]
[2018-03-28 13:09:06.359101486] process done ExitSuccess
[2018-03-28 13:09:06.360005167] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2018-03-28 13:09:06.361006846] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
addurl https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ
[2018-03-28 13:09:06.409163359] call: wget ["-nv","--show-progress","--clobber","-c","-O","/tmp/testyt/.git/annex/tmp/URL--https&c%%www.youtube.com%playlis-5bc73cdf8dc8bd73b13addc290b160e6","https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ","--user-agent","git-annex/6.20180316+gitg308f3ecf6-1~ndall+1"]
/tmp/testyt/.git/ann 100%[===================>] 211.54K --.-KB/s in 0.04s
2018-03-28 13:09:06 URL:https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ [216616/216616] -> "/tmp/testyt/.git/annex/tmp/URL--https&c%%www.youtube.com%playlis-5bc73cdf8dc8bd73b13addc290b160e6" [1]
[2018-03-28 13:09:06.844196402] process done ExitSuccess
[2018-03-28 13:09:06.845246148] read: youtube-dl ["https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ","--get-filename","--no-warnings"]
[2018-03-28 13:09:40.643440496] process done ExitSuccess
[2018-03-28 13:09:40.644420015] call: youtube-dl ["https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ","--no-playlist","--playlist-items","0","--max-filesize","9046842816"]
[youtube:playlist] PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ: Downloading webpage
[download] Downloading playlist: Canonical Computation in Brains and Machines
[youtube:playlist] playlist Canonical Computation in Brains and Machines: Downloading 0 videos
[download] Finished downloading playlist: Canonical Computation in Brains and Machines
[2018-03-28 13:09:41.925771815] process done ExitSuccess
youtube-dl did not put any media in its work directory, perhaps it's been configured to store files somewhere else?
failed
[2018-03-28 13:09:41.926669401] process done ExitSuccess
[2018-03-28 13:09:41.926951446] process done ExitSuccess
git-annex: addurl: 1 failed
git annex addurl --debug 6.49s user 0.32s system 18% cpu 35.834 total
"""]]

View file

@ -0,0 +1,26 @@
[[!comment format=mdwn
username="andrew"
avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435"
subject="comment 1"
date="2018-03-28T00:59:41Z"
content="""
I think if you don't mind scripting checkout Joey's comment in this thread: [\"Preseeding\" a special remote](https://git-annex.branchable.com/forum/__34__Preseeding__34___a_special_remote/).
Otherwise maybe the following will work. You'll still need to download all of the files to hash them.
UNTESTED:
Lets say your Windows share data is at mnt/share/data
* create a new repo on your Windows share at mnt/share/repo1
* [git-annex-import](https://git-annex.branchable.com/git-annex-import/) all of your files into the new repo. This probably would download all of the files to generate the hash, then hopefully do a file rename avoiding the re-upload step.
* then create a new [directory remote](http://git-annex.branchable.com/special_remotes/directory/) for repo1 at say, mnt/share/remote1
* then do a [git-annex-move](https://git-annex.branchable.com/git-annex-move/) of all the files from mnt/share/repo1 to your new empty remote
* you could then make a clone of that repo locally, now you have a nice looking working tree locally
* then add mnt/share/remote1 as a new remote to your local clone
If you don't care about hashing you could use the [WORM](http://git-annex.branchable.com/backends/) backend avoiding the initial download.
I think, If you don't care about having a nice working tree you could probably use the web special remote and just leave all the files in their messy rsynced state. IE create a local repo, then do `git annex addurl --fast <path>` for each file on your Windows share. See joey's comment on addurl in this [DVD thread](http://git-annex.branchable.com/forum/Managing_a_large_number_of_files_archived_on_many_pieces_of_read-only_medium___40__E.G._DVDs__41__/)
"""]]

View file

@ -0,0 +1,34 @@
[[!comment format=mdwn
username="andrew"
avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435"
subject="numcopies"
date="2018-03-27T22:25:30Z"
content="""
||| *When I add a new file to my client clone and do git annex sync --content is this respecting the … content of each remote or is it only trying to satisfy the numcopy?*
Both.
Running `git annex sync --content` will copy content to any remote where that content is wanted (IE look at the preferred content settings), and will drop files that are not wanted and don't violate numcopies totals (at the moment of the drop?).
||| *One of the remotes is set as archive and two as smallarchive yet sync --content only ever copies to one so as to satisfy numcopies.*
I don't believe that `git annex sync --content` is ever trying to “satisfy numcopies,” I would think of numcopies as more of a limit or restriction on when git-annex is allowed to drop content that is not wanted by a remote.
||| *Shouldn't it always try to make two copies in archive or smallarchive?*
Hmmmm. I would guess there is some issue with your archive or smallarchive expressions, or they aren't actually set (being used) or you have discovered an issue… You have overridden the standard groups, and can see your overrides with `git annex groupwanted archive` and `git annex groupwanted smallarchive`? And your remotes have `git annex group archive` and `git annex wanted groupwanted` set?
So `(include=*/archive/* or include=archive/*) and` means you are only copying files in the archive directory, was that your intention?
If so, the rest of your content expression seems like it should want 1 copy in an archive and 1 copy in a smallarchive, or 2 copies in 2 remotes marked archive or 2 copies in two remotes marked smallarchive.
||| *Interestingly, it copies data to my backup*
Right. If you are using a standard content group backup means “All content is wanted. Even content of old/deleted files.” This expression will want all content and never drop content.
"""]]

View file

@ -0,0 +1,16 @@
[[!comment format=mdwn
username="MatusGoljer"
avatar="http://cdn.libravatar.org/avatar/8152eed1d594c570563ed46e7fd8356f"
subject="comment 2"
date="2018-03-28T23:16:51Z"
content="""
Hi Andrew!
Actually, you are entirely correct, I just misinterpreted the `smallarchive` formula. What I was thinking it would do was it would upload the file in case there are not two copies in any combination of `archive` or `smallarchive` and as soon as there are two `archive` copies it would ideally drop the file if not in `*/archive/*` directory.
But now I see that the formula does exactly what you say it would do, the whole long `and not` condition is additional to the first, which means a file will not get uploaded if not in those directories.
I will play around with the expression, I'm sure it can be modified to do what I want.
Thanks! It was mostly me being silly, sometimes having other people re-state the obvious helps!
"""]]

View file

@ -0,0 +1,9 @@
[[!comment format=mdwn
username="cristianpolanco2015@04c392af1063e289871c7a1ae208794e1105ba65"
nickname="cristianpolanco2015"
avatar="http://cdn.libravatar.org/avatar/8787a59fc83d3a4d4ef0fd131c59e83d"
subject="Play Store App Download Free"
date="2018-03-29T15:27:19Z"
content="""
Download Google Play Store app free - Play Store https://playstore.zone/
"""]]

View file

@ -0,0 +1,24 @@
[[!comment format=mdwn
username="achilleas.k@14be77d42a1252fab5ec9dbf4e5ea03c5833e8c8"
nickname="achilleas.k"
avatar="http://cdn.libravatar.org/avatar/ed6c67c4d8e6c6850930e16eaf85a771"
subject="dmg (possibly) requires rebuild"
date="2018-03-29T15:31:23Z"
content="""
On macOS High Sierra (10.13.3) trying to use git-annex from the latest .dmg shows the following error:
> git annex version
dyld: Symbol not found: _inflateValidate
Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
Expected in: /Applications/git-annex.app/Contents/MacOS/bundle/B
in /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
error: git-annex died of signal 6
I ran it by opening the .app directory (Show Package Contents) and running the `runshell` script to set up the environment:
> which git
/Applications/git-annex.app/Contents/MacOS/bundle/git
> which git-annex
/Applications/git-annex.app/Contents/MacOS/bundle/git-annex
"""]]