diff --git a/doc/bugs/file_field_omitted_from_import_--json-progress.mdwn b/doc/bugs/file_field_omitted_from_import_--json-progress.mdwn new file mode 100644 index 0000000000..ce060104c3 --- /dev/null +++ b/doc/bugs/file_field_omitted_from_import_--json-progress.mdwn @@ -0,0 +1,37 @@ +### Please describe the problem. + +When importing from a directory special remote created with importtree=yes and the --json-progress option is specified, the file field is omitted from the progress objects. + +### What steps will reproduce the problem? + +The command + +git annex import --from import-special-remote --json-progress main:Music + +generates output with the file field omitted from the nested action object in the progress objects on line 2 and 4, below. + +### What version of git-annex are you using? On what operating system? + +10.20231228-g5540f42e21afce6947d5410c5f18b178de6c336a + +on Linux + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + +{\"command\":\"list\",\"error-messages\":[],\"input\":[],\"success\":true} +{\"action\":{\"command\":\"import import-special-remote\",\"input\":[]},\"byte-progress\":9972,\"percent-progress\":\"100%\",\"total-size\":9972} +{\"command\":\"import import-special-remote\",\"error-messages\":[],\"input\":[],\"success\":true} +{\"action\":{\"command\":\"import import-special-remote\",\"input\":[]},\"byte-progress\":18,\"percent-progress\":\"100%\",\"total-size\":18} +{\"command\":\"import import-special-remote\",\"error-messages\":[],\"input\":[],\"success\":true} +{\"command\":\"update\",\"error-messages\":[],\"input\":[],\"success\":true} + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Yes, I am developing the easy-git-annex interface. diff --git a/doc/todo/documentation__58___improve_on_special_remotes.mdwn b/doc/todo/documentation__58___improve_on_special_remotes.mdwn new file mode 100644 index 0000000000..ed44caec40 --- /dev/null +++ b/doc/todo/documentation__58___improve_on_special_remotes.mdwn @@ -0,0 +1,9 @@ +Even though there is a [special_remotes](https://git-annex.branchable.com/special_remotes/) page, description there is quite superficial and does not provide a good overview of various aspects on special remotes + +- life cycle, e.g. how they relate to Git remotes -- that they become listed among `git remote`s whenever enabled, that git remotes get the corresponding `annex-uuid` assigned when "sensed" or get annex-ignore, how to "disable" an enabled remote (just `git remote remove`) +- clearly list commands to operate on the special remotes (initremote, enableremote) or interrogate them (e.g. how to figure out if there is already a special remote with a target uuid but may be not enabled yet, etc) + +Without such a documentation it is hard to "on board" new git-annex users and developers. + +[[!meta author=yoh]] +[[!tag projects/dandi]] diff --git a/doc/todo/yt-dlp__58___parse__47__handle___40__error__41_____34__Video_unavailable__34__.mdwn b/doc/todo/yt-dlp__58___parse__47__handle___40__error__41_____34__Video_unavailable__34__.mdwn new file mode 100644 index 0000000000..b9bc7181a3 --- /dev/null +++ b/doc/todo/yt-dlp__58___parse__47__handle___40__error__41_____34__Video_unavailable__34__.mdwn @@ -0,0 +1,91 @@ +ATM `git-annex addurl` would happily download HTML from youtube instead of errorring, which would probably a better behavior, if video was removed from YouTube or made private. E.g. + +
+here is a detailed git annex --debug addurl output + +```shell +$> git annex --debug addurl --file Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv https://www.youtube.com/watch?v=jy01CnsQ9ec +[2024-01-29 16:55:36.511153059] (Utility.Process) process [2205617] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"] +[2024-01-29 16:55:36.515415282] (Utility.Process) process [2205617] done ExitSuccess +[2024-01-29 16:55:36.515888181] (Utility.Process) process [2205618] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"] +[2024-01-29 16:55:36.519652103] (Utility.Process) process [2205618] done ExitSuccess +[2024-01-29 16:55:36.521989271] (Utility.Process) process [2205619] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"] +addurl https://www.youtube.com/watch?v=jy01CnsQ9ec +[2024-01-29 16:55:36.546921281] (Utility.Url) Request { + host = "www.youtube.com" + port = 443 + secure = True + requestHeaders = [("Accept-Encoding",""),("User-Agent","git-annex/10.20231227-1~ndall+1")] + path = "/watch" + queryString = "?v=jy01CnsQ9ec" + method = "HEAD" + proxy = Nothing + rawBody = False + redirectCount = 10 + responseTimeout = ResponseTimeoutDefault + requestVersion = HTTP/1.1 + proxySecureMode = ProxySecureWithConnect +} + +[2024-01-29 16:55:36.767345074] (Utility.Process) process [2205641] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","symbolic-ref","-q","HEAD"] +[2024-01-29 16:55:36.771989935] (Utility.Process) process [2205641] done ExitSuccess +[2024-01-29 16:55:36.772619847] (Utility.Process) process [2205642] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","refs/heads/master"] +[2024-01-29 16:55:36.777260151] (Utility.Process) process [2205642] done ExitSuccess +[2024-01-29 16:55:36.778775886] (Utility.Process) process [2205643] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","check-attr","-z","--stdin","annex.backend","annex.largefiles","annex.numcopies","annex.mincopies","--"] +[2024-01-29 16:55:36.78290662] (Utility.Url) Request { + host = "www.youtube.com" + port = 443 + secure = True + requestHeaders = [("Accept-Encoding","identity"),("User-Agent","git-annex/10.20231227-1~ndall+1")] + path = "/watch" + queryString = "?v=jy01CnsQ9ec" + method = "GET" + proxy = Nothing + rawBody = False + redirectCount = 10 + responseTimeout = ResponseTimeoutDefault + requestVersion = HTTP/1.1 + proxySecureMode = ProxySecureWithConnect +} + +[2024-01-29 16:55:36.982919787] (Utility.Process) process [2205644] read: yt-dlp ["https://www.youtube.com/watch?v=jy01CnsQ9ec","--get-filename","--no-warnings","--no-playlist"] +[2024-01-29 16:55:38.035161914] (Utility.Process) process [2205644] done ExitFailure 1 +[2024-01-29 16:55:38.036097212] (Utility.Process) p(to Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv) "--stdin","--verbose","--non-matching"] +[2024-01-29 16:55:38.054030685] (Messages.explain) [ Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv does not match annex.largefiles: mimeencoding=binary[FALSE] ] + (non-large file; adding content to git repository) +[2024-01-29 16:55:38.054895156] (Utility.Process) process [2205761] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","hash-object","-w","--no-filters","--stdin-paths"] +[2024-01-29 16:55:38.072695867] (Utility.Process) process [2205762] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","write-tree"] +[2024-01-29 16:55:38.078372317] (Utility.Process) process [2205762] done ExitSuccess +[2024-01-29 16:55:38.07887887] (Utility.Process) process [2205763] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/annex/last-index"] +addurl https://www.youtube.com/watch?v=jy01CnsQ9ec (to Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv) (non-large file; adding content to git repository) ok +(recording state in git...) +[2024-01-29 16:55:38.083368582] (Utility.Process) process [2205764] feed: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","update-index","-z","--index-info"] +[2024-01-29 16:55:38.129969795] (Utility.Process) process [2205764] done ExitSuccess +[2024-01-29 16:55:38.130183526] (Database.Handle) commitDb start +[2024-01-29 16:55:38.130668768] (Database.Handle) commitDb done +[2024-01-29 16:55:38.131339954] (Utility.Process) process [2205619] done ExitSuccess +[2024-01-29 16:55:38.131872474] (Utility.Process) process [2205643] done ExitSuccess +[2024-01-29 16:55:38.132363417] (Utility.Process) process [2205761] done ExitSuccess +[2024-01-29 16:55:38.132786755] (Utility.Process) process [2205760] done ExitFailure 1 + +$> file -L Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv +Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv: HTML document, ASCII text, with very long lines (56909) + +``` +
+ +whenever that video is now announced to be Private if you go to that youtube url or `yt-dlp` saying that + +```shell +❯ yt-dlp https://www.youtube.com/watch?v=jy01CnsQ9ec --get-filename --no-warnings --no-playlist +ERROR: [youtube] jy01CnsQ9ec: Video unavailable. This video has been removed by the uploader +❯ yt-dlp --version +2023.11.16 +``` + +note that if I fake up that youtube url by changing letter to the next `d` - response does not include any reason + +```shell +❯ yt-dlp https://www.youtube.com/watch?v=jy01CnsQ9ed --get-filename --no-warnings --no-playlist +ERROR: [youtube] jy01CnsQ9ed: Video unavailable +```