Commit graph

67 commits

Author SHA1 Message Date
Joey Hess
7d9f0e0fbe
Added INFO to external special remote protocol.
It's left up to the special remote to detect when git-annex is new enough
to support the message; an old git-annex will blow up.

This commit was supported by the NSF-funded DataLad project.
2018-02-06 13:03:55 -04:00
Joey Hess
f5edb16729
Display progress meter when uploading a key without size information
Getting the size by statting the content file.

This commit was supported by the NSF-funded DataLad project.
2017-11-14 16:40:49 -04:00
Joey Hess
174ce991f9
typo 2017-05-19 10:57:59 -04:00
Joey Hess
37060c326a
fix build failure w/o concurrent-output
Also removed warning on old version, since the autobuilder highlights
warnings as errors and this is too minor for that.
2017-05-19 10:25:40 -04:00
Joey Hess
3ec0be7e1a
avoid warning when the concurrent-output flag is disabled 2017-05-16 21:06:34 -04:00
Joey Hess
1d45e47e3f
clear regions before ssh prompt
When built with concurrent-output 1.9, ssh password prompts will no longer
interfere with the -J display.

To avoid flicker, only done when ssh actually does need to prompt;
ssh is first run in batch mode and if that succeeds the connection is up
and no need to clear regions.

This commit was supported by the NSF-funded DataLad project.
2017-05-16 15:50:11 -04:00
Joey Hess
a1730cd6af
adeiu, MissingH
Removed dependency on MissingH, instead depending on the split
library.

After laying groundwork for this since 2015, it
was mostly straightforward. Added Utility.Tuple and
Utility.Split. Eyeballed System.Path.WildMatch while implementing
the same thing.

Since MissingH's progress meter display was being used, I re-implemented
my own. Bonus: Now progress is displayed for transfers of files of
unknown size.

This commit was sponsored by Shane-o on Patreon.
2017-05-16 01:03:52 -04:00
Joey Hess
6992fe133b
Ssh password prompting improved when using -J
When ssh connection caching is enabled (and when GIT_ANNEX_USE_GIT_SSH is
not set), only one ssh password prompt will be made per host, and only one
ssh password prompt will be made at a time.

This also fixes a race in prepSocket's stale ssh connection stopping
when run with -J. It was possible for one thread to start a cached ssh
connection, and another thread to immediately stop it, resulting in excess
connections being made.

This commit was supported by the NSF-funded DataLad project.
2017-05-11 17:36:03 -04:00
Joey Hess
ca0daa8bb8
factor non-type stuff out of Key 2017-02-24 13:42:30 -04:00
Joey Hess
672e53bded
add missing case for unknown size with normal output
This was lost in previous change, causing a crash in that case.
2016-10-05 15:10:24 -04:00
Joey Hess
28c6209f55
Make --json-progress output be shown even when the size of a object is not known. 2016-09-29 16:59:48 -04:00
Joey Hess
161d891508
Add "total-size" field to --json-progress output. 2016-09-29 16:29:54 -04:00
Joey Hess
d7ea6a5684
drop incremental json object display; clean up code
This gets rid of quite a lot of ugly hacks around json generation.

I doubt that any real-world json parsers can parse incomplete objects, so
while it's not as nice to need to wait for the complete object, especially
for commands like `git annex info` that take a while, it doesn't seem worth
the added complexity.

This also causes the order of fields within the json objects to be
reordered. Since any real json parser shouldn't care, the only possible
problem would be with ad-hoc parsers of the old json output.
2016-09-09 18:13:55 -04:00
Joey Hess
e2a69c2cee
refactor 2016-09-09 16:39:21 -04:00
Joey Hess
d4fbc3b460
make --json-progress work for url downloads 2016-09-09 16:15:39 -04:00
Joey Hess
312ef4dfae
make --json-progress update meter when getting from git remote with rsync 2016-09-09 16:05:45 -04:00
Joey Hess
a108235565
better locking for json with -J
Avoid threads emitting json at the same time and scrambling, which was
still possible even with the buffering, just less likely.

Converted json IO actions to JSONChunk data too.
2016-09-09 15:51:34 -04:00
Joey Hess
05d4438383
addurl, get: Added --json-progress option, which adds progress objects to the json output.
This doesn't work right when used with -J yet, and there is some really
ugly hand-crafting of part of the json output.
2016-09-09 15:06:54 -04:00
Joey Hess
f421a7f001
Remove key:null from git-annex add --json output. 2016-09-09 14:26:34 -04:00
Joey Hess
4a09b4bbbd
make maybeShowJSON also add to the buffer 2016-09-09 14:21:06 -04:00
Joey Hess
089c592977
buffer json output until done when in concurrent mode 2016-09-09 13:21:38 -04:00
Joey Hess
8ef494a833
disentangle concurrency and message type
This makes -Jn work with --json and --quiet, where before
setting -Jn disabled those options.

Concurrent json output is currently a mess though since threads output
chunks over top of one-another.
2016-09-09 12:57:42 -04:00
Joey Hess
d4c9711844
still update othermeter when not displaying progress at console
See no reason not to do this; the othermeter will be updating a transfer
info file or the like.
2016-09-08 13:18:07 -04:00
Joey Hess
e0fae28c72
Rate limit console progress display updates to 10 per second. Was updating as frequently as changes were reported, up to hundreds of times per second, which used unncessary bandwidth when running git-annex over ssh etc. 2016-09-08 13:17:43 -04:00
Joey Hess
03b031554f
avoid build warnings when building w/o concurrent-output 2016-09-06 14:36:15 -04:00
Joey Hess
38fffc6406
fix build with old ghc 2016-08-08 10:49:49 -04:00
Joey Hess
5235fb1185
avoid using Strings for JSON output; keep it ByteString throughout 2016-07-26 21:43:05 -04:00
Joey Hess
928fbb162d
improved use of Aeson for JSONActionItem 2016-07-26 19:50:02 -04:00
Joey Hess
870873bdaa
Removed dependency on json library; all JSON is now handled by aeson.
I've eyeballed all --json commands, and the only difference should be
that some fields are re-ordered.
2016-07-26 19:15:34 -04:00
Joey Hess
880674020f
aeson parser for --json output lines 2016-07-26 14:10:29 -04:00
Joey Hess
a030d0a8b7
allow using Aeson for streaming JSON output
Keeping Text.JSON use for now, because it seems a better fit for most of
the commands, which don't use very structured JSON objects, but just output
whatever fields suites them. But this lets Aeson be used when a more
structured data type is available to serialize to JSON.
2016-07-26 13:30:07 -04:00
Joey Hess
acf74ae945
improve json when showStart' is given only a key
Before, the json contained file:key; change that to key:

If a file and a key are given, inclue both file: and key:
2016-03-06 12:57:24 -04:00
Joey Hess
6021538332
fix build w/o concurrent-output 2016-02-15 15:29:41 -04:00
Joey Hess
4034de2be6
quiet warning on windows 2016-02-15 15:06:54 -04:00
Joey Hess
0f18636c8a
Work around problem with concurrent-output when in a non-unicode locale by avoiding use of it in such a locale.
Instead -J will behave as if it was built without concurrent-output support
in this situation. Ie, it will be mostly quiet, except when there's an
error.

Note that it's not a problem for a filename to contain invalid utf-8 when
in a utf-8 locale. That is handled ok by concurrent-output. It's only
displaying unicode characters in a non-unicode locale that doesn't work.
2016-02-14 15:02:42 -04:00
Joey Hess
3449c0e8ec
avoid spawning file size polling thread when not in -J mode 2015-11-16 21:21:58 -04:00
Joey Hess
e97fce35a6
Display progress meter in -J mode when downloading from the web.
Including in addurl, and get --from web, but also in S3 and External
special remotes when a web url is known for content in those remotes.
2015-11-16 21:00:54 -04:00
Joey Hess
8a352c3378
fix build warning when built w/o concurrent-output 2015-11-10 13:42:39 -04:00
Joey Hess
416c7656a1
Concurrent progress bars are now displayed when using -J with a command that moves file contents around. 2015-11-06 13:44:57 -04:00
Joey Hess
17b173dec9
don't display console regions untill there is output in them 2015-11-05 20:18:27 -04:00
Joey Hess
ab6b1edfee
join back threads before ending concurrent output so display works
I didn't really want to put allowConcurrentOutput in CmdLine.Action,
but there were dep loops and that was the best place available.
2015-11-05 17:23:26 -04:00
Joey Hess
e620073936
don't display blank regions once done 2015-11-04 16:58:19 -04:00
Joey Hess
340208f86b
remove some debug prints 2015-11-04 16:30:55 -04:00
Joey Hess
c0c595345c
arrange for regional output manager to run when -J is enabled
Commands that want to use it have to run their seek action inside
allowConcurrentOutput. Which seems reasonable; perhaps some future command
will want to support the -J flag but not use regions.

The region state moved from Annex to MessageState. This makes sense
organizationally, and note that some uses of onLocal use a different Annex
state, but pass the MessageState into it, which is what is needed.
2015-11-04 16:22:43 -04:00
Joey Hess
a4dd8503b8
add regions to concurrent output
still no progress displays when getting files etc, but a big improvement
2015-11-04 14:52:07 -04:00
Joey Hess
4fd03ccd7b
concurrent-output, first pass
Output without -Jn should be unchanged from before. With -Jn,
concurrent-output is used for messages, but regions are not used yet, so
it's a mess.
2015-11-04 13:45:34 -04:00
Joey Hess
67f7f1b1cb info: Added json output for "backend usage", "numcopies stats", "repositories containing these files", and "transfers in progress". 2015-06-16 13:50:28 -04:00
Joey Hess
505d540360 allow building without ascii-progress, since it is not ready yet
No progress bars with -J unless built with ascii-progress.
2015-05-12 13:54:16 -04:00
Joey Hess
5e9f0a3493 reuse strings 2015-04-14 16:46:06 -04:00
Joey Hess
a5d89633c3 scale progress bar to terminal size 2015-04-14 16:39:30 -04:00