Commit graph

30855 commits

Author SHA1 Message Date
Joey Hess
4a45f34fe1
don't support removing content from export with removeKey
There does not seem to be a use case for supporting that, and it would
need a lot of complication to support it in a way that allows eventual
consistency when two repositories are updating the same export.

This commit was sponsored by Henrik Riomar on Patreon.
2017-09-17 17:56:33 -04:00
Joey Hess
ac4c8df2ca
Merge branch 'master' of ssh://git-annex.branchable.com 2017-09-16 16:44:38 -04:00
Joey Hess
494b4066db
clarification 2017-09-16 16:44:27 -04:00
Joey Hess
18ba1be26f
design for next steps on exports 2017-09-16 16:41:04 -04:00
Joey Hess
fa73387345
add example, including use of branch:subdir to export only a subdir 2017-09-16 16:40:20 -04:00
Joey Hess
7960870e48
add link to git-annex-remote-gvfs for smb / sftp 2017-09-16 13:57:37 -04:00
gleachkr@7c488e398809299a1100b93f8884de43dee83674
5ec74f1cdb Added a comment: Thanks! 2017-09-16 16:46:48 +00:00
Joey Hess
dbe8c42fdc
update 2017-09-16 12:09:11 -04:00
Joey Hess
b7dafb4f66
response for gleachkr 2017-09-16 12:07:12 -04:00
gleachkr@7c488e398809299a1100b93f8884de43dee83674
63dfd6cde2 Added a comment 2017-09-16 16:02:41 +00:00
Joey Hess
260d1061cd
followup for gleachkr 2017-09-16 11:26:00 -04:00
gleachkr@7c488e398809299a1100b93f8884de43dee83674
97eacf3a7a Added a comment: More data points 2017-09-16 01:16:59 +00:00
Joey Hess
e1f5c90c92
split out Types.Export 2017-09-15 16:46:03 -04:00
Joey Hess
e54a05612e
avoid unncessary db queries when exported directory can't be empty
In rename foo/bar to foo/baz, foo can't be empty.

In delete zxyyz, there's no exported directory (top doesn't count).
2017-09-15 16:30:49 -04:00
Joey Hess
af82b2229c
devblog 2017-09-15 15:54:20 -04:00
Joey Hess
cf51f40f0e
webdav: Changed path used on webdav server for temporary files.
Done to avoid a "tmp" directory appearing in webdav exports.

Also affects non-export webdav remotes, so interrupted uploads using the
old path will not overwrite it. However, PUT is quite likely to be
implemented atomically on web servers anyway, so I doubt this will cause
problems.
2017-09-15 15:52:31 -04:00
Joey Hess
1223960294
empty directory removal working 2017-09-15 15:24:45 -04:00
Joey Hess
c633144d28
remove empty directories when removing from export
The subtle part of this is what happens when the remote fails to remove
an empty directory. The removal from the export needs to fail in that
case, so the removal will be tried again later. However, removeExportLocation
has already been run and changed the export db, so if the next run
checks getExportLocation, it might decide nothing remains to be done,
leaving the empty directory.

Dealt with that by making removeEmptyDirectories, handle a failure
by calling addExportLocation, reverting the database changes so the next
run will be guaranteed to try deleting the empty directory again.

This commit was sponsored by Thomas Hochstein on Patreon.
2017-09-15 15:22:53 -04:00
Joey Hess
e223cf568f
add table to keep track of what subdirectories are populated in the export
So empty subdirectories can be identified and removed.

This commit was sponsored by Jochen Bartl on Patreon.
2017-09-15 14:35:22 -04:00
Joey Hess
bdcf19b095
add missing case 2017-09-15 14:32:56 -04:00
Joey Hess
3435a47245
Merge branch 'master' of ssh://git-annex.branchable.com 2017-09-15 13:20:09 -04:00
Joey Hess
9f4ffe65e9
implement removeExportDirectory
Not yet called by Command.Export.

WebDAV needs this to clean up empty collections. Also, example.sh turned
out to not be cleaning up directories when removing content
from them, so it made sense for it to use this.

Remote.Directory did not need it, and since its cleanup method for empty
directories is more efficient than what Command.Export will need to do
to find empty directories, it uses Nothing so that extra work can be
avoided.

This commit was sponsored by Thom May on Patreon.
2017-09-15 13:18:21 -04:00
Joey Hess
78a67f29f8
forwarded from irc 2017-09-15 13:13:43 -04:00
Joey Hess
5fe803e14e
update 2017-09-15 12:22:11 -04:00
tomasz
dd2c97d8ef 2017-09-15 14:33:31 +00:00
Joey Hess
589184ccc3
/dev/null stderr when generating completions
avoid usage display when too old optparse-applicative
2017-09-14 13:10:45 -04:00
benjamin.poldrack@d09ccff6d42dd20277610b59867cf7462927b8e3
849a5d4771 Added a comment: @joey: Sorry ... 2017-09-14 12:00:13 +00:00
Joey Hess
268a0cc664
update 2017-09-13 15:52:19 -04:00
Joey Hess
86fe31c767
Merge branch 'master' of ssh://git-annex.branchable.com 2017-09-13 15:36:17 -04:00
Joey Hess
01068d8280
fix build with old http-client 2017-09-13 15:35:42 -04:00
Joey Hess
bf48ba4ef7
work around box.com webdav rename bug
Apparently box.com renaming is just buggy. I tried a couple of fixes:

* In case the http Manager was opening multiple connections and reaching
  different backend servers, I tried limiting the number of connections
  to 1. Didn't help.
* To make sure it was not a http connection reuse problem, I tried
  rewriting how exportAction works, so that the same http connection
  is clearly open. Didn't help.

So, disable renaming of exports for box.com. It would be good to test it
with some other webdav server.

This commit was sponsored by John Peloquin on Patreon.
2017-09-13 15:26:56 -04:00
Joey Hess
955c616956
fix exporting files in subdirectories to webdav
Use tmp/key when exporting, so the whole export directory structure does
not have to be created under tmp/

This commit was sponsored by Denis Dzyubenko on Patreon.
2017-09-13 15:09:19 -04:00
Joey Hess
969ac892a8
add Command.Export 2017-09-13 14:21:08 -04:00
yarikoptic
afac1c46bc Added a comment 2017-09-13 17:58:54 +00:00
Joey Hess
b31d682539
cleanup imports 2017-09-13 12:53:19 -04:00
Joey Hess
fd062dafc5
comment 2017-09-13 12:34:45 -04:00
Joey Hess
de58dce116
Merge branch 'master' of ssh://git-annex.branchable.com 2017-09-13 12:09:38 -04:00
Joey Hess
ab271ba6ca
trust level overridden message adjusted for forced untrusted export remotes 2017-09-13 12:08:46 -04:00
Joey Hess
28ba158a24
clear exportSupported for non-export remotes
Non-export remotes were being treated as untrusted, so the test suite
failed, and probably other things broke.
2017-09-13 12:05:53 -04:00
yarikoptic
0e4504cd2f Added a comment 2017-09-13 14:32:57 +00:00
yarikoptic
896e516941 crossed away comment on hash levels 2017-09-13 14:27:19 +00:00
yarikoptic
15ebb707fc initial finding about incorrect permissions ignored by fsck 2017-09-13 14:25:16 +00:00
Joey Hess
f8fd66d3f8
fix compaction of export.log
It was not getting old lines removed, because the tree graft confused
the updater, so it union merged from the previous git-annex branch,
which still contained the old lines. Fixed by carefully using setIndexSha.

This commit was supported by the NSF-funded DataLad project.
2017-09-12 18:30:36 -04:00
Joey Hess
0fadb620d3
devblog 2017-09-12 18:30:07 -04:00
Joey Hess
c8ed941a26
change export.log format to support multiple export remotes
This breaks backwards compatibility, but only with unreleased versions of
git-annex, which I think is acceptable.

This commit was supported by the NSF-funded DataLad project.
2017-09-12 17:45:52 -04:00
Joey Hess
301c959edf
remove debug print 2017-09-12 17:00:39 -04:00
Joey Hess
63ba764923
bug 2017-09-12 17:00:15 -04:00
Joey Hess
9c3622882b
export: cache connections for S3 and webdav 2017-09-12 16:59:04 -04:00
Joey Hess
7ad8e8b889
more box.com strangeness 2017-09-12 15:45:43 -04:00
Joey Hess
e177bb1e25
webdav: Fix lack of url-escaping of filenames.
inDAVLocation does not url-escape, and so exporting a filename with spaces
to box.com at least resulted in a error 400.

It might also have affected storing keys on a webdav remote, if the key
contained a space or other problem character. Pretty unlikely.

I emailed Clint about the inDAVLocation gotcha, but seems best to fix it
here.

This commit was supported by the NSF-funded DataLad project.
2017-09-12 15:45:03 -04:00