Commit graph

45412 commits

Author SHA1 Message Date
Spencer
949be665c0 Added contributions section to track my bugs and inquiries 2024-08-26 20:02:03 +00:00
yarikoptic
efdee386c0 initial report on desire to do handle pathspecs 2024-08-24 01:35:31 +00:00
yarikoptic
c3877f648c initial idea on another ability for get 2024-08-24 01:23:04 +00:00
gauss@055c9051f507c97fa5612f46c74ce636f5ecde10
d71ca87bc9 Added a comment: No root privileges server - annex-shell replaced by git-annex-shell 2024-08-23 01:51:49 +00:00
Spencer
acaa8e9cd5 Added a comment: Precise Workflow 2024-08-22 00:18:28 +00:00
Matthew
4a9e637d36 Added a comment: Help with .nfsXXXX files 2024-08-19 21:20:59 +00:00
matrss
9cfdae4c3b Added a comment 2024-08-19 10:25:13 +00:00
xentac
74b953cded Added a comment 2024-08-18 03:17:12 +00:00
Spencer
40b49e2ddd Added a comment: Remote Helper? 2024-08-17 05:33:01 +00:00
matrss
bcf876e3a0 2024-08-16 15:52:32 +00:00
matrss
f057010086 Added a comment 2024-08-16 15:45:45 +00:00
Atemu
e8997d8899 Added a comment 2024-08-15 15:40:20 +00:00
pedro-lopes-de-azevedo
c75ecc5350 Added a comment: parameter --from not accepted 2024-08-14 14:27:54 +00:00
bvaa
11eb2ae6ec Added a comment 2024-08-14 07:18:26 +00:00
Spencer
05a62e4e5f Added a comment: Workaround: --force-small 2024-08-13 07:05:57 +00:00
Spencer
3d252da06c Added a comment: Exact Moment Things Go Wrong 2024-08-13 06:22:11 +00:00
Spencer
ab5f920d77 .md linting 2024-08-13 04:46:53 +00:00
Spencer
8a91a8c208 2024-08-13 04:46:10 +00:00
Spencer
c4296fbd45 Added a comment: Still a Problem (on Mac?) 2024-08-13 04:21:33 +00:00
ewen
491cf67ce2 Added a comment: Most servers upgraded to TLS v1.2 EMS / TLS v1.3 2024-08-13 00:01:05 +00:00
xentac
fb186ab0a8 Added a comment 2024-08-09 19:31:12 +00:00
xentac
55a5cb7904 2024-08-09 19:22:19 +00:00
Joey Hess
152c87140b
update 2024-08-08 16:06:02 -04:00
Joey Hess
bda23daa6c
update 2024-08-08 15:54:22 -04:00
Joey Hess
fd03b31633
update 2024-08-08 15:53:36 -04:00
Joey Hess
7e48e712b2
update 2024-08-08 15:52:52 -04:00
Joey Hess
0959bfe5d3
update for exporttree=yes 2024-08-08 15:51:36 -04:00
Joey Hess
727b6a0b6d
update 2024-08-08 15:34:36 -04:00
Joey Hess
2616056cde
Merge branch 'exportreeplus' 2024-08-08 15:31:57 -04:00
Joey Hess
3b758aaad6
add news item for git-annex 10.20240808 2024-08-08 15:27:11 -04:00
Joey Hess
c15c32b5f8
releasing package git-annex version 10.20240808 2024-08-08 15:27:04 -04:00
Joey Hess
349b1e443b
proxied importtree=yes remotes are untrustworthy
Even without exporttree=yes.
2024-08-08 15:26:02 -04:00
Joey Hess
3ea835c7e8
proxied exporttree=yes versionedexport=yes remotes are not untrusted
This removes versionedExport, which was only used by the S3 special
remote. Instead, versionedexport=yes is a common way for remotes to
indicate that they are versioned.
2024-08-08 15:24:19 -04:00
Joey Hess
5c36177e58
proxied exporttree=yes remotes are untrustworthy
This is not perfect because it does not handle versioned special
remotes, which should not be untrustworthy, but now are when proxied.

The implementation turned out to be easy, because the exporttree field
is a default field, so is available in RemoteConfig even for git
remotes.
2024-08-08 14:43:53 -04:00
Joey Hess
b23c7f769e
update 2024-08-08 14:25:18 -04:00
Joey Hess
9663888c77
update 2024-08-08 14:05:05 -04:00
Joey Hess
c84d1a9462
update export db after rename from annexobjects location
This allows git-annex post-receive, on the first push to
the remote to see that it is able to get a key from it in
order to upload it back.

Also avoided actively checking if the source remote contains a key.
The location log is good enough. If the location log is wrong,
the export of that file will fail with an informative message.
2024-08-08 14:03:02 -04:00
Joey Hess
a2eb3b450a
post-receive: use the exporttree=yes remote as a source
This handles cases where a single key is used by multiple files in the
exported tree. When using `git-annex push`, the key's content gets
stored in the annexobjects location, and then when the branch is pushed,
it gets renamed from the annexobjects location to the first exported
file. For subsequent exported files, a copy of the content needs to be
made. This causes it to download the key from the remote in order to
upload another copy to it.

This is not needed when using `git push` followed by `git-annex copy --to`
the proxied remote, because the received key is stored at all export
locations then.

Also, fixed handling of the synced branch push, it was exporting master
when synced/master was pushed.

Note that currently, the first push to the remote does not see that it
is able to get a key from it in order to upload it back. It displays
"(not available)". The second push is able to. Since git-annex push
pushes first the synced branch and then the branch, this does end up
with a full export being made, but it is not quite right.
2024-08-08 13:49:53 -04:00
Joey Hess
7294d23d78
export: Added --from option
This is similar to git-annex copy --from --to, in that it downloads a
local copy, locks it for removal, uploads it, and drops it. Removal of
the temporary local copy is done without verifying numcopies for the
same reason as that command.

I do wonder, looking at this, if there's a race where the local copy
gets used as a copy to allow some other drop in the narrow window after
it is downloaded and before it gets locked for removal. That would need
some other repository to have an out of date location log that says the
repository contains a copy of the key, in order for it to try to use it
as a copy. If there is such a race, git-annex copy/move would also be
vulnerable to it. It would be better to lock it for removal before
starting to download it! That is possible in v10 repositories, which do
use a separate content lock file.

Note that, when the exported tree contains several files that use the
same key, it will be downloaded repeatedly, once per time needed to
upload it. It would be possible to avoid that extra work, but it would
complicate this since the local copy would need to be preserved, locked
for removal, until the end. Also, that would mean that interrupting the
export would leave possibly a lot of temporarily downloaded keys in the
local repository, while currently it can only leave one.
2024-08-08 12:08:55 -04:00
Joey Hess
bd677bb65a
avoid warning in startDispose
When a file never got exported to the remote, and is now being removed
from the exported tree, it tried to rename, which failed, and displayed
an ugly warning:

	unexport d m8
	  rename failed (/home/joey/tmp/bench2/d/m8: renameFile:renamePath:rename: does not exist (No such file or directory)); deleting instead
	ok
2024-08-08 11:59:16 -04:00
Joey Hess
01edd186e9
update proxied exporttree=yes remote on receive of sync branch
Since git-annex sync sends the sync branch first, and only displays the
output of the push to the sync branch, this makes git-annex
post-retrieve's output when updating the exported tree be visible when
syncing.

This also makes syncing with a non-bare repository still update the
exported tree, even when the checked out branch is not able to be
updated. The sync branch gets sent regardless.
2024-08-07 13:11:06 -04:00
Joey Hess
55adbb6694
avoid trying to export tree to proxied exporttree=yes remotes
This avoids a lot of ugly messages when syncing with such a remote.
The export tree happens on the proxy side.
2024-08-07 13:00:19 -04:00
Joey Hess
6d96734128
updateproxy, updatecluster check annexobjects=yes
updateproxy, updatecluster: Prevent using an exporttree=yes special remote
that does not have annexobjects=yes, since it will not work.
2024-08-07 12:27:24 -04:00
Joey Hess
8864a9e353
update 2024-08-07 11:49:53 -04:00
Joey Hess
1e0f13ad7f
comment 2024-08-07 11:39:29 -04:00
Joey Hess
b8f8c38e88
Merge branch 'master' into exportreeplus 2024-08-07 11:28:21 -04:00
Joey Hess
509b23fa00
catch ClientError from withClientM
When getting from a P2P HTTP remote, prompt for credentials when required,
instead of failing.

This feels like it might be a bug in servant-client. withClientM's type
suggests it would not throw a ClientError. But it does in this case.
2024-08-07 11:24:34 -04:00
Joey Hess
43e1f590c9
comment 2024-08-07 10:47:47 -04:00
Joey Hess
1038567881
proxy stores received keys to known export locations
This handles the workflow where the branch is first pushed to the proxy,
and then files in the exported tree are later are copied to the proxied remote.

Turns out that the way the export log is structured, nothing needs
to be done to finalize the export once the last key is sent to it. Which
is great because that would have been a lot of complication. On
receiving the push, Command.Export runs and calls recordExportBeginning,
does as much as it can to update the export with the files currently
on it, and then calls recordExportUnderway. At that point, the
export.log records the export as "complete", but it's not really. And
that's fine. The same happens when using `git-annex export` when some
files are not available to send. Other repositories that have
access to the special remote can already retrieve files from it. As
the missing files get copied to the exported remote, all that needs
to be done is record each in the export db.

At this point, proxying to exporttree=yes annexobjects=yes special remotes
is fully working. Except for in the case where multiple files in the
tree use the same key, and the files are sent to the proxied remote
before pushing the tree.

It seems that even special remotes without annexobjects=yes will work if
used with the workflow where the git-annex branch is pushed before
copying files. But not with the `git-annex push` workflow.
2024-08-07 09:47:34 -04:00
matrss
3ccbcc5662 2024-08-07 12:12:29 +00:00