From 5f1aa512725be3ce3af646b41f7be041eca30191 Mon Sep 17 00:00:00 2001 From: aaron Date: Mon, 25 Nov 2024 00:55:07 +0000 Subject: [PATCH 1/9] --- ...sting_of_files_across_disconnected_devices.mdwn | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/forum/Requesting_of_files_across_disconnected_devices.mdwn diff --git a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn new file mode 100644 index 0000000000..aaed669834 --- /dev/null +++ b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn @@ -0,0 +1,14 @@ +Has anyone done something where devices may not have a direct connection to other git-annex devices that allows them to post file requests that other devices then pickup and relay so that a shared endpoint can be populated with the requested file? I'm currently thinking of something where a cloud node might be 'expensive' and purges itself once some other devices contain the file; but at a later point in time a mobile device may only be able to connect to the cloud node, but wants to request the file. It could populate a 'git-annex-requests' file with contents similar to the following: +```text +file-shasum, requester-id, (optional endpoint1), (optional endpoint2),... +``` +This 'git-annex-requests' file would require a minimum of the file-shasum and requester id, with the endpoints helping other devices (containing the desired file) to know where to best push the file (other than guessing/all available remotes). So, for the attached diagram, where the mobile laptop attached to the cellphone wants a file from the remote-office nas/server, a flow would look like: laptop updates request file -> syncs file to phone -> syncs file to homelab server -> allows home office computer to sync file -> syncs file to home-office nas/server; the home-office computer gets the file from the nas and pushes it to the homelab server -> the mobile phone downloads the file from the homelab server -> the mobile laptop gets the file from the phone and removes the request from the requests file (which then triggers the reverse propagation of the acknowledgement/removal of the request and allows the devices to proceed with any garbage cleanup). + +Additionally, a 'git-annex-routing' file could optionally be added that includes netlist details describing routing chains where certain 'static' devices may be able to easily push to each other so that other git-annex clients can make more informed decisions on where to push a file. + +[[!img git_annex_request.png align="right" size="" alt="Network diagram"]] +[[!img git_annex_request_routing.png align="right" size="" alt="Network diagram showing file routing"]] + +Alternate links for images: +Example diagram with mixed network connections: https://imgur.com/gallery/network-diagram-with-mixed-inbound-outbound-connections-3q76OzI +Example diagram with network request across mixed network connections: https://imgur.com/gallery/network-diagram-file-request-across-mixed-inbound-outbound-connections-hev94Kj From 59c555584dbd24da131561628f04269293f334e4 Mon Sep 17 00:00:00 2001 From: aaron Date: Mon, 25 Nov 2024 00:56:17 +0000 Subject: [PATCH 2/9] --- doc/forum/Requesting_of_files_across_disconnected_devices.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn index aaed669834..322efcca52 100644 --- a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn +++ b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn @@ -10,5 +10,5 @@ Additionally, a 'git-annex-routing' file could optionally be added that includes [[!img git_annex_request_routing.png align="right" size="" alt="Network diagram showing file routing"]] Alternate links for images: -Example diagram with mixed network connections: https://imgur.com/gallery/network-diagram-with-mixed-inbound-outbound-connections-3q76OzI +Example diagram with mixed network connections: https://imgur.com/gallery/network-diagram-with-mixed-inbound-outbound-connections-3q76OzI Example diagram with network request across mixed network connections: https://imgur.com/gallery/network-diagram-file-request-across-mixed-inbound-outbound-connections-hev94Kj From 9934d210eaaf06e8c9d04613b19cc16bc38f33dc Mon Sep 17 00:00:00 2001 From: aaron Date: Mon, 25 Nov 2024 00:56:41 +0000 Subject: [PATCH 3/9] --- doc/forum/Requesting_of_files_across_disconnected_devices.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn index 322efcca52..f136ed2286 100644 --- a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn +++ b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn @@ -6,7 +6,7 @@ This 'git-annex-requests' file would require a minimum of the file-shasum and re Additionally, a 'git-annex-routing' file could optionally be added that includes netlist details describing routing chains where certain 'static' devices may be able to easily push to each other so that other git-annex clients can make more informed decisions on where to push a file. -[[!img git_annex_request.png align="right" size="" alt="Network diagram"]] +[[!img git_annex_request.png align="right" size="" alt="Network diagram"]] [[!img git_annex_request_routing.png align="right" size="" alt="Network diagram showing file routing"]] Alternate links for images: From f4293618310baf21144e966e14625416f116f6cb Mon Sep 17 00:00:00 2001 From: aaron Date: Mon, 25 Nov 2024 01:02:53 +0000 Subject: [PATCH 4/9] --- ...sting_of_files_across_disconnected_devices.mdwn | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn index f136ed2286..226fc53c07 100644 --- a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn +++ b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn @@ -1,14 +1,18 @@ -Has anyone done something where devices may not have a direct connection to other git-annex devices that allows them to post file requests that other devices then pickup and relay so that a shared endpoint can be populated with the requested file? I'm currently thinking of something where a cloud node might be 'expensive' and purges itself once some other devices contain the file; but at a later point in time a mobile device may only be able to connect to the cloud node, but wants to request the file. It could populate a 'git-annex-requests' file with contents similar to the following: +Has anyone done something where devices may not have a direct connection to other git-annex devices that allows them to post file requests that other devices then pickup and relay so that a shared endpoint can be populated with the requested file? I'm currently thinking of something where a cloud node might be 'expensive' and purges itself once some other devices contain the file; but at a later point in time a mobile device may only be able to connect to the cloud node, but wants to request the file. + +## Possible behavior +The requestor could populate a 'git-annex-requests' file at the root of the repositor with contents similar to the following: ```text file-shasum, requester-id, (optional endpoint1), (optional endpoint2),... ``` + This 'git-annex-requests' file would require a minimum of the file-shasum and requester id, with the endpoints helping other devices (containing the desired file) to know where to best push the file (other than guessing/all available remotes). So, for the attached diagram, where the mobile laptop attached to the cellphone wants a file from the remote-office nas/server, a flow would look like: laptop updates request file -> syncs file to phone -> syncs file to homelab server -> allows home office computer to sync file -> syncs file to home-office nas/server; the home-office computer gets the file from the nas and pushes it to the homelab server -> the mobile phone downloads the file from the homelab server -> the mobile laptop gets the file from the phone and removes the request from the requests file (which then triggers the reverse propagation of the acknowledgement/removal of the request and allows the devices to proceed with any garbage cleanup). Additionally, a 'git-annex-routing' file could optionally be added that includes netlist details describing routing chains where certain 'static' devices may be able to easily push to each other so that other git-annex clients can make more informed decisions on where to push a file. [[!img git_annex_request.png align="right" size="" alt="Network diagram"]] -[[!img git_annex_request_routing.png align="right" size="" alt="Network diagram showing file routing"]] +[[!img git_annex_request_routing.png align="right" size="" alt="Network diagram showing file routing"]] -Alternate links for images: -Example diagram with mixed network connections: https://imgur.com/gallery/network-diagram-with-mixed-inbound-outbound-connections-3q76OzI -Example diagram with network request across mixed network connections: https://imgur.com/gallery/network-diagram-file-request-across-mixed-inbound-outbound-connections-hev94Kj +## Alternate links for images +Example diagram with mixed network connections: [[https://imgur.com/gallery/network-diagram-with-mixed-inbound-outbound-connections-3q76OzI]] +Example diagram with network request across mixed network connections: [[https://imgur.com/gallery/network-diagram-file-request-across-mixed-inbound-outbound-connections-hev94Kj]] From 9a96925e29f4368063f0ecae790f5ab2f463f618 Mon Sep 17 00:00:00 2001 From: aaron Date: Mon, 25 Nov 2024 01:05:32 +0000 Subject: [PATCH 5/9] --- doc/forum/Requesting_of_files_across_disconnected_devices.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn index 226fc53c07..7b043ca033 100644 --- a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn +++ b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn @@ -1,4 +1,4 @@ -Has anyone done something where devices may not have a direct connection to other git-annex devices that allows them to post file requests that other devices then pickup and relay so that a shared endpoint can be populated with the requested file? I'm currently thinking of something where a cloud node might be 'expensive' and purges itself once some other devices contain the file; but at a later point in time a mobile device may only be able to connect to the cloud node, but wants to request the file. +Has anyone done something where devices may not have a direct connection to other git-annex devices, but where they can push out a request for a file? Basically, something that allows them to post file requests that other devices then pickup and relay to a shared endpoint that can be populated with the requested file? I'm currently thinking of a situation where a cloud node might be 'expensive' and purges itself once some other devices contain the file; but at a later point in time a mobile device (without the file) may only be able to connect to the cloud node and request the file. ## Possible behavior The requestor could populate a 'git-annex-requests' file at the root of the repositor with contents similar to the following: From 0e04031208ffb28c178a13643feb0ff74adae203 Mon Sep 17 00:00:00 2001 From: aaron Date: Mon, 25 Nov 2024 01:06:54 +0000 Subject: [PATCH 6/9] --- doc/forum/Requesting_of_files_across_disconnected_devices.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn index 7b043ca033..a37dbe2593 100644 --- a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn +++ b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn @@ -1,4 +1,4 @@ -Has anyone done something where devices may not have a direct connection to other git-annex devices, but where they can push out a request for a file? Basically, something that allows them to post file requests that other devices then pickup and relay to a shared endpoint that can be populated with the requested file? I'm currently thinking of a situation where a cloud node might be 'expensive' and purges itself once some other devices contain the file; but at a later point in time a mobile device (without the file) may only be able to connect to the cloud node and request the file. +Has anyone done something where devices may not have a direct connection to other git-annex devices, but where they can push out a request for a file? Basically, something that allows them to post file requests that other devices then pickup and relay to a shared endpoint that can be populated with the requested file? I'm currently thinking of a situation where a cloud node might be 'expensive' and purges itself once some other devices contain the file; but at a later point in time a mobile device (without the file or a direct connection to devices containing the file) may connect to the cloud node and request the file. ## Possible behavior The requestor could populate a 'git-annex-requests' file at the root of the repositor with contents similar to the following: From edacdcc30dafe79bab1eb007867d35187a945543 Mon Sep 17 00:00:00 2001 From: aaron Date: Mon, 25 Nov 2024 01:07:20 +0000 Subject: [PATCH 7/9] --- doc/forum/Requesting_of_files_across_disconnected_devices.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn index a37dbe2593..338ad68162 100644 --- a/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn +++ b/doc/forum/Requesting_of_files_across_disconnected_devices.mdwn @@ -1,7 +1,7 @@ Has anyone done something where devices may not have a direct connection to other git-annex devices, but where they can push out a request for a file? Basically, something that allows them to post file requests that other devices then pickup and relay to a shared endpoint that can be populated with the requested file? I'm currently thinking of a situation where a cloud node might be 'expensive' and purges itself once some other devices contain the file; but at a later point in time a mobile device (without the file or a direct connection to devices containing the file) may connect to the cloud node and request the file. ## Possible behavior -The requestor could populate a 'git-annex-requests' file at the root of the repositor with contents similar to the following: +The requestor could populate a 'git-annex-requests' file at the root of the repository with contents similar to the following: ```text file-shasum, requester-id, (optional endpoint1), (optional endpoint2),... ``` From 6514040b0abf71e530e5354000d1b04bf80645f6 Mon Sep 17 00:00:00 2001 From: kyle Date: Mon, 25 Nov 2024 02:23:25 +0000 Subject: [PATCH 8/9] Added a comment: re: How to get a list of all NOT unused files --- ..._6dfcbe06c7ea184e9a8ff3137584b07f._comment | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 doc/forum/How_to_get_a_list_of_all_NOT_unused_files/comment_1_6dfcbe06c7ea184e9a8ff3137584b07f._comment diff --git a/doc/forum/How_to_get_a_list_of_all_NOT_unused_files/comment_1_6dfcbe06c7ea184e9a8ff3137584b07f._comment b/doc/forum/How_to_get_a_list_of_all_NOT_unused_files/comment_1_6dfcbe06c7ea184e9a8ff3137584b07f._comment new file mode 100644 index 0000000000..14cbe7a44b --- /dev/null +++ b/doc/forum/How_to_get_a_list_of_all_NOT_unused_files/comment_1_6dfcbe06c7ea184e9a8ff3137584b07f._comment @@ -0,0 +1,50 @@ +[[!comment format=mdwn + username="kyle" + avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" + subject="re: How to get a list of all NOT unused files" + date="2024-11-25T02:23:25Z" + content=""" +> How to get a list of all NOT unused files + +There may be a simpler way, but one idea: + + * list all unused keys + * list all present keys + * filter out the unused keys from the present keys + +So something like this: + +``` +$ git annex findkeys | sort >present-keys +$ git annex unused --json | jq -r '.\"unused-list\" | to_entries | map(.value) | .[]' | sort >unused-keys +$ comm -2 -3 present-keys unused-keys +``` + +> Those that should be saved are tagged + +If you wanted to focus just on keys referenced from tags, you could +generate a list of those keys with + +``` +$ git rev-list --objects --no-object-names --no-walk --tags | \ + git annex lookupkey --ref --batch | grep -v '^$' +``` + +--- + +After generating a list of keys with either of those approaches, you +could copy them to your new repo with + +``` +git annex copy --to=NEW-REMOTE --batch-keys ... +``` + +For example, the full pipeline for the second approach could be + +``` +$ git rev-list --objects --no-object-names --no-walk --tags | \ + git annex lookupkey --ref --batch | grep -v '^$' | \ + git annex copy --to=NEW-REMOTE --batch-keys +``` + +"""]] From 372852875f0ccde21d28186d33512df476d98d01 Mon Sep 17 00:00:00 2001 From: aaron Date: Mon, 25 Nov 2024 02:55:25 +0000 Subject: [PATCH 9/9] Added a comment: Overriding git folder --- ...t_5_b1c2d9fa5167e01a7dc95c8fd91709a2._comment | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/tips/using_nested_git_repositories/comment_5_b1c2d9fa5167e01a7dc95c8fd91709a2._comment diff --git a/doc/tips/using_nested_git_repositories/comment_5_b1c2d9fa5167e01a7dc95c8fd91709a2._comment b/doc/tips/using_nested_git_repositories/comment_5_b1c2d9fa5167e01a7dc95c8fd91709a2._comment new file mode 100644 index 0000000000..1597dfa32c --- /dev/null +++ b/doc/tips/using_nested_git_repositories/comment_5_b1c2d9fa5167e01a7dc95c8fd91709a2._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="aaron" + avatar="http://cdn.libravatar.org/avatar/8a07e2f7af4bbf1bfcb48bbc53e00747" + subject="Overriding git folder" + date="2024-11-25T02:55:25Z" + content=""" +It seems to be that git has gotten smarter and now actively prevents you from adding a `.git` folder (I did this many years ago when before I learned about submodules); I'd like to do something like the following: +```bash +git init --separate-git-dir=.gitannex . +git --git-dir=.gitannex annex init +git clone some_repo # A repo I'm pulling from GitHub/wherever and don't want a submodule of as it's not my personal project +git --git-dir=.gitannex add some_repo +``` + +Essentially, I can override that `.git` folder name, but it still checks for other `.git` folders; is there a way to remove this check? +"""]]