From 806c5dc937c58b88ce5fb49d72527feb48c3e2b4 Mon Sep 17 00:00:00 2001 From: "tastabirta@e5349d873c7906025d7db2cc5b86e2529798b640" Date: Tue, 21 Mar 2023 21:28:21 +0000 Subject: [PATCH 1/6] --- .../gcrypt_remotes_using_relative_paths.mdwn | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 doc/bugs/gcrypt_remotes_using_relative_paths.mdwn diff --git a/doc/bugs/gcrypt_remotes_using_relative_paths.mdwn b/doc/bugs/gcrypt_remotes_using_relative_paths.mdwn new file mode 100644 index 0000000000..68edcc74da --- /dev/null +++ b/doc/bugs/gcrypt_remotes_using_relative_paths.mdwn @@ -0,0 +1,38 @@ +### Please describe the problem. +`git annex sync` is not automatically run for gcrypt remotes using rsync with a relative path + +### What steps will reproduce the problem? +Flow 1 (relative path, broken) + +* `git remote add test gcrypt::rsync://user@user.rsync.net:relative/path/to/repo` +* `git annex sync` -> DOES NOT SYNC to test remote +* Nothing has been synced so I CANNOT successfully clone from the test remote with `git clone gcrypt::rsync://user@user.rsync.net:relative/path/to/repo` +* `git push test git-annex master` +* I can successfully clone from the test remote with `git clone gcrypt::rsync://user@user.rsync.net:relative/path/to/repo` + +Flow 2 (absolute path, working) + +* `git remote add test gcrypt::rsync://user@user.rsync.net/full/path/to/repo` +* `git annex sync` -> DOES SYNC to test remote +* I can successfully clone from the test remote with `git clone gcrypt::rsync://user@user.rsync.net:relative/path/to/repo` + + +### What version of git-annex are you using? On what operating system? +* Debian 11 +* git-annex version: 10.20230227 +* git-remote-gcrypt version 1.5 + + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) +I am VERY happy with git annex and am using it successfully with a gcrypt remote using an absolute path :) + From d29ef26a6878549d966c39fb9353b608b093e6ca Mon Sep 17 00:00:00 2001 From: "carlos@4c213b52601d57b650b22d9a246c59aea2c8f859" Date: Tue, 21 Mar 2023 21:34:01 +0000 Subject: [PATCH 2/6] Added a comment --- ..._2_539dbce8852a9b6ae20ff0e55e74fc2a._comment | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/forum/git_annex_copy_to_google_drive_is_very_slow/comment_2_539dbce8852a9b6ae20ff0e55e74fc2a._comment diff --git a/doc/forum/git_annex_copy_to_google_drive_is_very_slow/comment_2_539dbce8852a9b6ae20ff0e55e74fc2a._comment b/doc/forum/git_annex_copy_to_google_drive_is_very_slow/comment_2_539dbce8852a9b6ae20ff0e55e74fc2a._comment new file mode 100644 index 0000000000..c87249890a --- /dev/null +++ b/doc/forum/git_annex_copy_to_google_drive_is_very_slow/comment_2_539dbce8852a9b6ae20ff0e55e74fc2a._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="carlos@4c213b52601d57b650b22d9a246c59aea2c8f859" + nickname="carlos" + avatar="http://cdn.libravatar.org/avatar/ef0152bec3818b24d9318f6e7013e104" + subject="comment 2" + date="2023-03-21T21:34:01Z" + content=""" +Hi Joey, + +This is my current effort to copy ~6.7 GB using Datalad (git annex with rclone+Gdrive in the background) + +``` +Total: 53%|█████████████████████████████████████████████████████████████████▏ | 3.49G/6.53G [8:25:38<7:21:03, 115k Bytes/s] +``` + +There are many small files in this repository for sure, but I haven't been able to get specific advice as to whether these very small speeds (even copying to an external drive took many hours) are expected given the bottlenecks you mention. Any feedback is welcome. +"""]] From a40a36495fff6f5472367544268843bf26b9ecf5 Mon Sep 17 00:00:00 2001 From: hurlebouc Date: Wed, 22 Mar 2023 05:47:16 +0000 Subject: [PATCH 3/6] Added a comment --- .../comment_8_1501223d25a03c27a7ceffc6b0ea32a3._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/bugs/SQlite_failed_when_copying_to_remote_repository/comment_8_1501223d25a03c27a7ceffc6b0ea32a3._comment diff --git a/doc/bugs/SQlite_failed_when_copying_to_remote_repository/comment_8_1501223d25a03c27a7ceffc6b0ea32a3._comment b/doc/bugs/SQlite_failed_when_copying_to_remote_repository/comment_8_1501223d25a03c27a7ceffc6b0ea32a3._comment new file mode 100644 index 0000000000..c9fb1de251 --- /dev/null +++ b/doc/bugs/SQlite_failed_when_copying_to_remote_repository/comment_8_1501223d25a03c27a7ceffc6b0ea32a3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="hurlebouc" + avatar="http://cdn.libravatar.org/avatar/bda734a6d937c1fe0c9778a6eaefffbc" + subject="comment 8" + date="2023-03-22T05:47:16Z" + content=""" +Thank you very much for the analysis! This reinforces my conviction that git-annex is a very high quality project. +"""]] From fc581d0c20caa5382a02d5ce61ac09393edd1d2b Mon Sep 17 00:00:00 2001 From: jg123h12jh3y12g3y Date: Wed, 22 Mar 2023 09:47:31 +0000 Subject: [PATCH 4/6] --- ...arifications_about_how_to_work_with_git-annex.mdwn | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 doc/forum/Clarifications_about_how_to_work_with_git-annex.mdwn diff --git a/doc/forum/Clarifications_about_how_to_work_with_git-annex.mdwn b/doc/forum/Clarifications_about_how_to_work_with_git-annex.mdwn new file mode 100644 index 0000000000..9f8943f00c --- /dev/null +++ b/doc/forum/Clarifications_about_how_to_work_with_git-annex.mdwn @@ -0,0 +1,11 @@ +Hello. I am a kinda-new git-annex user. I think git-annex is very difficult to understand. Thus, in this topic, I would like to write out my explanation of how git-annex works and how I can work with it, so that you can tell me if I am misunderstanding something. + +I am a machine learning engineer. I use git-annex to store code together data (large images) and artefacts (large images and model weights). Basically, I am trying to use git-annex as normal git, but with the ability to store large binary files. There is a central bare repository and a bunch of non-bare repositories cloned from it on different computers. Usually, the non-bare repositories connect only to the central bare repository, but sometimes I push/fetch between non-bare repositories directly. + +I can fetch/push/merge/pull the normal branches I work in just as I do with normal git (without annex). However, the git-annexed files aren't fetched/pushed this way because the branch actually contains only symlinks, not the files themselves. I have disabled all the automatic mergin, pulling, etc. functionality of git because I like to have total control. So, I merge or rebase everything manually when there is a need for it. And I never use pull. + +In each repository, there is a branch called git-annex. It contains some metadata that git-annex uses. While in a repository repo1, I can do `git annex sync --only-annex --no-content --no-commit --no-pull --no-push --no-resolvemerge repo2` and git-annex will use some magic (consisting of pushing, pulling, and some kind of automatic merging) to sync the git-annex branch of repo1 and repo2, i.e. it will make them contain the same metadata. The options `--no-commit --no-pull --no-push --no-resolvemerge` are needed to disable the dark magic that is useful for casual users but not for software developers who use git-annex as a git addon. The option `--only-annex` prevents git-annex from creating "synced" branches which are, as far as I understand, another piece of dark magic useful for casual users but not for software developers. If I want, I can remove the `--no-content` flag and git-annex will also download and/or upload the annexed data (does it affect only the data available in the current branch? or is it all data? i'm not sure). This is the only command I need to know to sync the git-annex branch. Supposedly, it's possible to do via normal git fetches, pushes, merges, and maybe pulls, but I don't know how to do that. + +The actual annexed data is stored somewhere in the .git directory. I don't need to worry where. What I need to know is that I can use `git annex copy`, `git annex get`, and `git annex sync --only-annex --no-commit --no-pull --no-push --no-resolvemerge otherrepo` with appropriate paths to copy the annexed data between repositories. + +Ok, so I can use `git annex sync` with a bunch of flags to sync the git-annex branch, I can use `git annex sync`, `git annex get`, `git annex copy` to copy the data around, and I don't need the synced branches. Is my understanding correct? From 4a2dfc4893eff4e3cfd44a0f6092eaa629e84193 Mon Sep 17 00:00:00 2001 From: "gioele@678b7c03f524f2669b179b603f65352fcc16774e" Date: Wed, 22 Mar 2023 10:06:26 +0000 Subject: [PATCH 5/6] --- ...ted_branch_update_after_error_in_drop.mdwn | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 doc/bugs/Failed_adjusted_branch_update_after_error_in_drop.mdwn diff --git a/doc/bugs/Failed_adjusted_branch_update_after_error_in_drop.mdwn b/doc/bugs/Failed_adjusted_branch_update_after_error_in_drop.mdwn new file mode 100644 index 0000000000..7e4f38e9cd --- /dev/null +++ b/doc/bugs/Failed_adjusted_branch_update_after_error_in_drop.mdwn @@ -0,0 +1,56 @@ +### Please describe the problem. + +git-annex failed to update the local adjusted branch after `git annex drop` stopped abruptly. In turn, `git annex drop` stopped abruptly because a `git annex copy --from X --to Y` had to be stopped (CTRL-C) because of lack of space. + +(Perhaps there are two issues here: `drop` being unable to cope with the unexpected situation, and the adjustment code being unable to cope with the weird stat left by `drop`'s abrupt interruption.) + +### What steps will reproduce the problem? + +``` +$ git copy -J4 --from X --to Y +[notice that git-annex starts saying "not enough free space, need 27.87 MB more"] +[kill process with CTRL-C] +$ git annex drop dirF/ +drop dirF/a/a1.txt ok +drop dirF/a/a2.txt ok +[...] +error: Your local changes to the following files would be overwritten by checkout: + dirF/a/a1.txt + dirF/a/a2.txt + [...] + +Aborting + + Updating adjusted branch failed. +(recording state in git...) +$ git status +HEAD detached at 9d92415fb +nothing to commit, working tree clean +$ git annex status +$ git branch +* (HEAD detached at 9d92415fb) + adjusted/master(unlocked) + adjusted/master(unlockpresent) + git-annex + master + synced/master +``` + +`dirF/` is the directory whose files were being copied when the process has been stopped with `CTRL-C`. + +### What version of git-annex are you using? On what operating system? + +``` +git-annex version: 10.20230215-gd24914f2a +[...] +operating system: linux x86_64 +supported repository versions: 8 9 10 +upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10 +local repository version: 8 +``` + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +git-annex is too good. It so rarely causes problems that one does not develop the "git-annex troubleshooting muscle". :) + + From 2888a724f97c63cd92b76b0a5785a5dd0324fdcf Mon Sep 17 00:00:00 2001 From: "gioele@678b7c03f524f2669b179b603f65352fcc16774e" Date: Wed, 22 Mar 2023 10:13:04 +0000 Subject: [PATCH 6/6] --- ...over_from_failed_branch_updates__63__.mdwn | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 doc/forum/How_to_recover_from_failed_branch_updates__63__.mdwn diff --git a/doc/forum/How_to_recover_from_failed_branch_updates__63__.mdwn b/doc/forum/How_to_recover_from_failed_branch_updates__63__.mdwn new file mode 100644 index 0000000000..6a227a99d3 --- /dev/null +++ b/doc/forum/How_to_recover_from_failed_branch_updates__63__.mdwn @@ -0,0 +1,28 @@ +Due to the issue reported in [[bugs/Failed_adjusted_branch_update_after_error_in_drop]], my git-annex repo is now is a limbo state: + + +``` +$ git branch --show-current +adjusted/master(unlockpresent) + +$ git annex drop dirF/ +drop dirF/a/a1.txt ok +drop dirF/a/a2.txt ok +[...] +error: Your local changes to the following files would be overwritten by checkout: + dirF/a/a1.txt + dirF/a/a2.txt + [...] + +Aborting + + Updating adjusted branch failed. +(recording state in git...) +$ git status +HEAD detached at 9d92415fb +nothing to commit, working tree clean +``` + +What is the recommended course of action to (safely) return to the `adjusted/master(unlockpresent)` branch? + +Is it safe to just use `git annex adjust --unlock-present`?