From 50b200b5e0112a73d6e7c11f2a913b7fb7bc1440 Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 18:51:50 +0000 Subject: [PATCH 01/25] --- doc/forum/how_to_get_into_git_annex....mdwn | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/forum/how_to_get_into_git_annex....mdwn b/doc/forum/how_to_get_into_git_annex....mdwn index 290428900c..84660c6aa7 100644 --- a/doc/forum/how_to_get_into_git_annex....mdwn +++ b/doc/forum/how_to_get_into_git_annex....mdwn @@ -1,9 +1,9 @@ So... I've been flirting with using git annex for literal years now, and if for some reason you are wanting to use it too here are some tips: -1) keep backups. seriously. just do it. it's possible to lose data, even though git annex is designed to avoid eating your data it will do it under certain circumstances. you aren't lucky enough to avoid it. trust me. -2) make a big fat git annex with too many files in it, and kick the tires, hard. run all the commands and try to break it, see what it does under certain circumstances before you run those same commands on your beloved data. (the documentation isn't always up to date, sometimes the options (which are complex) operate differently than the website says and differently than you expect, this is most likely due to code changes that haven't propagated to the website. -3) git annex bogs down fast when you are dealing with a large number of objects, there are ways to get that under control, but nothing is going to make managing an annex with millions of files "fast" for many operations. -4) now that you are a pro at git annex, STILL *keep* backups. git annex isn't a backup. it just isn't. nothing beats a simple usb hard drive stuffed in your safe with all your files on it and without the complexity that is git annex in the way. +* keep backups. seriously. just do it. it's possible to lose data, even though git annex is designed to avoid eating your data it will do it under certain circumstances. you aren't lucky enough to avoid it. trust me. +* make a big fat git annex with too many files in it, and kick the tires, hard. run all the commands and try to break it, see what it does under certain circumstances before you run those same commands on your beloved data. (the documentation isn't always up to date, sometimes the options (which are complex) operate differently than the website says and differently than you expect, this is most likely due to code changes that haven't propagated to the website. +* git annex bogs down fast when you are dealing with a large number of objects, there are ways to get that under control, but nothing is going to make managing an annex with millions of files "fast" for many operations. +* now that you are a pro at git annex, STILL *keep* backups. git annex isn't a backup. it just isn't. nothing beats a simple usb hard drive stuffed in your safe with all your files on it and without the complexity that is git annex in the way. From 935f8991cb9f5c6fa95a74ee553cffd99d0b5fc0 Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 18:55:30 +0000 Subject: [PATCH 02/25] Added a comment --- ...ent_2_cde005ffa911f72e6d37ec8ad9bf76f8._comment | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_2_cde005ffa911f72e6d37ec8ad9bf76f8._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_2_cde005ffa911f72e6d37ec8ad9bf76f8._comment b/doc/forum/how_to_get_into_git_annex.../comment_2_cde005ffa911f72e6d37ec8ad9bf76f8._comment new file mode 100644 index 0000000000..89470ab024 --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_2_cde005ffa911f72e6d37ec8ad9bf76f8._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" + nickname="eric.w" + avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" + subject="comment 2" + date="2020-12-31T18:55:29Z" + content=""" +the most recent example I've run across is the use of +git.annex=thin +in the link here: https://git-annex.branchable.com/tips/unlocked_files/ +it didn't result in a hardlink being made of the content for either git annex unlock or git annex unannex +instead I ended up getting the same functionality by use --fast. + +"""]] From 861b3b3088c1ba97d471298409779efe14351d13 Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 19:34:56 +0000 Subject: [PATCH 03/25] Added a comment --- .../comment_3_20d36c31731303291e4e43af1241692f._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_3_20d36c31731303291e4e43af1241692f._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_3_20d36c31731303291e4e43af1241692f._comment b/doc/forum/how_to_get_into_git_annex.../comment_3_20d36c31731303291e4e43af1241692f._comment new file mode 100644 index 0000000000..711ba5947b --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_3_20d36c31731303291e4e43af1241692f._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" + nickname="eric.w" + avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" + subject="comment 3" + date="2020-12-31T19:34:56Z" + content=""" +right now I am driving myself crazy trying to understand why I have objects that *nothing is pointing to*, yet git annex unused fails to report them. these objects report 1 hardlink and they are from a migrated backend. I'll try git annex forget, but I really don't understand what is keeping these objects from being reported as unused. +"""]] From 82b4916c311e6e91dc7d04f585f23ac891e434b8 Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 19:40:01 +0000 Subject: [PATCH 04/25] Added a comment --- ...nt_4_9a96e13da383c9b61145dd7ec16421c9._comment | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_4_9a96e13da383c9b61145dd7ec16421c9._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_4_9a96e13da383c9b61145dd7ec16421c9._comment b/doc/forum/how_to_get_into_git_annex.../comment_4_9a96e13da383c9b61145dd7ec16421c9._comment new file mode 100644 index 0000000000..ea8f2a0367 --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_4_9a96e13da383c9b61145dd7ec16421c9._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" + nickname="eric.w" + avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" + subject="comment 4" + date="2020-12-31T19:40:00Z" + content=""" +https://git-annex.branchable.com/bugs/migrated_files_not_showing_up_in_unused_list/ + +according to the link above it should be hardlinked to the new key for the new backend, but this isn't the case. this is on btrfs btw. +this is a test repo with no remotes as another data point. +also I migrated from SHA256E to SHA256. + +I tried git annex forget; git annex sync; git annex unused, still it isn't showing the objects as unused. +"""]] From 3643705e2e1262e5587384c2de01f988edc41692 Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 19:40:22 +0000 Subject: [PATCH 05/25] Added a comment --- ...nt_5_6ed2e755fb7f738d75468f1c7c4d5555._comment | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_5_6ed2e755fb7f738d75468f1c7c4d5555._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_5_6ed2e755fb7f738d75468f1c7c4d5555._comment b/doc/forum/how_to_get_into_git_annex.../comment_5_6ed2e755fb7f738d75468f1c7c4d5555._comment new file mode 100644 index 0000000000..4f14ac711c --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_5_6ed2e755fb7f738d75468f1c7c4d5555._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" + nickname="eric.w" + avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" + subject="comment 5" + date="2020-12-31T19:40:22Z" + content=""" +https://git-annex.branchable.com/bugs/migrated_files_not_showing_up_in_unused_list/ + +according to the link above it should be hardlinked to the new key for the new backend, but this isn't the case. this is on btrfs btw. +this is a test repo with no remotes as another data point. +also I migrated from SHA256E to SHA256. + +I tried git annex forget --force; git annex sync; git annex unused, still it isn't showing the objects as unused. +"""]] From 073cdd62276f108edeafd18dd00ad537dabd4c7a Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 19:41:18 +0000 Subject: [PATCH 06/25] removed --- ...nt_4_9a96e13da383c9b61145dd7ec16421c9._comment | 15 --------------- 1 file changed, 15 deletions(-) delete mode 100644 doc/forum/how_to_get_into_git_annex.../comment_4_9a96e13da383c9b61145dd7ec16421c9._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_4_9a96e13da383c9b61145dd7ec16421c9._comment b/doc/forum/how_to_get_into_git_annex.../comment_4_9a96e13da383c9b61145dd7ec16421c9._comment deleted file mode 100644 index ea8f2a0367..0000000000 --- a/doc/forum/how_to_get_into_git_annex.../comment_4_9a96e13da383c9b61145dd7ec16421c9._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" - nickname="eric.w" - avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" - subject="comment 4" - date="2020-12-31T19:40:00Z" - content=""" -https://git-annex.branchable.com/bugs/migrated_files_not_showing_up_in_unused_list/ - -according to the link above it should be hardlinked to the new key for the new backend, but this isn't the case. this is on btrfs btw. -this is a test repo with no remotes as another data point. -also I migrated from SHA256E to SHA256. - -I tried git annex forget; git annex sync; git annex unused, still it isn't showing the objects as unused. -"""]] From 6e467a38072a23a1b445fa810de40e8273c720df Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 20:34:55 +0000 Subject: [PATCH 07/25] Added a comment --- .../comment_5_05f1010f442de6e6e34c1f8ea5c250c7._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_5_05f1010f442de6e6e34c1f8ea5c250c7._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_5_05f1010f442de6e6e34c1f8ea5c250c7._comment b/doc/forum/how_to_get_into_git_annex.../comment_5_05f1010f442de6e6e34c1f8ea5c250c7._comment new file mode 100644 index 0000000000..53e42fa5e2 --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_5_05f1010f442de6e6e34c1f8ea5c250c7._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" + nickname="eric.w" + avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" + subject="comment 5" + date="2020-12-31T20:34:55Z" + content=""" +I am digging into this further, and it looks like git annex uses cp --reflog=auto, confirmed with filefrag -v, but even if the object from the old backend isn't taking up space, its still frustrating that I can't figure out why git annex is keeping old files around and not reporting them via git annex unused. +"""]] From e82bbb09a2f85b8beecff9f06684f3aed1e0538d Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 20:51:12 +0000 Subject: [PATCH 08/25] Added a comment --- ..._142a83bd4f5b1729635bf891f75bd94b._comment | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_6_142a83bd4f5b1729635bf891f75bd94b._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_6_142a83bd4f5b1729635bf891f75bd94b._comment b/doc/forum/how_to_get_into_git_annex.../comment_6_142a83bd4f5b1729635bf891f75bd94b._comment new file mode 100644 index 0000000000..57417bb289 --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_6_142a83bd4f5b1729635bf891f75bd94b._comment @@ -0,0 +1,21 @@ +[[!comment format=mdwn + username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" + nickname="eric.w" + avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" + subject="comment 6" + date="2020-12-31T20:51:12Z" + content=""" +after (re)reading the following: + +https://git-annex.branchable.com/forum/switching_backends/ + +https://git-annex.branchable.com/bugs/migrated_files_not_showing_up_in_unused_list/ + +I confirmed again that git annex sync was re-ran, there are no remotes, so that isn't a thing here. I checked out each git branch and did a + +```find ./???/ -lname '*c0ade___this_is_a_long_hash___566fd3*'``` + +and nothing in any branch is pointed to this old backend key. + +so I am both stymied and befuddled... any tips are appreciated. +"""]] From 89ac4e68bb8211f273db582e23775ccbcfe076e9 Mon Sep 17 00:00:00 2001 From: Lukey Date: Thu, 31 Dec 2020 21:46:33 +0000 Subject: [PATCH 09/25] Added a comment --- ...mment_7_96147f452229d09d4fa8ed4d606341aa._comment | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_7_96147f452229d09d4fa8ed4d606341aa._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_7_96147f452229d09d4fa8ed4d606341aa._comment b/doc/forum/how_to_get_into_git_annex.../comment_7_96147f452229d09d4fa8ed4d606341aa._comment new file mode 100644 index 0000000000..f266af8de6 --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_7_96147f452229d09d4fa8ed4d606341aa._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 7" + date="2020-12-31T21:46:33Z" + content=""" +Hmm, you seem to have mixed a lot of things up here:
+1. You are not supposed to use `git annex unannex` to unlock a file. Just pretend this command doesn't exist for now and use `git annex unlock` instead. In general, look at the manpages of the commands. For example `man git-annex-unannex`.
+2. Before doing anything further, clean up your repository from the mistake above. First, add all unannexed files back to the annex with `git annex add .` (from the root of your repo) and then commit everything with `git annex sync`. `git status` should now output `nothing to commit, working tree clean`.
+3. After setting `git config annex.thin true` you are supposed to run `git annex fix`. That's exactly what the link you gave says. But as you are using btrfs, I suggest you not to use hard-links, as git annex makes use of reflinks already.
+4. Now that you have a clean worktree, try `git annex unused` again. If it still doesn't work post the full output of `git annex unused` here. +"""]] From cc1e8897e62b1381654647e040dbf8382538bb80 Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 22:00:08 +0000 Subject: [PATCH 10/25] Added a comment --- ..._48bddaee71ad5f246199d66be12709d3._comment | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_8_48bddaee71ad5f246199d66be12709d3._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_8_48bddaee71ad5f246199d66be12709d3._comment b/doc/forum/how_to_get_into_git_annex.../comment_8_48bddaee71ad5f246199d66be12709d3._comment new file mode 100644 index 0000000000..8288e6379a --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_8_48bddaee71ad5f246199d66be12709d3._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" + nickname="eric.w" + avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" + subject="comment 8" + date="2020-12-31T22:00:07Z" + content=""" +thanks for responding... + +I used git annex unannex because I tried using git annex uninit and it DELETED my entire multi TB ./.git/annex/objects, even though I only had a handful of symlinks on in that repo, I wanted to find another way to unannex files that wouldn't delete my technically \"unused\" data. + +and git annex unannex was what I tried when git annex unlock would not hardlink the files via annex.thin=true. it was only with toying with the 2 commands and finally --fast that I was able to get it to hardlink the files + +my end goal was to be able to remove my data reliably from git annex entirely without it purging the object store. + +and now as I read about hardlinks=true or whatever I see that git annex doesn't really love to hardlink multiple files past 2 because then multiple, independent files being modified would corrupt the object store. + +I just want this thing to be reliable at scale. I put all my data into it but the speed is killing me, so I want to be able to get it out or split off data types to secondary git annexes, while having some idea of what it's doing under the covers so I don't get surprised. +"""]] From 31eef3d213e694aa4eabc04b7a0ea0e36d8b1be9 Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 22:05:19 +0000 Subject: [PATCH 11/25] Added a comment --- ...omment_9_9ccb674318f87a63849e654f789722cf._comment | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_9_9ccb674318f87a63849e654f789722cf._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_9_9ccb674318f87a63849e654f789722cf._comment b/doc/forum/how_to_get_into_git_annex.../comment_9_9ccb674318f87a63849e654f789722cf._comment new file mode 100644 index 0000000000..7c70ffb8a8 --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_9_9ccb674318f87a63849e654f789722cf._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" + nickname="eric.w" + avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" + subject="comment 9" + date="2020-12-31T22:05:19Z" + content=""" +I'll chew on the rest of your response, I was bent on hardlinks because I haven't messed with btrfs reflog COW thing much, but its likely clearly the way to go here, so all of my consternation with hardlinks is likely getting me nowhere. I am just always at 90% full and so I don't want to do anything that is going to run me out of space in the middle of an expensive operation. + +anyways, thanks. I guess I just wish I had only put big files into my annex at first, though I would never have known how badly it fails at scale (on my hardware, etc.) +"""]] From c7939d226eaa5474f538214488d501760eebdd47 Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Thu, 31 Dec 2020 22:45:03 +0000 Subject: [PATCH 12/25] Added a comment --- ...ment_10_42c2754ddb58ab47a442909f9be494a7._comment | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_10_42c2754ddb58ab47a442909f9be494a7._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_10_42c2754ddb58ab47a442909f9be494a7._comment b/doc/forum/how_to_get_into_git_annex.../comment_10_42c2754ddb58ab47a442909f9be494a7._comment new file mode 100644 index 0000000000..62a6be9174 --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_10_42c2754ddb58ab47a442909f9be494a7._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" + nickname="eric.w" + avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" + subject="comment 10" + date="2020-12-31T22:45:03Z" + content=""" +You will be unsurprised to hear that what you suggested worked. not sure what helped other than me cleaning up my working tree and doing a solid git annex add .; git annex sync. I also removed annex.thin since its evidently not helping me. +thanks a ton. what got me here was me basically running through the \"splitting a repo\" process of making a new git repo, doing a cp -rl ./.git/annex/objects to the new repo and then running various tests on it. I just want to make sure I don't step on my own feet here. + +thanks a ton. +"""]] From f972c5c132106f7cde1e2df00147dfac4d46acef Mon Sep 17 00:00:00 2001 From: "eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" Date: Fri, 1 Jan 2021 04:08:44 +0000 Subject: [PATCH 13/25] Added a comment --- ...ment_11_ad9858a76d5a66b2619cec3e3c17abd2._comment | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 doc/forum/how_to_get_into_git_annex.../comment_11_ad9858a76d5a66b2619cec3e3c17abd2._comment diff --git a/doc/forum/how_to_get_into_git_annex.../comment_11_ad9858a76d5a66b2619cec3e3c17abd2._comment b/doc/forum/how_to_get_into_git_annex.../comment_11_ad9858a76d5a66b2619cec3e3c17abd2._comment new file mode 100644 index 0000000000..f6c5870036 --- /dev/null +++ b/doc/forum/how_to_get_into_git_annex.../comment_11_ad9858a76d5a66b2619cec3e3c17abd2._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="eric.w@eee65cd362d995ced72640c7cfae388ae93a4234" + nickname="eric.w" + avatar="http://cdn.libravatar.org/avatar/8d9808c12db3a3f93ff7f9e74c0870fc" + subject="comment 11" + date="2021-01-01T04:08:43Z" + content=""" +couple of final notes: + +* ```--reflog=always``` isn't a cp option, its reflink, and I am a moron. +* that same options on btrfs is the bomb. all of the advantages of hardlinks without the disadvantages. +"""]] From 497d45d04c95d5a780e50d3395a140a58ba2067e Mon Sep 17 00:00:00 2001 From: Lukey Date: Fri, 1 Jan 2021 15:43:43 +0000 Subject: [PATCH 14/25] --- doc/bugs/git_annex_fsck_--time-limit_broken.mdwn | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 doc/bugs/git_annex_fsck_--time-limit_broken.mdwn diff --git a/doc/bugs/git_annex_fsck_--time-limit_broken.mdwn b/doc/bugs/git_annex_fsck_--time-limit_broken.mdwn new file mode 100644 index 0000000000..93c20bba0e --- /dev/null +++ b/doc/bugs/git_annex_fsck_--time-limit_broken.mdwn @@ -0,0 +1,13 @@ +### Please describe the problem. +`git annex fsck --time-limit=` is broken.
+For one, there is a large delay between the specified time limit until something actually happens. With 20 seconds, `git annex fsck` always runs more than 5 minutes. And then something of the following happens:
+Sometimes it works as intended.
+Sometimes it prints "Time limit (20s) reached!" but hangs without exiting.
+Sometimes it prints "Time limit (20s) reached!" but continues fscking.
+ +### What steps will reproduce the problem? +In a sufficiently large repo run `git annex fsck --time-limit=20s`. + +### What version of git-annex are you using? On what operating system? +8.20201127 + From c7bcc32e37c6eba260fbf86e695298600d418928 Mon Sep 17 00:00:00 2001 From: AlbertZeyer Date: Fri, 1 Jan 2021 21:25:44 +0000 Subject: [PATCH 15/25] Added a comment: annex.largefiles --- ...omment_1_0e6b855afb4fba540ea5560df26839c5._comment | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 doc/git-annex-add/comment_1_0e6b855afb4fba540ea5560df26839c5._comment diff --git a/doc/git-annex-add/comment_1_0e6b855afb4fba540ea5560df26839c5._comment b/doc/git-annex-add/comment_1_0e6b855afb4fba540ea5560df26839c5._comment new file mode 100644 index 0000000000..d233d96d15 --- /dev/null +++ b/doc/git-annex-add/comment_1_0e6b855afb4fba540ea5560df26839c5._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="AlbertZeyer" + avatar="http://cdn.libravatar.org/avatar/b37d71961a6a5abf9b7184ed77b5a941" + subject="annex.largefiles" + date="2021-01-01T21:25:43Z" + content=""" +Does annex.largefiles has some documentation? It would be nice to link to that on the doc of git-annex-add. + +Esp, after reading this, I wonder about the default value of annex.largefiles. (I assume/hope it is disabled?) + +"""]] From 62a6d40f1c5e128990cf35daed11f169b9382c71 Mon Sep 17 00:00:00 2001 From: AlbertZeyer Date: Fri, 1 Jan 2021 21:30:38 +0000 Subject: [PATCH 16/25] Added a comment: Adding external files --- ...omment_2_43cf725964c63a2d2545d9f204316a57._comment | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 doc/git-annex-add/comment_2_43cf725964c63a2d2545d9f204316a57._comment diff --git a/doc/git-annex-add/comment_2_43cf725964c63a2d2545d9f204316a57._comment b/doc/git-annex-add/comment_2_43cf725964c63a2d2545d9f204316a57._comment new file mode 100644 index 0000000000..2b390d980d --- /dev/null +++ b/doc/git-annex-add/comment_2_43cf725964c63a2d2545d9f204316a57._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="AlbertZeyer" + avatar="http://cdn.libravatar.org/avatar/b37d71961a6a5abf9b7184ed77b5a941" + subject="Adding external files" + date="2021-01-01T21:30:38Z" + content=""" +Let's assume I have some external files in my `~/Pictures` and I want to import them. + +Should I use `git annex import ~/Pictures/BestPics2020` or `cp -r ~/Pictures/BestPics2020 .; git annex add BestPics2020`? Is there a difference? Which way would be recommended or preferred? + +"""]] From 754327b2c4823aaa4bee1bb5daba840289baf7ac Mon Sep 17 00:00:00 2001 From: AlbertZeyer Date: Fri, 1 Jan 2021 21:33:41 +0000 Subject: [PATCH 17/25] Added a comment: rename or move files --- ...ment_4_013e20add9a007d3f9a9c2a2ceb6cb06._comment | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 doc/git-annex-move/comment_4_013e20add9a007d3f9a9c2a2ceb6cb06._comment diff --git a/doc/git-annex-move/comment_4_013e20add9a007d3f9a9c2a2ceb6cb06._comment b/doc/git-annex-move/comment_4_013e20add9a007d3f9a9c2a2ceb6cb06._comment new file mode 100644 index 0000000000..8b9f2effc7 --- /dev/null +++ b/doc/git-annex-move/comment_4_013e20add9a007d3f9a9c2a2ceb6cb06._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="AlbertZeyer" + avatar="http://cdn.libravatar.org/avatar/b37d71961a6a5abf9b7184ed77b5a941" + subject="rename or move files" + date="2021-01-01T21:33:41Z" + content=""" +Is this command also for renaming or moving files, like `git mv`? + +If not, I think this should be explained more clearly in the documentation. + +If not, how would I move/rename files then? As I understand, annexed files are just symlinks. So if I would move the file to another directory (e.g. via `git mv` or just `mv`), the symlink might break. + +"""]] From 90a026b33d4b4f1c13801a6aefc6c6f04fa8077e Mon Sep 17 00:00:00 2001 From: AlbertZeyer Date: Fri, 1 Jan 2021 21:49:29 +0000 Subject: [PATCH 18/25] Added a comment: Difference to import/add? --- .../comment_2_d1a04e31fea877ae5fe873fbd01fdcaa._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/git-annex-reinject/comment_2_d1a04e31fea877ae5fe873fbd01fdcaa._comment diff --git a/doc/git-annex-reinject/comment_2_d1a04e31fea877ae5fe873fbd01fdcaa._comment b/doc/git-annex-reinject/comment_2_d1a04e31fea877ae5fe873fbd01fdcaa._comment new file mode 100644 index 0000000000..1a65fc6e01 --- /dev/null +++ b/doc/git-annex-reinject/comment_2_d1a04e31fea877ae5fe873fbd01fdcaa._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="AlbertZeyer" + avatar="http://cdn.libravatar.org/avatar/b37d71961a6a5abf9b7184ed77b5a941" + subject="Difference to import/add?" + date="2021-01-01T21:49:29Z" + content=""" +Considering `git annex reinject /tmp/foo.iso foo.iso`, what is the difference to `git import `/tmp/foo.iso` or `cp /tmp/foo.iso; git annex add foo.iso`? +"""]] From 065cea35687e4efa87db83e22be1442a0f347494 Mon Sep 17 00:00:00 2001 From: AlbertZeyer Date: Fri, 1 Jan 2021 22:30:34 +0000 Subject: [PATCH 19/25] Added a comment --- ..._362ce3b030970db82c8dd0d98791186b._comment | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 doc/forum/Import_existing_files/comment_2_362ce3b030970db82c8dd0d98791186b._comment diff --git a/doc/forum/Import_existing_files/comment_2_362ce3b030970db82c8dd0d98791186b._comment b/doc/forum/Import_existing_files/comment_2_362ce3b030970db82c8dd0d98791186b._comment new file mode 100644 index 0000000000..45887ce220 --- /dev/null +++ b/doc/forum/Import_existing_files/comment_2_362ce3b030970db82c8dd0d98791186b._comment @@ -0,0 +1,35 @@ +[[!comment format=mdwn + username="AlbertZeyer" + avatar="http://cdn.libravatar.org/avatar/b37d71961a6a5abf9b7184ed77b5a941" + subject="comment 2" + date="2021-01-01T22:30:34Z" + content=""" +Hi, thanks for the answer. + +What if I would want to leave `~/Pictures` as-is, and not move it, nor change it? I would prefer that. I just want to add its content to a Git Annex repo, and easily sync future changes as well to the repo (e.g. after I added more files, or renamed some files, or updated some files). + +Why `git annex sync` and not `git commit`? I always did only `git commit` so far. + +Why `git annex reinject` and not `git annex import` or `cp|mv` & `git annex add`? +Also, why would I not add files which were/are not part of the original `~/Pictures`? The original `~/Pictures` would not have contained all of the pictures, as they are somewhat distributed. So I want to add unknown files as well. + +Why would I import the DVD to a dummy branch? I would want it all in my master/main branch, or not? (I also don't quite understand why I would want branches at all?) +I also potentially want to `git annex get` such a file at some point. + +What are \"too many files\" for a single repo? And why is that a problem? +I am just adding a Google Takeout archive to Git Annex ([via](https://github.com/albertz/chrome-ext-google-takeout-downloader/)), and it will contain also many of the files of `~/Pictures` (although not all; and sometimes, but not always, in smaller quality, but often also in original quality), but also many other files. So it's already pretty mixed up. +Or does it make sense to just share the Annex object storage (`.git/annex/objects`) in multiple repos? +Or do you mean that as the intended use case for branches actually? + +What dotfiles does `annex.dotfiles` include? Just all `.*`? +Why would I not want to add dotfiles? I think I would want to just archive the whole directory as-is. + +Also, after reading a bit further, and trying it out a bit, I don't quite understand: + +Given some file path (e.g. `Picture/BestPics2020/a.jpg`), how can I find other paths of the same file? (E.g. I would also have the file stored under `Picture/2020/01/a.jpg` or so.) Is that with `git annex list`? I'm not sure this lists all paths. So far I only see a single path always. + +I'm not really sure how to use `git annex import` properly, in case the file is already annexed under a different path. In any case, I also want to add the new path (new name). + +Sorry for the many follow-up questions, but this is still all somewhat unclear to me. + +"""]] From ed412d2406c2ad418514a49a43d18be43eeafc0d Mon Sep 17 00:00:00 2001 From: Lukey Date: Sat, 2 Jan 2021 15:05:01 +0000 Subject: [PATCH 20/25] Added a comment --- ..._69188f669e6fe5ca1a8c34c3dc3ec201._comment | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/forum/Import_existing_files/comment_3_69188f669e6fe5ca1a8c34c3dc3ec201._comment diff --git a/doc/forum/Import_existing_files/comment_3_69188f669e6fe5ca1a8c34c3dc3ec201._comment b/doc/forum/Import_existing_files/comment_3_69188f669e6fe5ca1a8c34c3dc3ec201._comment new file mode 100644 index 0000000000..aed6dac420 --- /dev/null +++ b/doc/forum/Import_existing_files/comment_3_69188f669e6fe5ca1a8c34c3dc3ec201._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 3" + date="2021-01-02T15:05:01Z" + content=""" +You can of course just use `~/Pictures` directly as a repository. So `cd ~/Pictures; git init; git annex init`. + +`git annex sync` does a little more things than just `git commit`. For example, it also automatically commits deletion of files. + +Sorry, I thought the existing copies of your Photos where just backups of your `~/Pictures`. In that case I suggest you to `mv` the files into the annex and then just `git annex add` them. For DVD's import to a sub-directory of your master branch instead of a dummy branch and without the `--no-content` option. + +\"Too many files\" depends on you liking. The more files the slower some operations get, like `git annex sync`. I suggest you to set something like `git annex config --set annex.largefiles 'largerthan=32kb'`. This way, small files get added to git itself instead of git-annex, which speeds up git-annex operations if there are a lot of small files. Note that these small files will be in every clone of the repo and can't be `git annex drop`ed. + +The various configuration options are documented in the main [[git-annex]] manpage, at the bottom. Without the `annex.dotfiles` option, dotfiles (any file starting with \".\" and anything inside directories starting with \".\") will still be added, but to git itself with the disadvantages mentioned above. + +You can get the key/hash for that file with `git annex info `, and then search for other files with the same content with `find . -lname '*'`. + +You can just `cp/mv` the files in the annex and `git annex add` them. Note that for duplicate files in the annex, only one copy of the data/file content will be stored. +"""]] From 420e704955338001c37907ee38dd00b20317a421 Mon Sep 17 00:00:00 2001 From: Lukey Date: Sat, 2 Jan 2021 15:12:49 +0000 Subject: [PATCH 21/25] Added a comment --- ...comment_3_8517f9634d217f731efd704405d3f2ca._comment | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/git-annex-add/comment_3_8517f9634d217f731efd704405d3f2ca._comment diff --git a/doc/git-annex-add/comment_3_8517f9634d217f731efd704405d3f2ca._comment b/doc/git-annex-add/comment_3_8517f9634d217f731efd704405d3f2ca._comment new file mode 100644 index 0000000000..1cc51ec209 --- /dev/null +++ b/doc/git-annex-add/comment_3_8517f9634d217f731efd704405d3f2ca._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 3" + date="2021-01-02T15:12:49Z" + content=""" +The various configuration options are documented in the main [[git-annex]] manpage, at the bottom. + +If it is a one-shot, just use `cp/mv` and `git annex add`. If you want to frequently import from that location, use directory special-remotes with importtree=yes. +"""]] From bd5c66811877843f4134cf9f989f8c5d9356c186 Mon Sep 17 00:00:00 2001 From: Lukey Date: Sat, 2 Jan 2021 15:16:04 +0000 Subject: [PATCH 22/25] Added a comment --- .../comment_3_2dcdd82efbd6dcac0f3b729d55a09386._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/git-annex-reinject/comment_3_2dcdd82efbd6dcac0f3b729d55a09386._comment diff --git a/doc/git-annex-reinject/comment_3_2dcdd82efbd6dcac0f3b729d55a09386._comment b/doc/git-annex-reinject/comment_3_2dcdd82efbd6dcac0f3b729d55a09386._comment new file mode 100644 index 0000000000..aa9f7b4aaf --- /dev/null +++ b/doc/git-annex-reinject/comment_3_2dcdd82efbd6dcac0f3b729d55a09386._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 3" + date="2021-01-02T15:16:04Z" + content=""" +The difference of `git annex reinject` to (`git annex import` or `cp/mv; git annex add`) is that only known file contents will be reinjected. +"""]] From 7f1116000219cd6931439440069e54165e68b9f6 Mon Sep 17 00:00:00 2001 From: Lukey Date: Sat, 2 Jan 2021 15:22:18 +0000 Subject: [PATCH 23/25] Added a comment --- .../comment_5_e7608b7fac4ec80781cb4281dc2bf596._comment | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/git-annex-move/comment_5_e7608b7fac4ec80781cb4281dc2bf596._comment diff --git a/doc/git-annex-move/comment_5_e7608b7fac4ec80781cb4281dc2bf596._comment b/doc/git-annex-move/comment_5_e7608b7fac4ec80781cb4281dc2bf596._comment new file mode 100644 index 0000000000..2395e2857f --- /dev/null +++ b/doc/git-annex-move/comment_5_e7608b7fac4ec80781cb4281dc2bf596._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Lukey" + avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" + subject="comment 5" + date="2021-01-02T15:22:18Z" + content=""" +`git annex move` belongs so the same class of commands as `git annex get`, `git annex drop` and `git annex copy` in that it manages file content. Git annex automatically registers a pre-commit hook to fixup symlinks. `git annex fix` can also be used to fixup symlinks, but it currently is broken. +"""]] From 0a264f1c983507ecf032161e614adc8d09eb592e Mon Sep 17 00:00:00 2001 From: Lukey Date: Sat, 2 Jan 2021 15:28:02 +0000 Subject: [PATCH 24/25] --- doc/bugs/git_annex_fix_broken.mdwn | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/bugs/git_annex_fix_broken.mdwn diff --git a/doc/bugs/git_annex_fix_broken.mdwn b/doc/bugs/git_annex_fix_broken.mdwn new file mode 100644 index 0000000000..642715c21d --- /dev/null +++ b/doc/bugs/git_annex_fix_broken.mdwn @@ -0,0 +1,17 @@ +### Please describe the problem. +`git annex fix` doesn't fix up broken symlinks afer moving a file. + +### What steps will reproduce the problem? + + git init + git annex init + mkdir dir + touch dir/a + git annex add . + git annex sync + mv dir/a . + git annex fix a + ls -alh + +### What version of git-annex are you using? On what operating system? +8.20201127 (I know I know... One year old version :) From 62f06649d749aed65753bd407268567e09b7e5ca Mon Sep 17 00:00:00 2001 From: AlbertZeyer Date: Sat, 2 Jan 2021 16:23:05 +0000 Subject: [PATCH 25/25] Added a comment --- .../comment_4_3bbbe94633b6cb4ef93b7942eb36cc6c._comment | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/git-annex-add/comment_4_3bbbe94633b6cb4ef93b7942eb36cc6c._comment diff --git a/doc/git-annex-add/comment_4_3bbbe94633b6cb4ef93b7942eb36cc6c._comment b/doc/git-annex-add/comment_4_3bbbe94633b6cb4ef93b7942eb36cc6c._comment new file mode 100644 index 0000000000..f13b663199 --- /dev/null +++ b/doc/git-annex-add/comment_4_3bbbe94633b6cb4ef93b7942eb36cc6c._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="AlbertZeyer" + avatar="http://cdn.libravatar.org/avatar/b37d71961a6a5abf9b7184ed77b5a941" + subject="comment 4" + date="2021-01-02T16:23:05Z" + content=""" +But is there a difference to `git annex import`? What is the difference? Why would you use `git annex add` instead of `git annex import`? + +"""]]