diff --git a/doc/bugs/Add_day_to_metadata.mdwn b/doc/bugs/Add_day_to_metadata.mdwn deleted file mode 100644 index 24188a4bcb..0000000000 --- a/doc/bugs/Add_day_to_metadata.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -### Please describe the problem. - -The metadata doesn't include the day for RSS feeds, which is probably only a problem for me as I've built something on top of the metadata to provide an interface for selecting podcasts. - -### Please provide any additional information below. - -The attached patch adds the day field to the metadata. - -There's also a tiny bit extra for stack.yaml which will only affect people who have nix enabled in Stack and will make the build successful for those people. - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -git-annex is awesome, I lean on it heavily nearly every single day. - -> [[merged|done]]. Thanks for the patch! --[[Joey]] diff --git a/doc/bugs/Add_day_to_metadata/comment_1_d46d9f085b7077cc95d71628e45c231d._comment b/doc/bugs/Add_day_to_metadata/comment_1_d46d9f085b7077cc95d71628e45c231d._comment deleted file mode 100644 index ec344a2d6e..0000000000 --- a/doc/bugs/Add_day_to_metadata/comment_1_d46d9f085b7077cc95d71628e45c231d._comment +++ /dev/null @@ -1,64 +0,0 @@ -[[!comment format=mdwn - username="seantparsons" - avatar="http://cdn.libravatar.org/avatar/616fb81847630239dd1ab099138cb685" - subject="Since the attachment doesn't appear to be there, here's the content." - date="2017-10-22T20:32:21Z" - content=""" - diff --git a/Annex/MetaData.hs b/Annex/MetaData.hs - index e22ed05a6..355c5124a 100644 - --- a/Annex/MetaData.hs - +++ b/Annex/MetaData.hs - @@ -60,10 +60,11 @@ dateMetaData :: UTCTime -> MetaData -> MetaData - dateMetaData mtime old = MetaData $ M.fromList $ filter isnew - [ (yearMetaField, S.singleton $ toMetaValue $ show y) - , (monthMetaField, S.singleton $ toMetaValue $ show m) - + , (dayMetaField, S.singleton $ toMetaValue $ show d) - ] - where - isnew (f, _) = S.null (currentMetaDataValues f old) - - (y, m, _d) = toGregorian $ utctDay mtime - + (y, m, d) = toGregorian $ utctDay mtime - - {- Parses field=value, field+=value, field-=value, field?=value -} - parseModMeta :: String -> Either String ModMeta - diff --git a/Annex/MetaData/StandardFields.hs b/Annex/MetaData/StandardFields.hs - index c91b53930..b9ea47e2f 100644 - --- a/Annex/MetaData/StandardFields.hs - +++ b/Annex/MetaData/StandardFields.hs - @@ -9,6 +9,7 @@ module Annex.MetaData.StandardFields ( - tagMetaField, - yearMetaField, - monthMetaField, - + dayMetaField, - lastChangedField, - mkLastChangedField, - isLastChangedField - @@ -27,6 +28,9 @@ yearMetaField = mkMetaFieldUnchecked \"year\" - monthMetaField :: MetaField - monthMetaField = mkMetaFieldUnchecked \"month\" - - +dayMetaField :: MetaField - +dayMetaField = mkMetaFieldUnchecked \"day\" - + - lastChangedField :: MetaField - lastChangedField = mkMetaFieldUnchecked lastchanged - - diff --git a/stack.yaml b/stack.yaml - index d84c4682e..ac601200e 100644 - --- a/stack.yaml - +++ b/stack.yaml - @@ -24,3 +24,11 @@ extra-deps: - explicit-setup-deps: - git-annex: true - resolver: lts-9.9 - +nix: - + packages: - + - ncurses - + - icu - + - libcxx - + - gcc - + - zlib - + - rsync - \ No newline at end of file - -"""]] diff --git a/doc/bugs/Add_day_to_metadata/comment_2_ea15c799c5abfc97eaa20424c15b73a4._comment b/doc/bugs/Add_day_to_metadata/comment_2_ea15c799c5abfc97eaa20424c15b73a4._comment deleted file mode 100644 index 73a5b56063..0000000000 --- a/doc/bugs/Add_day_to_metadata/comment_2_ea15c799c5abfc97eaa20424c15b73a4._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2017-11-08T18:51:20Z" - content=""" -Unfortunately the added nix section broke the i386-ancient build, -which uses stack 1.0.4. That version of stack complains: - -Executable named nix-shell not found on path: ["/usr/local/sbin","/usr/local/bin","/sbin","/bin","/usr/sbin","/usr/bin"] - -So, I've reverted the addition of that section. -"""]] diff --git a/doc/bugs/Adding_files_fails_with_--jobs_more_than_2.mdwn b/doc/bugs/Adding_files_fails_with_--jobs_more_than_2.mdwn deleted file mode 100644 index 8dbecc7105..0000000000 --- a/doc/bugs/Adding_files_fails_with_--jobs_more_than_2.mdwn +++ /dev/null @@ -1,66 +0,0 @@ -### Please describe the problem. - -I'm running into an issue where adding files to a git annex repository fails when I specify a number of jobs equal to the number of cores on my machine, resulting in the following message: - -`git-annex: getCurrentDirectory:getWorkingDirectory: resource exhausted (Too many open files)` - - -### What steps will reproduce the problem? - -Running the command below on a directory with less than 1000 files. - -``` -git init -git annex init -git annex add --jobs 16 . -``` - -### What version of git-annex are you using? On what operating system? - -Mac OSX 10.15.5 - -``` -# brew install git-annex -20:16 $ git annex version -git-annex version: 8.20200617 -build flags: Assistant Webapp Pairing S3 WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.1 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: darwin x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 0 1 2 3 4 5 6 7 -local repository version: 8 -``` - -### Please provide any additional information below. - -The log was empty, but here is a snippet of the stdout. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - -add test-app/test/app.R -git-annex: getCurrentDirectory:getWorkingDirectory: resource exhausted (Too many open files) -failed -add tool_args/d1b89bbf441df390a60d787af813817579d2b760 -git-annex: getCurrentDirectory:getWorkingDirectory: resource exhausted (Too many open files) -failed -... -add CCLE_expression_full.csv -git-annex: cp: createProcess: runInteractiveProcess: pipe: resource exhausted (Too many open files) -failed -add Achilles_20Q1/CCLE_expression_full.csv -git-annex: cp: createProcess: runInteractiveProcess: pipe: resource exhausted (Too many open files) -failed - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I've been having quite a bit of fun revisiting git-annex and datalad after quite a while, and I'm really excited to see how far things have come. I'm pretty close to adopting these tools in my data science group after a pretty exhaustive search of related technologies like quilt, dvc, and attempts to role my own solution. Using Github + the S3 special remote w/versioning enabled is just about the best solution to dataset tracking I've come across. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/Adding_files_fails_with_--jobs_more_than_2/comment_1_77cfd006625edc78e8c3bf6cbb4e1f09._comment b/doc/bugs/Adding_files_fails_with_--jobs_more_than_2/comment_1_77cfd006625edc78e8c3bf6cbb4e1f09._comment deleted file mode 100644 index 2a4c839168..0000000000 --- a/doc/bugs/Adding_files_fails_with_--jobs_more_than_2/comment_1_77cfd006625edc78e8c3bf6cbb4e1f09._comment +++ /dev/null @@ -1,24 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-07-21T14:09:48Z" - content=""" -It looks to me like git-annex add is leaking a FD to -.git/annex/othertmp.lck - -I ran git-annex add -J16 in a repo with 1000 files, and around half way -through it had 832 open FDs, and 737 of them were that lock file. 55 more -were pipes (to and from git cat-file etc; recent work reduced the number of -those a lot), and the remaining 37 were event and timer stuff used by the -RTS. - -The number of -J does not change that leak, same number with -J2. -J2 will -open slightly less other files though. Without -J, the FD leak does not -happen. - -Also, git-annex 8.20200330 does not have this problem, it seems to be a -recent bug. - -[[!commit 3334130368829ad2041006560e578f1f876f68e4]] claimed to fix a leak -in this same code path. -"""]] diff --git a/doc/bugs/Adding_files_fails_with_--jobs_more_than_2/comment_2_45702c5a87e891a0dde5459b5156c57e._comment b/doc/bugs/Adding_files_fails_with_--jobs_more_than_2/comment_2_45702c5a87e891a0dde5459b5156c57e._comment deleted file mode 100644 index 9b89f2eaff..0000000000 --- a/doc/bugs/Adding_files_fails_with_--jobs_more_than_2/comment_2_45702c5a87e891a0dde5459b5156c57e._comment +++ /dev/null @@ -1,25 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""analysis""" - date="2020-07-21T18:35:08Z" - content=""" -Utility.LockPool.STM.releaseLock seems to be where the problem is. - -It waits to close a lock FD if some other thread is using the lock. -But, this means that, if enough threads are run that the lock is -always in use by one thread, it will never close it. Meanwhile, each -lockShared call opens the lock file anew, accumilating another FD. - -3334130368829ad2041006560e578f1f876f68e4 is at fault, indeed. - -That commit mentions that it would be better to have two calls to -lockShared only open the file once, but that it would be difficult to do -that atomically. Perhaps there is a way to do that... It didn't seem easy -to do this time either. - -Alternatively, registerCloseLockFile currently takes an action that closes -one lock FD, and just combines that to its existing close action with `>>`. -So it builds up one big action that closes all the FDs. Instead, make each -lock handle contain its close action, and have releaseLock only release the -one it was called with. Implemented this, and it solved it. -"""]] diff --git a/doc/bugs/Adding_files_fails_with_--jobs_more_than_2/comment_3_8f94f5d1ee5c92e945ba2344cdea8307._comment b/doc/bugs/Adding_files_fails_with_--jobs_more_than_2/comment_3_8f94f5d1ee5c92e945ba2344cdea8307._comment deleted file mode 100644 index 493f63732d..0000000000 --- a/doc/bugs/Adding_files_fails_with_--jobs_more_than_2/comment_3_8f94f5d1ee5c92e945ba2344cdea8307._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="nicholsn" - nickname="nolan.nichols" - avatar="http://cdn.libravatar.org/avatar/12e81148e905851b4794288e8bf336e9" - subject="Thank you" - date="2020-07-28T03:57:32Z" - content=""" -This fix is much appreciated! -"""]] diff --git a/doc/bugs/Build_fails_with_type_error_on_Windows.mdwn b/doc/bugs/Build_fails_with_type_error_on_Windows.mdwn deleted file mode 100644 index c7e1e8d652..0000000000 --- a/doc/bugs/Build_fails_with_type_error_on_Windows.mdwn +++ /dev/null @@ -1,44 +0,0 @@ -### Please describe the problem. - -Trying to build git-annex on Windows by running "stack build" as instructed by https://git-annex.branchable.com/install/Windows/ currently fails with the following error: - -[[!format sh """ - -[305 of 636] Compiling Annex.PidLock - -Annex\PidLock.hs:70:9: error: - * Couldn't match expected type `Annex a' with actual type `IO a' - * In a stmt of a 'do' block: gonopidlock - In the expression: - do let p = f (proc cmd ps) - let gonopidlock = withCreateProcess p a - gonopidlock - In an equation for `pidLockChildProcess': - pidLockChildProcess cmd ps f a - = do let p = ... - let gonopidlock = ... - gonopidlock - * Relevant bindings include - gonopidlock :: IO a (bound at Annex\PidLock.hs:47:13) - a :: Maybe Handle - -> Maybe Handle -> Maybe Handle -> ProcessHandle -> IO a - (bound at Annex\PidLock.hs:45:30) - pidLockChildProcess :: FilePath - -> [String] - -> (CreateProcess -> CreateProcess) - -> (Maybe Handle - -> Maybe Handle -> Maybe Handle -> ProcessHandle -> IO a) - -> Annex a - (bound at Annex\PidLock.hs:45:1) - | -70 | gonopidlock - | ^^^^^^^^^^^ - -"""]] - -This happens when trying to build both the most recent commit (20c7e6c) and the most recent tag (8.20200908). - -- stack version: 2.3.3 -- Windows version: Windows 7 Enterprise, version 6.1 - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/Build_fails_with_type_error_on_Windows/comment_1_344be786838e34f65a6a5d1eec866c5f._comment b/doc/bugs/Build_fails_with_type_error_on_Windows/comment_1_344be786838e34f65a6a5d1eec866c5f._comment deleted file mode 100644 index a33122f8a4..0000000000 --- a/doc/bugs/Build_fails_with_type_error_on_Windows/comment_1_344be786838e34f65a6a5d1eec866c5f._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-10-07T16:03:04Z" - content=""" -Thanks for reporting this. I usually only find out when the Windows CI -fails, but it's not been running for a while. - -I've fixed this particular problem, but don't have a windows build env -handy to test the rest of the build. Please followup if there's a -subsequent build failure. -"""]] diff --git a/doc/bugs/Build_failures_with_tasty_1.3.mdwn b/doc/bugs/Build_failures_with_tasty_1.3.mdwn deleted file mode 100644 index 0ca3c12f03..0000000000 --- a/doc/bugs/Build_failures_with_tasty_1.3.mdwn +++ /dev/null @@ -1,149 +0,0 @@ -### Please describe the problem. -git-annex failed to build with latest tasty 1.3 - -### What steps will reproduce the problem? -Build git-annex with tasty-1.3 - -### What version of git-annex are you using? On what operating system? -8.20200501 on Arch Linux. - -### Please provide any additional information below. - [545 of 638] Compiling Test ( Test.hs, dist/build/git-annex/git-annex-tmp/Test.dyn_o ) - - Test.hs:98:13: error: - • Couldn't match type ‘(,) [String]’ with ‘Parser’ - Expected type: Parser TestOptions - Actual type: ([String], TestOptions) - • In the expression: - TestOptions - <$> suiteOptionParser ingredients (tests False True mempty) - <*> - switch - (long "keep-failures" - <> help "preserve repositories on test failure") - <*> switch (long "fakessh" <> internal) - <*> cmdParams "non-options are for internal use only" - In an equation for ‘optParser’: - optParser - = TestOptions - <$> suiteOptionParser ingredients (tests False True mempty) - <*> - switch - (long "keep-failures" - <> help "preserve repositories on test failure") - <*> switch (long "fakessh" <> internal) - <*> cmdParams "non-options are for internal use only" - | - 98 | optParser = TestOptions - | ^^^^^^^^^^^... - - Test.hs:99:13: error: - • Couldn't match type ‘Parser Test.Tasty.Options.OptionSet’ - with ‘Test.Tasty.Options.OptionSet’ - Expected type: ([String], Test.Tasty.Options.OptionSet) - Actual type: ([String], Parser Test.Tasty.Options.OptionSet) - • In the second argument of ‘(<$>)’, namely - ‘suiteOptionParser ingredients (tests False True mempty)’ - In the first argument of ‘(<*>)’, namely - ‘TestOptions - <$> suiteOptionParser ingredients (tests False True mempty)’ - In the first argument of ‘(<*>)’, namely - ‘TestOptions - <$> suiteOptionParser ingredients (tests False True mempty) - <*> - switch - (long "keep-failures" - <> help "preserve repositories on test failure")’ - | - 99 | <$> suiteOptionParser ingredients (tests False True mempty) - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - Test.hs:100:13: error: - • Couldn't match type ‘Parser’ with ‘(,) [String]’ - Expected type: ([String], Bool) - Actual type: Parser Bool - • In the second argument of ‘(<*>)’, namely - ‘switch - (long "keep-failures" - <> help "preserve repositories on test failure")’ - In the first argument of ‘(<*>)’, namely - ‘TestOptions - <$> suiteOptionParser ingredients (tests False True mempty) - <*> - switch - (long "keep-failures" - <> help "preserve repositories on test failure")’ - In the first argument of ‘(<*>)’, namely - ‘TestOptions - <$> suiteOptionParser ingredients (tests False True mempty) - <*> - switch - (long "keep-failures" - <> help "preserve repositories on test failure") - <*> switch (long "fakessh" <> internal)’ - | - 100 | <*> switch - | ^^^^^^... - - Test.hs:104:13: error: - • Couldn't match type ‘Parser’ with ‘(,) [String]’ - Expected type: ([String], Bool) - Actual type: Parser Bool - • In the second argument of ‘(<*>)’, namely - ‘switch (long "fakessh" <> internal)’ - In the first argument of ‘(<*>)’, namely - ‘TestOptions - <$> suiteOptionParser ingredients (tests False True mempty) - <*> - switch - (long "keep-failures" - <> help "preserve repositories on test failure") - <*> switch (long "fakessh" <> internal)’ - In the expression: - TestOptions - <$> suiteOptionParser ingredients (tests False True mempty) - <*> - switch - (long "keep-failures" - <> help "preserve repositories on test failure") - <*> switch (long "fakessh" <> internal) - <*> cmdParams "non-options are for internal use only" - | - 104 | <*> switch - | ^^^^^^... - - Test.hs:108:13: error: - • Couldn't match type ‘Parser’ with ‘(,) [String]’ - Expected type: ([String], Types.Command.CmdParams) - Actual type: Parser Types.Command.CmdParams - • In the second argument of ‘(<*>)’, namely - ‘cmdParams "non-options are for internal use only"’ - In the expression: - TestOptions - <$> suiteOptionParser ingredients (tests False True mempty) - <*> - switch - (long "keep-failures" - <> help "preserve repositories on test failure") - <*> switch (long "fakessh" <> internal) - <*> cmdParams "non-options are for internal use only" - In an equation for ‘optParser’: - optParser - = TestOptions - <$> suiteOptionParser ingredients (tests False True mempty) - <*> - switch - (long "keep-failures" - <> help "preserve repositories on test failure") - <*> switch (long "fakessh" <> internal) - <*> cmdParams "non-options are for internal use only" - | - 108 | <*> cmdParams "non-options are for internal use only" - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -Yes. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/Cannot_integrate_git-annex_with_SourceTree_in_Catalina.mdwn b/doc/bugs/Cannot_integrate_git-annex_with_SourceTree_in_Catalina.mdwn deleted file mode 100644 index 5512abae2c..0000000000 --- a/doc/bugs/Cannot_integrate_git-annex_with_SourceTree_in_Catalina.mdwn +++ /dev/null @@ -1,35 +0,0 @@ -### Please describe the problem. -Historically (in older versions of MacOS) to get git-annex working with SourceTree I've done the following: - -When committing in SourceTree or SmartGit after adding annex, you may get error "git: 'annex' is not a git command". To fix: -First make sure your client is using the system git -In SourceTree preferences, go to Git tab and use system git (likely in /usr/bin/git) -Disable SIP (needed step starting from Mac OSX El Capitan), by doing the following: -Restart your Mac. -Then, make a symbolic link so SourceTree/SmartGit can see git-annex (look at which git-annex to find real location): -sudo ln -s /usr/local/bin/git-annex /usr/bin/git-annex -Re-enable SIP, by following the same steps for disabling, but rather issuing the command csrutil enable in the Terminal window - -But now, I still can't write the symbolic link with SIP disabled. Obviously this isn't git-annex's fault. But I cannot figure out how to integrate git-annex so that SourceTree can work with it. - -### What steps will reproduce the problem? -See above. - -### What version of git-annex are you using? On what operating system? -git-annex version: 8.20200908 -macOS 10.15.6 - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - - -> [[notabug|done]] --[[Joey]] diff --git a/doc/bugs/Cannot_integrate_git-annex_with_SourceTree_in_Catalina/comment_1_ebf44f50f952a87d1af06ea24877baae._comment b/doc/bugs/Cannot_integrate_git-annex_with_SourceTree_in_Catalina/comment_1_ebf44f50f952a87d1af06ea24877baae._comment deleted file mode 100644 index 2bfea18452..0000000000 --- a/doc/bugs/Cannot_integrate_git-annex_with_SourceTree_in_Catalina/comment_1_ebf44f50f952a87d1af06ea24877baae._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="mark@6b90344cdab3158eacb94a3944460d138afc9bef" - nickname="mark" - avatar="http://cdn.libravatar.org/avatar/780625d9a11f1840abd94822dfe3e9f5" - subject="comment 1" - date="2020-09-28T21:10:56Z" - content=""" -And BTW, we really value git-annex and use it for keeping all of our golden reference outputs for regression testing in place and available for others who need to run tests. -"""]] diff --git a/doc/bugs/Cannot_integrate_git-annex_with_SourceTree_in_Catalina/comment_2_c3a86ef4972b1a2db985fd03750aa449._comment b/doc/bugs/Cannot_integrate_git-annex_with_SourceTree_in_Catalina/comment_2_c3a86ef4972b1a2db985fd03750aa449._comment deleted file mode 100644 index 8585de6df5..0000000000 --- a/doc/bugs/Cannot_integrate_git-annex_with_SourceTree_in_Catalina/comment_2_c3a86ef4972b1a2db985fd03750aa449._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="mark@6b90344cdab3158eacb94a3944460d138afc9bef" - nickname="mark" - avatar="http://cdn.libravatar.org/avatar/780625d9a11f1840abd94822dfe3e9f5" - subject="comment 2" - date="2020-09-28T21:40:21Z" - content=""" -Switching SourceTree to look at /usr/local/git/bin fixed the problem. - -This can be CLOSED. -"""]] diff --git a/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead.mdwn b/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead.mdwn deleted file mode 100644 index 34fb1c4f35..0000000000 --- a/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead.mdwn +++ /dev/null @@ -1,49 +0,0 @@ -I'm not sure if this is a bug per se but a small bother. - -In [deleting unwanted files](https://git-annex.branchable.com/tips/deleting_unwanted_files/) it is mentioned that you can use `git annex drop --force $file` to make it go away. But after that when fsck is run it will warn that there are no copies and that it should be marked as dead. And if it's deleted from the branch it will still complain if fsck is run with `--all`. - -Shouldn't they be marked dead automatically if the copy count reaches 0? - -Example: - -[[!format sh """ -$ git annex fsck -fsck test - ** No known copies exist of test -failed -(recording state in git...) -git-annex: fsck: 1 failed -$ rm test -$ git annex fsck --all -$ fsck SHA256E-s16--c21f3ac6b5f6e45b1c0b292bcd5cc806298ecb033bc7030a6071e3c894d73054 - ** No known copies exist of SHA256E-s16--c21f3ac6b5f6e45b1c0b292bcd5cc806298ecb033bc7030a6071e3c894d73054 - - (Avoid this check by running: git annex dead --key ) -failed -(recording state in git...) -git-annex: fsck: 1 failed -$ git annex forget --force -$ git annex repair --force -$ git annex fsck --all -fsck SHA256E-s16--c21f3ac6b5f6e45b1c0b292bcd5cc806298ecb033bc7030a6071e3c894d73054 - ** No known copies exist of SHA256E-s16--c21f3ac6b5f6e45b1c0b292bcd5cc806298ecb033bc7030a6071e3c894d73054 - - (Avoid this check by running: git annex dead --key ) -failed -(recording state in git...) -git-annex: fsck: 1 failed -"""]] - -From this, not even forget,fsck or repair can make this go away but only `git annex dead --key` which can be a pain to call it on a ton of different keys (there doesn't appear to be an option to batch mark them all as dead either). - - - - -Additionally the key will still remain in the `git-annex` branch, I'm not sure about the internal optimizations in place but I'd assume leaving unused keys around in the branch can cause slowdowns when it's checked out/updated in repositories with a lot of add/update/delete activity. - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I'm using git annex to manage all my personal data, thanks for making such an amazing piece of software :) - -> [[notabug|done]] --[[Joey]] diff --git a/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_1_234aa73f3463c35a0f3004c6524d0464._comment b/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_1_234aa73f3463c35a0f3004c6524d0464._comment deleted file mode 100644 index 7e4523ccf8..0000000000 --- a/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_1_234aa73f3463c35a0f3004c6524d0464._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="CandyAngel" - avatar="http://cdn.libravatar.org/avatar/15c0aade8bec5bf004f939dd73cf9ed8" - subject="comment 1" - date="2018-10-03T08:01:25Z" - content=""" -Regarding git-annex-dead: - -> This command exists to deal with situations where data has been lost, and **you know it has**, and you want to stop being reminded of that fact. - -Dead-ifying keys automatically could result in the situation where a key is marked as dead and the user doesn't (and wouldn't) know about it (until trying to use the file) because git-annex would never complain about it. - -Adding --batch support for `git-annex-dead --keys` would be good. I suggest making a todo for just that and/or for adding support for `git annex drop --force --dead`? -"""]] diff --git a/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_2_745d70d2b7fea257727465281e16701e._comment b/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_2_745d70d2b7fea257727465281e16701e._comment deleted file mode 100644 index 951282c994..0000000000 --- a/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_2_745d70d2b7fea257727465281e16701e._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2018-10-04T17:03:42Z" - content=""" -I agree with CandyAngel, this is not something that should be done -automatically. - -While `git annex dead --batch` could be worth adding, it might be more -useful to support `git annex dead --files file|dir ...` so you can just run -it on the same files you're dropping. -"""]] diff --git a/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_3_9cc112dc5814b45966f48ecdb6376180._comment b/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_3_9cc112dc5814b45966f48ecdb6376180._comment deleted file mode 100644 index b4c35097a9..0000000000 --- a/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_3_9cc112dc5814b45966f48ecdb6376180._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="glasserc" - avatar="http://cdn.libravatar.org/avatar/8e865c04033751520601e9c15e58ddc4" - subject="Is `dead` really the solution here?" - date="2020-05-06T20:29:03Z" - content=""" -I encountered this behavior as well on an annex where I had decided to move a file into a different, unrelated repository. I used `dropunused` to get rid of the contents. Now I have a repository for which `git annex get` constantly complains that a file is not known to be in any repository. - -As a naive user, I wouldn't have considered \"dead\" to be the description for this file. It isn't lost in any way, just untracked in the context of this repository. It's a bit surprising that once a file is ever tracked, it can never be untracked by any means -- even marking it as dead is still tracking the deadness of the file. - -I guess from a theoretical point of view that some branch or some repo somewhere might still refer to the file in some way and its absence might therefore have consequences later on. From a practical point of view, once the commit removing the last link to the file is on master on all known repositories, it seems like we can consider this file \"deleted\" in a way that is distinct from \"dead\". - -Leaving aside what the status is called, how about detecting it automatically on `dropunused`? This is a bit different from detecting it on `drop` because we have already concluded that the file is \"gone\" in some way. - -"""]] diff --git a/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_4_39e375d942289b07116600f1a22da5be._comment b/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_4_39e375d942289b07116600f1a22da5be._comment deleted file mode 100644 index 6185eef978..0000000000 --- a/doc/bugs/Deleting_last_copy_of_files_doesn__39__t_mark_them_dead/comment_4_39e375d942289b07116600f1a22da5be._comment +++ /dev/null @@ -1,25 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-05-07T15:56:38Z" - content=""" -`git annex get` will only complain about files that are still in the -current branch. If you want to use `--all` though, you will have to mark -such files as dead. git-annex otherwise has no way to differentiate between -the last copy of a file being lost inaverdently and intentionally removed. - -Auto-deading files would hide problems. - -> even marking it as dead is still tracking the deadness of the file - -You can use git-annex forget to prune that information from the git-annex -branch, but it's completely reasonable for a git repository to retain -history by default. - -> Leaving aside what the status is called, how about detecting it -> automatically on `dropunused` - -Unused files are only unused in the head of branches, not in all the -history of the git repository. Again, git is all about preserving the -history of files. -"""]] diff --git a/doc/bugs/Doesn__39__t_build_with_hinotify__44__0.3.10___47___fsnotify__44__0.2.1.2.mdwn b/doc/bugs/Doesn__39__t_build_with_hinotify__44__0.3.10___47___fsnotify__44__0.2.1.2.mdwn deleted file mode 100644 index 0d91c4a440..0000000000 --- a/doc/bugs/Doesn__39__t_build_with_hinotify__44__0.3.10___47___fsnotify__44__0.2.1.2.mdwn +++ /dev/null @@ -1,198 +0,0 @@ -### Please describe the problem. -Build failure with hinotify,0.3.10 / fsnotify,0.2.1.2 - -### What version of git-annex are you using? On what operating system? -git-annex 6.20180427 on Arch x86_64. - -### Please provide any additional information below. - -[[!format sh """ -[124 of 594] Compiling Utility.DirWatcher.INotify ( Utility/DirWatcher/INotify.hs, dist/build/git-annex/git-annex-tmp/Utility/DirWatcher/INotify.dyn_o ) - -Utility/DirWatcher/INotify.hs:58:54: error: - • Couldn't match type ‘[Char]’ - with ‘Data.ByteString.Internal.ByteString’ - Expected type: System.Posix.ByteString.FilePath.RawFilePath - Actual type: FilePath - • In the third argument of ‘addWatch’, namely ‘dir’ - In the first argument of ‘void’, namely - ‘(addWatch i watchevents dir handler)’ - In the first argument of ‘catchIO’, namely - ‘void (addWatch i watchevents dir handler)’ - | -58 | void (addWatch i watchevents dir handler) - | ^^^ - -Utility/DirWatcher/INotify.hs:97:41: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the first argument of ‘indir’, namely ‘f’ - In the second argument of ‘($)’, namely ‘indir f’ - In the expression: recurse $ indir f - | -97 | | isd = recurse $ indir f - | ^ - -Utility/DirWatcher/INotify.hs:99:41: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the first argument of ‘getstatus’, namely ‘f’ - In a stmt of a 'do' block: ms <- getstatus f - In the expression: - do ms <- getstatus f - case ms of - Just s - | isSymbolicLink s - -> when (hashook addSymlinkHook) $ runhook addSymlinkHook f ms - | isRegularFile s -> when (hashook addHook) $ runhook addHook f ms - _ -> noop - | -99 | ms <- getstatus f - | ^ - -Utility/DirWatcher/INotify.hs:104:80: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the second argument of ‘runhook’, namely ‘f’ - In the second argument of ‘($)’, namely - ‘runhook addSymlinkHook f ms’ - In the expression: - when (hashook addSymlinkHook) $ runhook addSymlinkHook f ms - | -104 | runhook addSymlinkHook f ms - | ^ - -Utility/DirWatcher/INotify.hs:107:73: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the second argument of ‘runhook’, namely ‘f’ - In the second argument of ‘($)’, namely ‘runhook addHook f ms’ - In the expression: when (hashook addHook) $ runhook addHook f ms - | -107 | runhook addHook f ms - | ^ - -Utility/DirWatcher/INotify.hs:112:67: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the third argument of ‘checkfiletype’, namely ‘f’ - In the expression: checkfiletype isRegularFile addHook f - In an equation for ‘go’: - go (Closed {isDirectory = False, maybeFilePath = Just f}) - = checkfiletype isRegularFile addHook f - | -112 | checkfiletype Files.isRegularFile addHook f - | ^ - -Utility/DirWatcher/INotify.hs:115:46: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the first argument of ‘scan’, namely ‘f’ - In the expression: scan f - In an equation for ‘go’: go (MovedIn {filePath = f}) = scan f - | -115 | go (MovedIn { filePath = f }) = scan f - | ^ - -Utility/DirWatcher/INotify.hs:117:44: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the second argument of ‘runhook’, namely ‘f’ - In the expression: runhook delDirHook f Nothing - In an equation for ‘go’: - go (MovedOut {isDirectory = isd, filePath = f}) - | isd = runhook delDirHook f Nothing - | otherwise = runhook delHook f Nothing - | -117 | | isd = runhook delDirHook f Nothing - | ^ - -Utility/DirWatcher/INotify.hs:118:47: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the second argument of ‘runhook’, namely ‘f’ - In the expression: runhook delHook f Nothing - In an equation for ‘go’: - go (MovedOut {isDirectory = isd, filePath = f}) - | isd = runhook delDirHook f Nothing - | otherwise = runhook delHook f Nothing - | -118 | | otherwise = runhook delHook f Nothing - | ^ - -Utility/DirWatcher/INotify.hs:124:54: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the second argument of ‘runhook’, namely ‘f’ - In the second argument of ‘($)’, namely - ‘runhook delDirHook f Nothing’ - In the expression: guarded $ runhook delDirHook f Nothing - | -124 | | isd = guarded $ runhook delDirHook f Nothing - | ^ - -Utility/DirWatcher/INotify.hs:125:57: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the second argument of ‘runhook’, namely ‘f’ - In the second argument of ‘($)’, namely ‘runhook delHook f Nothing’ - In the expression: guarded $ runhook delHook f Nothing - | -125 | | otherwise = guarded $ runhook delHook f Nothing - | ^ - -Utility/DirWatcher/INotify.hs:127:58: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the second argument of ‘filetype’, namely ‘f’ - In the first argument of ‘unlessM’, namely - ‘(filetype (const True) f)’ - In the expression: unlessM (filetype (const True) f) - | -127 | guarded = unlessM (filetype (const True) f) - | ^ - -Utility/DirWatcher/INotify.hs:130:50: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the second argument of ‘runhook’, namely ‘f’ - In the expression: runhook modifyHook f Nothing - In an equation for ‘go’: - go (Modified {isDirectory = isd, maybeFilePath = Just f}) - | isd = noop - | otherwise = runhook modifyHook f Nothing - | -130 | | otherwise = runhook modifyHook f Nothing - | ^ - -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Yes! - -> [[fixed|done]] a long time ago --[[Joey]] diff --git a/doc/bugs/Error_attempting_to_create_adb_special_remote.mdwn b/doc/bugs/Error_attempting_to_create_adb_special_remote.mdwn deleted file mode 100644 index 7cf9854c49..0000000000 --- a/doc/bugs/Error_attempting_to_create_adb_special_remote.mdwn +++ /dev/null @@ -1,46 +0,0 @@ -### Please describe the problem. -I get error while attempting to create an adb special remote -Problem appears to be with gvfs (mtp) mount point that my phone is getting mounted to. - -### What steps will reproduce the problem? -Attach android phone to computer via USB. (eg Pixel3A, Samsung Galaxy S6 etc.) -Phone mounts using gvfs-mtp. (GNOME virt file system) - Specifically in my case at the following path: - -/run/user/1000/gvfs/mtp\:host\=Google_Pixel_3a_94MBYKEOEJS/Internal\ shared\ storage/DCIM - -Attempt to create adb special remote as follows: - -git annex initremote android type=adb -androiddirectory=/run/user/1000/gvfs/mtp\:host\=Google_Pixel_3a_94MBYKEOEJS/Internal\ shared\ storage/DCIM -encryption=none exporttree=yes importtree=yes - -Results in following error message: -initremote android git-annex: adb: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory) failed git-annex: initremote: 1 failed - -I can successfully 'cd' into the path via xterm and browse the phone's DCIM directory etc as normal -so the path itself if fine. I also tried enclosing the dir name in quotes, double quotes etc but that didn't help -Also tried by creating a symbolic link to the path name and specifying the link name, but no joy -I was hoping that it was the ":" between mtp and host that was the issue) - -I have also attempted just to use the directory special remote as well - but I get same error. - - - -### What version of git-annex are you using? On what operating system? -8.20200226 (but I have also seen same issue on earlier ver 7) -I upgraded to 8 in hope of rectifying the issue. - -### Please provide any additional information below. - -Running Debian 10.3 - - -### Have you had any luck using git-annex before? -Yes, have been using git-annex successfully for several years, but this is my first attempt at using the adb special remote. - -Thanks -M. - - -> [[notabug|done]] diff --git a/doc/bugs/Error_attempting_to_create_adb_special_remote/comment_1_ac03f6ddcb801d85047f8d81d0505d75._comment b/doc/bugs/Error_attempting_to_create_adb_special_remote/comment_1_ac03f6ddcb801d85047f8d81d0505d75._comment deleted file mode 100644 index 80c783a974..0000000000 --- a/doc/bugs/Error_attempting_to_create_adb_special_remote/comment_1_ac03f6ddcb801d85047f8d81d0505d75._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-04-02T00:36:55Z" - content=""" -Your phone is being mounted with mtp. That is not the protocol used by the -adb special remote. - -The androiddirectory parameter cannot be used to specify a path on -your local computer. I think the documentation of that parameter is clear -about that? - -You seem to not have the adb command installed, based on the error message. -If you install it and follow the documentation, I'd expect it will work. - -Or, since the phone is already getting mounted by mtp, you can, instead of -making an adb special remote, simply make a -[directory special remote](https://git-annex.branchable.com/special_remotes/directory/) -"""]] diff --git a/doc/bugs/Error_cloning_repository_on_Windows.mdwn b/doc/bugs/Error_cloning_repository_on_Windows.mdwn deleted file mode 100644 index 6b41f908d7..0000000000 --- a/doc/bugs/Error_cloning_repository_on_Windows.mdwn +++ /dev/null @@ -1,20 +0,0 @@ -### Please describe the problem. - -Running "git clone git://git-annex.branchable.com/" in git-bash on Windows 7 fails with: - -[[!format sh """ -error: invalid path 'doc/bugs/Add_day_to_metadata./comment_1_d46d9f085b7077cc95d71628e45c231d._comment' -fatal: unable to checkout working tree -warning: Clone succeeded, but checkout failed. -You can inspect what was checked out with 'git status' -and retry with 'git restore --source=HEAD :/' -"""] - -I believe the failure is because "Add_day_to_metadata." (a path component with an empty extension) is invalid on Windows. - -Because of this, developing and building on Windows is harder and more needlessly complicated than it should be. Please rename the "doc/bugs/Add_day_to_metadata." directory along with the other directories whose names also end in a period (They can be found with "git ls-tree -r --name-only HEAD | grep '\./'"). -### What steps will reproduce the problem? - -Cloning the git-annex repository on Windows 7 — or any other Windows, I believe. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/Error_cloning_repository_on_Windows/comment_1_3c95fbb637e8043f988822069518fd3a._comment b/doc/bugs/Error_cloning_repository_on_Windows/comment_1_3c95fbb637e8043f988822069518fd3a._comment deleted file mode 100644 index 06f71efba5..0000000000 --- a/doc/bugs/Error_cloning_repository_on_Windows/comment_1_3c95fbb637e8043f988822069518fd3a._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-10-07T16:11:23Z" - content=""" -Thanks for reporting that. We used to have problems with : getting into -filenames in the website, fixed that, so I've gone ahead and fixed this -too, including (I hope) configuring it to reject a trailing period. -"""]] diff --git a/doc/bugs/Error_cloning_repository_on_Windows/comment_2_0f857ecd589d4ff78646f46ddbd78465._comment b/doc/bugs/Error_cloning_repository_on_Windows/comment_2_0f857ecd589d4ff78646f46ddbd78465._comment deleted file mode 100644 index 3639d60f78..0000000000 --- a/doc/bugs/Error_cloning_repository_on_Windows/comment_2_0f857ecd589d4ff78646f46ddbd78465._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="jwodder" - avatar="http://cdn.libravatar.org/avatar/b06e01332c949b895c681cc92934f36a" - subject="comment 2" - date="2020-10-08T12:44:51Z" - content=""" -You overlooked the directory `doc/bugs/Very_not_graceful_exit_on_cwd_deletion_while_git_annex_adding_files.` -"""]] diff --git a/doc/bugs/Error_cloning_repository_on_Windows/comment_3_2e291373a2f3cd3e90e2c0bbad2cda7d._comment b/doc/bugs/Error_cloning_repository_on_Windows/comment_3_2e291373a2f3cd3e90e2c0bbad2cda7d._comment deleted file mode 100644 index dd5a2ab767..0000000000 --- a/doc/bugs/Error_cloning_repository_on_Windows/comment_3_2e291373a2f3cd3e90e2c0bbad2cda7d._comment +++ /dev/null @@ -1,7 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-10-22T17:57:49Z" - content=""" -So I did, fixed now. -"""]] diff --git a/doc/bugs/Error_cloning_repository_on_Windows/comment_4_70595ac6d747eb0cabffb2844fc036a4._comment b/doc/bugs/Error_cloning_repository_on_Windows/comment_4_70595ac6d747eb0cabffb2844fc036a4._comment deleted file mode 100644 index 15826e5aad..0000000000 --- a/doc/bugs/Error_cloning_repository_on_Windows/comment_4_70595ac6d747eb0cabffb2844fc036a4._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="jwodder" - avatar="http://cdn.libravatar.org/avatar/b06e01332c949b895c681cc92934f36a" - subject="comment 4" - date="2020-10-22T18:13:40Z" - content=""" -Unfortunately, it turns out that empty extensions aren't the only problem. Trying to check out the latest commit (577af1b) on Windows 7 now gives: - -``` -error: unable to create file doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_1_8a6cd8dc6e7c2cdefd60a07356be1d27._comment: Filename too long -error: unable to create file doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails_with___34__git__58___createProcess__58___runInteractiveProcess__58___chdir__58___inappropriate_type___40__Not_a_directory__41____34__/comment_3_fd3a287ee6e73946889a4dd406fae0fc._comment: Filename too long -error: unable to create file doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_1_aaea812a8f2b04d9b17d1ba9c6fad39f._comment: Filename too long -error: unable to create file doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_2_c55f2d95f7f08e6f6439f36790e5a538._comment: Filename too long -error: unable to create file doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_3_b49a3df3b13f564e588824910a07bc43._comment: Filename too long -error: unable to create file doc/bugs/crippled_fs___40__pidlock__41___leads_to_git-annex__58___SQLite3_returned_ErrorIO_while_attempting_to_perform_prepare___34__PRAGMA_journal__95__mode__61__WAL__59____34____58___disk_I__47__O_error/comment_4_a9e6e37dbaa1664d825bdbd64de4fbdd._comment: Filename too long -``` -"""]] diff --git a/doc/bugs/Error_cloning_repository_on_Windows/comment_5_6ff246a8cda6bc964fc30e1ccee5ab6f._comment b/doc/bugs/Error_cloning_repository_on_Windows/comment_5_6ff246a8cda6bc964fc30e1ccee5ab6f._comment deleted file mode 100644 index dee9a765e3..0000000000 --- a/doc/bugs/Error_cloning_repository_on_Windows/comment_5_6ff246a8cda6bc964fc30e1ccee5ab6f._comment +++ /dev/null @@ -1,30 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2020-11-02T17:38:34Z" - content=""" -I think windows has a limit of around 256 characters on the total length of -a path. (vs 4096 on linux) - -So, even if ikiwiki had a way to limit the length of a page name, -which could perhaps be done with `wiki_file_regexp`, that would not help, -because several short page names can be nested into a path that ends up too -long for windows. As is the case here; neither of the bug report filenames -is too long, only the comments on them end up too long. Allowing posting a -bug report but not commenting on it would not be good. - -Since I am flatly NOT going to decouple the bug tracking repo from the -source code repo, this seems to put me in the position of going around -cleaning up things that happen to be too long for windows manually. Which I -frankly, also do not want to do. I guess I'll take patches if people want -to send them, but is this really how we want to spend our time? - -(Also, renaming someone's bug report out from under where it was risks them -not seeing a request for more information.) - -Note that, windows provides ways to avoid these length limits, by prefixing -paths with a sequence that avoids using a compatability layer that imposes -them, and instead uses a more modern layer that has more reasonable limits. -I think git-annex actually uses that itself, thanks to some improvements to -ghc. git for windows could be made to also, I suppose. -"""]] diff --git a/doc/bugs/Error_cloning_repository_on_Windows/comment_6_2474b1cc6256afefb3d0530c3a0a24e8._comment b/doc/bugs/Error_cloning_repository_on_Windows/comment_6_2474b1cc6256afefb3d0530c3a0a24e8._comment deleted file mode 100644 index 0ef6e775aa..0000000000 --- a/doc/bugs/Error_cloning_repository_on_Windows/comment_6_2474b1cc6256afefb3d0530c3a0a24e8._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 6""" - date="2020-11-02T19:17:01Z" - content=""" -Hmm, the form has size=40, but people are pasting entire long error -messages into it. Although special characters also being escaped increases -the length, if it were really limited to around that length, it would still -be short enough. - -Ok, added maxlength=50 to it. -"""]] diff --git a/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match.mdwn b/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match.mdwn deleted file mode 100644 index 63c8d16c3c..0000000000 --- a/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match.mdwn +++ /dev/null @@ -1,33 +0,0 @@ -### Please describe the problem. - -Trying to replace my rrsync setup with a restricted git-annex-shell, I ran into the following problem (slightly obfuscated): - - ± git annex get . - get secret (from secrets...) - git-annex-shell: Only allowed to access ~/store not ~/store/secrets/6a8/d0c/GPGHMACSHA1--0000000000000000000000000000000000000000/GPGHMACSHA1--0000000000000000000000000000000000000000 - rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] - rsync error: error in rsync protocol data stream (code 12) at io.c(235) [Receiver=3.1.2] - -It does not matter if GIT_ANNEX_SHELL_DIRECTORY is just "store", "store/secrets" or the full absolute path to any of the two directories, the output and the results are the same. - - -### What steps will reproduce the problem? - -Create an encrypted rsync remote in ~/store/secrets, add - - restrict,command="GIT_ANNEX_SHELL_DIRECTORY=store GIT_ANNEX_SHELL_LIMITED=true GIT_ANNEX_SHELL_READONLY=true git-annex-shell -c \"$SSH_ORIGINAL_COMMAND\"" ssh-rsa ... - -to ~/.ssh/authorized keys and try to retrieve files. - - -### What version of git-annex are you using? On what operating system? - -git-annex version: 6.20180227-g32d682dd8 (standalone version) on Debian stretch - - -### Have you had any luck using git-annex before? - -As always, I'm a fan :> - - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_1_9523af238c6cd2ab66f83474dd580427._comment b/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_1_9523af238c6cd2ab66f83474dd580427._comment deleted file mode 100644 index 9d567f3762..0000000000 --- a/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_1_9523af238c6cd2ab66f83474dd580427._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2018-03-07T19:46:27Z" - content=""" -A rsync special remote needs ssh to run rsync --server, -you have it running git-annex-shell. - -That can't work. It is possible to lock down generic rsync over -ssh to only operate in one directory, but git-annex-shell -is not the tool to do it. - -If there's a bug here, it's whatever led you to expect this would work.. - -(But, if you make a gcrypt special remote, git-annex-shell can be used -to access it, and it'll be all encrypted and files stored with rsync, -and can be locked down.) -"""]] diff --git a/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_2_888d17279dc33de32d7433f1f88e94e1._comment b/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_2_888d17279dc33de32d7433f1f88e94e1._comment deleted file mode 100644 index ba7b1fdedd..0000000000 --- a/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_2_888d17279dc33de32d7433f1f88e94e1._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="https://tribut.de/" - nickname="Felix" - avatar="http://cdn.libravatar.org/avatar/d7508a030fd9902a76b02f7feff1327e80400a1317ee98e463c68ef4c9c40bb8" - subject="comment 2" - date="2018-03-07T20:02:07Z" - content=""" -OK, sorry for making this weird bug report then. I was getting errors like - - rrsync: SSH_ORIGINAL_COMMAND='git-annex-shell' is not rsync - -so I assumed that my rrsync setup was incorrect and maybe it had something to do with your [recent blog](https://git-annex.branchable.com/devblog/day_486__time_to_ditch_rsync/). Unfortunately I cannot reproduce it right now, so I guess I'll wait until it happens again. -"""]] diff --git a/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_3_56eeb148889bc95a464f00208775aae1._comment b/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_3_56eeb148889bc95a464f00208775aae1._comment deleted file mode 100644 index 10ed73864e..0000000000 --- a/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_3_56eeb148889bc95a464f00208775aae1._comment +++ /dev/null @@ -1,34 +0,0 @@ -[[!comment format=mdwn - username="https://tribut.de/" - nickname="Felix" - avatar="http://cdn.libravatar.org/avatar/d7508a030fd9902a76b02f7feff1327e80400a1317ee98e463c68ef4c9c40bb8" - subject="comment 3" - date="2018-03-07T20:15:00Z" - content=""" -Got it to happen at least (though I do not believe this is was caused me to investigate this)... - -This works: - - ± git annex sync --content - commit - On branch master - Your branch is up to date with 'origin/master'. - - nothing to commit, working tree clean - ok - pull origin - ok - -This gives the error: - - ± git annex sync -J4 --content - /usr/local/bin/rrsync: SSH_ORIGINAL_COMMAND='git-annex-shell inannex .' is not rsync - - Unable to run git-annex-shell on remote . - On branch master - Your branch is up to date with 'origin/master'. - - nothing to commit, working tree clean - commit ok - pull origin ok -"""]] diff --git a/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_4_57d694df6c22c42cbdc98ffc5dbc0945._comment b/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_4_57d694df6c22c42cbdc98ffc5dbc0945._comment deleted file mode 100644 index e344830b01..0000000000 --- a/doc/bugs/GIT__95__ANNEX__95__SHELL__95__DIRECTORY_won__39__t_match/comment_4_57d694df6c22c42cbdc98ffc5dbc0945._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2018-03-07T20:25:10Z" - content=""" -Ok, this involves the fairly recently added support for -avoding overlapping ssh password prompts when running multiple ssh -processes concurrently. - -In -J mode, git-annex warms up the ssh connection by running -"git-annex-shell inannex". Your rrsync config prevents that being run. - -This doesn't actually cause a problem, the connection is still warmed up. -If you needed a password to connect, it would have prompted for it before -the error message from rrsync. - -But it's ugly.. I've committed a fix that will avoid the ugly messages. -"""]] diff --git a/doc/bugs/Importfeed_replaces_all___34__.__34___characters_with___34____95____34___in_filename.mdwn b/doc/bugs/Importfeed_replaces_all___34__.__34___characters_with___34____95____34___in_filename.mdwn deleted file mode 100644 index db444ff43d..0000000000 --- a/doc/bugs/Importfeed_replaces_all___34__.__34___characters_with___34____95____34___in_filename.mdwn +++ /dev/null @@ -1,31 +0,0 @@ -### Please describe the problem. - -Beginning with 8.20200522, importfeed replaces all dot (".") characters in filenames with "_". As an example, before that release, the TWIT Security Now podcast had filenames that looked like this: - -SN_767__WiFi_6.mp3 - -After the release, the filenames look like this: - -SN_769__Zoom_s_E2EE_Design_mp3 - -This change occurred across all my downloaded podcasts. - -I suspect the bug has something to do with the following changes described in the news for 8.20200522: - -- addurl, importfeed: Avoid adding filenames with leading '.', instead it will be replaced with '_'. -- addurl, importfeed: Allow '-' in filenames, as long as it's not the first character. - - -### What steps will reproduce the problem? - -git-annex importfeed --fast http://leoville.tv/podcasts/sn.xml - -### What version of git-annex are you using? On what operating system? - -Arch Linux - -git annex version 8.20200720.1-g1ccb6699a1 - - - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/Incorrect_install_dir_for_fish_completion.mdwn b/doc/bugs/Incorrect_install_dir_for_fish_completion.mdwn deleted file mode 100644 index d400715579..0000000000 --- a/doc/bugs/Incorrect_install_dir_for_fish_completion.mdwn +++ /dev/null @@ -1,39 +0,0 @@ -### Please describe the problem. - -[git-annex Makefile: install-completions](http://source.git-annex.branchable.com/?p=source.git;a=blob;f=Makefile;h=965f53e1fc4a8f6d69041eabaccd759268f6490f;hb=HEAD#l87) - -git-annex installs fish completions to the wrong directory. `$(SHAREDIR)/fish/completions` is the directory documented as being exclusive to completions which are shipped in the fish source code; third-party applications installing their own completions are intended to use `$(SHAREDIR)/fish/vendor_completions.d` instead. - -See [https://fishshell.com/docs/current/index.html#completion-path](https://fishshell.com/docs/current/index.html#completion-path) - -Note that this location can also be obtained in a similar manner to bash-completion: - -``` -$ pkg-config bash-completion --variable=completionsdir -/usr/share/bash-completion/completions -``` - -``` -$ pkg-config fish --variable=completionsdir -/usr/share/fish/vendor_completions.d -``` - -### What steps will reproduce the problem? - -Run "make install-completions", or install a linux distribution package of git-annex that builds with the current Makefile (Arch Linux or Debian will both show the same issue). - -### What version of git-annex are you using? On what operating system? - -Arch Linux - -git-annex 7.20191230-7 - -### Please provide any additional information below. - -Apparently this is a very common mistake :/ so far I've seen many more projects do this wrong than do it right. - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Not a user, just here to help improve cross-distro packaging. :) - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/Incorrect_install_dir_for_fish_completion/comment_1_c5601e6da07e39a0c23d29ee0a728970._comment b/doc/bugs/Incorrect_install_dir_for_fish_completion/comment_1_c5601e6da07e39a0c23d29ee0a728970._comment deleted file mode 100644 index 6de076e590..0000000000 --- a/doc/bugs/Incorrect_install_dir_for_fish_completion/comment_1_c5601e6da07e39a0c23d29ee0a728970._comment +++ /dev/null @@ -1,32 +0,0 @@ -[[!comment format=mdwn - username="Chel" - avatar="http://cdn.libravatar.org/avatar/a42feb5169f70b3edf7f7611f7e3640c" - subject="comment 1" - date="2020-02-03T23:34:02Z" - content=""" -I've got an error from `make install` on version 7.20200202.7: - -~~~ -./git-annex --fish-completion-script git-annex 2>/dev/null \ - > dest/usr/local/share/fish/completions/git-annex.fish -make: *** [Makefile:87: install-completions] Error 2 -~~~ - -I think there should be: - -[[!format diff \"\"\" -diff --git a/Makefile b/Makefile -index eb3a34e6a..722921e00 100644 ---- a/Makefile -+++ b/Makefile -@@ -86,7 +86,7 @@ install-completions: build - > $(DESTDIR)$(ZSH_COMPLETIONS_PATH)/_git-annex - install -d $(DESTDIR)$(PREFIX)/$(SHAREDIR)/fish/vendor_completions.d - ./git-annex --fish-completion-script git-annex 2>/dev/null \ -- > $(DESTDIR)$(PREFIX)/$(SHAREDIR)/fish/completions/git-annex.fish -+ > $(DESTDIR)$(PREFIX)/$(SHAREDIR)/fish/vendor_completions.d/git-annex.fish - - test: git-annex git-annex-shell - ./git-annex test -\"\"\"]] -"""]] diff --git a/doc/bugs/Incorrect_install_dir_for_fish_completion/comment_2_7607d3170194f8ad46e0c0178b3ea317._comment b/doc/bugs/Incorrect_install_dir_for_fish_completion/comment_2_7607d3170194f8ad46e0c0178b3ea317._comment deleted file mode 100644 index 87dfd3d2f5..0000000000 --- a/doc/bugs/Incorrect_install_dir_for_fish_completion/comment_2_7607d3170194f8ad46e0c0178b3ea317._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="eschwartz@5abb721e66990e478c7d1caf96beb4f9794eb168" - nickname="eschwartz" - avatar="http://cdn.libravatar.org/avatar/16ec8475b4e3507f8d1a71101c16b208" - subject="Partial fix only." - date="2020-02-03T23:47:35Z" - content=""" -Looks like the install -d was changed to create the new directory, but the actual file write got left left out of the commit. -"""]] diff --git a/doc/bugs/Incorrect_install_dir_for_fish_completion/comment_3_020854d8b66be6b17f850075fa8d95ac._comment b/doc/bugs/Incorrect_install_dir_for_fish_completion/comment_3_020854d8b66be6b17f850075fa8d95ac._comment deleted file mode 100644 index 3e18d4c192..0000000000 --- a/doc/bugs/Incorrect_install_dir_for_fish_completion/comment_3_020854d8b66be6b17f850075fa8d95ac._comment +++ /dev/null @@ -1,7 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-02-04T16:09:04Z" - content=""" -So it did. Fixed now. Thanks for checking. -"""]] diff --git a/doc/bugs/Missing_sanity_check_of_group_names._Special_group_names_should_not_be_allowed.mdwn b/doc/bugs/Missing_sanity_check_of_group_names._Special_group_names_should_not_be_allowed.mdwn deleted file mode 100644 index bf2972257b..0000000000 --- a/doc/bugs/Missing_sanity_check_of_group_names._Special_group_names_should_not_be_allowed.mdwn +++ /dev/null @@ -1,37 +0,0 @@ -### Please describe the problem. - -As documented in [[git-annex-matching-options]], `--copies` accepts `trustlevel` or `groupname` in the following format: - -* `--copies=trustlevel:number` -* `--copies=groupname:number` - -This is ambiguous in the unlikely case where a user might come up with the idea to create a group called `trusted`, `semitrusted` or `untrusted` (`remote..annex-trustlevel`). It should thereby not be allowed to use such special groups. - -### What steps will reproduce the problem? - - -[[!format sh """ -## In an annex repo: - -git annex group here trusted - -## What does this command do now? -## For reference, it interprets it as `--copies=trustlevel:number` -git annex find --copies=trusted:1 - -"""]] - -### What version of git-annex are you using? On what operating system? - - git-annex version: 6.20180509 - build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite - dependency versions: aws-0.14.1 bloomfilter-2.0.1.0 cryptonite-0.20 DAV-1.3.1 feed-0.3.11.1 ghc-8.0.1 http-client-0.4.31.1 persistent-sqlite-2.6 torrent-10000.0.0 uuid-1.3.12 yesod-1.4.3 - key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL - remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external - operating system: linux x86_64 - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Yes! Thanks so much for your work! - -> Closing per comments. [[done]] --[[Joey]] diff --git a/doc/bugs/Missing_sanity_check_of_group_names._Special_group_names_should_not_be_allowed/comment_1_b4ba41b6e0e611d90ddda92bbe991266._comment b/doc/bugs/Missing_sanity_check_of_group_names._Special_group_names_should_not_be_allowed/comment_1_b4ba41b6e0e611d90ddda92bbe991266._comment deleted file mode 100644 index 4bd9247927..0000000000 --- a/doc/bugs/Missing_sanity_check_of_group_names._Special_group_names_should_not_be_allowed/comment_1_b4ba41b6e0e611d90ddda92bbe991266._comment +++ /dev/null @@ -1,22 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2018-06-04T16:06:57Z" - content=""" -Thanks for pointing out this ambiguity, which I don't remember having -ever considered. - -The code for this does look for the name of a trust level first, -and only a group name if it's not a trust level. So, adding a group -with a colliding name will never change the behavior of such a command -or preferred content expression. - -The only problem then would be that you couldn't match -on a group by that name. But if you run into that problem, -you can simply rename your group. - -So I don't think that git-annex needs a sanity check, really. - -(I have added a comment in the code to make clear that the order of -parsing this does matter.) -"""]] diff --git a/doc/bugs/Missing_sanity_check_of_group_names._Special_group_names_should_not_be_allowed/comment_2_73b2d20a793a8a17cf0a3d593b3b4ed4._comment b/doc/bugs/Missing_sanity_check_of_group_names._Special_group_names_should_not_be_allowed/comment_2_73b2d20a793a8a17cf0a3d593b3b4ed4._comment deleted file mode 100644 index c9b6f38a71..0000000000 --- a/doc/bugs/Missing_sanity_check_of_group_names._Special_group_names_should_not_be_allowed/comment_2_73b2d20a793a8a17cf0a3d593b3b4ed4._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="ypid" - avatar="http://cdn.libravatar.org/avatar/4286841cddc77176758dde874afe9a3a" - subject="comment 2" - date="2018-06-04T19:42:30Z" - content=""" -Works for me, thanks. I also think this is not a real issue, I just noticed the edge case. -"""]] diff --git a/doc/bugs/Parallel_fsck_on_files_with_same_content_in_bup_remote_can_fail.mdwn b/doc/bugs/Parallel_fsck_on_files_with_same_content_in_bup_remote_can_fail.mdwn deleted file mode 100644 index d631ae5a4f..0000000000 --- a/doc/bugs/Parallel_fsck_on_files_with_same_content_in_bup_remote_can_fail.mdwn +++ /dev/null @@ -1,149 +0,0 @@ -### Please describe the problem. - -If there are multiple files with the same keys in the repository and they are copied to bup special remote, -then `git annex fsck --from=bup` with `--jobs=N` option (N >= 2) can show an error and remove these keys from bup. - -Based on the error message (about locked .git/annex/tmp/ file), this problem is probably not specific to bup, -but I tested it with bup only. - -### What steps will reproduce the problem? - -1. Configure a bup special remote. -2. Add files with the same content to annex (and with the same backend). -3. Copy these files to bup. -4. Run `git annex fsck --from=bup -JN` several times, until it removes these keys from bup. - -### What version of git-annex are you using? On what operating system? - -git-annex 7.20191230-g985373f8e, build from source, on Debian GNU/Linux buster. - -bup 0.29.3-2 from Debian sid. Also tried with bup 0.30, build from source. - -### Please provide any additional information below. - -[[!format txt """ -~ $ mkdir testdir -~ $ cd testdir -~/testdir $ -~/testdir $ git init -Initialized empty Git repository in /home/test/testdir/.git/ -~/testdir $ -~/testdir $ git annex init testrepo -init testrepo (scanning for unlocked files...) -ok -(recording state in git...) -~/testdir $ -~/testdir $ ls ~/.bup/index-cache/ -~/testdir $ -~/testdir $ git annex initremote bup type=bup buprepo=~/testdir/.bup encryption=none -initremote bup (bup init...) -Reinitialized existing Git repository in /home/test/.bup/ -Initialized empty Git repository in /home/test/testdir/.bup/ -ok -(recording state in git...) -~/testdir $ -~/testdir $ ls ~/.bup/index-cache/ -None__home_test_testdir__bup -~/testdir $ -~/testdir $ echo aaa >file1 -~/testdir $ echo aaa >file2 -~/testdir $ -~/testdir $ git annex add . -add file1 -ok -add file2 -ok -(recording state in git...) -~/testdir $ -~/testdir $ git commit -m files -[master (root-commit) 7a03b66] files - 2 files changed, 2 insertions(+) - create mode 120000 file1 - create mode 120000 file2 -~/testdir $ -~/testdir $ git -C .bup show-ref -~/testdir $ -~/testdir $ git annex whereis -whereis file1 (1 copy) - 5d9b0df2-000b-4273-bc4a-fb3b9d8319bd -- testrepo [here] -ok -whereis file2 (1 copy) - 5d9b0df2-000b-4273-bc4a-fb3b9d8319bd -- testrepo [here] -ok -~/testdir $ -~/testdir $ git annex copy --to=bup . -copy file1 (to bup...) - -bloom: creating from 1 file (3 objects).ing: 0 kbytes -Receiving index from server: 1156/1156, done. -bloom: creating from 1 file (3 objects). -ok -copy file2 ok -(recording state in git...) -~/testdir $ -~/testdir $ git annex lookupkey file1 file2 -SHA256E-s4--17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76 -SHA256E-s4--17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76 -~/testdir $ -~/testdir $ git -C .bup show-ref -2076647ee23ad632c8cf96caf51febbd0604452c refs/heads/SHA256E-s4--17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76 -~/testdir $ -~/testdir $ git annex fsck --from=bup -fsck file1 -(checksum...) ok -fsck file2 -(checksum...) ok -(recording state in git...) -~/testdir $ -~/testdir $ git -C .bup show-ref -2076647ee23ad632c8cf96caf51febbd0604452c refs/heads/SHA256E-s4--17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76 -"""]] - -Now run `git annex fsck --from=bup -J2` multiple times, until it drops the key from bup... - -[[!format txt """ -~/testdir $ git annex fsck --from=bup -J2 -fsck file1 fsck file2 - -100% 4 B 5 B/s 0s - content cannot be completely removed from bup remote - - file2: Bad file size (4 B smaller); dropped from bup -(checksum...) -git-annex: .git/annex/tmp/fsck14654.SHA256E-s4--17e682f060b5f8e47ea04c5c4855908b0a5ad612022260fe50e11ecb0cc0ab76: openBinaryFile: resource busy (file is locked) -failed -(fixing location log) (checksum...) ok -(recording state in git...) -git-annex: fsck: 1 failed -~/testdir $ -~/testdir $ git -C .bup show-ref -~/testdir $ -~/testdir $ git annex whereis -whereis file1 (2 copies) - 5d9b0df2-000b-4273-bc4a-fb3b9d8319bd -- testrepo [here] - 88cc362a-f87a-43c7-b194-e79b2ee91828 -- [bup] -ok -whereis file2 (2 copies) - 5d9b0df2-000b-4273-bc4a-fb3b9d8319bd -- testrepo [here] - 88cc362a-f87a-43c7-b194-e79b2ee91828 -- [bup] -ok -~/testdir $ -~/testdir $ git annex fsck --from=bup -fsck file1 (fixing location log) - ** Based on the location log, file1 - ** was expected to be present, but its content is missing. -failed -fsck file2 ok -(recording state in git...) -git-annex: fsck: 1 failed -~/testdir $ -~/testdir $ git annex whereis -whereis file1 (1 copy) - 5d9b0df2-000b-4273-bc4a-fb3b9d8319bd -- testrepo [here] -ok -whereis file2 (1 copy) - 5d9b0df2-000b-4273-bc4a-fb3b9d8319bd -- testrepo [here] -ok -"""]] - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/Parallel_fsck_on_files_with_same_content_in_bup_remote_can_fail/comment_1_8cc0d742cd59046b038879f4823c9639._comment b/doc/bugs/Parallel_fsck_on_files_with_same_content_in_bup_remote_can_fail/comment_1_8cc0d742cd59046b038879f4823c9639._comment deleted file mode 100644 index 755f2bbf5f..0000000000 --- a/doc/bugs/Parallel_fsck_on_files_with_same_content_in_bup_remote_can_fail/comment_1_8cc0d742cd59046b038879f4823c9639._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-02-14T18:49:07Z" - content=""" -Ugh, I think this could potentially result in data loss. Not when using bup, -but other special remotes. - -I've fixed it in git and will think about moving the date of the next -release up. -"""]] diff --git a/doc/bugs/Q__58___is_there_a_reason_to___34__sense__34___git_annex_post-receive_in_a_hook__63__.mdwn b/doc/bugs/Q__58___is_there_a_reason_to___34__sense__34___git_annex_post-receive_in_a_hook__63__.mdwn deleted file mode 100644 index 6705c87307..0000000000 --- a/doc/bugs/Q__58___is_there_a_reason_to___34__sense__34___git_annex_post-receive_in_a_hook__63__.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -I looked into post-receive hook of an arbitrary git/git-annex repo to find - -[[!format sh """ -(git-annex)hopa:/tmp/testds/.git/hooks[master] -$> cat post-receive -#!/bin/sh -# automatically configured by git-annex -if git annex post-receive --help >/dev/null 2>&1; then git annex post-receive; fi -"""]] - -and wondered why it actually needs conditioning? `--help` runs for me longer (40-50ms) than actual `git annex post-receive` call (20ms) when there is nothing todo. So I wondered why wasting time on `--help`? - -[[!meta author=yoh]] - -> [[done]], there does not seem to be a bug here. --[[Joey]] diff --git a/doc/bugs/Q__58___is_there_a_reason_to___34__sense__34___git_annex_post-receive_in_a_hook__63__/comment_1_dabe4b26c60d9d7d392e441480a1ef07._comment b/doc/bugs/Q__58___is_there_a_reason_to___34__sense__34___git_annex_post-receive_in_a_hook__63__/comment_1_dabe4b26c60d9d7d392e441480a1ef07._comment deleted file mode 100644 index a55abe1294..0000000000 --- a/doc/bugs/Q__58___is_there_a_reason_to___34__sense__34___git_annex_post-receive_in_a_hook__63__/comment_1_dabe4b26c60d9d7d392e441480a1ef07._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-09-27T16:57:30Z" - content=""" -Old versions of git-annex didn't have a post-receive command, -so initailizing with a new git-annex and then using an old one would be a -problem without this check. -"""]] diff --git a/doc/bugs/Q__58___is_there_a_reason_to___34__sense__34___git_annex_post-receive_in_a_hook__63__/comment_2_07f9665bf0acea333dafb2db81e87619._comment b/doc/bugs/Q__58___is_there_a_reason_to___34__sense__34___git_annex_post-receive_in_a_hook__63__/comment_2_07f9665bf0acea333dafb2db81e87619._comment deleted file mode 100644 index 22a84f2893..0000000000 --- a/doc/bugs/Q__58___is_there_a_reason_to___34__sense__34___git_annex_post-receive_in_a_hook__63__/comment_2_07f9665bf0acea333dafb2db81e87619._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2019-09-30T17:36:40Z" - content=""" -Of course if you're sure you'll never use an old version of git-annex -(or don't care if the hook breaks if you do), you can modify the hook -script. - -I don't see any changes I should make here, can this bug report be closed? -"""]] diff --git a/doc/bugs/Remote_Tests__58___storeKey_fails_for_http_remotes___40__skip_instead__63____41__.mdwn b/doc/bugs/Remote_Tests__58___storeKey_fails_for_http_remotes___40__skip_instead__63____41__.mdwn deleted file mode 100644 index 217385e652..0000000000 --- a/doc/bugs/Remote_Tests__58___storeKey_fails_for_http_remotes___40__skip_instead__63____41__.mdwn +++ /dev/null @@ -1,85 +0,0 @@ -### Please describe the problem. -When running `git annex testremote origin --test-readonly=filename` on a git http remote that supports git-annex, the `storeKey` test fails with error: - -``` -storeKey: FAIL -./Command/TestRemote.hs:277: -(got: Left "copying to non-ssh repo not supported") -``` - -(Full example output below) - - -### What steps will reproduce the problem? - -Using https://downloads.kitenet.net/.git/ as an example of a public repository with https git annex support: - -``` -$ git clone https://downloads.kitenet.net/.git/ -Cloning into 'downloads.kitenet.net'... -$ cd downloads.kitenet.net/debug-me/linux/current/ -$ git annex get debug-me-standalone-amd64.tar.gz -get debug-me-standalone-amd64.tar.gz (from origin...) -(checksum...) ok -(recording state in git...) -$ git annex testremote origin --test-readonly debug-me-standalone-amd64.tar.gz -testremote origin Remote Tests - unavailable remote - removeKey: dropping from http remote not supported -OK - storeKey: FAIL - ./Command/TestRemote.hs:277: - (got: Left "copying to non-ssh repo not supported") - checkPresent: OK (0.02s) - retrieveKeyFile: download failed: ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 6, addrAddress = , addrCanonName = }, host name: Just "!dne!", service name: Just "443"): does not exist (Name or service not known) - download failed: ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 6, addrAddress = , addrCanonName = }, host name: Just "!dne!", service name: Just "443"): does not exist (Name or service not known) -OK (0.01s) - retrieveKeyFileCheap: OK - key size Just 13600699; NoChunks; encryption none - present True: OK (0.61s) - retrieveKeyFile: OK (2.39s) - fsck downloaded object: OK - retrieveKeyFile resume from 33%: OK (1.95s) - fsck downloaded object: OK - retrieveKeyFile resume from 0: OK (1.94s) - fsck downloaded object: OK - retrieveKeyFile resume from end: OK (0.68s) - fsck downloaded object: OK - -1 out of 14 tests failed (7.61s) -failed -git-annex: testremote: 1 failed -``` - -### What version of git-annex are you using? On what operating system? - -``` -git-annex version: 7.20191230-g985373f8e -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.2.0.1 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.10.5 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: linux x86_64 -supported repository versions: 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -``` - -OS: Arch Linux (5.5.1-arch1-1) - -### Please provide any additional information below. - -I came across this while trying to make our hosted git and git-annex service (gin.g-node.org) open to public https git annex downloads. I was using the `testremote` command to make sure everything works as intended and saw the error thinking, at first, that the server was serving something incorrectly. The error message does clearly state that `copying to non-ssh repo not supported`, so I thought I'd try with kitenet to see if the same happens and it does. - -I don't know if this is a bug or intentional—perhaps the test should be failing to indicate that the remote doesn't support `storeKey`? On the other hand, the `removeKey` test displays the "not supported" message and then passes, so maybe the `storeKey` test should behave the same way. - -It's possible there's another issue here I'm not entirely aware of. - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Of course! I use and rely on it daily :) - -> This is not a bug in the test suite, it turns out, but in -> git-annex's handling of a http remote. It was throwing fatal errors -> rather than the correct behavior of displaying a warning. [[fixed|done]] -> --[[Joey]] diff --git a/doc/bugs/Running_fsck_on_a_remote_drops_bad_files.mdwn b/doc/bugs/Running_fsck_on_a_remote_drops_bad_files.mdwn deleted file mode 100644 index 1fe67798e7..0000000000 --- a/doc/bugs/Running_fsck_on_a_remote_drops_bad_files.mdwn +++ /dev/null @@ -1,39 +0,0 @@ -### Please describe the problem. -Running fsck on a remote completely drops the file - -### What steps will reproduce the problem? -* Create a standard git annex repo -* Set up another remote (I used a standard git annex repo) in it -* Corrupt a file in the remote repo. Changing one byte is sufficient for this -* Run `git-annex fsck` -* git-annex will notice the corruption and completely drop the file from the remote - -I expect git-annex to never drop data unless specified as also mentioned [here](https://git-annex.branchable.com/walkthrough/fsck__58___when_things_go_wrong) - -I would rather look at the corrupted file myself and figure out the best course of action instead of losing it completely. - -At this point, I'm working around it by not running fsck on remotes, since local fsck seems to work as expected. - - -### What version of git-annex are you using? On what operating system? -8.20200309, Mac - - -### Please provide any additional information below. - -~~~ -$ git-annex fsck --from=origin -fsck dir1/dir1_file1 (checksum...) ok -fsck dir1/dir1_file2 (checksum...) ok -fsck file1 (checksum...) - file1: Bad file content; dropped from origin -failed -fsck file2 (checksum...) ok -(recording state in git...) -git-annex: fsck: 1 failed -~~~ - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -Yes it stores most of my data - -> [[notabug|done]] --[[Joey]] diff --git a/doc/bugs/Running_fsck_on_a_remote_drops_bad_files/comment_1_c75030329f9769e557aba68193ce1801._comment b/doc/bugs/Running_fsck_on_a_remote_drops_bad_files/comment_1_c75030329f9769e557aba68193ce1801._comment deleted file mode 100644 index 61476408ce..0000000000 --- a/doc/bugs/Running_fsck_on_a_remote_drops_bad_files/comment_1_c75030329f9769e557aba68193ce1801._comment +++ /dev/null @@ -1,21 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-06-30T14:33:08Z" - content=""" -The corrupted file is downloaded from the remote, and that copy of the file -gets moved to .git/annex/bad/ when fsck determines it's corrupt. A message -is displayed giving the path where the content is left. - -The only time that's not done is if the local repository has its own copy -of the file. In that case, there's not much benefit to accumulating bad -copies in .git/annex/bad since there's still a (presumably) good copy. - -If fsck left corrupted content on the remote, that would risk actual data -loss. Consider if a second repository also had access to the remote, and -in that repository you ran git-annex drop. It would see that the remote -still contains the file, so assume the content is good, and so go ahead and -drop the content from the second repository. That might be the only -uncorrupted copy of the file it's lost. So once we know that the content is -corrupt, the safe thing to do is move it out of the way. -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines.mdwn b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines.mdwn deleted file mode 100644 index 9bc2aed3a0..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines.mdwn +++ /dev/null @@ -1,69 +0,0 @@ -### Please describe the problem. - -I created a dataset with datalad and added multiple remotes/buckets on a private S3-like server (minio). -On the HPC I am able to get the data and push it. -On another machine, I cannot get the data, here are the debug info. - - -$ git-annex get -v -d --from s3unf.phantom.anat.mri sub-01/ses-ana001/anat/sub-01_ses-ana001_T1w.nii.gz -[2020-02-21 16:20:09.399086] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] -[2020-02-21 16:20:09.405552] process done ExitSuccess -[2020-02-21 16:20:09.405884] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] -[2020-02-21 16:20:09.411446] process done ExitSuccess -[2020-02-21 16:20:09.413223] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..050f99f2712d25b581b58a3adc6af83b8d5345b0","--pretty=%H","-n1"] -[2020-02-21 16:20:09.419815] process done ExitSuccess -[2020-02-21 16:20:09.420835] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"] -[2020-02-21 16:20:09.421976] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] -[2020-02-21 16:20:09.456813] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"] -[2020-02-21 16:20:09.462354] process done ExitSuccess -[2020-02-21 16:20:09.462577] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"] -[2020-02-21 16:20:09.468639] process done ExitSuccess -[2020-02-21 16:20:09.468923] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--","sub-01/ses-ana001/anat/sub-01_ses-ana001_T1w.nii.gz"] -get sub-01/ses-ana001/anat/sub-01_ses-ana001_T1w.nii.gz (from s3unf.phantom.anat.mri...) -[2020-02-21 16:20:09.585446] String to sign: "GET\n\n\nFri, 21 Feb 2020 21:20:09 GMT\n/phantom.anat.mri/SHA256E-s28509481-S1073741824-C1--7eb345c88a120d934cb390ad0385149cb6ae8540a2ed6689251cba22b7832306.nii.gz" -[2020-02-21 16:20:09.585844] Host: "" -[2020-02-21 16:20:09.586001] Path: "/phantom.anat.mri/SHA256E-s28509481-S1073741824-C1--7eb345c88a120d934cb390ad0385149cb6ae8540a2ed6689251cba22b7832306.nii.gz" -[2020-02-21 16:20:09.586261] Query string: "" -[2020-02-21 16:20:09.586423] Header: [("Date","Fri, 21 Feb 2020 21:20:09 GMT"),("Authorization","AWS manager:k/aaaaaaaaaaaaaaaaaaaaaa=")] -[2020-02-21 16:20:09.593542] Response metadata: S3: request ID=, x-amz-id-2= -[2020-02-21 16:20:09.594433] String to sign: "GET\n\n\nFri, 21 Feb 2020 21:20:09 GMT\n/phantom.anat.mri/SHA256E-s28509481--7eb345c88a120d934cb390ad0385149cb6ae8540a2ed6689251cba22b7832306.nii.gz" -[2020-02-21 16:20:09.594667] Host: "" -[2020-02-21 16:20:09.594819] Path: "/phantom.anat.mri/SHA256E-s28509481--7eb345c88a120d934cb390ad0385149cb6ae8540a2ed6689251cba22b7832306.nii.gz" -[2020-02-21 16:20:09.595006] Query string: "" -[2020-02-21 16:20:09.595153] Header: [("Date","Fri, 21 Feb 2020 21:20:09 GMT"),("Authorization","AWS manager:wwwwwwwwwwwwwwwwwww ")] -[2020-02-21 16:20:09.596938] Response metadata: S3: request ID=, x-amz-id-2= -failed -[2020-02-21 16:20:09.600838] process done ExitSuccess -[2020-02-21 16:20:09.601651] process done ExitSuccess -git-annex: get: 1 failed - - -When running that command, no request is sent to the server!? (in minio trace logs), but I don't know why. - - - -### What steps will reproduce the problem? - - - -### What version of git-annex are you using? On what operating system? - -latest package in conda-forge on ubuntu - - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Yes. - -> [[closing|done]] since bug reporter says it works for them now after some -> change, and is not responding to my followups. --[[Joey]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_10_41720da2eebb09b23cc739a45ca5d653._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_10_41720da2eebb09b23cc739a45ca5d653._comment deleted file mode 100644 index 001986ae4d..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_10_41720da2eebb09b23cc739a45ca5d653._comment +++ /dev/null @@ -1,53 +0,0 @@ -[[!comment format=mdwn - username="basile.pinsard@f1a7fae9f3bd9d5282fca11f62ad53b45a8eb317" - nickname="basile.pinsard" - avatar="http://cdn.libravatar.org/avatar/87e1f73acf277ad0337b90fc0253c62e" - subject="comment 10" - date="2020-02-28T15:44:50Z" - content=""" -@joey - -I am now trying from scratch, created a new repository and trying to create the remote from the server to the S3 on the same network. - -``` -git-annex config --set annex.security.allowed-ip-addresses 10.10.10.20 -git annex initremote -d s3.test type=S3 encryption=none autoenable=true host=s3.unf-montreal.ca port=443 protocol=https chunk=1GiB bucket=test requeststyle=path -``` - -I get the following error despite relaxing the security option. (same thing with 'all' instead of 10.10.10.20). - -Maybe this security check is broken, and this is why I have problem just getting data from the local network? - - -``` - (InternalException (ConnectionRestricted \"Configuration of annex.security.allowed-ip-addresses does not allow accessing address 10.10.10.20\")) -``` - -Also, when I add the option exporttree=no - -``` -git-annex: Unexpected parameters: exporttree -failed -``` - -while the option exists in the doc, and I used it in the past. - -Version - -``` -$ git-annex version -git-annex version: 7.20200219-g5094a3f -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: linux x86_64 -supported repository versions: 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -local repository version: 7 -``` - - -Thank you for your help. - -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_11_8badfb9c3e63f198d0e115d6ef64176d._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_11_8badfb9c3e63f198d0e115d6ef64176d._comment deleted file mode 100644 index 8ec95ca9b5..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_11_8badfb9c3e63f198d0e115d6ef64176d._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 11""" - date="2020-02-28T17:51:47Z" - content=""" -You need to use git config to set that, not git-annex config. - -git-annex config can only be used with a few carefully -chosen configuration settings that make sense to -apply to all clones of the repository. annex.security.allowed-ip-addresses -is not amoung those. This is clearly documented in the git-annex config man -page IMHO, but I'm always interested to learn about problems with -documentation. -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_12_e820e19c1737ecd0740800483e51df00._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_12_e820e19c1737ecd0740800483e51df00._comment deleted file mode 100644 index ffc2967b4d..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_12_e820e19c1737ecd0740800483e51df00._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 12""" - date="2020-02-28T17:55:48Z" - content=""" -I'd be interested in a separate bug report about exporttree=no not being -accepted by git-annex. (I just tried it myself with a S3 remote and it was -accepted, but maybe I don't completely understand what you did.) -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_13_6b13804fc0e8e55691a86a5f9a2c8568._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_13_6b13804fc0e8e55691a86a5f9a2c8568._comment deleted file mode 100644 index 4c8e282428..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_13_6b13804fc0e8e55691a86a5f9a2c8568._comment +++ /dev/null @@ -1,31 +0,0 @@ -[[!comment format=mdwn - username="basile.pinsard@f1a7fae9f3bd9d5282fca11f62ad53b45a8eb317" - nickname="basile.pinsard" - avatar="http://cdn.libravatar.org/avatar/87e1f73acf277ad0337b90fc0253c62e" - subject="comment 13" - date="2020-02-28T20:20:09Z" - content=""" -> You need to use git config to set that, not git-annex config. - -Unless I am getting crazy, I am pretty sure I managed to make that command work before with git-annex (the --set option is not in the git config syntax), so I wrote it in my own doc a few months back. In the git-repo I ran it before, the .git/config file contains the option. - -The command returns a nice message that made me think it worked: - -``` -$ git-annex config --set annex.security.allowed-ip-addresses 10.10.10.20 -annex.security.allowed-ip-addresses 10.10.10.20 ok -(recording state in git...) -``` - -- About the doc, I am not sure if I got the command from the man page. Maybe an example of how to set these variables with git config (`git config --add `) in the man page would avoid this kind of mistakes from dumb users like me ;). Maybe I thought that git-annex-config also change .git/config file. - -- The ConnectionRestricted for annex.security.allowed-ip-addresses do not show-up when trying to get data from a clone of the git where the remote was configured after correctly setting that option. That is why there was no traffic out except the DNS request. - -- Maybe git-annex config should not return a nice message when trying to set an unknown option. - -I will open a separate bug for the exporttree shortly. - -Now it works! -Thanks for your help! - -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_14_a908b0e465a08d60030c3e401022dad7._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_14_a908b0e465a08d60030c3e401022dad7._comment deleted file mode 100644 index 841f0dc086..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_14_a908b0e465a08d60030c3e401022dad7._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 14""" - date="2020-03-02T19:03:44Z" - content=""" -It's true that you can currently use git-annex config to set absolutely any -value, but git-annex only looks at the values that are listed in the -documentation. - -I suppose I'll change it to reject other values. - ----- - -So, you say it works, but I still don't understand the behavior you -originally reported. Have you somehow fixed it? Or just not reproduced it -with your new from scratch repository? -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_1_d8395bbf0e512c09f6e5a668cdff6313._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_1_d8395bbf0e512c09f6e5a668cdff6313._comment deleted file mode 100644 index 46aabe316b..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_1_d8395bbf0e512c09f6e5a668cdff6313._comment +++ /dev/null @@ -1,21 +0,0 @@ -[[!comment format=mdwn - username="basile.pinsard@f1a7fae9f3bd9d5282fca11f62ad53b45a8eb317" - nickname="basile.pinsard" - avatar="http://cdn.libravatar.org/avatar/87e1f73acf277ad0337b90fc0253c62e" - subject="no traffic getting out" - date="2020-02-25T16:55:23Z" - content=""" -Things are getting weirder. -There is no traffic coming out of my machine (`tcpdump`) when running the `git-annex get`. -When I run curl pointing to the same https address, it works, the server receives the request. -I tried the standalone version, different versions from ubuntu repos, conda-forge, all the same on Ubuntu 18.04 machines, finally from a docker-container with anaconda (debian-based I think), same thing. - -When I connect to my cellphone as a hotspot, it works! -Is there any way to have more debug from the S3 remote code, or from haskell S3 or network packages? - -I am loosing my mind here. -Help :/ - -Thanks! - -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_1_f4046c8ff337e823d7b69e42feefdc11._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_1_f4046c8ff337e823d7b69e42feefdc11._comment deleted file mode 100644 index 161ced293c..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_1_f4046c8ff337e823d7b69e42feefdc11._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-02-25T17:24:51Z" - content=""" -It certianly seems to have sent a request to *somewhere*, -since it gets back a response: - - [2020-02-21 16:20:09.596938] Response metadata: S3: request ID=, x-amz-id-2= - -Kind of looks like that response is not the response that a S3 server -should generate, since it's missing the usual metadata. -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_3_6a75b7e50a4d40134a58d3c8e60073ed._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_3_6a75b7e50a4d40134a58d3c8e60073ed._comment deleted file mode 100644 index 83b277f7fd..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_3_6a75b7e50a4d40134a58d3c8e60073ed._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-02-25T17:35:02Z" - content=""" -Unfortunately --debug is all the debug info that's available. - -One possibility that comes to mind is a http proxy. Perhaps curl is -using some proxy that is necessary to get data out, and git-annex is not. - -By default, git-annex avoids connections to local IP adddresses, -for security reasons. If you have a local http proxy, it won't use it -either. You could try this: - - git config annex.security.allowed-ip-addresses all -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_4_6d1b19d795f5d335c710b00874906d07._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_4_6d1b19d795f5d335c710b00874906d07._comment deleted file mode 100644 index 2ca27259da..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_4_6d1b19d795f5d335c710b00874906d07._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-02-25T17:45:07Z" - content=""" -Hmm, git-annex does display a big warning message if there's a local proxy -that its security does not allow it to use. - -Still, annex.security.allowed-ip-addresses seems like the only likely -reason git-annex would not connect to a server while curl does. -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_5_6d1d45fee3dd4ae82b5791335fb548b0._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_5_6d1d45fee3dd4ae82b5791335fb548b0._comment deleted file mode 100644 index f3c7b0cf20..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_5_6d1d45fee3dd4ae82b5791335fb548b0._comment +++ /dev/null @@ -1,20 +0,0 @@ -[[!comment format=mdwn - username="basile.pinsard@f1a7fae9f3bd9d5282fca11f62ad53b45a8eb317" - nickname="basile.pinsard" - avatar="http://cdn.libravatar.org/avatar/87e1f73acf277ad0337b90fc0253c62e" - subject="comment 5" - date="2020-02-26T13:46:22Z" - content=""" -Yes the local IP is already listed in the security parameters, the error message was quite informative. -There are no proxy configured on any of the machines. -tcpdump should show me any data going out to the server. - -I looked at the AWS.S3 package, and it seems some bucket resolve function might rely on the DNS, correct me if I am wrong. -The requeststyle is set to path as this is what minio can provide, so the -Could it be that the local DNS do not support some functions that S3 rely on? -However, I already created remotes and pushed data from these servers in the past and I don't think the DNS server has changed since. - -As the hotspot test shows, this is an interaction of the local network configuration with git-annex+dependencies code, but I can't figure out how to debug that. - -Thanks. -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_6_f9880977cd07ed71e9515c45b10d09c9._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_6_f9880977cd07ed71e9515c45b10d09c9._comment deleted file mode 100644 index 158168d3d9..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_6_f9880977cd07ed71e9515c45b10d09c9._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="basile.pinsard@f1a7fae9f3bd9d5282fca11f62ad53b45a8eb317" - nickname="basile.pinsard" - avatar="http://cdn.libravatar.org/avatar/87e1f73acf277ad0337b90fc0253c62e" - subject="strace" - date="2020-02-26T15:50:50Z" - content=""" -Running strace to monitor network, I get the following errors regarding nscd, though this socket is not here either when running the hotspot test: -``` -[pid 23666] connect(18, {sa_family=AF_UNIX, sun_path=\"/var/run/nscd/socket\"}, 110) = -1 ENOENT (No such file or directory) -[pid 23666] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 17 -[pid 23666] connect(17, {sa_family=AF_UNIX, sun_path=\"/var/run/nscd/socket\"}, 110) = -1 ENOENT (No such file or directory) -[pid 23666] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 17 -[pid 23666] connect(17, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr(\"127.0.0.53\")}, 16) = 0 -``` -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_7_157d54b3858b7841e1aa366c90736202._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_7_157d54b3858b7841e1aa366c90736202._comment deleted file mode 100644 index 12b671996d..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_7_157d54b3858b7841e1aa366c90736202._comment +++ /dev/null @@ -1,27 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 7""" - date="2020-02-26T17:58:05Z" - content=""" -DNS is typically needed yes. That it gets a http response -from some server seems to imply it's able to resolve the IP address of -the server. So I'd guess DNS is not the problem, unless it's resolving the -*wrong* IP address and connecting to the wrong server. Which would be -consistent with tcpdump not showing any traffic to the server. - -ncsd can be configured on some systems to do DNS lookups. It's normal -for glibc to try to connect to nscd for a DNS lookup and fall back to a -regular lookup when it's not installed. That seems to be all your strace is -showing. - -The next line of your strace shows it connecting to 127.0.0.53 -That seems to be an address systemd sets up to use resolvd for dns. -That connection seems to succeed, so I guess dns resolution proceeds using -resolved. - -All this DNS stuff is completely normal glibc name resolution as far as I -can see. - -My suggestion is to tcpdump the entire network traffic when git-annex is -running, and analize that. -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_8_dd40cb01999b4d05db0df84c07b3d67b._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_8_dd40cb01999b4d05db0df84c07b3d67b._comment deleted file mode 100644 index 79fffe6da2..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_8_dd40cb01999b4d05db0df84c07b3d67b._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="basile.pinsard@f1a7fae9f3bd9d5282fca11f62ad53b45a8eb317" - nickname="basile.pinsard" - avatar="http://cdn.libravatar.org/avatar/87e1f73acf277ad0337b90fc0253c62e" - subject="comment 8" - date="2020-02-26T19:11:53Z" - content=""" -The only relevant traffic when running the `git-annex get` is the DNS call for the A record, the answer from the DNS is right (and AAAA record but no IPV6 on our network so 0 results). -"""]] diff --git a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_9_ce0d2cd9b4a1ca205144d294c1e021da._comment b/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_9_ce0d2cd9b4a1ca205144d294c1e021da._comment deleted file mode 100644 index 0760801d57..0000000000 --- a/doc/bugs/S3-remote_fail_to_get_data_on_some_clones__47__machines/comment_9_ce0d2cd9b4a1ca205144d294c1e021da._comment +++ /dev/null @@ -1,21 +0,0 @@ -[[!comment format=mdwn - username="basile.pinsard@f1a7fae9f3bd9d5282fca11f62ad53b45a8eb317" - nickname="basile.pinsard" - avatar="http://cdn.libravatar.org/avatar/87e1f73acf277ad0337b90fc0253c62e" - subject="comment 9" - date="2020-02-26T19:15:11Z" - content=""" -The only difference in the `dig` answer when running from the outside is additional answer sections: -``` -;; AUTHORITY SECTION: -. 3600 IN NS ns22.domaincontrol.com. -. 3600 IN NS ns21.domaincontrol.com. - -;; ADDITIONAL SECTION: -ns22.domaincontrol.com. 21 IN A 173.201.68.11 -ns22.domaincontrol.com. 21 IN AAAA 2603:5:2241::b -ns21.domaincontrol.com. 21 IN A 97.74.100.11 -ns21.domaincontrol.com. 21 IN AAAA 2603:5:2141::b - -``` -"""]] diff --git a/doc/bugs/Small_patch_to_make_93520790a_compile_on_Windows.mdwn b/doc/bugs/Small_patch_to_make_93520790a_compile_on_Windows.mdwn deleted file mode 100644 index 1437d10d1a..0000000000 --- a/doc/bugs/Small_patch_to_make_93520790a_compile_on_Windows.mdwn +++ /dev/null @@ -1,82 +0,0 @@ -### Please describe the problem. - -Latest git-annex commit 93520790a still doesn't quite compile without a small patch :) - -### What steps will reproduce the problem? - -`stack setup && stack build` - -### What version of git-annex are you using? On what operating system? - -[[!format sh """ -git-annex version: 8.20201117-g93520790a -build flags: Assistant Webapp Pairing TorrentParser Feeds Testsuite S3 WebDAV -dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external -operating system: mingw32 x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 2 3 4 5 6 7 -"""]] - -Windows version 20H2 (build 19042.630), 64 bit. - -### Please provide any additional information below. - -Relevant parts of the build log: - -[[!format sh """ -jkniiv@AINESIS:/c/Projektit/git-annex.branchable.com/git-annex--BUILD-201125-93520790a$ tail -n 25 stack.build.LOG~102 -[326 of 644] Compiling P2P.IO -[327 of 644] Compiling CmdLine.GitRemoteTorAnnex -[328 of 644] Compiling Annex.CheckIgnore -[329 of 644] Compiling Annex.CheckAttr -[330 of 644] Compiling Backend -[331 of 644] Compiling Annex.Transfer -[332 of 644] Compiling Annex.CatFile -[333 of 644] Compiling RemoteDaemon.Common -[334 of 644] Compiling Database.Keys -[335 of 644] Compiling Upgrade.V7 -[336 of 644] Compiling Annex.Content.Presence - -Annex\Content\Presence.hs:122:85: error: - * Variable not in scope: removeLink :: FilePath -> IO () - * Perhaps you meant one of these: - `R.removeLink' (imported from Utility.RawFilePath), - `removeFile' (imported from Annex.Common) - | -122 | void $ tryIO $ removeWhenExistsWith removeLink (fromRawFilePath lockfile) - | ^^^^^^^^^^ - - --- While building package git-annex-8.20201116 (scroll up to its section to see the error) using: - C:\Users\jkniiv\Projektit\git-annex.branchable.com\git-annex--BUILD-201125-93520790a\.stack-work\dist\29cc6475\setup\setup --builddir=.stack-work\dist\29cc6475 build exe:git-annex --ghc-options " -fdiagnostics-color=always" - Process exited with code: ExitFailure 1 -# End of transcript. -"""]] - -The change I ended up making was: - -[[!format diff """ -diff --git a/Annex/Content/Presence.hs b/Annex/Content/Presence.hs -index 4b33bcefb..05ff5715e 100644 ---- a/Annex/Content/Presence.hs -+++ b/Annex/Content/Presence.hs -@@ -119,7 +119,7 @@ inAnnexSafe key = inAnnex' (fromMaybe True) (Just False) go key - Nothing -> return is_locked - Just lockhandle -> do - dropLock lockhandle -- void $ tryIO $ removeWhenExistsWith removeLink (fromRawFilePath lockfile) -+ void $ tryIO $ removeWhenExistsWith R.removeLink lockfile - return is_unlocked - , return is_missing - ) -"""]] - -This then compiled cleanly. - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Git Annex is great. It works with multi-gigabyte backup files via the MD5E backend just dandy :) - -> [[fixed|done]], thanks for the patch --[[Joey]] diff --git a/doc/bugs/Test_prop__95__viewedFile__95__rountrips_fails_occasionally.mdwn b/doc/bugs/Test_prop__95__viewedFile__95__rountrips_fails_occasionally.mdwn deleted file mode 100644 index 065ae8a0db..0000000000 --- a/doc/bugs/Test_prop__95__viewedFile__95__rountrips_fails_occasionally.mdwn +++ /dev/null @@ -1,104 +0,0 @@ -### Please describe the problem. - -Running the test QuickCheck.prop_viewedFile_rountrips fails in a particular case (but only occasionally). - -### What steps will reproduce the problem? - -Running the following fails: - -[[!format sh """ -jkniiv@AINESIS MINGW64 /c/annx -$ ./git-annex.exe test -p 'QuickCheck.prop_viewedFile_rountrips' --quickcheck-replay=819461 --quickcheck-verbose 2>&1 | tee git-annex.test--p-QuickCheck_prop_viewedFile_rountrips.LOG~201 -[...] -$ cat git-annex.test--p-QuickCheck_prop_viewedFile_rountrips.LOG~201 -Tests - QuickCheck - prop_viewedFile_rountrips: FAIL (0.01s) - Passed: - TestableFilePath {fromTestableFilePath = "{\DC4"} - - Passed: - TestableFilePath {fromTestableFilePath = ":"} - - Passed: - TestableFilePath {fromTestableFilePath = "\SI"} - - Passed: - TestableFilePath {fromTestableFilePath = "\"78"} - - Passed: - TestableFilePath {fromTestableFilePath = "[:"} - - Passed: - TestableFilePath {fromTestableFilePath = "\ax0"} - - Passed: - TestableFilePath {fromTestableFilePath = "_\SUB"} - - Passed: - TestableFilePath {fromTestableFilePath = "\DC1"} - - Passed: - TestableFilePath {fromTestableFilePath = "wP["} - - Passed: - TestableFilePath {fromTestableFilePath = "8gp\v\DC38"} - - Passed: - TestableFilePath {fromTestableFilePath = "cO\ESC\FS]"} - - Passed: - TestableFilePath {fromTestableFilePath = "("} - - Passed: - TestableFilePath {fromTestableFilePath = "3\ESCLK=\SOH"} - - Passed: - TestableFilePath {fromTestableFilePath = "21U'd(\DEL("} - - Failed: - TestableFilePath {fromTestableFilePath = "C:"} - - *** Failed! Falsified (after 15 tests): - TestableFilePath {fromTestableFilePath = "C:"} - Use --quickcheck-replay=819461 to reproduce. - -1 out of 1 tests failed (0.01s) - (Failures above could be due to a bug in git-annex, or an incompatibility - with utilities, such as git, installed on this system.) -# End of transcript. -"""]] - -If you remove the option `--quickcheck-replay=819461`, the test usually passes. - -As you can see, the test above supposedly generated a file named "C:" which in its particular use case caused problems. -N.B. Git Bash (and thus MSYS2 or Cygwin) doesn't have any problem creating a file named "C:" (`touch 'C:'`) nor deleting it (`rm 'C:'`) -so I guess it isn't an illegal name in Windows as such but somehow git-annex managed to ruffle its feathers nonetheless. - -### What version of git-annex are you using? On what operating system? - -[[!format sh """ -git-annex version: 8.20201117-g93520790a -build flags: Assistant Webapp Pairing TorrentParser Feeds Testsuite S3 WebDAV -dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external -operating system: mingw32 x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 2 3 4 5 6 7 -"""]] - -This is built with the additional patch mentioned in [[bugs/Small_patch_to_make_93520790a_compile_on_Windows]]. - -Windows version 20H2 (build 19042.630), 64 bit. - -### Please provide any additional information below. - -(The transcript is above.) - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Git Annex is great. It works with multi-gigabyte backup files via the MD5E backend just dandy :) - -> [[fixed|done]] in [[!commit e92117bfd0d28b24f51a10903158b95010defffd]] -> --[[Joey]] diff --git a/doc/bugs/Upgrader_crashed.mdwn b/doc/bugs/Upgrader_crashed.mdwn deleted file mode 100644 index b9b8238f41..0000000000 --- a/doc/bugs/Upgrader_crashed.mdwn +++ /dev/null @@ -1,42 +0,0 @@ -### Please describe the problem. -I get the following **warning** on my git-annex dashboard - -``` -Upgrader crashed: C:\Users\alexasus\.config\git-annex\program: openFile: does not exist (No such file or directory) -``` - -### What steps will reproduce the problem? -- Install git-annex with the Windows 10 installer. -- Create a repository with the default path. - - -### What version of git-annex are you using? On what operating system? -- git-annex version: git-annex-installer_8.20201007+git171-g7e24b2587 (this is the Windows 10 installer filename) -- Windows 10 Version 2004 (OS Build 19041.630) - - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - -[2020-11-18 11:29:22.6950775] main: starting assistant version 8.20201008-g7e24b2587 -Launching web browser on file://C:\Users\alexasus\Desktop\annex\.git\annex\webapp.html -[2020-11-18 11:29:22.7940807] Cronner: You should enable consistency checking to protect your data. -(scanning...) [2020-11-18 11:29:23.278062] Watcher: Performing startup scan -(started...) -Upgrader crashed: C:\Users\alexasus\.config\git-annex\program: openFile: does not exist (No such file or directory) -[2020-11-18 11:29:24.1010593] Upgrader: warning Upgrader crashed: C:\Users\alexasus\.config\git-annex\program: openFile: does not exist (No such file or directory) - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -I've barely just installed. So I'm gonna get going these days. That warning doesn't seem to be an issue so far. I've looked it up but couldn't find any other reports for that warning. - - - -> [[fixed|done]] (by skipping checking upgrades on OS's where it's not -> supported) --[[Joey]] diff --git a/doc/bugs/Upgrader_crashed/comment_1_3c2bbcd929b0364a6fb0b9f7bcf59550._comment b/doc/bugs/Upgrader_crashed/comment_1_3c2bbcd929b0364a6fb0b9f7bcf59550._comment deleted file mode 100644 index 5883eee050..0000000000 --- a/doc/bugs/Upgrader_crashed/comment_1_3c2bbcd929b0364a6fb0b9f7bcf59550._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-11-19T16:46:50Z" - content=""" -Thanks for reporting. You do not need to worry about this, it seems we -accidentially enabled trying to upgrade in the recent release for windows, -but git-annex has never actually supported upgrading itself on windows, -so of course it fails. -"""]] diff --git a/doc/bugs/Very_not_graceful_exit_on_cwd_deletion_while_git_annex_adding_files.mdwn b/doc/bugs/Very_not_graceful_exit_on_cwd_deletion_while_git_annex_adding_files.mdwn deleted file mode 100644 index c1fe61532d..0000000000 --- a/doc/bugs/Very_not_graceful_exit_on_cwd_deletion_while_git_annex_adding_files.mdwn +++ /dev/null @@ -1,68 +0,0 @@ -I was git-annex-adding a folder of big files. -I ran the command while inside a folder I didn't need anymore. -I deleted the folder with nautilus (moved it to trash) which caused the cwd to change and that really confused git-annex. - -Root of repo is audio-recordings/ - -While inside audio-recordings/a I ran `git annex add ../audio-files` - -Then I deleted audio-recordings/a with Nautilus which caused its path to change to /.Trash-1000/files/a - -As soon as git-annex finished hashing the file it was hashing, this happened: - -``` -add ../audio-files/moto-maxx/2018.04.17 1438.wav -100% 579.95 MiB 160 MiB/s 0s - ../audio-files/moto-maxx/2018.04.17 1438.wav changed while it was being added -failed -fatal: not a git repository: '../.git' -error: unknown option `cached' -usage: git diff --no-index [] -... -[the entire help message] -... ---find-object - look for differences that change the number of occurrences of the specified object - --diff-filter [(A|C|D|M|R|T|U|X|B)...[*]] - select files by diff type - --output Output to a specific file - -(recording state in git...) -fatal: not a git repository: '../.git' -^Ceral-pathspecs","add","--"] exited 123) - -``` - -The file that was being added was left like this: - -``` -ll "audio-files/moto-maxx/2018.04.17 1438.wav" - --r--r--r-- 2 ### ### 608121644 abr 17 2018 'audio-files/moto-maxx/2018.04.17 1438.wav' -``` - -It appears git-annex just chmoded it, but didn't symlinkify it. - -### What version of git-annex are you using? On what operating system? - -`Ubuntu 20.04 focal` - -`Linux 5.8.0-050800-generic` - -`git-annex version: 8.20200226` - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I am absolutely in love with it. I even built a special remote for it and I'm now building another because that one was stupid. - -Seriously, git-annex is amazing. - -*** - -## edit: - -Turns out this "bug" affects pretty much everything. - -TL;DR: relative path access will be made relative to the new location if you move a directory that's got something running in it, so don't. - - [[done]] diff --git a/doc/bugs/Very_not_graceful_exit_on_cwd_deletion_while_git_annex_adding_files/comment_1_04201aa4345db10dbb5fb4637946d9c1._comment b/doc/bugs/Very_not_graceful_exit_on_cwd_deletion_while_git_annex_adding_files/comment_1_04201aa4345db10dbb5fb4637946d9c1._comment deleted file mode 100644 index 29954e8dad..0000000000 --- a/doc/bugs/Very_not_graceful_exit_on_cwd_deletion_while_git_annex_adding_files/comment_1_04201aa4345db10dbb5fb4637946d9c1._comment +++ /dev/null @@ -1,52 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-08-06T18:01:23Z" - content=""" -Thanks, I was able to reproduce this easily: - - git init r - cd r - mkdir a - mkdir b - dd if=/dev/zero of=a/big bs=1M count=1000 - cd b - git annex add ../a & (sleep 1; mv ../b ~/trash) - -By moving the directory the process is running in, all relative path -accesses it does after the move are relative to the new location. This is -super unsafe, but I don't know if it's super unsafe in a way that's unique to -git-annex. If a process does any relative path accesses with ../ in them, -it's going to be vulnerable to being flung around in this way and will start -to do unexpected things. - -git seems to avoid this being a problem by starting with a chdir to the top -of the repo, and then uses relative paths without ../. (Mostly.. --git-dir -with ../ in it will make git use such paths.) - -Other commands.. not so much. `rm -rf ../foo` ends by unlinking "../foo" -so if it's flung around it will delete the wrong thing. (Though its -directory tree traversal uses openat() and so avoids deleting a whole wrong -directory tree.) And `vim ../foo` overwrote an existing file after being -moved, bypassing its usual protections about overwriting a modified file. - -So, I think the super unsafe thing is generally moving directories around -when they have processes running in them. Unfortunately, the file manager -has a good reason to want to do it to handle deletions too.. Well, my -opinion of unix's safety was already not great, but it's now gone down some -more. - -(Fun thing to consider: What if you have the ability to move a directory -that a root-owned process is running in, and the root-owned process does -relative paths accesses? This seems like it could be a fertile source of -security holes.) - ----- - -The git error message about the cached option happen when "git diff ---cached" is run outside a git repository. Why is it outside a git -repository? See above. So I don't think it can be avoided. - -Only improvement that seems feasible is, to keep an open handle to the -file, so it can fchmod the fix the permissions back. -"""]] diff --git a/doc/bugs/Very_not_graceful_exit_on_cwd_deletion_while_git_annex_adding_files/comment_2_18501567083a4cdb1beaab401b9e7d4f._comment b/doc/bugs/Very_not_graceful_exit_on_cwd_deletion_while_git_annex_adding_files/comment_2_18501567083a4cdb1beaab401b9e7d4f._comment deleted file mode 100644 index 7d1f185e92..0000000000 --- a/doc/bugs/Very_not_graceful_exit_on_cwd_deletion_while_git_annex_adding_files/comment_2_18501567083a4cdb1beaab401b9e7d4f._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="cardoso-neto" - avatar="http://cdn.libravatar.org/avatar/d90a656df072f3a29da54302c190c696" - subject="Understood." - date="2020-08-07T02:24:43Z" - content=""" -Thank you for the detailed and timely response. - -That was some good thinking there with this being a possible attack vector. Food for thought right there. - -Btw, I'm sorry for explaining what I did instead of providing you with commands to reproduce the bug. I don't know what I was thinking. Git-annex has been a very big part of my life these last few years and I got a little nervous while writing the bug report. lol - -I shall be more careful with from where I run my commands henceforth. -"""]] diff --git a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II.mdwn b/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II.mdwn deleted file mode 100644 index 9d5c4cff98..0000000000 --- a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II.mdwn +++ /dev/null @@ -1,66 +0,0 @@ -### Please describe the problem. - -When running git-annex tests on a specific machine I get the following error: - -``` -$ ./dist-newstyle/build/x86_64-linux/ghc-8.6.5/git-annex-8.20200226/build/git-annex/git-annex test -v -p Tests.QuickCheck.prop_hashes_stable -Tests - QuickCheck - prop_hashes_stable: Illegal instruction (core dumped) -``` - -(I get the same error when just running `./dist-newstyle/build/x86_64-linux/ghc-8.6.5/git-annex-8.20200226/build/git-annex/git-annex test`, but after successful passes of previous tests.) - -### What steps will reproduce the problem? - -Run the following on a machine with the CPU "AMD Phenom(tm) II X3 720 Processor": - -``` -git clone git://git-annex.branchable.com/ git-annex -cd git-annex -cabal install -j --only-dependencies -cabal configure -cabal build -j -./dist-newstyle/build/x86_64-linux/ghc-8.6.5/git-annex-8.20200226/build/git-annex/git-annex test -v -p Tests.QuickCheck.prop_hashes_stable -``` - -### What version of git-annex are you using? On what operating system? - -I'm trying to build git-annex from the current "master". However, the same problem exists in 7.20190819, 7.20191230 and 7.20200204. - -I'm building on NixOS, release 19.09, using GHC 8.6.5. - -### Please provide any additional information below. - -I believe the problem was initially encountered on NixOS' build farm, https://hydra.nixos.org/ , when trying to build the git-annex package version 7.20190819 for NixOS/nixpkgs. (As a result, the built package was not available when I tried to update my installation of NixOS, which led me to investigating this problem.) - -So I tried to build the same package (and other versions, see above) on my own machine running NixOS (which has AMD Phenom II CPU). It consistently failed during testing on numerous attempts with different configurations. However, another machine (with AMD FX-8300 CPU) was able to build the same package, and never had the same test failures. - -Then I built git-annex on both machines from source (not as a Nix package), using cabal, and tried to run tests, as shown in the beginning. I got the same results: Phenom II failed the test Tests.QuickCheck.prop_hashes_stable with "Illegal instruction", while FX-8300 passed it. - -I do **not** think that my Phenom II machine has hardware problems: I ran MemTest recently, and this machine is able to build other packages (including GCC and GHC) just fine. - -Moreover, the same problem occurs on hydra.hixos.org , and it looks like it's sporadic (repeating the build can fix it). I guess that this depends on which machine is assigned to build the package: some machines are able to build it, while others are not. - -So, my hypothesis is that git-annex's tests (and maybe non-test code pieces as well) use some CPU instructions that are not supported on Phenom II (and other) CPUs, but supported on later CPUs such as FX-8300. Probably this is even a problem of GHC rather than of git-annex, but I'm not qualified to tell (I know nothing relevant about Haskell compilation or code generation). - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - -[nix-shell:~/build/git-annex]$ ./dist-newstyle/build/x86_64-linux/ghc-8.6.5/git-annex-8.20200226/build/git-annex/git-annex test -v -p Tests.QuickCheck.prop_hashes_stable -Illegal instruction (core dumped) - -[nix-shell:~/build/git-annex]$ dmesg | tail -n1 -[151755.340563] traps: git-annex:w[20112] trap invalid opcode ip:2d6c567 sp:7fefa0a1fd98 error:0 in git-annex[407000+313b000] - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I'm using git-annex as a core of the simple in-house system that handles storage, transfer and tracking of multi-GB data sets, taking advantage of git for version tracking, git-annex for data storage and transfer, and a single SSH access point to both of them. - -I must admit that it was a pain to adapt git-annex to my needs, and probably I did it in a manner that wasn't an intended/supported way of using git-annex. But it does the job. - -> [[closing|done]] as it's a bug in cryptonite --[[Joey]] diff --git a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_1_3932b6954cc42d2c376f48a6cdf0f879._comment b/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_1_3932b6954cc42d2c376f48a6cdf0f879._comment deleted file mode 100644 index e1ea388d83..0000000000 --- a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_1_3932b6954cc42d2c376f48a6cdf0f879._comment +++ /dev/null @@ -1,22 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-02T17:48:17Z" - content=""" -The failing test case is testing all the hashes that git-annex uses. -Quite likely what has happened is that the hash implementation has been -built with a CPU-specific optimisation that is not available on your AMD. - -The hash implementations are in http://hackage.haskell.org/package/cryptonite - -While cryptonite has build flags for CPU-specific features, -there are also, in its C code, some checks of gcc defines that indicate -specific CPU features. I found #defines checking `__SSE2__` and `__SSE4__` -and `__AVX__` (mostly in blake2) and there might be others. That seems -likely to be a bug in cryptonite, if its build flags are not fully -controlling use of things like SSE. - -I've made a change to git-annex's test suite to narrow down the problem; -it now tests each hash individually, so you'll see which specific ones -fail. -"""]] diff --git a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_2_4aa4d250bf1050fd9854ca7478e28888._comment b/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_2_4aa4d250bf1050fd9854ca7478e28888._comment deleted file mode 100644 index b40e46b371..0000000000 --- a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_2_4aa4d250bf1050fd9854ca7478e28888._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="http://id.pvgoran.name/" - nickname="pvgoran" - avatar="http://cdn.libravatar.org/avatar/e32a61d9c49989ae31d7a30d6af27f5c73d8d46ba03c840a612791fe5c820b87" - subject="comment 2" - date="2020-03-03T02:11:30Z" - content=""" -I run the tests on the new `master`, and I can see that all `blake2` tests (`Tests.QuickCheck.blake2s_160` and so on) fail, whereas all other hash-related QuickCheck tests succeed. -"""]] diff --git a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_3_28ede555e6a372ca0278148016cae4b0._comment b/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_3_28ede555e6a372ca0278148016cae4b0._comment deleted file mode 100644 index cd5995b27d..0000000000 --- a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_3_28ede555e6a372ca0278148016cae4b0._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="http://id.pvgoran.name/" - nickname="pvgoran" - avatar="http://cdn.libravatar.org/avatar/e32a61d9c49989ae31d7a30d6af27f5c73d8d46ba03c840a612791fe5c820b87" - subject="comment 3" - date="2020-03-03T03:16:25Z" - content=""" -After rebuilding `cryptonite` without AES-NI support as suggested in https://github.com/haskell-crypto/cryptonite/issues/260#issuecomment-484185981 (by passing `--constraint=\"cryptonite -support_aesni\"` to `cabal install --dependencies-only` and to `cabal build`), all of QuickCheck tests pass on the machine in question. (Many other tests fail, but it's probably because I run tests from a not-yet-installed `git-annex` binary.) - -So `cryptonite` apparently uses these AES-NI instructions (if configured to support them) without runtime checks (or at least without working runtime checks). Which constitutes a problem for any binary distributions that include `cryptonite` code: either they won't work on older CPUs, or they will not work as fast as they could on newer CPUs. - -"""]] diff --git a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_4_422e16904033ff063a7a4881097197fd._comment b/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_4_422e16904033ff063a7a4881097197fd._comment deleted file mode 100644 index 28d3ef849b..0000000000 --- a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_4_422e16904033ff063a7a4881097197fd._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="http://id.pvgoran.name/" - nickname="pvgoran" - avatar="http://cdn.libravatar.org/avatar/e32a61d9c49989ae31d7a30d6af27f5c73d8d46ba03c840a612791fe5c820b87" - subject="comment 4" - date="2020-03-03T03:19:01Z" - content=""" -joey, can you tell what problems I will face if I run a git-annex binary with broken blake2 implementation? What functionality of git-annex will fail? -"""]] diff --git a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_5_8e0165a7473220e3a87fee9f1e27fed3._comment b/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_5_8e0165a7473220e3a87fee9f1e27fed3._comment deleted file mode 100644 index 6aafe243a1..0000000000 --- a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_5_8e0165a7473220e3a87fee9f1e27fed3._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="http://id.pvgoran.name/" - nickname="pvgoran" - avatar="http://cdn.libravatar.org/avatar/e32a61d9c49989ae31d7a30d6af27f5c73d8d46ba03c840a612791fe5c820b87" - subject="comment 5" - date="2020-03-03T04:16:35Z" - content=""" -I also posted an issue for `cryptonite`: https://github.com/haskell-crypto/cryptonite/issues/314 -"""]] diff --git a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_6_c87be8241a5facb94cb9146ebb31a7de._comment b/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_6_c87be8241a5facb94cb9146ebb31a7de._comment deleted file mode 100644 index 411ab6b594..0000000000 --- a/doc/bugs/__34__Illegal_instruction__34___when_running_certain_tests_on_AMD_Phenom_II/comment_6_c87be8241a5facb94cb9146ebb31a7de._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 6""" - date="2020-03-09T19:00:35Z" - content=""" -git-annex does not use blake2 by default. If someone has configured it to -use blake2 (probably because they want the speed) and added some files to -a repo, it will then use blake2 to hash those files going forward. - -So, in balance, it seems better to disable the unsafe optimisation, -and have it be a little bit slower, than not work on some machines. - -Thanks for fileing the issue on cryptonite. I don't see anything further to -be done in git-annex, so I'll close this bug now. -"""]] diff --git a/doc/bugs/__34__Variable_not_in_scope__34___when_compiling_on_Windows.mdwn b/doc/bugs/__34__Variable_not_in_scope__34___when_compiling_on_Windows.mdwn deleted file mode 100644 index 6b2069d57d..0000000000 --- a/doc/bugs/__34__Variable_not_in_scope__34___when_compiling_on_Windows.mdwn +++ /dev/null @@ -1,22 +0,0 @@ -Attempting to build commit 7fcbb92 on Windows 7 by running "stack build" fails with the following error: - -[[!format sh """ -[571 of 636] Compiling Utility.WebApp - -Utility\WebApp.hs:93:17: error: - Variable not in scope: inet_addr :: [Char] -> IO HostAddress - | -93 | addr <- inet_addr "127.0.0.1" - | ^^^^^^^^^ - -Utility\WebApp.hs:96:33: error: - Variable not in scope: aNY_PORT :: PortNumber - | -96 | bind sock (SockAddrInet aNY_PORT addr) - | ^^^^^^^^ -"""]] - -- stack version: 2.3.3 -- Windows version: Windows 7 Enterprise, version 6.1 - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/__34__Variable_not_in_scope__34___when_compiling_on_Windows/comment_1_8822bbeffb9a89daceb2900ecfe045f5._comment b/doc/bugs/__34__Variable_not_in_scope__34___when_compiling_on_Windows/comment_1_8822bbeffb9a89daceb2900ecfe045f5._comment deleted file mode 100644 index 00789c9c68..0000000000 --- a/doc/bugs/__34__Variable_not_in_scope__34___when_compiling_on_Windows/comment_1_8822bbeffb9a89daceb2900ecfe045f5._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-10-08T14:38:26Z" - content=""" -Those came from the network library, but were removed in network-3.0. - -I've updated it to work with the new library. -"""]] diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails.mdwn b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails.mdwn deleted file mode 100644 index 873a233c44..0000000000 --- a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails.mdwn +++ /dev/null @@ -1,32 +0,0 @@ -### Please describe the problem. - -In a repository with submodules, if I want to initialise git-annex on all submodules, I expect that `git submodule foreach git annex init` should work. Other `git-annex` commands seem to work with the `submodule foreach` command just fine. - - -### What steps will reproduce the problem? - - 1. Clone a repository with submodules (recursively): `git clone --recursive ` - 2. Initialise the main repository's annex: `git annex init` - 3. Try to initialise git-annex in all submodules: `git submodule foreach git annex init` - -This produces the following error: -``` -git-annex: git: createProcess: runInteractiveProcess: chdir: inappropriate type (Not a directory) -fatal: run_command returned non-zero status for scripts -. -``` - -As a workaround, manually changing directory into each submodule and running `git annex init` works, but there are cases where having a general script that can iterate submodules is convenient. - - -### What version of git-annex are you using? On what operating system? - -- git-annex version: 8.20201007-gcf33be21ac -- OS: Arch Linux - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I use it regularly with great success :) - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_1_8a6cd8dc6e7c2cdefd60a07356be1d27._comment b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_1_8a6cd8dc6e7c2cdefd60a07356be1d27._comment deleted file mode 100644 index 6e21268bb1..0000000000 --- a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_1_8a6cd8dc6e7c2cdefd60a07356be1d27._comment +++ /dev/null @@ -1,37 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="comment 1" - date="2020-10-20T15:59:26Z" - content=""" -I can trigger this issue on my end too (tried with -7.20190503-g4da50456a and 8.20201008-gad86a25c5). In case it's -helpful, here is a minimal reproducer: - -[[!format sh \"\"\" -set -ue - -cd \"$(mktemp -d \"${TMPDIR:-/tmp}\"/gx-XXXXXXX)\" - -git annex version -pwd - -git init src -( - cd src - git init sub - git -C sub annex init - git -C sub commit --allow-empty -m sub-c0 - git submodule add ./sub - git commit -m sub -) - -git clone --recursive src dest -( - cd dest - git annex init - git submodule foreach 'git annex init' -) -\"\"\"]] - -"""]] diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_3_23f3dac4a22cf7202a0bb849ee499766._comment b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_3_23f3dac4a22cf7202a0bb849ee499766._comment deleted file mode 100644 index 27a3c5bd0e..0000000000 --- a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_3_23f3dac4a22cf7202a0bb849ee499766._comment +++ /dev/null @@ -1,37 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-10-23T16:21:12Z" - content=""" -Let's see, sub/.git is a file not a directory, and that's what it's trying -to chdir to, when running git. - -fixupUnusualRepos usually converts that file to a symlink to a directory, -but when the annex has not been initialized yet, it avoids doing so, because -users would be unhappy if an accidential git-annex run in a submodule not -initialized for git-annex started messing with it. - -git-annex does not normally chdir when running git, and the only place that -does is Config.read'. Which only does it to make git read the right config, -for when it's reading some remote's config. So, using --git-dir there will -have the same effect, and avoid the problem. - -Which it did, but then there's the new problem not much further along: - - git-annex: /home/joey/tmp/gx-KHSxawT/sub: changeWorkingDirectory: does not exist (No such file or directory) - -This is from Git.CurrentRepo.get, which does a setCurrentDirectory. -Where does that path that does not exist come from? -.git/modules/sub/config has "core.worktree = ../../../sub" - -git clone sets that. And it's a path from ../.git/modules/sub/ -back to the work tree. git-annex interprets it as relative to `GIT_DIR`, -which git submodule sets to ".git". - -So git-annex needs to check if .git is a gitdir: reference, and -then interpret core.worktree relative to what it points to. - -Or, alternatively, fixupUnusualRepos could learn to tell if the *current* -command is git-annex init, and go ahead with the fixes. Which would avoid -needing to chase more of this weird stuff. -"""]] diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_3_fd3a287ee6e73946889a4dd406fae0fc._comment b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_3_fd3a287ee6e73946889a4dd406fae0fc._comment deleted file mode 100644 index 0b12ba684d..0000000000 --- a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_3_fd3a287ee6e73946889a4dd406fae0fc._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="achilleas.k@14be77d42a1252fab5ec9dbf4e5ea03c5833e8c8" - avatar="http://cdn.libravatar.org/avatar/ed6c67c4d8e6c6850930e16eaf85a771" - subject="comment 3" - date="2020-10-21T07:51:37Z" - content=""" -Great! Thanks for checking and for the minimal reproducible example! -"""]] diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_4_e772639c09c1089700b2c2c170ea458e._comment b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_4_e772639c09c1089700b2c2c170ea458e._comment deleted file mode 100644 index 423a42244b..0000000000 --- a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_4_e772639c09c1089700b2c2c170ea458e._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-10-23T18:19:03Z" - content=""" -Tried to implement that idea of detecting if the repo is in the process of -initializing, but automatic initialization makes that impractical to -implement, I think. -"""]] diff --git a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_5_167ade7bcd58354280df31442b586024._comment b/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_5_167ade7bcd58354280df31442b586024._comment deleted file mode 100644 index 7656acb431..0000000000 --- a/doc/bugs/__34__git_submodule_foreach_git_annex_init__34___fails/comment_5_167ade7bcd58354280df31442b586024._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="achilleas.k@14be77d42a1252fab5ec9dbf4e5ea03c5833e8c8" - avatar="http://cdn.libravatar.org/avatar/ed6c67c4d8e6c6850930e16eaf85a771" - subject="comment 5" - date="2020-10-24T14:39:38Z" - content=""" -Awesome! Thanks for the quick response and fix Joey! -"""]] diff --git a/doc/bugs/__34__no_such_file_or_directory__34___when_using_bundle.mdwn b/doc/bugs/__34__no_such_file_or_directory__34___when_using_bundle.mdwn deleted file mode 100644 index b8069b8151..0000000000 --- a/doc/bugs/__34__no_such_file_or_directory__34___when_using_bundle.mdwn +++ /dev/null @@ -1,49 +0,0 @@ -### Please describe the problem. - -The bundle provided at https://git-annex.branchable.com/install/Linux_standalone/ doesn't work - -### What steps will reproduce the problem? - -1. I downloaded this file: https://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz - -2. I untared it in /usr/local and renamed the resulting directory to '/usr/local/git-annex' - -3. I added '/usr/local/git-annex' to my PATH - -4. I ran 'git-annex' and got: - -mv: cannot move '/home/lyderic/.cache/git-annex/locales.tmp/ec977e22ef909b450a3a84bd55783865.1075139' to '/home/lyderic/.cache/git-annex/locales/ec977e22ef909b450a3a84bd55783865': No such file or directory - -5. I fixed the problem by doing this: - -$ mkdir ~/.cache/git-annex/locales - -[[done]] - -### What version of git-annex are you using? On what operating system? - -git-annex version: 8.20200909-g2d5036e44 -OS: Ubuntu 20.04.1 - -### Please provide any additional information below. - -This is a transcript of the commands I ran to show and fix the bug: - -[[!format sh """ -~$ sudo tar -xzf Downloads/git-annex-standalone-amd64.tar.gz -C /usr/local -~$ sudo mv /usr/local/git-annex.linux /usr/local/git-annex -~$ sudo chown -R 0.0 /usr/local/git-annex -~$ export PATH=$PATH:/usr/local/git-annex -~$ git-annex version -mv: cannot move '/home/lyderic/.cache/git-annex/locales.tmp/ec977e22ef909b450a3a84bd55783865.1075139' to '/home/lyderic/.cache/git-annex/locales/ec977e22ef909b450a3a84bd55783865': No such file or directory -~$ mkdir ~/.cache/git-annex/locales -~$ git-annex version | head -1 -git-annex version: 8.20200909-g2d5036e44 -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I have never used it. I am learning it / evaluating it right now. I am very impressed so far! - -> [[fixed|done]], the bundle will be updated in the upcoming release. -> --[[Joey]] diff --git a/doc/bugs/__96__git_annex_direct__96___failed_and_will_not_resume.mdwn b/doc/bugs/__96__git_annex_direct__96___failed_and_will_not_resume.mdwn deleted file mode 100644 index 3f4fe29d86..0000000000 --- a/doc/bugs/__96__git_annex_direct__96___failed_and_will_not_resume.mdwn +++ /dev/null @@ -1,32 +0,0 @@ -### Please describe the problem. - - -### What steps will reproduce the problem? - -``` -chymera@silenthost /mnt/data/ni_data $ git annex direct -[...] -direct ofM.vta/20180730_070313_6592_1_1/8/pdata/1/reco ok -direct ofM.vta/20180730_070313_6592_1_1/8/pdata/1/visu_pars ok -direct ofM.vta/20180730_070313_6592_1_1/8/pulseprogram ok -direct ofM.vta/20180730_070313_6592_1_1/8/specpar ok -direct ofM.vta/20180730_070313_6592_1_1/8/uxnmr.info ok -error: git-annex died of signal 11 -chymera@silenthost /mnt/data/ni_data $ git annex direct -commit - -^C -``` - -I let it try to resume for a very long time (about a day) and it did nothing. - - -### What version of git-annex are you using? On what operating system? - -6.20170818 on Gentoo Linux - -### Please provide any additional information below. - -This is a fairly large repository which was only just cloned and with all data transferred (`git clone` and `git annex get .` worked fine). - -> [[closing|done]] per my comment --[[Joey]] diff --git a/doc/bugs/__96__git_annex_direct__96___failed_and_will_not_resume/comment_1_2aa8d50cef4f85d958371e2177f87d3a._comment b/doc/bugs/__96__git_annex_direct__96___failed_and_will_not_resume/comment_1_2aa8d50cef4f85d958371e2177f87d3a._comment deleted file mode 100644 index a1226cd2c3..0000000000 --- a/doc/bugs/__96__git_annex_direct__96___failed_and_will_not_resume/comment_1_2aa8d50cef4f85d958371e2177f87d3a._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-05-04T17:01:56Z" - content=""" -git-annex no longer supports direct mode in current versions, -so whatever code might have caused that behavior is removed. - -The signal 11 is, however, suprising. If a current version of git-annex -segfaults, it would be worth filing a bug report and chasing that down, -although only after you test your machine's memory for problems that might -lead to a sefault. -"""]] diff --git a/doc/bugs/add_--largerthan_reversion.mdwn b/doc/bugs/add_--largerthan_reversion.mdwn deleted file mode 100644 index c94205688f..0000000000 --- a/doc/bugs/add_--largerthan_reversion.mdwn +++ /dev/null @@ -1,17 +0,0 @@ -`git annex add --largerthan=200mb` of a new file -does not add a file that is large enough. - -This is a reversion, introduced last year in -[[!commit 3066bdb1fb60e80f40b5badc150fb6eb51a922bb]]. - -`git annex import /dir --largerthan=100mb` is also affected, and has an -ugly fail mode as it tries to look up an annexed file outside the repo: - - joey@darkstar:~/tmp/t>git annex import --largerthan=100mb ../d - fatal: './../d/foo' is outside repository at '/home/joey/tmp/t' - -That commit was otherwise right, eg `git-annex get --largerthan` should -look at the size of the annexed file, not of the file on disk, which could -be a small pointer file. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/addurl_--file_only_works_with_youtube-dl_with_--fast_or_--relaxed.mdwn b/doc/bugs/addurl_--file_only_works_with_youtube-dl_with_--fast_or_--relaxed.mdwn deleted file mode 100644 index 2b86ae7a81..0000000000 --- a/doc/bugs/addurl_--file_only_works_with_youtube-dl_with_--fast_or_--relaxed.mdwn +++ /dev/null @@ -1,54 +0,0 @@ -### Please describe the problem. - -The `--file` option doesn't work with youtube-dl downloaded content unless --fast or --relaxed are passed. - -### What steps will reproduce the problem? - -[[!format sh """ -$ git annex addurl --file out.m4a https://www.bbc.co.uk/programmes/p08jsctb -addurl https://www.bbc.co.uk/programmes/p08jsctb -... (cut most youtube-dl output) -[download] Destination: Bloodsport, Introducing Bloodsport-p08jscdd.m4a -[download] 100% of 909.06KiB in 00:03 -[ffmpeg] Correcting container in "Bloodsport, Introducing Bloodsport-p08jscdd.m4a" -(to Bloodsport, Introducing Bloodsport-p08jscdd.m4a) ok -(recording state in git...) -$ ls -'Bloodsport, Introducing Bloodsport-p08jscdd.m4a' -"""]] - -Whereas I expected the created file to be 'out.m4a'. - -However it works correctly when using --fast or --relaxed to set the URL, and then using git annex get separately: - -[[!format sh """ -$ git annex addurl --relaxed --file out.m4a https://www.bbc.co.uk/programmes/p08jsctb -addurl https://www.bbc.co.uk/programmes/p08jsctb (to out.m4a) ok -(recording state in git...) -$ ls -out.m4a -$ git annex get out.m4a -get out.m4a (from web...) -... (cut most youtube-dl output) -[download] Destination: Bloodsport, Introducing Bloodsport-p08jscdd.m4a -[download] 100% of 909.06KiB in 00:01 -[ffmpeg] Correcting container in "Bloodsport, Introducing Bloodsport-p08jscdd.m4a" -ok -(recording state in git...) -$ ls -out.m4a -"""]] - -### What version of git-annex are you using? On what operating system? - -8.20200330 (Debian sid) - -### Please provide any additional information below. - -I only recently started using the web special remote stuff, and really like it; thanks! - -### Have you had any luck using git-annex before? - -Yes, I've used it for several years for all sorts of things, and it just gets better and better. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/aeson_bound_can_be_bumped_to___60___1.6.mdwn b/doc/bugs/aeson_bound_can_be_bumped_to___60___1.6.mdwn deleted file mode 100644 index 018f7cd909..0000000000 --- a/doc/bugs/aeson_bound_can_be_bumped_to___60___1.6.mdwn +++ /dev/null @@ -1,14 +0,0 @@ -### Please describe the problem. - -git-lfs has an aeson upper bound of < 1.5, but I have built it successfully with aeson 1.5.4 - -### What version of git-annex are you using? On what operating system? - -git-lfs 1.1.0 on nixos - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I really like git-annex. Although I have to admit I am not an active user myself. I just want to keep it working for all nixos users, I know some that rely on it. - -> [[done]] --[[Joey]] diff --git a/doc/bugs/aeson_bound_can_be_bumped_to___60___1.6/comment_1_d139540cafd5e74f85ce28fa8cb83a4b._comment b/doc/bugs/aeson_bound_can_be_bumped_to___60___1.6/comment_1_d139540cafd5e74f85ce28fa8cb83a4b._comment deleted file mode 100644 index e58b25b74f..0000000000 --- a/doc/bugs/aeson_bound_can_be_bumped_to___60___1.6/comment_1_d139540cafd5e74f85ce28fa8cb83a4b._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-11-19T15:58:05Z" - content=""" -Fixed that. Do note this is not a bug tracker for other software, even if I -happen to have written it; you can contact me by email for git-lfs bugs. -"""]] diff --git a/doc/bugs/all_files_are_annexed_by_default.mdwn b/doc/bugs/all_files_are_annexed_by_default.mdwn deleted file mode 100644 index 6f587985ab..0000000000 --- a/doc/bugs/all_files_are_annexed_by_default.mdwn +++ /dev/null @@ -1,20 +0,0 @@ -Hello, - -I am using git annex for three years with no issues :-), but now I am facing a problem I’ve never seen before. - -I have a repository with several annexed files since 2017. Usually, I add big files with ``git annex add`` while for regular small files, I always use a simple ``git add``. -My problem is, now, all files added with ``git annex add`` are always considered as annexed files, which is undesirable. If I clone a new copy of the repository, ``git add`` works as expected, but from the moment I run a simple ``git annex get somefile``, all ``git add`` is executed as if it were a ``git annex add``. - -I don't know if this is a bug or just a change in the default behavior. - -Any idea about how to solve it? - -Thank you - - -Versions: -git 2.20.1 / -git annex 7.20190912-1 / -ubuntu 19.10 - -> [[done]] diff --git a/doc/bugs/all_files_are_annexed_by_default/comment_1_7a15de0d33751507dcd6292230c940a0._comment b/doc/bugs/all_files_are_annexed_by_default/comment_1_7a15de0d33751507dcd6292230c940a0._comment deleted file mode 100644 index 4d7debe8b3..0000000000 --- a/doc/bugs/all_files_are_annexed_by_default/comment_1_7a15de0d33751507dcd6292230c940a0._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="git add behavior" - date="2020-03-27T20:49:19Z" - content=""" -It's a change in default behavior, but the old default was restored in the more recent versions. (Detailed discussion [[here|forum/lets_discuss_git_add_behavior]]). Can you upgrade to the most recent version? [[install/conda]] lets you install one without being root. You might also want to set `annex.gitaddtoannex` to `false`. -"""]] diff --git a/doc/bugs/all_files_are_annexed_by_default/comment_2_8ff76c57cfddc3ed745ca16a4100d5a6._comment b/doc/bugs/all_files_are_annexed_by_default/comment_2_8ff76c57cfddc3ed745ca16a4100d5a6._comment deleted file mode 100644 index c52ea7a425..0000000000 --- a/doc/bugs/all_files_are_annexed_by_default/comment_2_8ff76c57cfddc3ed745ca16a4100d5a6._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="comment: resolved by 7.20191024" - date="2020-03-27T20:49:44Z" - content=""" -> git annex 7.20190912-1 - -The `git add` behavior that you describe changed in 7.20191024. The -CHANGELOG entries under that release provide a good description of the -current behavior. -"""]] diff --git a/doc/bugs/all_files_are_annexed_by_default/comment_3_b7fee790993867ff83af440f0b37aec4._comment b/doc/bugs/all_files_are_annexed_by_default/comment_3_b7fee790993867ff83af440f0b37aec4._comment deleted file mode 100644 index 4b25eec8ee..0000000000 --- a/doc/bugs/all_files_are_annexed_by_default/comment_3_b7fee790993867ff83af440f0b37aec4._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 3" - date="2020-03-27T21:11:13Z" - content=""" -My \"upgrade to most recent version\" advice should have mentioned that the most recent version is 8.*, which will auto-upgrade your repositories to version 8. If for whatever reason you don't want to do the upgrade right now, 7.* versions from 7.20191024 restore old behavior, as Kyle suggested. Note that if you have `annex.largefiles` configured, `git add` will add matching files to the annex, unless `annex.gitaddtoannex`. -"""]] diff --git a/doc/bugs/all_files_are_annexed_by_default/comment_4_da7a858e0be1ff45dea0b6e8d05da89f._comment b/doc/bugs/all_files_are_annexed_by_default/comment_4_da7a858e0be1ff45dea0b6e8d05da89f._comment deleted file mode 100644 index ae24942a0a..0000000000 --- a/doc/bugs/all_files_are_annexed_by_default/comment_4_da7a858e0be1ff45dea0b6e8d05da89f._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 4" - date="2020-03-27T21:12:18Z" - content=""" -I meant, unless `annex.gitaddtoannex=false`. -"""]] diff --git a/doc/bugs/all_files_are_annexed_by_default/comment_5_a7d0f94e7df2d838d1f8042866081915._comment b/doc/bugs/all_files_are_annexed_by_default/comment_5_a7d0f94e7df2d838d1f8042866081915._comment deleted file mode 100644 index 6814d42adb..0000000000 --- a/doc/bugs/all_files_are_annexed_by_default/comment_5_a7d0f94e7df2d838d1f8042866081915._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="vinicius.vin@6d4d58c59c394cd744d469c9d7c41e264331dfcd" - nickname="vinicius.vin" - avatar="http://cdn.libravatar.org/avatar/b332bfc1d3f49c196e1bff84b53d0f8b" - subject="comment 5" - date="2020-03-27T23:04:34Z" - content=""" -Both suggestions solve my issue. -Thank you for the quick reply! -"""]] diff --git a/doc/bugs/all_files_are_annexed_by_default/comment_6_ff42a4c90da62b7588ec058dc8ede812._comment b/doc/bugs/all_files_are_annexed_by_default/comment_6_ff42a4c90da62b7588ec058dc8ede812._comment deleted file mode 100644 index 0662bc606d..0000000000 --- a/doc/bugs/all_files_are_annexed_by_default/comment_6_ff42a4c90da62b7588ec058dc8ede812._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="clarification re: upgrades" - date="2020-03-30T15:24:01Z" - content=""" -p.s. the italics in my comment re: upgrading to v8 were an accident of formatting (`7.*` / `8.*` got interpreted as italics), not some emphasis of caution about the upgrade. v8 has been working fine for me, and on this forum there has been only [[one|bugs/upgrade_to_V8_fails]] report of an issue with the upgrade, which has since been fixed. You can also guard against any possible upgrade issues by backing up `.git/annex/` (excluding `.git/annex/objects`) and git config files before the upgrade. - -One small v8 change to note is the [more uniform handling of dotfiles](https://hackage.haskell.org/package/git-annex-8.20200226/changelog); old scripts using `--include-dotfiles` flag of `git-annex-add` would need to be updated. -"""]] diff --git a/doc/bugs/all_files_are_annexed_by_default/comment_7_0dce9bb98a7fca55a80b10fb7d130133._comment b/doc/bugs/all_files_are_annexed_by_default/comment_7_0dce9bb98a7fca55a80b10fb7d130133._comment deleted file mode 100644 index 8bcca35fb6..0000000000 --- a/doc/bugs/all_files_are_annexed_by_default/comment_7_0dce9bb98a7fca55a80b10fb7d130133._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="vinicius.vin@6d4d58c59c394cd744d469c9d7c41e264331dfcd" - nickname="vinicius.vin" - avatar="http://cdn.libravatar.org/avatar/b332bfc1d3f49c196e1bff84b53d0f8b" - subject="comment 7" - date="2020-03-30T21:02:52Z" - content=""" -Thanks, Ilya. -I will wait for the major upgrade of my distro to migrate to git-annex 8. -For now, I just updated git-annex to 7.20191024, where I can get back the ``git add`` behavior without update all the dependency chain. -"""]] diff --git a/doc/bugs/annex.genmetadata_should_default_to_true.mdwn b/doc/bugs/annex.genmetadata_should_default_to_true.mdwn deleted file mode 100644 index 5aef50d45b..0000000000 --- a/doc/bugs/annex.genmetadata_should_default_to_true.mdwn +++ /dev/null @@ -1,13 +0,0 @@ -### Please describe the problem. -annex.genmetadata is off by default, which will potentially lose you a lot of timestamp metadata if you don't pay attention. It should be on by default. - -### What steps will reproduce the problem? -Add a file to a fresh annex, observe it has no metadata. - -### What version of git-annex are you using? On what operating system? -6.20180227 on Debian - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -Yes I love it! (Except for its spotty timestamp support) - -> [[notabug|done]] --[[Joey]] diff --git a/doc/bugs/annex.genmetadata_should_default_to_true/comment_1_215ca5e67a1bc04bcb8bea6062aebf3f._comment b/doc/bugs/annex.genmetadata_should_default_to_true/comment_1_215ca5e67a1bc04bcb8bea6062aebf3f._comment deleted file mode 100644 index 48822f22a8..0000000000 --- a/doc/bugs/annex.genmetadata_should_default_to_true/comment_1_215ca5e67a1bc04bcb8bea6062aebf3f._comment +++ /dev/null @@ -1,23 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2018-04-04T17:50:02Z" - content=""" -I think there are two potential problems with this change: - -1. Bloat. If someone is not going to use that metadata, they may not want - the overhead of attaching it to every file. - -2. It's entirely possible that a user has chosen to use year/month/day - metadata fields for their own purposes. Perhaps they are using it to - keep track of the original publication date of files, and so would not - want "wrong" values to be automatically added. - -And that's essentially the reason why all parts of git-annex that deal -with specific metadata fields are optional, so there's no default -hard-coded semantics to the metadata fields. - -(With the exception of git-annex importfeed, which always -stores the itemid from the rss feed in the metadata, but only because -it's needed to detect buggy feeds that change their item urls.) -"""]] diff --git a/doc/bugs/annex.genmetadata_should_default_to_true/comment_2_0966f8bf944ded0a5199285ed3c12930._comment b/doc/bugs/annex.genmetadata_should_default_to_true/comment_2_0966f8bf944ded0a5199285ed3c12930._comment deleted file mode 100644 index cea41ad1db..0000000000 --- a/doc/bugs/annex.genmetadata_should_default_to_true/comment_2_0966f8bf944ded0a5199285ed3c12930._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="vrs+annex@ea5fa24dbb279be61a8e50adb638bf8366300717" - nickname="vrs+annex" - avatar="http://cdn.libravatar.org/avatar/74219abcec6eece8e2c9d4351c2c912c" - subject="comment 2" - date="2018-04-05T01:32:32Z" - content=""" -1. Not losing data (or alternatively having a great big warning in the manual, or requiring it as a configuration step) should be the default in software that manages files, especially software that advertises a backup usecase on its front page. I do think the current implementation could be improved, which is what I opened for. If that's not enough, a simple binary format should do the trick at about the same overhead as regular filesystems while staying future-proof. -2. Implementing timestamps with a field name that doesn't clash with existing fields should avoid this issue. -"""]] diff --git a/doc/bugs/annex.genmetadata_should_default_to_true/comment_3_7322a9f7d8d01e8daf67bfb08ce7042c._comment b/doc/bugs/annex.genmetadata_should_default_to_true/comment_3_7322a9f7d8d01e8daf67bfb08ce7042c._comment deleted file mode 100644 index 8d0efd3ecb..0000000000 --- a/doc/bugs/annex.genmetadata_should_default_to_true/comment_3_7322a9f7d8d01e8daf67bfb08ce7042c._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="CandyAngel" - avatar="http://cdn.libravatar.org/avatar/15c0aade8bec5bf004f939dd73cf9ed8" - subject="comment 3" - date="2018-04-05T15:45:52Z" - content=""" -As far as I am aware, git-annex doesn't store *any* information about any files (including their content!) without fairly explicit instruction to do so.. this would be exception to that general behaviour. -"""]] diff --git a/doc/bugs/annex.genmetadata_should_default_to_true/comment_4_615bb36fb8e2958c001fc6df997b2928._comment b/doc/bugs/annex.genmetadata_should_default_to_true/comment_4_615bb36fb8e2958c001fc6df997b2928._comment deleted file mode 100644 index 2e6f134778..0000000000 --- a/doc/bugs/annex.genmetadata_should_default_to_true/comment_4_615bb36fb8e2958c001fc6df997b2928._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="reasons not to have annex.genmetadata default to true" - date="2020-02-10T18:40:52Z" - content=""" -Some reasons `annex.genmetadata` should *not* default to true: (1) ordinary git does not preserve file modtimes, probably on purpose: if you have some kind of `make` process, you want `git update` to cause updated files to have updated modtimes, not the modtimes from when the files were added to git, so that `make` can detect the change and update downstream files; (2) as @joey noted, potentially wasteful bloat, especially for repos with many files; (3) two different copies of a file may have different modtimes, but all copies must have the same git-annex metadata, because metadata is attached to the [[key|backends]], which for most backends is computed from file contents. - -The WOM backend stores the modtime in the key, but then does not store checksums. If [[todo/external_backends]] are implemented, you could make one that includes both the checksum and the modtime in the key. -"""]] diff --git a/doc/bugs/annex.genmetadata_should_default_to_true/comment_5_8cd47851ca76005e0011da7c2f62d77d._comment b/doc/bugs/annex.genmetadata_should_default_to_true/comment_5_8cd47851ca76005e0011da7c2f62d77d._comment deleted file mode 100644 index 69f6444e69..0000000000 --- a/doc/bugs/annex.genmetadata_should_default_to_true/comment_5_8cd47851ca76005e0011da7c2f62d77d._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2020-02-17T16:47:33Z" - content=""" -I agree with Ilya's points. - -And it's unresonable to characterize this as data loss, because git itself -does not store file timestamp data. Making such mischaracterizations does, -however, cause me, as the maintainer, to wonder if this is a feature that -is worth keeping, since I have no interest in descending that rabbit hole -or fighting such accusations. Generally, going to the strongest possible -argument, when requesting a change, is not actually your best move. - -Closing this bug report. -"""]] diff --git a/doc/bugs/annex_get_fails_with___39__not_available__39___for_git__58____47____47___protocol.mdwn b/doc/bugs/annex_get_fails_with___39__not_available__39___for_git__58____47____47___protocol.mdwn deleted file mode 100644 index 863d9eb5f5..0000000000 --- a/doc/bugs/annex_get_fails_with___39__not_available__39___for_git__58____47____47___protocol.mdwn +++ /dev/null @@ -1,180 +0,0 @@ -### Please describe the problem. - -A very simple scenario where a git annex is cloned via git:// protocol results in the cloned repo being not functional. - -Specifically, in a repo that had been cloned via git:// protocol, I encounter a 'not available' error when attempting git annex get - -[[!format sh """ -shaddy@jazz-deb:~/tmp/gitresearch/bar$ git annex get motd -get motd (not available) - Try making some of these repositories available: - f75d1e03-3b95-45be-9d30-ca8996e0b69a -- foo -failed -git-annex: get: 1 failed -"""]] - - -### What steps will reproduce the problem? - -The first step is to run git daemon for the git:// protocol: - -[[!format sh """ -shaddy@jazz-deb:~/tmp/gitresearch$ git daemon --export-all --enable=receive-pack --base-path=. & -[1] 10519 -"""]] - -Now make a reference git annex repo: - -[[!format sh """ -shaddy@jazz-deb:~/tmp/gitresearch$ git init foo -Initialized empty Git repository in /home/shaddy/tmp/gitresearch/foo/.git/ -shaddy@jazz-deb:~/tmp/gitresearch$ cd foo -shaddy@jazz-deb:~/tmp/gitresearch/foo$ git annex init foo -init foo ok -(recording state in git...) -shaddy@jazz-deb:~/tmp/gitresearch/foo$ cp /etc/motd . -shaddy@jazz-deb:~/tmp/gitresearch/foo$ git annex add motd -add motd ok -(recording state in git...) -shaddy@jazz-deb:~/tmp/gitresearch/foo$ git commit -m foo -[master (root-commit) 16dbab7] foo - 1 file changed, 1 insertion(+) - create mode 120000 motd -"""]] - -Now create a clone via git://: - -[[!format sh """ -shaddy@jazz-deb:~/tmp/gitresearch/foo$ cd .. -shaddy@jazz-deb:~/tmp/gitresearch$ git clone git://localhost/foo bar -Cloning into 'bar'... -remote: Enumerating objects: 13, done. -remote: Counting objects: 100% (13/13), done. -remote: Compressing objects: 100% (9/9), done. -remote: Total 13 (delta 1), reused 0 (delta 0) -Receiving objects: 100% (13/13), done. -Resolving deltas: 100% (1/1), done. -shaddy@jazz-deb:~/tmp/gitresearch$ cd bar -shaddy@jazz-deb:~/tmp/gitresearch/bar$ git annex init bar -init bar (merging origin/git-annex into git-annex...) -(recording state in git...) -ok -(recording state in git...) -shaddy@jazz-deb:~/tmp/gitresearch/bar$ git annex sync -commit -On branch master -Your branch is up to date with 'origin/master'. - -nothing to commit, working tree clean -ok -pull origin -ok -push origin -Enumerating objects: 8, done. -Counting objects: 100% (8/8), done. -Delta compression using up to 2 threads -Compressing objects: 100% (5/5), done. -Writing objects: 100% (6/6), 675 bytes | 168.00 KiB/s, done. -Total 6 (delta 1), reused 1 (delta 0) -To git://localhost/foo - * [new branch] git-annex -> synced/git-annex - * [new branch] master -> synced/master -ok -"""]] - - -[[!format sh """ -shaddy@jazz-deb:~/tmp/gitresearch/bar$ git annex get motd -get motd (not available) - Try making some of these repositories available: - f75d1e03-3b95-45be-9d30-ca8996e0b69a -- foo -failed -git-annex: get: 1 failed -"""]] - -Now the simple attempt to get the motd file fails: - -[[!format sh """ -shaddy@jazz-deb:~/tmp/gitresearch/bar$ git annex get motd -get motd (not available) - Try making some of these repositories available: - f75d1e03-3b95-45be-9d30-ca8996e0b69a -- foo -failed -git-annex: get: 1 failed -"""]] - -The confirmation that using local remotes in this simple way works file: - -[[!format sh """ -shaddy@jazz-deb:~/tmp/gitresearch/bar$ cd .. -shaddy@jazz-deb:~/tmp/gitresearch$ git clone foo xyz -Cloning into 'xyz'... -done. -shaddy@jazz-deb:~/tmp/gitresearch$ cd xyz -shaddy@jazz-deb:~/tmp/gitresearch/xyz$ git annex init xyz -init xyz (merging origin/git-annex into git-annex...) -(recording state in git...) -ok -(recording state in git...) -shaddy@jazz-deb:~/tmp/gitresearch/xyz$ git annex sync -commit -On branch master -Your branch is up to date with 'origin/master'. - -nothing to commit, working tree clean -ok -pull origin -ok -push origin -Enumerating objects: 8, done. -Counting objects: 100% (8/8), done. -Delta compression using up to 2 threads -Compressing objects: 100% (5/5), done. -Writing objects: 100% (6/6), 707 bytes | 353.00 KiB/s, done. -Total 6 (delta 1), reused 0 (delta 0) -To /home/shaddy/tmp/gitresearch/foo - 9be7f98..1a2bd51 git-annex -> synced/git-annex -ok -shaddy@jazz-deb:~/tmp/gitresearch/xyz$ ls -lL motd -ls: cannot access 'motd': No such file or directory -shaddy@jazz-deb:~/tmp/gitresearch/xyz$ git annex get motd -get motd (from origin...) (checksum...) ok -(recording state in git...) -shaddy@jazz-deb:~/tmp/gitresearch/xyz$ ls -lL motd --r--r--r-- 1 shaddy shaddy 286 May 3 12:08 motd -"""]] - - -### What version of git-annex are you using? On what operating system? - -[[!format sh """ -shaddy@jazz-deb:~/tmp/gitresearch/bar$ lsb_release -a -No LSB modules are available. -Distributor ID: Debian -Description: Debian GNU/Linux 10 (buster) -Release: 10 -Codename: buster -shaddy@jazz-deb:~/tmp/gitresearch/bar$ uname -a -Linux jazz-deb 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux -shaddy@jazz-deb:~/tmp/gitresearch/bar$ git annex version -git-annex version: 7.20190129 -build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external -operating system: linux x86_64 -supported repository versions: 5 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -local repository version: 5 -"""]] - -### Please provide any additional information below. - -Already posted above - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Well... it did work for git's local protocol... - -> [[done]], this was not really a bug, but I have added a warning message -> --[[Joey]] diff --git a/doc/bugs/annex_get_fails_with___39__not_available__39___for_git__58____47____47___protocol/comment_1_f8cc3215ebcafcef7b2c8829ad08928e._comment b/doc/bugs/annex_get_fails_with___39__not_available__39___for_git__58____47____47___protocol/comment_1_f8cc3215ebcafcef7b2c8829ad08928e._comment deleted file mode 100644 index 1b324831aa..0000000000 --- a/doc/bugs/annex_get_fails_with___39__not_available__39___for_git__58____47____47___protocol/comment_1_f8cc3215ebcafcef7b2c8829ad08928e._comment +++ /dev/null @@ -1,68 +0,0 @@ -[[!comment format=mdwn - username="beryllium@5bc3c32eb8156390f96e363e4ba38976567425ec" - nickname="beryllium" - avatar="http://cdn.libravatar.org/avatar/62b67d68e918b381e7e9dd6a96c16137" - subject="Still fails with git-annex 8.20200330 via backports" - date="2020-05-04T09:02:03Z" - content=""" -Just in case there is a concern that version 7 was too old to include potential fixes, I upgraded another machine via Debian backports, to have git-annex 8.20200330. - -Here's the same test playing out: - - -[[!format sh \"\"\" -shaddy@notjazz-debbackports:~/git-research01$ git daemon --export-all --enable=receive-pack --base-path=. & -[1] 4965 -shaddy@notjazz-debbackports:~/git-research01$ git clone git://localhost/foo xyz -Cloning into 'xyz'... -remote: Enumerating objects: 34, done. -remote: Counting objects: 100% (34/34), done. -remote: Compressing objects: 100% (29/29), done. -remote: Total 34 (delta 9), reused 0 (delta 0), pack-reused 0 -Receiving objects: 100% (34/34), done. -Resolving deltas: 100% (9/9), done. -shaddy@notjazz-debbackports:~/git-research01$ cd xyz/ -shaddy@notjazz-debbackports:~/git-research01/xyz$ git annex sync -(merging origin/git-annex into git-annex...) -(recording state in git...) -(scanning for unlocked files...) -commit (recording state in git...) - -On branch master -Your branch is up to date with 'origin/master'. - -nothing to commit, working tree clean -ok -pull origin -ok -push origin -Enumerating objects: 8, done. -Counting objects: 100% (8/8), done. -Delta compression using up to 4 threads -Compressing objects: 100% (5/5), done. -Writing objects: 100% (6/6), 704 bytes | 64.00 KiB/s, done. -Total 6 (delta 1), reused 1 (delta 0), pack-reused 0 -To git://localhost/foo - 2e4b34f..b5ce3d8 git-annex -> synced/git-annex -ok -shaddy@notjazz-debbackports:~/git-research01/xyz$ git annex get motd -get motd (not available) - Try making some of these repositories available: - 02ebc1c9-52d3-4377-bf29-1e4319ce2cf1 -- shaddy@notjazz-debbackports:~/git-research01/foo -failed -git-annex: get: 1 failed -shaddy@notjazz-debbackports:~/git-research01/xyz$ git annex version -git-annex version: 8.20200330 -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: linux x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 0 1 2 3 4 5 6 7 -local repository version: 8 -shaddy@notjazz-debbackports:~/git-research01/xyz$ - -\"\"\"]] - -"""]] diff --git a/doc/bugs/annex_get_fails_with___39__not_available__39___for_git__58____47____47___protocol/comment_2_874c2c2dc99362b559c8fe1715d3c0c4._comment b/doc/bugs/annex_get_fails_with___39__not_available__39___for_git__58____47____47___protocol/comment_2_874c2c2dc99362b559c8fe1715d3c0c4._comment deleted file mode 100644 index 8867addac1..0000000000 --- a/doc/bugs/annex_get_fails_with___39__not_available__39___for_git__58____47____47___protocol/comment_2_874c2c2dc99362b559c8fe1715d3c0c4._comment +++ /dev/null @@ -1,26 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-05-04T16:13:02Z" - content=""" -git-annex cannot possibly support operation over the git protocol, because -that protocol does not have any way to add an extension such as git-annex -to it. - -So git-annex currently completely ignores git:// repositories, the same as -it would any git repository using some other foreign protocol. -(eg rsync or a protocol added by a git extension) - -This is actually identical behavior to if the remote had "url = -/media/whatever" and the drive was unmounted after the repo was cloned. -git-annex is not able to discover a uuid for the remote, and so it does -not know the remote contains the content, and so it does not say anything -about not being able to get the content from the remote, because it -never tried. IOW, this is a "beware of the leopard" situation. - -So, it would better for this to be handled the same way that a ssh remote -that lacks git-annex-shell is handled, by explicitly setting annex-ignore -on first use and displaying a warning. (Although it should not set -annex-ignore for the /media/whatever case, because the drive may later get -mounted again.) -"""]] diff --git a/doc/bugs/async_external_special_remote__39__s_stdin_not_closed.mdwn b/doc/bugs/async_external_special_remote__39__s_stdin_not_closed.mdwn deleted file mode 100644 index a4e7059be9..0000000000 --- a/doc/bugs/async_external_special_remote__39__s_stdin_not_closed.mdwn +++ /dev/null @@ -1,171 +0,0 @@ -### Please describe the problem. - -It seems that git-annex does not close the stdin of an external special remote -when the ASYNC extension is in use. Sometimes git-annex goes ahead anyway, but -sometimes it hangs until it or the remote process is killed. - -### What steps will reproduce the problem? - -Save the following code (a minimal dummy external special remote) into -`git-annex-remote-test` somewhere on PATH and make it executable: - -[[!format python """ -#!/usr/bin/env python3 - -import os -import sys - - -def write(s): - print(s) - sys.stdout.flush() - - -write("VERSION 1") - -while True: - line = sys.stdin.readline().strip() - if not line: - print("got EOF", file=sys.stderr) - break - - cmd, *args = line.split() - prefix = "" - if cmd == "J": - prefix = f"J {args[0]} " - cmd, *args = args[1:] - - if cmd == "EXTENSIONS": - if os.environ.get("TEST_ASYNC_ON") == "1": - write("EXTENSIONS INFO ASYNC") - else: - write("EXTENSIONS INFO") - elif cmd == "INITREMOTE": - write(prefix + "INITREMOTE-SUCCESS") - elif cmd == "PREPARE": - write(prefix + "PREPARE-SUCCESS") - elif cmd == "TRANSFER": - tp, key, fn = args - write(prefix + f"TRANSFER-SUCCESS {tp} {key}") - elif cmd == "CHECKPRESENT": - (key,) = args - write(prefix + f"CHECKPRESENT-SUCCESS {key}") - elif cmd == "REMOVE": - (key,) = args - write(prefix + f"REMOVE-SUCCESS {key}") - else: - write(prefix + "UNSUPPORTED-REQUEST") -"""]] - -Run the following script: - -[[!format sh """ -#!/bin/sh - -PS4='================ ' -set -ex - -cd "$(mktemp -d)" -git init -git annex init -git annex initremote test type=external externaltype=test encryption=none -touch file -git annex add file -git annex copy --to test file -"""]] - -If the `TEST_ASYNC_ON` environment variable is set to `1`, the dummy remote will -advertise ASYNC support. In that case, the `git annex copy` command will hang -until it or the remote process is manually killed. It looks like the -`initremote` command also does not close the stdin of the remote processes that -it spawns (as seen by the lack of `got EOF` lines in the terminal output), but -it is able to complete successfully anyway. - -If ASYNC is not used, all commands complete quickly and successfully. - -### What version of git-annex are you using? On what operating system? - -git-annex 8.20201116-g864af53a2d on Arch Linux. - -[[!format txt """ -$ git-annex version -git-annex version: 8.20201116-g864af53a2d -build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV -dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.10.2 http-client-0.7.3 persistent-sqlite-2.11.0.0 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external -operating system: linux x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 0 1 2 3 4 5 6 7 - -$ uname -a -Linux dzhu-p52 5.9.9-arch1-1 #1 SMP PREEMPT Wed, 18 Nov 2020 19:52:04 +0000 x86_64 GNU/Linux -"""]] - -### Please provide any additional information below. - -I've seen the same behavior with a compiled Go executable as the special remote, -so this does not appear to be a Python-specific buffering issue, at least. - -Output I see from running the above code (`asynctest.sh` is the shell script): - -[[!format txt """ -$ ./asynctest.sh -================= mktemp -d -================ cd /tmp/tmp.5QyEIgAUUJ -================ git init -Initialized empty Git repository in /tmp/tmp.5QyEIgAUUJ/.git/ -================ git annex init -init (scanning for unlocked files...) -ok -(recording state in git...) -================ git annex initremote test type=external externaltype=test encryption=none -initremote test got EOF -got EOF -got EOF -ok -(recording state in git...) -================ touch file -================ git annex add file -add file -ok -(recording state in git...) -================ git annex copy --to test file -copy file ok -(recording state in git...) -got EOF - -$ TEST_ASYNC_ON=1 ./asynctest.sh -================= mktemp -d -================ cd /tmp/tmp.cyiP80y3kE -================ git init -Initialized empty Git repository in /tmp/tmp.cyiP80y3kE/.git/ -================ git annex init -init (scanning for unlocked files...) -ok -(recording state in git...) -================ git annex initremote test type=external externaltype=test encryption=none -initremote test ok -(recording state in git...) -================ touch file -================ git annex add file -add file -ok -(recording state in git...) -================ git annex copy --to test file -copy file ok -(recording state in git...) -"""]] -and then it gets stuck there. - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Yes! My photo collection and other miscellaneous files were getting out of hand, -with tens of thousands of files spread across various drives and cloud services -with varying amounts of ad hoc duplication. git-annex is helping me get -everything under control for the first time; I'm not all the way there yet, but -I finally have an idea of how to handle it all. I started developing something -to do the file management myself, but embraced git-annex wholeheartedly once I -grokked it. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/async_external_special_remote__39__s_stdin_not_closed/comment_1_76518f4ba7aec2f5ea0b14bf371f488e._comment b/doc/bugs/async_external_special_remote__39__s_stdin_not_closed/comment_1_76518f4ba7aec2f5ea0b14bf371f488e._comment deleted file mode 100644 index e60ec017f4..0000000000 --- a/doc/bugs/async_external_special_remote__39__s_stdin_not_closed/comment_1_76518f4ba7aec2f5ea0b14bf371f488e._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="comment 1" - date="2020-11-30T15:18:57Z" - content=""" -Thanks for the reproducer. I tested it out locally, hoping I could -report that the hang was resolved in the 8.20201127 release -(specifically by 8.20201116-47-gd8b7f6721 [*]). Unfortunately, your -example still hangs for me on master (708181175). As with the linked -report below, it looks like that hang starts with 7245a9ed5 (Improve -shutdown process for external special remotes and external backends, -2020-11-02). - -[*] related report: - https://git-annex.branchable.com/bugs/Buggy_external_special_remote_stalls_after_7245a9e/ - -"""]] diff --git a/doc/bugs/async_external_special_remote__39__s_stdin_not_closed/comment_2_db65c2eeb11a9922ca35872a45acd61c._comment b/doc/bugs/async_external_special_remote__39__s_stdin_not_closed/comment_2_db65c2eeb11a9922ca35872a45acd61c._comment deleted file mode 100644 index 73251fdd45..0000000000 --- a/doc/bugs/async_external_special_remote__39__s_stdin_not_closed/comment_2_db65c2eeb11a9922ca35872a45acd61c._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-11-30T16:40:41Z" - content=""" -Analysis: It's hanging on 'hClose hout'. - -I think that's because there's a thread that is waiting for the next line -of output from the remote. hClose ends up taking a MVar, if the read -keeps that MVar unpopulate hClose can then hang. This is surprising; I've -never seen hClose hang, and the docs don't mention it can. - -Solution will be to cleanly shutdown the async IO threads before shutting -down the external process. -"""]] diff --git a/doc/bugs/async_external_special_remote__39__s_stdin_not_closed/comment_3_481734c0f34ab14129dc313bcf575ff1._comment b/doc/bugs/async_external_special_remote__39__s_stdin_not_closed/comment_3_481734c0f34ab14129dc313bcf575ff1._comment deleted file mode 100644 index fbb2b3b9c2..0000000000 --- a/doc/bugs/async_external_special_remote__39__s_stdin_not_closed/comment_3_481734c0f34ab14129dc313bcf575ff1._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="dzhu" - avatar="http://cdn.libravatar.org/avatar/798e5065b15706c694d2b5ab56f9a12e" - subject="comment 3" - date="2020-11-30T19:10:53Z" - content=""" -Thanks for the very fast response and fix! I look forward to using and developing async external special remotes. -"""]] diff --git a/doc/bugs/automerge_leaves_cruft_behind.mdwn b/doc/bugs/automerge_leaves_cruft_behind.mdwn deleted file mode 100644 index 53e4e79670..0000000000 --- a/doc/bugs/automerge_leaves_cruft_behind.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -Seeing `git-annex sync` after resolving a merge conflict between a -directory and an annexed file leave behind -files like `foo~refs_remotes_origin_master`, which are not checked into -git. - -Also happens with merge.directoryRenames=conflict - -Seems to only happen sometimes, unsure why.. Maybe git doesn't always -write those files, or sometimes something done with git deletes them? - -Root cause seems to be that it's making an InodeCache, but that can't -be done of a broken symlink. So, it's probably a reversion introduced with -v7 unlocked files or so. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/behavior_change_on_unstaged_deleted_files.mdwn b/doc/bugs/behavior_change_on_unstaged_deleted_files.mdwn deleted file mode 100644 index 94296af19e..0000000000 --- a/doc/bugs/behavior_change_on_unstaged_deleted_files.mdwn +++ /dev/null @@ -1,17 +0,0 @@ -Files that have been deleted but not staged are being listed by git-annex -commands. This is a recent reversion. - -Does not seem to be due to the changes to how ls-files is used, as the old -way also listed deleted files -- how did they get filtered out before and -why did it break? --[[Joey]] - -> It's from [[!commit 88a7fb5cbb7358f8d395f6c306fb9e94e1f1a724]]. -> Things used to use whenAnnexed, which uses lookupKey, which checks -> doesFileExist, so deleted files were filtered out there. So -> seekFilteredKeys should also check that. -> -> Also, there should be a test case for this, IIRC there was one past case -> of a bug involving this and it's a bit of an easy edge case to forget -> about. --[[Joey]] - -[[done]] diff --git a/doc/bugs/borg_uses_non_unique_contentidentifier_which_gets_lost.mdwn b/doc/bugs/borg_uses_non_unique_contentidentifier_which_gets_lost.mdwn deleted file mode 100644 index a5a4febacd..0000000000 --- a/doc/bugs/borg_uses_non_unique_contentidentifier_which_gets_lost.mdwn +++ /dev/null @@ -1,28 +0,0 @@ -borg uses a non-unique ContentIdentifier ("") for everything. -I think this is why, it eventually gets lost from the sqlite database for -some keys, preventing retrieval of content from the remote. - -Repositories affected by this problem can be fixed up by just: -`rm -rf .git/annex/cidsdb` - -The ContentIdentifiers table has a -"ContentIndentifiersCidRemoteIndex cid remote", and that's not just an -index, it's a uniqueness constraint. - -And that makes sense generally, the point of a ContentIdentifier is that -wherever a remote uses it, it identifies the same content. - -I think sqlite probably lets things be added -that violate the constraint at first, but then later writes it removes -the "non-unique" row. Which in this case associates the same cid with -a different key. - -I'm thinking this was a mistaken optimisation. getContentIdentifierKeys -is supposed to return a [Key] for a ContentIdentifier; there can be more -than one and it contains code that assumes it will get back all of them. -And if a remote uses a hash for generating ContentIdentifiers, two different -Key can have the same content in edge cases. - -So, need to upgrade the database, removing this constraint from it. - ->> [[done]] --[[Joey]] diff --git a/doc/bugs/brew_install_git-annex_failed.mdwn b/doc/bugs/brew_install_git-annex_failed.mdwn deleted file mode 100644 index e75a66cdcd..0000000000 --- a/doc/bugs/brew_install_git-annex_failed.mdwn +++ /dev/null @@ -1,50 +0,0 @@ -### Please describe the problem. - -brew install git-annex failed. - -### What steps will reproduce the problem? - - -### What version of git-annex are you using? On what operating system? -Mac OS X 10.11 El Capitan - -### Please provide any additional information below. - -[[!format sh """ -95b3d57ce7--git-annex-7.20191230.tar.gz -==> cabal v1-sandbox init -==> cabal v1-update -==> cabal v1-install --jobs=4 --max-backjumps=100000 alex -==> cabal v1-install --jobs=4 --max-backjumps=100000 happy -==> cabal v1-install --jobs=4 --max-backjumps=100000 c2hs -==> cabal v1-install --jobs=4 --max-backjumps=100000 --only-dependencies --constraint http-conduit>=2.3 --constraint net -==> cabal v1-configure --flags=s3 webapp -==> cabal v1-install --jobs=4 --max-backjumps=100000 --prefix=/usr/local/Cellar/git-annex/7.20191230 --constraint http-c -Last 15 lines from /Users/choi/Library/Logs/Homebrew/git-annex/08.cabal: -StandaloneDeriving - -Please enable the extensions by copy/pasting these lines into the top of your file: - -{-# LANGUAGE DerivingStrategies #-} -{-# LANGUAGE StandaloneDeriving #-} - | -31 | share [mkPersist sqlSettings, mkMigrate "migrateKeysDb"] [persistLowerCase| - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^... -cabal: Leaving directory '.' -cabal: Error: some packages failed to install: -git-annex-7.20191230-JGm7b2Gk5I8w0hi2BDCiw failed during the building phase. -The exception was: -ExitFailure 1 - - -Do not report this issue to Homebrew/brew or Homebrew/core! - -These open issues may also help: -git-annex-remote-rclone 0.6 (new formula) https://github.com/Homebrew/homebrew-core/pull/49468 -git-annex: add OBJC_DISABLE_INITIALIZE_FORK_SAFETY environment variable https://github.com/Homebrew/homebrew-core/pull/48411 -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - - -> [[fixed|done]] in git-annex master --[[Joey]] diff --git a/doc/bugs/brew_install_git-annex_failed/comment_1_57b2c2c989311b4db01aca8d07802207._comment b/doc/bugs/brew_install_git-annex_failed/comment_1_57b2c2c989311b4db01aca8d07802207._comment deleted file mode 100644 index f8e923f11b..0000000000 --- a/doc/bugs/brew_install_git-annex_failed/comment_1_57b2c2c989311b4db01aca8d07802207._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="nrg@bd619d1ebf16e6324c546adea8be8fe1cc2b4325" - nickname="nrg" - avatar="http://cdn.libravatar.org/avatar/428b6c95b52769cf9eecdd351018eacb" - subject="Confirmed with macOS 10.14.6 building git-annex-7.20200202.7" - date="2020-02-03T23:10:22Z" - content=""" -The issue is also seen [here](https://github.com/Homebrew/homebrew-core/pull/49731). -"""]] diff --git a/doc/bugs/brew_install_git-annex_failed/comment_2_a398db56168954d43620620104215b14._comment b/doc/bugs/brew_install_git-annex_failed/comment_2_a398db56168954d43620620104215b14._comment deleted file mode 100644 index d6931cda0f..0000000000 --- a/doc/bugs/brew_install_git-annex_failed/comment_2_a398db56168954d43620620104215b14._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="nrg@bd619d1ebf16e6324c546adea8be8fe1cc2b4325" - nickname="nrg" - avatar="http://cdn.libravatar.org/avatar/428b6c95b52769cf9eecdd351018eacb" - subject="Change introduced by persistent-sqlite and persistent-template" - date="2020-02-04T15:09:59Z" - content=""" -The change was introduced [here](https://github.com/yesodweb/persistent/commit/6ca1c2401f228293c64ae05e6109d4936b98c4b9). -"""]] diff --git a/doc/bugs/brew_install_git-annex_failed/comment_3_627876a558898bb9f622b80dbacee051._comment b/doc/bugs/brew_install_git-annex_failed/comment_3_627876a558898bb9f622b80dbacee051._comment deleted file mode 100644 index 373cb3e662..0000000000 --- a/doc/bugs/brew_install_git-annex_failed/comment_3_627876a558898bb9f622b80dbacee051._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-02-04T15:56:00Z" - content=""" -I think I've fixed this in master now, but have not been able to test the fix -yet since I don't have the bandwidth to upgrade. -"""]] diff --git a/doc/bugs/checkpresentkey_fails_on_read-only_remote.mdwn b/doc/bugs/checkpresentkey_fails_on_read-only_remote.mdwn deleted file mode 100644 index cf253e1dce..0000000000 --- a/doc/bugs/checkpresentkey_fails_on_read-only_remote.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -In an external special remote, if I set annex-readonly=true, I get - -[[!format sh """ - git annex --verbose --debug checkpresentkey MD5E-s18932--663e2e38c829d7b3fff12fce3a6fdb6d.fasta 380286ac-2e8f-4285-94da-406eca323411 -... -(checking dnanexus...) Configuration does not allow accessing dx://file-BXF0vYQ0QyBF509G9J12g927/contaminants.clip_db.fasta -"""]] - -Removing the annex-readonly setting lets checkpresentkey work. But isn't checkpresentkey a read-only operation? - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_1_6222abd4764f397db7a1912cecb637d3._comment b/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_1_6222abd4764f397db7a1912cecb637d3._comment deleted file mode 100644 index 8c1fe491d6..0000000000 --- a/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_1_6222abd4764f397db7a1912cecb637d3._comment +++ /dev/null @@ -1,23 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-05-28T14:29:45Z" - content=""" -The error message is git-annex trying to access a "dx://" url, which is not -allowed by the default configuration. (Also which git-annex would have no -idea how to access even if it were allowed.) - -Setting readonly=true for a special remote prevents using the external -special remote program at all, which would presumably do something that -worked better if it ran. - -This might be a bug in the special remote you're using, if its -documentation suggests it will work in readonly mode and it only registers -these "dx://" urls that can't work in readonly mode. - -Did you set remote.name.annex-readonly manually, or did you run -git annex enableremote with readonly=true? Only the latter is documented to -disable use of the special remote program, which could be considered a -documentation bug in git-annex. That's the only potential git-annex bug I'm -seeing here. -"""]] diff --git a/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_2_ac6dc78235484ee20906dfadaf7506e5._comment b/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_2_ac6dc78235484ee20906dfadaf7506e5._comment deleted file mode 100644 index 74978945fe..0000000000 --- a/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_2_ac6dc78235484ee20906dfadaf7506e5._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="readonly special remotes]" - date="2019-05-28T17:25:19Z" - content=""" -\"Setting readonly=true for a special remote prevents using the external special remote program at all\" -- that's the part I didn't realize. If I understand correctly: - -1. for all remotes, setting `readonly=true` \"prevents git-annex from making changes to a remote... this both prevents git-annex sync from pushing changes, and prevents storing or removing files from read-only remote.\" -2. additionally, for special remotes, this \"prevents using the external special remote program at all\", and instead causes `git-annex` to be [downloading the content of a file using a regular http connection, with no authentication](https://git-annex.branchable.com/design/external_special_remote_protocol/#index9h2)? - -If so, it would be best to add a separate config setting for (2). I'm not sure, though, why that case needs special handling. If a key's contents is available at a regular `http` URL, shouldn't the key be recorded as present in the built-in web special remote, rather than the user-defined special remote? - -The external special remote here is one I wrote myself, for accessing data on [DNAnexus](https://www.dnanexus.com/). -"""]] diff --git a/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_3_04859f6abcb548f32ff6ce12d858fae5._comment b/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_3_04859f6abcb548f32ff6ce12d858fae5._comment deleted file mode 100644 index 93f35a8057..0000000000 --- a/doc/bugs/checkpresentkey_fails_on_read-only_remote/comment_3_04859f6abcb548f32ff6ce12d858fae5._comment +++ /dev/null @@ -1,50 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-04-23T17:13:42Z" - content=""" -So, enableremote readonly=true sets remote.name.annex-readonly -in git config. And it assumes that, if that is set, you don't want to use -the external special remote program at all, but instead a previous use of -it elsewhere should have stored the urls where git-annex can download content -stored in that remote. - -The use case for enableremote readonly=true is that you want to provide a -way for users to get content you have stored in your remote without the -bother of installing a third-party program to access it. While you could -set remote.name.annex-readonly in git config after enableremote, -you would need to have the program installed for the enableremote step, -and readonly=true avoids that. - -It's the same as `git annex copy --to s3` registering the content as stored -in the s3 remote, not the web remote, even if git-annex knows there's a -publically available url that can be used to access it. - -If git-annex treated that as also storing the content to the web remote, -then it would be maintaining two sets of books for the same copy of the file. -So instead the user needs to enable use of the s3 remote -(even if without any S3 creds in a necessarily read-only mode) in order for -git-annex to access files stored on it. And same with these external -special remotes, except their code is not built into git-annex, so -readonly=true provides a way to not need to run their code at all. - -Now, you did not pass readonly=true to enableremote from what I -understand, but instead came along later and set -remote.name.annex-readonly=true in git config. And I think your goal was to -keep using the special remote program for eg downloads, but prevent -writing to the remote. - -So yes, it would be better if enableremote readonly=true set some other -config than remote.name.annex-readonly. As it is, there's no way to -distinguish between the two use cases. - ---- - -Ok, I'm gonna make remote.name.annex-externaltype=readonly be a special -case that avoids running the external special remote program. - -In the case where no program is available, it will check if -remote.name.annex-readonly is set, and if so when it fails it will -suggest the user might want to set annex-externaltype=readonly to deal with -this change. -"""]] diff --git a/doc/bugs/checkpresentkey_false_negatives_in_bare_repos.mdwn b/doc/bugs/checkpresentkey_false_negatives_in_bare_repos.mdwn deleted file mode 100644 index 1082fb62cf..0000000000 --- a/doc/bugs/checkpresentkey_false_negatives_in_bare_repos.mdwn +++ /dev/null @@ -1,49 +0,0 @@ -### Please describe the problem. -git annex checkpresentkey does not refuse to run in bare repos, as many commands do, but seems to report all keys as absent. - - -### What steps will reproduce the problem? -This is a simple example starting from nothing, but in every bare repo I test, regardless if I use batch or not, all keys are reported absent. Observe that the key can be cat'd directly out of the git annex directory in both bare and non-bare, was moved to the bare via git annex sync, yet shows 1 for present in the non-bare and 0 for the bare: - -[[!format sh """ -mkdir repro -cd repro -mkdir bare nonbare -cd bare -git init --bare -cd ../nonbare -git clone ../bare . -git annex init -echo hello > greeting -git annex add greeting -git commit -m update -git annex sync --content -git annex sync --content -echo SHA256E-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03 | git annex checkpresentkey --batch -cat .git/annex/objects/*/*/SHA256E-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03/SHA256E-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03 -cd ../bare -echo SHA256E-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03 | git annex checkpresentkey --batch -cat annex/objects/*/*/SHA256E-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03/SHA256E-s6--5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03 -"""]] - - - -### What version of git-annex are you using? On what operating system? -Debian Testing, git annex version 8.20200330-1 - - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Oh it's great, especially the low level plumbing commands. I use it for all my backups by first backup up each machine into a Bup repo, and then storing the bup packs/indices in git-annex, which I use for sync'ing across machines. - -> [[notabug|done]] --[[Joey]] diff --git a/doc/bugs/checkpresentkey_false_negatives_in_bare_repos/comment_1_a081b3e4bd25f6d18fe8cdcd3385ae94._comment b/doc/bugs/checkpresentkey_false_negatives_in_bare_repos/comment_1_a081b3e4bd25f6d18fe8cdcd3385ae94._comment deleted file mode 100644 index fb60e1c882..0000000000 --- a/doc/bugs/checkpresentkey_false_negatives_in_bare_repos/comment_1_a081b3e4bd25f6d18fe8cdcd3385ae94._comment +++ /dev/null @@ -1,24 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-06-16T16:37:24Z" - content=""" -git-annex checkpresentkey checks if a key is present in remotes. - -Your bare repository does not have any remotes that contain the key -(due to not having any remotes at all). This is why checkpresentkey is -behaving like this. - -The same behavior happens if you remove the origin remote from the nonbare -repo and run the command there. - -The documentation seems clear enough about this behavior, although there -is the use of "somewhere" in this part: - - When no remote is specified, it verifies if the key's content is - present somewhere, checking accessible remotes until it finds the con‐ - tent. - -Which I suppose could be interpreted to mean, if the key -is present in the local repo, it's "somewhere". -"""]] diff --git a/doc/bugs/checkpresentkey_wrongly_reports_key_absense.mdwn b/doc/bugs/checkpresentkey_wrongly_reports_key_absense.mdwn deleted file mode 100644 index ebc08e239b..0000000000 --- a/doc/bugs/checkpresentkey_wrongly_reports_key_absense.mdwn +++ /dev/null @@ -1,86 +0,0 @@ -(master_env_v082_py36) 00:55 [ilya-work] $ git annex checkpresentkey MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam 19c05b2b-b155-4ac3-a9e7-fa7c8d67e430 -(master_env_v082_py36) 00:55 [ilya-work] $ echo $? -0 -(master_env_v082_py36) 00:55 [ilya-work] $ git annex checkpresentkey MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam -(master_env_v082_py36) 00:55 [ilya-work] $ echo $? -1 - -(master_env_v082_py36) 00:55 [ilya-work] $ git annex whereis --key MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam -whereis MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam (1 copy) - 19c05b2b-b155-4ac3-a9e7-fa7c8d67e430 -- [dnanexus] - - dnanexus: dx://file-FBf5qFj06KBfVygf2pz3pXPQ/1375.cleaned.bam -ok -(master_env_v082_py36) 00:57 [ilya-work] $ git annex readpresentkey MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam 19c05b2b-b155-4ac3-a9e7-fa7c8d67e430 -(master_env_v082_py36) 00:57 [ilya-work] $ echo $? -0 - -(master_env_v082_py36) 00:57 [ilya-work] $ git annex --verbose --debug checkpresentkey MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam 19c05b2b-b155-4ac3-a9e7-fa7c8d67e430 -[2019-02-14 00:58:32.575463] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] -[2019-02-14 00:58:32.583744] process done ExitSuccess -[2019-02-14 00:58:32.583809] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] -[2019-02-14 00:58:32.592023] process done ExitSuccess -[2019-02-14 00:58:32.592656] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..5dde7d61a8183a50acf98e0847650df20547a0c5","--pretty=%H","-n1"] -[2019-02-14 00:58:32.608022] process done ExitSuccess -[2019-02-14 00:58:32.608086] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..aee29af11409cba48bb52ef5893249f5c964180c","--pretty=%H","-n1"] -[2019-02-14 00:58:32.830193] process done ExitSuccess -[2019-02-14 00:58:32.830306] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..818a062162bbac26adfd3d30f5ab076d3d0b77e0","--pretty=%H","-n1"] -[2019-02-14 00:58:32.844831] process done ExitSuccess -[2019-02-14 00:58:32.84522] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"] -[2019-02-14 00:58:32.845628] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] -[2019-02-14 00:58:32.860656] read: git ["config","--null","--list"] -[2019-02-14 00:58:32.874021] process done ExitSuccess -[2019-02-14 00:58:32.881706] chat: /data/ilya-work/viral-ngs/is-1808252024-add-benchmarks/tools/git-annex-remotes/git-annex-remote-dnanexus [] -[2019-02-14 00:58:32.924132] git-annex-remote-dnanexus[1] --> VERSION 1 -[2019-02-14 00:58:32.924215] git-annex-remote-dnanexus[1] <-- EXTENSIONS INFO -[2019-02-14 00:58:32.924288] git-annex-remote-dnanexus[1] --> EXTENSIONS -[2019-02-14 00:58:32.924322] git-annex-remote-dnanexus[1] <-- PREPARE -[2019-02-14 00:58:32.924372] git-annex-remote-dnanexus[1] --> PREPARE-SUCCESS -[2019-02-14 00:58:32.9244] git-annex-remote-dnanexus[1] <-- CHECKPRESENT MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam -[2019-02-14 00:58:32.924461] git-annex-remote-dnanexus[1] --> DEBUG CHECKING PRESENCE OF KEY MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam -[2019-02-14 00:58:32.924491] CHECKING PRESENCE OF KEY MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam -[2019-02-14 00:58:32.924512] git-annex-remote-dnanexus[1] --> GETURLS MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam dx:// -[2019-02-14 00:58:32.925707] git-annex-remote-dnanexus[1] <-- VALUE dx://file-FBf5qFj06KBfVygf2pz3pXPQ/1375.cleaned.bam -[2019-02-14 00:58:32.925755] git-annex-remote-dnanexus[1] <-- VALUE -[2019-02-14 00:58:33.399433] git-annex-remote-dnanexus[1] --> CHECKPRESENT-SUCCESS MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995.bam -(master_env_v082_py36) 00:58 [ilya-work] $ --(master_env_v082_py36) 00:58 [ilya-work] $ git annex --verbose --debug checkpresentkey MD5E-s622177--a8c1aebb39d2093b08ce81a13661a995\ -.bam -[2019-02-14 01:00:22.921333] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] -[2019-02-14 01:00:22.929444] process done ExitSuccess -[2019-02-14 01:00:22.929505] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-ann\ -ex"] -[2019-02-14 01:00:22.937493] process done ExitSuccess -[2019-02-14 01:00:22.938154] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..5dde7d61a8\ -183a50acf98e0847650df20547a0c5","--pretty=%H","-n1"] -[2019-02-14 01:00:22.952747] process done ExitSuccess -[2019-02-14 01:00:22.952809] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..aee29af114\ -09cba48bb52ef5893249f5c964180c","--pretty=%H","-n1"] -[2019-02-14 01:00:23.174093] process done ExitSuccess -[2019-02-14 01:00:23.174196] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..818a062162\ -bbac26adfd3d30f5ab076d3d0b77e0","--pretty=%H","-n1"] -[2019-02-14 01:00:23.189329] process done ExitSuccess -[2019-02-14 01:00:23.189644] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"] -[2019-02-14 01:00:23.189965] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname)\ - %(objecttype) %(objectsize)"] -[2019-02-14 01:00:23.204081] read: git ["config","--null","--list"] -[2019-02-14 01:00:23.21793] process done ExitSuccess - -(master_env_v082_py36) 01:00 [ilya-work] $ git annex version -git-annex version: 7.20181211-g1cdb7a2 -build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.2 feed-1.0.0.0 ghc-8.2.2 http-client-0.5.13.1 persistent-sql\ -ite-2.8.1.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_2\ -24 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE\ -2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256\ - BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external -operating system: linux x86_64 -supported repository versions: 5 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -local repository version: 5 - -Linux ip-172-31-85-193 4.14.97-74.72.amzn1.x86_64 #1 SMP Tue Feb 5 20:59:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/checkpresentkey_wrongly_reports_key_absense/comment_1_e5f07ee71004e9d6358035cbfb9d1651._comment b/doc/bugs/checkpresentkey_wrongly_reports_key_absense/comment_1_e5f07ee71004e9d6358035cbfb9d1651._comment deleted file mode 100644 index 4e16b60c23..0000000000 --- a/doc/bugs/checkpresentkey_wrongly_reports_key_absense/comment_1_e5f07ee71004e9d6358035cbfb9d1651._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-03-18T17:52:03Z" - content=""" -I see a transcript of .. something .. with no explanation of what's going -on. Everything in the transcript looks more or less reasonable? - -I could spend an hour or ten trying to guess at what bug is, but it seems more -efficient for you to explain what you expected it do and why you think -the actual behavior is wrong? -"""]] diff --git a/doc/bugs/checkpresentkey_wrongly_reports_key_absense/comment_2_0a7bb6b12a4eb49addf443db2d16553f._comment b/doc/bugs/checkpresentkey_wrongly_reports_key_absense/comment_2_0a7bb6b12a4eb49addf443db2d16553f._comment deleted file mode 100644 index e454d38bf7..0000000000 --- a/doc/bugs/checkpresentkey_wrongly_reports_key_absense/comment_2_0a7bb6b12a4eb49addf443db2d16553f._comment +++ /dev/null @@ -1,26 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-06-16T17:35:41Z" - content=""" -So I guess the problem description is: - -checkpresentkey with the uuid of a remote verifies the key is on the -remote, return 1. But, with no uuid, it's supposed to check all remotes, -and instead it seems to fail and return 0 without checking the remote -whose uuid the earlier run of it showed does contain the key. - -And I think I see why it would do that. When no remote is given, it reads -the location log, and checks each remote that the location log says -contains the key. So, if the location log is out of date and a remote does -contain the key, it will return 0 w/o checking it. - -It would probably make sense for it to actually try all accessible remotes, -on the off chance one of them has the key. That's what the documentation -says it does, although I don't think it ever has. - -The original use case for this was -[[todo/checkpresentkey_without_explicit_remote]], and it does seem like -that use case intended to check all remotes, not only ones the location log -has it. -"""]] diff --git a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails.mdwn b/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails.mdwn deleted file mode 100644 index 3a358edaf7..0000000000 --- a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails.mdwn +++ /dev/null @@ -1,65 +0,0 @@ -### Please describe the problem. - -I'm trying to copy a directory full of photos (~9GB) to an S3 special remote. It seems to fail when running multiple jobs. The machine I am using has two cores (4 logical due to hyperthreading) and I initially used -J8 and that failed quickly. I haven't retried that yet to see if it always fails quickly but I immediately retried with -J4 and that seemed to run okay for a little over an hour and a half before failing again. This is the second large directory I've tried to copy to this remote and had a similar problem the last time (many retries seemed to allow me to get through). - - -### What steps will reproduce the problem? - - -### What version of git-annex are you using? On what operating system? - -git-annex was installed via homebrew on macos 10.14.5 - -[[!format sh """ - -15:23 $ git annex version -git-annex version: 7.20190322 -build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.19 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.2.2 http-client-0.5.14 persistent-sqlite-2.6.4 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLA -KE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external -operating system: darwin x86_64 -supported repository versions: 5 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 - -"""]] - - -### Please provide any additional information below. - -[[!format sh """ - -22:56 $ git annex copy -J8 photos/ --to s3 -copy photos/2001/01/IMAG0003.JPG (checking s3...) (to s3...) ok -copy photos/2001/01/IMAG0006.JPG (checking s3...) (to s3...) -88% 319.84 KiB 68 KiB/s 0s -copy photos/2001/01/IMAG0001-1.JPG (checking s3...) (to s3...) -63% 255.88 KiB 25 KiB/s 6s -copy photos/2001/01/IMAG0002.JPG (checking s3...) (to s3...) -63% 223.89 KiB 17 KiB/s 7s -copy photos/2001/01/IMAG0004.JPG (checking s3...) (to s3...) -copy photos/2001/01/IMAG0005.JPG (checking s3...) (to s3...) -80% 287.86 KiB 159 KiB/s 0s -copy photos/2001/01/IMAG0001.JPG (checking s3...) (to s3...) -74% 255.88 KiB 49 KiB/s 1s -copy photos/2001/01/IMAG0002-1.JPG (checking s3...) (to s3...) - ParseError {errorContexts = [], errorMessage = "Failed reading: takeWhile1", errorPosition = 4:2 (285)} -copy photos/2001/01/IMAG0007.JPG (checking s3...) (to s3...) -10% 31.98 KiB 102 KiB/s 2s -git-annex(21112,0x70000dd5b000) malloc: Incorrect checksum for freed object 0x7fe0b3e02bf8: probably modified after being freed. -Corrupt value: 0x6574636f2f6e6f69 -git-annex(21112,0x70000dd5b000) malloc: *** set a breakpoint in malloc_error_break to debug -error: git-annex died of signal 6 - - -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - - -I'm a new git-annex user and besides these occasional issues with concurrent with git-annex-copy I found it really easy to get going with git-annex and to manage my imported files. In particular I really appreciate the extensive documentation! - -> [[done]], at least the parts of this that I was able to reproduce are -> fixed and it seems quite likely the same underlying problem could cause -> other strange behavior. --[[Joey]] diff --git a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_1_31242c64bc6d9760bca99687d4a5c1c8._comment b/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_1_31242c64bc6d9760bca99687d4a5c1c8._comment deleted file mode 100644 index e7a93d9aa2..0000000000 --- a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_1_31242c64bc6d9760bca99687d4a5c1c8._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="issues with concurrent copy" - date="2019-08-05T17:53:37Z" - content=""" -I've seen similar issues with concurrent copy: [[bugs/parallel_copy_fails]] -"""]] diff --git a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_2_9dfe48a6ee8247d1db60ca6bde5dae16._comment b/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_2_9dfe48a6ee8247d1db60ca6bde5dae16._comment deleted file mode 100644 index cdb27ee872..0000000000 --- a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_2_9dfe48a6ee8247d1db60ca6bde5dae16._comment +++ /dev/null @@ -1,50 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="another trace of concurrent copy failure" - date="2019-09-05T06:09:13Z" - content=""" -Some more concurrent copy failures when non-concurrent works: - -[[!format sh \"\"\" -(master_env_v156_py36) 23:35 [viral-ngs-benchmarks] $ git annex copy -J16 --to viral-ngs-benchmarks-s3 --not --in=viral-ngs-benchmark\ -s-s3 --and --not --in=mygs --and --not --in=dnanexus -copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades-3.13.0/files/calls/assemble_denovo/assemb\ -le/0/outputs/subsampBam/LASV_NGA_2016_0014.ll2.subsamp.bam (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok -copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades-3.13.0/files/calls/assemble_denovo/refine\ -_2x_and_plot/0/outputs/aligned_bam_idx/LASV_NGA_2016_0014.ll2.all.bai (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s\ -3...) ok -copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades-3.13.0/files/calls/assemble_denovo/refine\ -_2x_and_plot/0/outputs/aligned_only_reads_bam/LASV_NGA_2016_0014.ll2.mapped.bam (checking viral-ngs-benchmarks-s3...) (to viral-ngs-be\ -nchmarks-s3...) ok -... - -copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades-3.13.0/cromwell_submit_output.txt (checki\ -ng viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) - S3Error {s3StatusCode = Status {statusCode = 403, statusMessage = \"Forbidden\"}, s3ErrorCode = \"SignatureDoesNotMatch\", s3ErrorMessag\ -e = \"The request signature we calculated does not match the signature you provided. Check your key and signing method.\", s3ErrorResour\ -ce = Nothing, s3ErrorHostId = Just \"7uk2qBwWawhwYhROH7SzY+i3Y/49OQ9OiIT81eRkoEvZowo56xVkATE9qPU4Zs78x4C7gPsu744=\", s3ErrorAccessKeyId \ -= Just \"AKIAXXXXXXXXXXXXXXX\", s3ErrorStringToSign = Just \"PUT\n\nin\nThu, 05 Sep 2019 03:35:47 GMT\nx-amz-storage-class:STANDARD\n/vi\ -ral-ngs-benchmarks/MD5E-s628-S16777216-C1--065d8c01e5f7bd3b997c6d24109005c7.txt\", s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothin\ -g, s3ErrorEndpoint = Nothing} -ok - -... -copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades/files/calls/assemble_denovo/refine_2x_and\ -_plot/0/outputs/aligned_only_reads_bam/LASV_NGA_2016_0014.ll2.mapped.bam (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmark\ -s-s3...) ok -copy benchmarks/intermed/analysis-FBGGXX00YJ80114YGQ0yKzFG/benchmark_variants/v1.23.0_spades/files/calls/assemble_denovo/assemble/0/st\ -dout/stdout (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok -double free or corruption (out) -error: git-annex died of signal 6 -(master_env_v156_py36) 23:35 [viral-ngs-benchmarks] $ git annex copy --to viral-ngs-benchmarks-s3 --not --in=viral-ngs-benchmarks-s3\ - --and --not --in=mygs --and --not --in=dnanexus -... -(recording state in git...) -(master_env_v156_py36) 01:17 [viral-ngs-benchmarks] $ echo $? -0 - - - -\"\"\"]] -"""]] diff --git a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_3_f244825dcd89cd6bc74deec7ac4bdd99._comment b/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_3_f244825dcd89cd6bc74deec7ac4bdd99._comment deleted file mode 100644 index f2b950348e..0000000000 --- a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_3_f244825dcd89cd6bc74deec7ac4bdd99._comment +++ /dev/null @@ -1,24 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2019-11-13T19:29:34Z" - content=""" ---debug might provide some clue in its http dump. - -The ParseError comes from attoparsec. Seems likely that aeson/aws is what's -using it there, and that it is failing to parse something from S3. - -Of course, the malloc error suggests a low-level memory problem, probably -from C code. I don't think git-annex contains anything like that, so it -must be from a dependency. - -The S3 signature being wrong again points to the aws library, or something -lower level. And then the following double free is another low-level memory -problem. - -So there's a pattern, and it seems to extend across linux and OSX. - -Kind of wondering if something in the library stack is somehow failing to -be concurrency safe. If two http requests end up using the same memory, -it would kind of explain all of this. -"""]] diff --git a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_4_af2e6683949d6168ff4ca606085cf870._comment b/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_4_af2e6683949d6168ff4ca606085cf870._comment deleted file mode 100644 index ebb25ad201..0000000000 --- a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_4_af2e6683949d6168ff4ca606085cf870._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-09-14T21:32:05Z" - content=""" -Thinking about this, one thing in git-annex that might somehow involve -unsafe memory access would be if Remote.S3's use of runResourceT returns -a value that depends on a resource that then gets freed. - -[[!commit ddf963d0194acf9f6f059fa37f3e89e59d682de9]] should avoid all ways that might possibly have happened, -so if anyone has a way to reproduce this reliably, it would be good to know -if it still happens after that fix. -"""]] diff --git a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_5_8ea6aacd9aa428725425d0a4cf5d4c89._comment b/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_5_8ea6aacd9aa428725425d0a4cf5d4c89._comment deleted file mode 100644 index fd73972cbc..0000000000 --- a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_5_8ea6aacd9aa428725425d0a4cf5d4c89._comment +++ /dev/null @@ -1,36 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2020-09-14T23:43:03Z" - content=""" -git-annex copy -A --to S3 -J40 hangs reproductibly for phibs (on irc). - -When it hangs, at least one job is stuck at 100%, and often several, -though some other jobs are hung at smaller percent. - -Without -J, it did not seem to hang. It also may have not been hung with --J2 (was interrupted before finishing). - -One attempt (with --json) had this message: - - magicFile: got error code but no error message - -Two attempts had this: - - Missing root element - -One time just before the hang, the other time 20+ files before the hang. -That message coes from Text.XML apparently, -which I think points to this being the same bug. - -phibs tried a similar copy to a directory special remote, did not hang. - -phibs tried the ResourceT strictness patch there and it did not help. -I think that pretty much rules out any problem in git-annex's code, -the problem is probably in the aws library or the http stack. - -The S3 remote has chunking (10 mb) and no encryption, and is on AWS. - -Some file sizes of files that hung at 100%: 64083380, 7668, 72792, 6952, -9584456, 21050948. -"""]] diff --git a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_6_6f986fe2e2ea9df19b0b30db553b2574._comment b/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_6_6f986fe2e2ea9df19b0b30db553b2574._comment deleted file mode 100644 index cd9692f012..0000000000 --- a/doc/bugs/concurrent_git-annex-copy_to_s3_special_remote_fails/comment_6_6f986fe2e2ea9df19b0b30db553b2574._comment +++ /dev/null @@ -1,90 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 6""" - date="2020-09-17T19:00:00Z" - content=""" -Have not reproduced a hang or segfault, but I did easily -reproduce the "Missing root element" - - joey@kite:~/tmp/awstest>git annex copy -A --to s3test2 -J40 - copy SHA256E-s6952--21f7b50313b7f85bbd97c1625b74559094c81be25223cd54ee692c063c5eebd7 (checking s3test2...) (to s3test2...) ok - copy SHA256E-s3872792--6fe6dc74f1388a5ffef061f0b16d167584de9dbe02f037892a36348703edddda (checking s3test2...) (to s3test2...) - Missing root element - - Missing root element - failed - copy SHA256E-s3772792--835b658d79060395049f50dfba8a7795e07ce90357bd5ae13abd593301b210c5 (checking s3test2...) (to s3test2...) ok - copy SHA256E-s2072792--f687b87a28d8ddbb4fc6492ca88f65cccd493409590cb0faab508fe1968c74b3 (checking s3test2...) (to s3test2...) ok - copy SHA256E-s2272792--c3d185d7ef57ac2846021d995578a22645b4cd4ad00b54b86fb6608b440e4f83 (checking s3test2...) (to s3test2...) ok - copy SHA256E-s72792--8a923aceb4279623450186f52571010999f94915e98ef9d149aeab0023647ab5 (checking s3test2...) (to s3test2...) - Missing root element - ok - copy SHA256E-s3572792--761d388643772967499796e0114f526908582fd173aacd136e3541fb5683fbb0 (checking s3test2...) (to s3test2...) ok - copy SHA256E-s2172792--b1ce7d642d1d8d3285a1718f2227def468b439d522f2a05d9cc47026d4a57743 (checking s3test2...) (to s3test2...) - Missing root element - - Missing root element - ok - [...] - (recording state in git...) - git-annex: copy: 1 failed - -Interesting that only 1 actually failed. And fsck confirms that the others did -get stored, despite the messages. So it seems that mostly the checkpresent -test (HEAD) is failing, so it thinks it's not present and stores it, -but in the 1 failur above, both checkpresent and the upload failed with that message. - --J4 is enough to reproduce it, although maybe less frequently. -Chunking is not needed to reproduce it. - -But `git-annex fsck --fast -J40 --from remote` does not reproduce it, -despite doing the same head operaton. Seems to maybe need a combination of -simulantaneous storeObject and headObject, or something wrong about the -timing when using fsck. This makes it harder to isolate a test case.. - ---debug shows this occuring once per "Missing Root Element" - - [2020-09-17 15:25:32.604667504] Response status: Status {statusCode = 400, statusMessage = "Bad Request"} - -So, S3 is for whatever reason rejecting a request, but its response didn't -get corrupted, it's just lacking the usual XML body, and 400 is not handled -like 404 is so it tries to parse it as XML. -If something is getting corrupted here, it must be the request. - -Took a tcpdump of this happening, and found the "Bad Request" packets in wireshark. -I asked wireshark to "Follow TCP stream", and saw this (I added directional -markers): - - >PUT /SHA256E-s3072792--827469d4593f92db48dc23c557bafc200ece505ed2b0d0c4a8b07a5f752af9bb HTTP/1.1 - >Host: s3test2-da497c39-25b8-4b4e-ae49-349364e440fd.s3.amazonaws.com - >Accept-Encoding: gzip - >Content-Length: 3072792 - >Date: Thu, 17 Sep 2020 19:42:49 GMT - >Content-Type: p.....p.....t-stream - >Authorization: AWS - >x-amz-storage-class: STANDARD - > - >a.y.|.Iz..:..G....s..~v........S.......D.......:B1.D....O{.@{`O.o..p....g...m.Y......^.oZK..K...L13...#S....w..6.O....4.;0./;Q.oVJ.....w..5.^....GdT..I7I}t.L...E\..(.........{pZ.K....r...8......~...cc.#6.......a.g....l.. - >[snip rest of object being sent] - >..... - >..NZ.3..m"..N..~.....b.....k..........MZF..).....M.['....e....EJ..n...E..c...~.?. - git annex copy -J4 --to=box-dartm3.com --json --json-progress video.mp4 -... -{"byte-progress":13166304,"action":{"command":"copy","note":"to box-dartm3.com...","key":"MD5E-s1073741824--e06da14afc6face003121641e60593bb.mp4","file":"video.mp4"},"total-size":1073741824,"percent-progress":"1.23%"} -{"command":"copy","note":"checking box-dartm3.com...","success":false,"key":"MD5E-s1073741824--e06da14afc6face003121641e60593bb.mp4","file":"video.mp4"} - DAV failure: 408 "Request Timeout" - CallStack (from HasCallStack): - error, called at ./Remote/WebDAV.hs:324:47 in main:Remote.WebDAV -git-annex: copy: 1 failed - -$> git annex copy -J4 --to=box-dartm3.com --json --json-progress video.mp4 -{"byte-progress":0,"action":{"command":"copy","note":"to box-dartm3.com...","key":"MD5E-s1073741824--e06da14afc6face003121641e60593bb.mp4","file":"video.mp4"},"total-size":1073741824,"percent-progress":"0%"} -{"byte-progress":32752,"action":{"command":"copy","note":"to box-dartm3.com...","key":"MD5E-s1073741824--e06da14afc6face003121641e60593bb.mp4","file":"video.mp4"},"total-size":1073741824,"percent-progress":"0%"} - -... -{"byte-progress":225153536,"action":{"command":"copy","note":"to box-dartm3.com...","key":"MD5E-s1073741824--e06da14afc6face003121641e60593bb.mp4","file":"video.mp4"},"total-size":1073741824,"percent-progress":"20.97%"} -{"command":"copy","note":"checking box-dartm3.com...","success":false,"key":"MD5E-s1073741824--e06da14afc6face003121641e60593bb.mp4","file":"video.mp4"} - DAV failure: 408 "Request Timeout" - CallStack (from HasCallStack): - error, called at ./Remote/WebDAV.hs:324:47 in main:Remote.WebDAV -git-annex: copy: 1 failed - -$> git annex copy -J4 --to=box-dartm3.com --json --json-progress video.mp4 -{"byte-progress":200000000,"action":{"command":"copy","note":"to box-dartm3.com...","key":"MD5E-s1073741824--e06da14afc6face003121641e60593bb.mp4","file":"video.mp4"},"total-size":1073741824,"percent-progress":"18.63%"} -... - -"""]] - -apparently it is actually timing out on checking (I guess after chunk completion?), not even when copying... could there be multiple attempts and some grace time period for webdav server possibly to "finish receiving" a particular file? - - -[[!meta author="yoh"]] - -> I see that the bug in the DAV library has been fixed (in 2018), -> so hopefully nothing more needs to be done. [[done]] --[[Joey]] diff --git a/doc/bugs/could_webdav_be_more_resilient_to_timeouts__63__/comment_1_bb4130ae787e19757b849a6301129c34._comment b/doc/bugs/could_webdav_be_more_resilient_to_timeouts__63__/comment_1_bb4130ae787e19757b849a6301129c34._comment deleted file mode 100644 index 8fd509a675..0000000000 --- a/doc/bugs/could_webdav_be_more_resilient_to_timeouts__63__/comment_1_bb4130ae787e19757b849a6301129c34._comment +++ /dev/null @@ -1,29 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2017-08-15T17:02:20Z" - content=""" -This last came up in 2014, at the time I disabled the http response timeout -entirely with setResponseTimeout Nothing. - -Looking at Network.Protocol.HTTP.DAV.setResponseTimeout: - - setResponseTimeout :: MonadIO m => Maybe Int -> DAVT m () - #if MIN_VERSION_http_client(0,5,0) - setResponseTimeout rt = baseRequest %= \x -> x { responseTimeout = maybe responseTimeoutDefault responseTimeoutMicro rt } - #else - setResponseTimeout rt = baseRequest %= \x -> x { responseTimeout = rt } - #endif - -Looks like with recent http-client versions in the case of Nothing, rather -than disabling the timeout, it now defaults to responseTimeoutDefault. -Which is 30 seconds. Since I'm passing it Nothing to try to disable the -timeout, that's kind of a problem! - -I feel this is a bug in the DAV library. I could try to work around it -with `Just maxBound`, that's many years worth of microseconds on 64 bit, but -on 32 bit, it's somewhere under 10 minutes, which is really not good -enough when the goal is to disable timeouts at this level entirely. - -Bug filed on DAV: -"""]] diff --git a/doc/bugs/directory_special_remote_export_file_mode.mdwn b/doc/bugs/directory_special_remote_export_file_mode.mdwn deleted file mode 100644 index 8f1f150d1b..0000000000 --- a/doc/bugs/directory_special_remote_export_file_mode.mdwn +++ /dev/null @@ -1,23 +0,0 @@ -When exporting a tree to a directory special remote, files are written mode -600. This prevents eg, publishing them by http, and then accessing them -with httpalso special remote. - -`viaTmp` creates a temp file, with temp file perms. Either it should use -umask perms, or all callers that don't explicitly set perms should. - -This also affects some other things, eg hook files written by git-annex -init, and some stuff in ~/.config/git-annex like autostart. - -> [[fixed|done]] --[[Joey]] - -`withTmpFileIn` also uses openTempFile, and probably its callers do need to -adjust perms if desired since it could be used with a real temp directory. - -> Audited and fixed these. It affected only directory special remote and -> bittorrent special remote if a download from it were interrupted and then -> resumed by a different user than the one who started it. --[[Joey]] - -There are also a couple of other uses of openTempFile, which need to be -audited for this problem. --[[Joey]] - -> Checked, all were ok. --[[Joey]] diff --git a/doc/bugs/does_not_handle_youtube_playlists.mdwn b/doc/bugs/does_not_handle_youtube_playlists.mdwn deleted file mode 100644 index 213fa96328..0000000000 --- a/doc/bugs/does_not_handle_youtube_playlists.mdwn +++ /dev/null @@ -1,43 +0,0 @@ -### Please describe the problem. - -Not really related to datalad -- thought to addurl youtube playlist - -youtube-dl seems might be capable of doing it - -### Please provide any additional information below. - -[[!format sh """ - -$> git annex addurl --debug 'https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ' [2018-03-28 13:09:06.337738339] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] -[2018-03-28 13:09:06.34708705] process done ExitSuccess -[2018-03-28 13:09:06.347202003] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] -[2018-03-28 13:09:06.35303272] process done ExitSuccess -[2018-03-28 13:09:06.353205536] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..734b368eab4f39d8494671657977952b02a35d9a","--pretty=%H","-n1"] -[2018-03-28 13:09:06.359101486] process done ExitSuccess -[2018-03-28 13:09:06.360005167] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"] -[2018-03-28 13:09:06.361006846] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] -addurl https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ -[2018-03-28 13:09:06.409163359] call: wget ["-nv","--show-progress","--clobber","-c","-O","/tmp/testyt/.git/annex/tmp/URL--https&c%%www.youtube.com%playlis-5bc73cdf8dc8bd73b13addc290b160e6","https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ","--user-agent","git-annex/6.20180316+gitg308f3ecf6-1~ndall+1"] -/tmp/testyt/.git/ann 100%[===================>] 211.54K --.-KB/s in 0.04s -2018-03-28 13:09:06 URL:https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ [216616/216616] -> "/tmp/testyt/.git/annex/tmp/URL--https&c%%www.youtube.com%playlis-5bc73cdf8dc8bd73b13addc290b160e6" [1] -[2018-03-28 13:09:06.844196402] process done ExitSuccess -[2018-03-28 13:09:06.845246148] read: youtube-dl ["https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ","--get-filename","--no-warnings"] -[2018-03-28 13:09:40.643440496] process done ExitSuccess -[2018-03-28 13:09:40.644420015] call: youtube-dl ["https://www.youtube.com/playlist?list=PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ","--no-playlist","--playlist-items","0","--max-filesize","9046842816"] -[youtube:playlist] PLBHioGD0U1Cjd-meZbEcz-9ZxK-mb50tZ: Downloading webpage -[download] Downloading playlist: Canonical Computation in Brains and Machines -[youtube:playlist] playlist Canonical Computation in Brains and Machines: Downloading 0 videos -[download] Finished downloading playlist: Canonical Computation in Brains and Machines -[2018-03-28 13:09:41.925771815] process done ExitSuccess - - youtube-dl did not put any media in its work directory, perhaps it's been configured to store files somewhere else? -failed -[2018-03-28 13:09:41.926669401] process done ExitSuccess -[2018-03-28 13:09:41.926951446] process done ExitSuccess -git-annex: addurl: 1 failed -git annex addurl --debug 6.49s user 0.32s system 18% cpu 35.834 total - -"""]] - - -> importfeed supports youtube playlists, once you look up their rss feed. [[done]] --[[Joey]] diff --git a/doc/bugs/does_not_handle_youtube_playlists/comment_1_3521c057eef592458704f3616ffc68a2._comment b/doc/bugs/does_not_handle_youtube_playlists/comment_1_3521c057eef592458704f3616ffc68a2._comment deleted file mode 100644 index 36d32ad886..0000000000 --- a/doc/bugs/does_not_handle_youtube_playlists/comment_1_3521c057eef592458704f3616ffc68a2._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="np.gan@f5b13d2a374edae9a4c8b2f76022d8835781b9a5" - nickname="np.gan" - avatar="http://cdn.libravatar.org/avatar/a9b2109945678289ae279aea36904c84" - subject="youtube feeds are limited" - date="2019-02-10T13:09:54Z" - content=""" -RSS feeds corresponding to playlists are limited to 15 results. - -Since youtube-dl can parse and show the playlist contents as JSON it would be awesome for git-annex importfeed to support youtube-dl. - -Ref: https://superuser.com/a/1359132 -"""]] diff --git a/doc/bugs/does_not_handle_youtube_playlists/comment_2_5bed8f612e182a5c53dd464aea9b505b._comment b/doc/bugs/does_not_handle_youtube_playlists/comment_2_5bed8f612e182a5c53dd464aea9b505b._comment deleted file mode 100644 index 04a75db01a..0000000000 --- a/doc/bugs/does_not_handle_youtube_playlists/comment_2_5bed8f612e182a5c53dd464aea9b505b._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 2" - date="2020-09-02T02:53:21Z" - content=""" -yep, would have been nice if git-annex either asked youtube-dl or just figured out itself to map URL to `https://www.youtube.com/feeds/videos.xml?playlist_id=ID` to get a feed for a copy/pasted playlist URL. -"""]] diff --git a/doc/bugs/dotfiles_handled_differently.mdwn b/doc/bugs/dotfiles_handled_differently.mdwn deleted file mode 100644 index cdfbad6d45..0000000000 --- a/doc/bugs/dotfiles_handled_differently.mdwn +++ /dev/null @@ -1,36 +0,0 @@ -For most files, whether they get annexed is controlled by `annex.largefiles`. But dotfiles are configured to *never* be annexed regardless of `annex.largefiles`. This is unexpected and confusing. More oddly, `git-annex-add` doesn't seem to add them to regular git, either. - -I'm guessing this flows from the new default of `git add` adding files to the annex, which [[I hope gets reversed|forum/lets_discuss_git_add_behavior]]. But separately, I'm not sure why `git-annex-add` does not add the dotfiles to regular git: - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - -(master_env_v163_py36) 22:32 [newtree02] $ git status -On branch newtree02 -Untracked files: - (use "git add ..." to include in what will be committed) - - mydir/ - -nothing added to commit but untracked files present (use "git add" to track) -(master_env_v163_py36) 22:32 [newtree02] $ ls -al mydir -total 12 -drwxrwxr-x 2 ilya ilya 4096 Oct 20 22:31 . -drwxrwxr-x 3 ilya ilya 4096 Oct 20 22:31 .. --rw-rw-r-- 1 ilya ilya 9 Oct 20 22:31 .mynewdot -(master_env_v163_py36) 22:32 [newtree02] $ git annex add mydir -(master_env_v163_py36) 22:32 [newtree02] $ git status -On branch newtree02 -Untracked files: - (use "git add ..." to include in what will be committed) - - mydir/ - - -# End of transcript or log. -"""]] - -> There have been several changes to dotfiles since this bug was filed. I -> think most relevantly, annex.dotfiles can be set to let annex.largefiles -> control them too. So this seems [[done]] --[[Joey]] diff --git a/doc/bugs/dotfiles_handled_differently/comment_1_d7fb4eeb05d72c07b8d6452a273f68e4._comment b/doc/bugs/dotfiles_handled_differently/comment_1_d7fb4eeb05d72c07b8d6452a273f68e4._comment deleted file mode 100644 index 08dc6cc37e..0000000000 --- a/doc/bugs/dotfiles_handled_differently/comment_1_d7fb4eeb05d72c07b8d6452a273f68e4._comment +++ /dev/null @@ -1,23 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-10-21T16:53:45Z" - content=""" -`git-annex add .foo` will add the file to the annex, unless your -annex.largefiles is otherwise configured. There is no special handling -of largefiles for dotfiles[1]. - -But `git-annex add` skips dotfiles unless explicitly listed. -It always has! There's a --include-dotfiles you can use to override that -default. - -I don't see a bug here, except possibly the description on the man page -could mention it skipping dotfiles. But the existance of the option -does make the skipping be documented. - -[1] Although, the default .git/info/attributes does only apply -the annex filter to non-dotfiles. So `git add .foo` adds it to git and -not to git-annex. That's a little bit cheezy, but the alternative was to -have a different annex.largefiles setting for `git add` than for `git-annex -add` -"""]] diff --git a/doc/bugs/dotfiles_handled_differently/comment_2_3b5835519b02ef607db3b534b6d2fefd._comment b/doc/bugs/dotfiles_handled_differently/comment_2_3b5835519b02ef607db3b534b6d2fefd._comment deleted file mode 100644 index d428cbcf08..0000000000 --- a/doc/bugs/dotfiles_handled_differently/comment_2_3b5835519b02ef607db3b534b6d2fefd._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="different annex.largefiles setting for git add than for git-annex add" - date="2019-10-21T20:29:06Z" - content=""" -\"the alternative was to have a different annex.largefiles setting for git add than for git-annex add\" -- [[that alternative|todo/separate_annex.largefiles.git-add_and_annex.largefiles.git-annex-add_settings]] would actually fix many of the [[issues I and others are having|forum/lets_discuss_git_add_behavior]] with v7 semantics. What are the reasons you think it would be bad? -"""]] diff --git a/doc/bugs/dotfiles_handled_differently/comment_3_3d0d47a6cda3a556bb34442d60df1402._comment b/doc/bugs/dotfiles_handled_differently/comment_3_3d0d47a6cda3a556bb34442d60df1402._comment deleted file mode 100644 index e8e79464b0..0000000000 --- a/doc/bugs/dotfiles_handled_differently/comment_3_3d0d47a6cda3a556bb34442d60df1402._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="skipping dotfiles" - date="2019-10-21T20:39:09Z" - content=""" -\" git-annex add skips dotfiles unless explicitly listed. It always has!\" -- not sure why it hasn't crashed my use case before (thought I'm re-running on same data); but maybe, reconsider this default? Since `git add .` adds everything, it's easy to assume (as I did) that `git annex add .` does too. Why not add dotfiles to regular git, rather than skipping them outright? -"""]] diff --git a/doc/bugs/download_failed__58___Requested_Range_Not_Satisfiable.mdwn b/doc/bugs/download_failed__58___Requested_Range_Not_Satisfiable.mdwn deleted file mode 100644 index 6cb41f9439..0000000000 --- a/doc/bugs/download_failed__58___Requested_Range_Not_Satisfiable.mdwn +++ /dev/null @@ -1,45 +0,0 @@ -### Please describe the problem. - -addurl fails with the following message: - - % git annex addurl --file=episodes/rtq-proteomics1.mp3 https://test.bioinformatics.chat/episodes/rtq-proteomics.mp3 - addurl https://test.bioinformatics.chat/episodes/rtq-proteomics.mp3 - - download failed: Requested Range Not Satisfiable - - - download failed: Requested Range Not Satisfiable - failed - git-annex: addurl: 1 failed - - -I configured the server to log the Range header as the last field, and this is what I observe: - - x.x.x.x - - [18/Nov/2020:13:32:19 +0000] "GET /episodes/rtq-proteomics.mp3 HTTP/1.1" 416 197 "-" "-" "bytes=30343208-" - -So the start of the range is exactly the file size, and the 416 response is warranted. It looks like git annex does not handle this edge case. - -### What steps will reproduce the problem? - -I'm not sure tbh — I think this happened because I interrupted a previous download of this file. - -### What version of git-annex are you using? On what operating system? - -Stock git-annex on Fedora 32: - - git-annex version: 8.20200330 - build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite - dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.4 feed-1.1.0.0 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.1 - key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL - remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external - operating system: linux x86_64 - supported repository versions: 8 - upgrade supported from repository versions: 0 1 2 3 4 5 6 7 - local repository version: 8 - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Yes! git-annex has been working great for me so far, and is powering the bioinformatics chat podcast (https://bioinformatics.chat/). Thanks! - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/download_failed__58___Requested_Range_Not_Satisfiable/comment_1_44bc0678a9978b62ab133d64d3614610._comment b/doc/bugs/download_failed__58___Requested_Range_Not_Satisfiable/comment_1_44bc0678a9978b62ab133d64d3614610._comment deleted file mode 100644 index 006eec9bdc..0000000000 --- a/doc/bugs/download_failed__58___Requested_Range_Not_Satisfiable/comment_1_44bc0678a9978b62ab133d64d3614610._comment +++ /dev/null @@ -1,30 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-11-19T16:56:45Z" - content=""" -To reproduce this, interrupt git-annex after it downloads the whole file, -but before it moves it from the download location into the annex. (Or, -let it get the file, then move the object back to the temp object location.) - -This is a tricky case, because if the total file size is not -known when resuming the download, how can it detect if it's got it all -already? And git-annex does not always know the total file size, eg when -git-annex addurl --relaxed is used, and then git-annex get is later used -to download the content. - -What git-annex already tried to do to detect this is, -when it got a 416 it looks for a Content-Range header "bytes */$size" -where $size is the same as the size of the file on disk. - -That relied on the http library throwing an exception for the 416. -Thing is, http may or may not throw exceptions for non-2xx -responses, depending on the input Request. IMHO that is a very bad design, -it leads to this kind of bug, rather than making it evident with the data -types what is going on. - -Currently downloadConduit takes a Request, and assumes it throws exceptions -for 416, but not for 401. Both can't be right. - -Ok, fixed this mess.. -"""]] diff --git a/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__.mdwn b/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__.mdwn deleted file mode 100644 index 7255a3e254..0000000000 --- a/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__.mdwn +++ /dev/null @@ -1,198 +0,0 @@ -### Please describe the problem. - -This new release of git-annex claims that content is required when I'm trying to drop it despite the fact that I'm using an `exclude=` -sub-expression *that matches* the exclusion in the required content expression. This works fine in my "gold" version aka. -8.20201103-g0cf77eb41 so somewhere in between there was a change that broke this. - -### What steps will reproduce the problem? - -The following transcript describes the problem well. I'm in an archive folder (in a hidemissing-unlocked branch) where I should be able -to drop annexed files as long as the archival remote (in this case "k-levyn-annex2") is connected and reachable by git-annex: - -[[!format sh """ -PS> git annex version | head -n 1 -git-annex version: 8.20201127-g1a1d671dd -PS> pwd - -Path ----- -G:\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A\arkistoidut - -PS> ls - - Directory: G:\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A\arkistoidut - -Mode LastWriteTime Length Name ----- ------------- ------ ---- --a--- 13.8.2020 13:27 103 .ankkuri --a--- 25.11.2020 12:11 20190345809 5D3DB6C10EAF0911-07-07.mrimg - -PS> git annex whereis .\5D3DB6C10EAF0911-07-07.mrimg -whereis 5D3DB6C10EAF0911-07-07.mrimg (2 copies) - 3362df51-1789-4471-96a0-d2267ada6aa4 -- Reflect-varmistukset [here] - 46a41e47-45c8-4a86-b348-db0c4cfb18f3 -- Reflect-varmistukset [k-levyn-annex2] -ok -PS> fsutil hardlink list .\5D3DB6C10EAF0911-07-07.mrimg -\Reflect-varmistukset\.git\annex\objects\692\4bf\MD5E-s20190345809--37ade0ff04b63b042f87d7fbc4f7b0a4.mrimg\MD5E-s20190345809--37ade0ff04b63b042f87d7fbc4f7b0a4.mrimg -\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A\arkistoidut\5D3DB6C10EAF0911-07-07.mrimg -PS> git annex drop .\5D3DB6C10EAF0911-07-07.mrimg -drop 5D3DB6C10EAF0911-07-07.mrimg - That file is required content, it cannot be dropped! - - (Use --force to override this check, or adjust required content configuration.) -failed -git-annex: drop: 1 failed -PS> git annex required . -(include=*.mrimg and exclude=*arkistoidut*) -PS> git annex find --want-drop --in . -5D3DB6C10EAF0911-07-07.mrimg -PS> git annex required . '(include=*.mrimg and exclude=*/arkistoidut/*)' -required . ok -(recording state in git...) -PS> git annex required . -(include=*.mrimg and exclude=*/arkistoidut/*) -PS> git annex find --want-drop --in . -[empty] -PS> git annex required . '(include=*.mrimg and exclude=*\arkistoidut\*)' -required . ok -(recording state in git...) -PS> git annex required . -(include=*.mrimg and exclude=*\arkistoidut\*) -PS> git annex find --want-drop --in . -5D3DB6C10EAF0911-07-07.mrimg -PS> git annex drop .\5D3DB6C10EAF0911-07-07.mrimg -drop 5D3DB6C10EAF0911-07-07.mrimg - That file is required content, it cannot be dropped! - - (Use --force to override this check, or adjust required content configuration.) -failed -git-annex: drop: 1 failed -# end of transcript -"""]] - -(I've edited the transcript above by removing the long paths in the PowerShell prompt (now just "PS> ") and adding the text "[empty]" where -responses have been just that, empty lines. The words "varmistukset", "arkistoidut" and "ankkuri" are in Finnish and mean "backups", -"archives" and "anchor" (ie. placeholder) respectively.) - -This is the contents of the `.git/config` file of the repo: - -``` -[core] - repositoryformatversion = 0 - filemode = false - bare = false - logallrefupdates = true - symlinks = false - ignorecase = true - pager = delta --theme='Monokai Extended' -[interactive] - diffFilter = delta --color-only -[annex] - thin = true - backend = MD5E - uuid = 3362df51-1789-4471-96a0-d2267ada6aa4 - sshcaching = false - crippledfilesystem = true - version = 8 - maxextensionlength = 8 -[filter "annex"] - smudge = git-annex smudge -- %f - clean = git-annex smudge --clean -- %f -[remote "k-levyn-annex2"] - url = k:\\Reflect-varmistukset - fetch = +refs/heads/*:refs/remotes/k-levyn-annex2/* - annex-uuid = 46a41e47-45c8-4a86-b348-db0c4cfb18f3 -[receive] - denyCurrentBranch = warn -[merge] - renames = true - directoryRenames = false -``` - -### What version of git-annex are you using? On what operating system? - -Version 8.20201127, two commits ahead (1a1d671dd "fix build") of the release tag, so built from source. This is on Windows 10 version 20H2 -(build 19042.630), 64 bit. - -[[!format sh """ -git-annex version: 8.20201127-g1a1d671dd -build flags: Assistant Webapp Pairing TorrentParser Feeds Testsuite S3 WebDAV -dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external -operating system: mingw32 x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 2 3 4 5 6 7 -"""]] - -### Please provide any additional information below. - -I'm going to show you when this works ok (well, I have another curious problem in that in many cases git-annex doesn't want to remove -the target file even when it has dropped the key / annexed blob already but that's beside the point for now). This transcript follows -the one above: - -
-a mostly working example (or: the works for me case) - -[[!format sh """ -PS> C:\git-annex-testsuite-results\last-working\git-annex.exe version | head -n 1 -git-annex version: 8.20201103-g0cf77eb41 -PS> C:\git-annex-testsuite-results\last-working\git-annex.exe required . -(include=*.mrimg and exclude=*\arkistoidut\*) -PS> C:\git-annex-testsuite-results\last-working\git-annex.exe find --want-drop --in . -5D3DB6C10EAF0911-07-07.mrimg -PS> C:\git-annex-testsuite-results\last-working\git-annex.exe drop .\5D3DB6C10EAF0911-07-07.mrimg -drop 5D3DB6C10EAF0911-07-07.mrimg ok -(recording state in git...) -PS> ls - - Directory: G:\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A\arkistoidut - -Mode LastWriteTime Length Name ----- ------------- ------ ---- --a--- 13.8.2020 13:27 103 .ankkuri --a--- 25.11.2020 12:11 20190345809 5D3DB6C10EAF0911-07-07.mrimg - -PS> C:\git-annex-testsuite-results\last-working\git-annex.exe info .\5D3DB6C10EAF0911-07-07.mrimg -file: 5D3DB6C10EAF0911-07-07.mrimg -size: 20.19 gigabytes -key: MD5E-s20190345809--37ade0ff04b63b042f87d7fbc4f7b0a4.mrimg -present: false -PS> C:\git-annex-testsuite-results\last-working\git-annex.exe contentlocation MD5E-s20190345809--37ade0ff04b63b042f87d7fbc4f7b0a4.mrimg -[empty] -PS> fsutil hardlink list .\5D3DB6C10EAF0911-07-07.mrimg -\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A\arkistoidut\5D3DB6C10EAF0911-07-07.mrimg -PS> rm .\5D3DB6C10EAF0911-07-07.mrimg -PS> git status -On branch adjusted/master(hidemissing-unlocked) -Changes not staged for commit: - (use "git add/rm ..." to update what will be committed) - (use "git restore ..." to discard changes in working directory) - deleted: 5D3DB6C10EAF0911-07-07.mrimg - -no changes added to commit (use "git add" and/or "git commit -a") -PS> git restore 5D3DB6C10EAF0911-07-07.mrimg -PS> ls - - Directory: G:\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A\arkistoidut - -Mode LastWriteTime Length Name ----- ------------- ------ ---- --a--- 13.8.2020 13:27 103 .ankkuri --a--- 29.11.2020 11:33 74 5D3DB6C10EAF0911-07-07.mrimg - -PS> cat .\5D3DB6C10EAF0911-07-07.mrimg -/annex/objects/MD5E-s20190345809--37ade0ff04b63b042f87d7fbc4f7b0a4.mrimg -# end of transcript -"""]] - -
- -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Yes, so far it has worked nicely archiving (and describing via git-annex metadata) my multi-gigabyte Macrium Reflect backup files in an -orderly fashion. - -[[!meta author=jkniiv]] - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_1_5d7428158cc90704b96a60eeb55efcd0._comment b/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_1_5d7428158cc90704b96a60eeb55efcd0._comment deleted file mode 100644 index 6bc441cbc0..0000000000 --- a/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_1_5d7428158cc90704b96a60eeb55efcd0._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-12-14T19:36:20Z" - content=""" -Looks like it was caused by [[!commit d032b0885d80d12c00fa8813e88deab1631eef8a]] which made MatchingKey be used -rather than MatchingFile. Which oops, mean the filename is left relative rather -than being made into a path from the top of the repo. - -Fixed that and your test case works. I do think this would be a better -expression for you to use though: - - (include=*.mrimg and exclude=*/arkistoidut/* and exclude=arkistoidut/*) - -Or maybe just exclude=arkistoidut/* rather than both, depending on if you -want to support subdirectories of subdirectories with that name, or only -the single subdirectory in the top of your repo. -"""]] diff --git a/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_2_f1f67b94a6a126c73e8b4d222afe09d9._comment b/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_2_f1f67b94a6a126c73e8b4d222afe09d9._comment deleted file mode 100644 index 2bfb567e0f..0000000000 --- a/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_2_f1f67b94a6a126c73e8b4d222afe09d9._comment +++ /dev/null @@ -1,163 +0,0 @@ -[[!comment format=mdwn - username="jkniiv" - avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d" - subject="yay, it's working now! :)" - date="2020-12-15T04:46:38Z" - content=""" -Excellent! Now it seems to work, indeed (passes the testsuite[^1] and this particular use case). And thanks for your quick response. :) -As for the specifics of the required content expression, it's true that my archive directories have so far been located in a subdirectory -of the backup set[^2] specific subdirectories of the repo but I guess I could use an archive directory situated at the top of the repo for truly old -backups or for a similar use case. Because my old required expression didn't use any path separators at all (and somehow it still worked!), -I'm now using your version, however slightly modified (see the transcript, it's a path separator issue). - -[^1]: The parts I'm interested in as I'm not yet doing the special remote tests (\"Tests.Remote Tests\"). -[^2]: In my usage a backup set comprises those disk image-based backups that I have taken from the same major Windows version (or \"feature update\") of my Windows installation. - -
-a transcript of a working session archiving one of my backups using version 8.20201128-g5e094d02d (lacks the git-annex sync part but you get the idea) - -[[!format sh \"\"\" -PS> df -h . -C:\scoop\apps\gow\current\bin\df.exe: Warning: cannot read table of mounted file systems: Invalid argument -Filesystem Size Used Avail Use% Mounted on -- 466G 402G 65G 87% G:\ -PS> git annex version | head -n 1 -git-annex version: 8.20201128-g5e094d02d -PS> pwd - -Path ----- -G:\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A - -PS> git annex required . -(include=*.mrimg and exclude=*\arkistoidut\*) -PS> git annex required . '(include=*.mrimg and exclude=*/arkistoidut/* and exclude=arkistoidut/*)' -required . ok -(recording state in git...) -PS> git annex required . -(include=*.mrimg and exclude=*/arkistoidut/* and exclude=arkistoidut/*) -PS> ls - - Directory: G:\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A - -Mode LastWriteTime Length Name ----- ------------- ------ ---- -d---- 14.12.2020 2:12 arkistoidut --a--- 9.11.2020 1:15 172453484741 5D3DB6C10EAF0911-00-00.mrimg --a--- 20.11.2020 21:56 17538095697 5D3DB6C10EAF0911-05-05.mrimg --a--- 30.11.2020 20:21 25282746961 5D3DB6C10EAF0911-11-11.mrimg --a--- 4.12.2020 0:33 5793369649 5D3DB6C10EAF0911-12-12.mrimg --a--- 7.12.2020 15:25 165950159045 AF13B339E6F3BC85-00-00.mrimg --a--- 9.12.2020 1:38 6567823953 AF13B339E6F3BC85-01-01.mrimg --a--- 11.12.2020 1:27 3972721183 AF13B339E6F3BC85-02-02.mrimg --a--- 14.12.2020 1:54 11383011921 AF13B339E6F3BC85-03-03.mrimg --a--- 14.11.2020 17:17 339 Justfile - -PS> git log -[redacted] -PS> git annex metadata (get-item 5d3*-1[12]-*.mrimg) -metadata 5D3DB6C10EAF0911-11-11.mrimg - -ok -metadata 5D3DB6C10EAF0911-12-12.mrimg - lastchanged=2020-12-03@22-36-50 - tag=inkrementaali - tag-lastchanged=2020-12-03@22-36-50 -ok -PS> # let's archive the incremental -PS> git mv 5D3DB6C10EAF0911-12-12.mrimg .\arkistoidut\ -PS> git status -On branch adjusted/master(hidemissing-unlocked) -Changes to be committed: - (use \"git restore --staged ...\" to unstage) - renamed: 5D3DB6C10EAF0911-12-12.mrimg -> arkistoidut/5D3DB6C10EAF0911-12-12.mrimg - -PS> git commit -m 'Arkistoitu: 5D3DB6C10EAF0911-12-12.mrimg' -[adjusted/master(hidemissing-unlocked) 18aa23e] Arkistoitu: 5D3DB6C10EAF0911-12-12.mrimg - 1 file changed, 0 insertions(+), 0 deletions(-) - rename Jarkon ThinkPad T450s (Win10 v2004) . A/{ => arkistoidut}/5D3DB6C10EAF0911-12-12.mrimg (100%) -PS> cd .\arkistoidut\ -PS> pwd - -Path ----- -G:\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A\arkistoidut - -PS> ls - - Directory: G:\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A\arkistoidut - -Mode LastWriteTime Length Name ----- ------------- ------ ---- --a--- 13.8.2020 13:27 103 .ankkuri --a--- 4.12.2020 0:33 5793369649 5D3DB6C10EAF0911-12-12.mrimg - -PS> # let's try to drop the archived file; 1st connecting the remote -PS> df -h k: -C:\scoop\apps\gow\current\bin\df.exe: Warning: cannot read table of mounted file systems: Invalid argument -Filesystem Size Used Avail Use% Mounted on -- 1,9T 1,8T 118G 94% K:\ -PS> git annex find --want-drop --in . -PS> # ouch, git-annex doesn't want to drop the file; for real now: -PS> git annex drop 5D3DB6C10EAF0911-12-12.mrimg -drop 5D3DB6C10EAF0911-12-12.mrimg - That file is required content, it cannot be dropped! - - (Use --force to override this check, or adjust required content configuration.) -failed -git-annex: drop: 1 failed -PS> # same result, maybe it's those forward slashes in the expr -PS> git annex required . -(include=*.mrimg and exclude=*/arkistoidut/* and exclude=arkistoidut/*) -PS> git annex required . '(include=*.mrimg and exclude=*\arkistoidut\* and exclude=arkistoidut\*)' -required . ok -(recording state in git...) -PS> git annex required . -(include=*.mrimg and exclude=*\arkistoidut\* and exclude=arkistoidut\*) -PS> git annex find --want-drop --in . -5D3DB6C10EAF0911-12-12.mrimg -PS> # that's better! and for real now: -PS> git annex drop 5D3DB6C10EAF0911-12-12.mrimg -drop 5D3DB6C10EAF0911-12-12.mrimg ok -(recording state in git...) -PS> git annex info 5D3DB6C10EAF0911-12-12.mrimg -file: 5D3DB6C10EAF0911-12-12.mrimg -size: 5.79 gigabytes -key: MD5E-s5793369649--14bf4083cd9c4859705d74ce86b3a354.mrimg -present: false -PS> # it seems to work! let's check the link count: -PS> fsutil hardlink list 5D3DB6C10EAF0911-12-12.mrimg -\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A\arkistoidut\5D3DB6C10EAF0911-12-12.mrimg -PS> # as it should be; how about the file size? -PS> ls - - Directory: G:\Reflect-varmistukset\Jarkon ThinkPad T450s (Win10 v2004) . A\arkistoidut - -Mode LastWriteTime Length Name ----- ------------- ------ ---- --a--- 13.8.2020 13:27 103 .ankkuri --a--- 4.12.2020 0:33 5793369649 5D3DB6C10EAF0911-12-12.mrimg - -PS> # for some reason the content file is still there; -PS> # no worries, I can work around that -PS> rm 5D3DB6C10EAF0911-12-12.mrimg -PS> git status -On branch adjusted/master(hidemissing-unlocked) -Changes not staged for commit: - (use \"git add/rm ...\" to update what will be committed) - (use \"git restore ...\" to discard changes in working directory) - deleted: 5D3DB6C10EAF0911-12-12.mrimg - -no changes added to commit (use \"git add\" and/or \"git commit -a\") -PS> git restore 5D3DB6C10EAF0911-12-12.mrimg -PS> cat 5D3DB6C10EAF0911-12-12.mrimg -/annex/objects/MD5E-s5793369649--14bf4083cd9c4859705d74ce86b3a354.mrimg -PS> # excellent! all is well :) -PS> df -h . -C:\scoop\apps\gow\current\bin\df.exe: Warning: cannot read table of mounted file systems: Invalid argument -Filesystem Size Used Avail Use% Mounted on -- 466G 397G 70G 86% G:\ -# end of transcript -\"\"\"]] -
-"""]] diff --git a/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_3_9f0415b3945ee80698b5ee50351f1a5c._comment b/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_3_9f0415b3945ee80698b5ee50351f1a5c._comment deleted file mode 100644 index b0e38eb212..0000000000 --- a/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_3_9f0415b3945ee80698b5ee50351f1a5c._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-12-15T16:16:21Z" - content=""" -Hmm, yes preferred content expressions on windows ought -to let `/` be used and still match on `\`. (And vice-versa, although then -the preferred content expression is windows-specific.) - -I've implemented that now. -"""]] diff --git a/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_4_af4426660e374017b33859dd1a955ba8._comment b/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_4_af4426660e374017b33859dd1a955ba8._comment deleted file mode 100644 index 37eccd3166..0000000000 --- a/doc/bugs/drop_claims_that_content_is_required___40__8.20201127__41__/comment_4_af4426660e374017b33859dd1a955ba8._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="jkniiv" - avatar="http://cdn.libravatar.org/avatar/05fd8b33af7183342153e8013aa3713d" - subject="well done!" - date="2020-12-21T18:14:26Z" - content=""" -Version 8.20201128-g6b1357482 seems to work exactly as it's written on the tin. Thanks, Joey! -"""]] diff --git a/doc/bugs/drop_file_denied_with_required__61__groupwanted_and_groupwanted__61__nothing.mdwn b/doc/bugs/drop_file_denied_with_required__61__groupwanted_and_groupwanted__61__nothing.mdwn deleted file mode 100644 index e8d2a33789..0000000000 --- a/doc/bugs/drop_file_denied_with_required__61__groupwanted_and_groupwanted__61__nothing.mdwn +++ /dev/null @@ -1,135 +0,0 @@ -### Please describe the problem. - -git-annex denies dropping a file because it be required, but `required` = `groupwanted` and `groupwanted` = `nothing`. - -It seems that this is regardless of the value of `groupwanted`. I also tried with other expressions. - -### What steps will reproduce the problem? - -[[!format sh """ -#!/bin/sh -set -x - -# init repos & add file -mkdir local remote -cd remote -git init && git annex init remote -cd ../local -git init && git annex init local -touch file -ls -git annex add . -git remote add remote ../remote - -# repo settings -git annex group remote g -git annex groupwanted g nothing -git annex required remote groupwanted - -# test -git annex whereis file -git annex copy file --to=remote --auto # doesn't copy, as expected -git annex whereis file -git annex copy file --to=remote # copies, as expected -git annex whereis file -git annex drop file --from=remote # ERROR: drop denied -git annex whereis file - -# test using required=nothing directly -git annex required remote nothing -git annex drop file --from=remote # drops, as expected -git annex whereis file -"""]] - - -### What version of git-annex are you using? On what operating system? - - -* version 7.20190819 on NixOS 19.09 as well as version 8.20200309 from unstable Nixpkgs -* local repository version: 7 (makes no difference whether 5 or 7) - - -### Please provide any additional information below. - -[[!format sh """ -# transcript -[user@host annex-test]$ mkdir local remote -[user@host annex-test]$ cd remote -[user@host remote]$ git init -Initialized empty Git repository in /home/johannes/annex-test/remote/.git/ -[user@host remote]$ git annex init remote -init remote ok -(recording state in git...) -[user@host remote]$ cd ../local -[user@host local]$ git init -Initialized empty Git repository in /home/johannes/annex-test/local/.git/ -[user@host local]$ git annex init local -init local ok -(recording state in git...) -[user@host local]$ touch file -[user@host local]$ ls -file -[user@host local]$ git annex add . -add file - - -ok -(recording state in git...) -[user@host local]$ git remote add remote ../remote -[user@host local]$ git annex group remote g -group remote ok -(recording state in git...) -[user@host local]$ git annex groupwanted g nothing -groupwanted g ok -(recording state in git...) -[user@host local]$ git annex required remote groupwanted -required remote ok -(recording state in git...) -[user@host local]$ git annex whereis file -whereis file (1 copy) - a886389d-ebd5-489c-804b-1bcbddf2c9c5 -- local [here] -ok -[user@host local]$ git annex copy file --to=remote --auto -[user@host local]$ git annex whereis file -whereis file (1 copy) - a886389d-ebd5-489c-804b-1bcbddf2c9c5 -- local [here] -ok -[user@host local]$ git annex copy file --to=remote -copy file (to remote...) (checksum...) ok -(recording state in git...) -[user@host local]$ git annex whereis file -whereis file (2 copies) - 343d87b7-e21a-4f38-b9b6-66b01aaa7995 -- remote - a886389d-ebd5-489c-804b-1bcbddf2c9c5 -- local [here] -ok -[user@host local]$ git annex drop file --from=remote -drop remote file - That file is required content, it cannot be dropped! - - (Use --force to override this check, or adjust required content configuration.) -failed -git-annex: drop: 1 failed -[user@host local]$ git annex whereis file -whereis file (2 copies) - 343d87b7-e21a-4f38-b9b6-66b01aaa7995 -- remote - a886389d-ebd5-489c-804b-1bcbddf2c9c5 -- local [here] -ok -[user@host local]$ git annex required remote nothing -required remote ok -(recording state in git...) -[user@host local]$ git annex drop file --from=remote -drop remote file ok -(recording state in git...) -[user@host local]$ git annex whereis file -whereis file (1 copy) - a886389d-ebd5-489c-804b-1bcbddf2c9c5 -- local [here] -ok -# End of transcript or log. -"""]] - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I had, some time ago I synced my phone with it. I stopped because of the painful crippled file system there, and the annex was so large that the phone was slow. But I always wanted to restart annexing, to archive and partially checkout my libraries, and maybe do the phone sync right (probably via a special remote)! It seems to be THE tool as soon as a repo is split over multiple disks. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/drop_file_denied_with_required__61__groupwanted_and_groupwanted__61__nothing/comment_1_ff2986a884988cdf3782db59325d2a9f._comment b/doc/bugs/drop_file_denied_with_required__61__groupwanted_and_groupwanted__61__nothing/comment_1_ff2986a884988cdf3782db59325d2a9f._comment deleted file mode 100644 index 24e09e6a18..0000000000 --- a/doc/bugs/drop_file_denied_with_required__61__groupwanted_and_groupwanted__61__nothing/comment_1_ff2986a884988cdf3782db59325d2a9f._comment +++ /dev/null @@ -1,24 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-04-28T17:06:39Z" - content=""" -This comes down to this part of preferredRequiredMapsLoad: - - pc <- genmap preferredContentLog =<< groupPreferredContentMapRaw - rc <- genmap requiredContentLog M.empty - -So for required content, it does not use the group preferred content map. - -Should it also use the groupwanted values for required content in this case? -It kind of makes sense, but I do wonder if someone might have a group that they -want to use one expression for its preferred content (groupwanted) -and a different expression for its required content ("grouprequired"). - -OTOH, I suppose that allowing for the former case now does not prevent -later adding support for the latter. - -(Also, setting required to "standard" already works, so that's a precident.) - -Confirmed that passing the map does fix the behavior, so I'm doing that. -"""]] diff --git a/doc/bugs/embedcreds_with_external_special_remotes.mdwn b/doc/bugs/embedcreds_with_external_special_remotes.mdwn deleted file mode 100644 index 396b7ee366..0000000000 --- a/doc/bugs/embedcreds_with_external_special_remotes.mdwn +++ /dev/null @@ -1,70 +0,0 @@ -### Please describe the problem. -git-annex-remote-googledrive uses SETCREDS and GETCREDS to let git-annex handle the credentials. According to the [documentation](https://git-annex.branchable.com/design/external_special_remote_protocol/) it should be stored inside the git-annex branch when using hyprid or pubkey encryption. However, this does not happen. Even setting embedcreds to yes does not change anything. - -### What steps will reproduce the problem? - git annex initremote testremote type=external externaltype=googledrive prefix=test keyid= encryption=pubkey -or - - git annex initremote testremote type=external externaltype=googledrive embedcreds=yes prefix=test keyid= encryption=pubkey - - -### What version of git-annex are you using? On what operating system? - git-annex version: 8.20200309-g14a4a9f4c - build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite - dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.0 ghc-8.8.3 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.1 - key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL - remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external - operating system: linux x86_64 - supported repository versions: 8 - upgrade supported from repository versions: 0 1 2 3 4 5 6 7 - local repository version: 8 - - -### Please provide any additional information below. - -[[!format sh """ -% git annex initremote g4 type=external externaltype=googledrive embedcreds=yes prefix=test keyid=***redacted*** encryption=pubkey --debug -[...] -[2020-06-11 12:14:11.622651546] git-annex-remote-googledrive[1] --> SETCREDS credentials ***redacted*** -[2020-06-11 12:14:11.623213097] chat: gpg ["--quiet","--trust-model","always","--decrypt"] -[2020-06-11 12:14:11.761559468] process done ExitSuccess -[2020-06-11 12:14:11.761663537] chat: gpg ["--quiet","--trust-model","always","--batch","--recipient","***redacted***","--encrypt","--no-encrypt-to","--no-default-recipient","--force-mdc","--no-textmode"] -[2020-06-11 12:14:11.765603345] process done ExitSuccess -[2020-06-11 12:14:11.765697179] git-annex-remote-googledrive[1] --> INITREMOTE-SUCCESS -[...] -% git show git-annex -commit abb4cf685439115dffc393bb73cd3bb499f6aaec (git-annex) -Author: Silvio Ankermann -Date: Thu Jun 11 12:14:11 2020 +0200 - - update - -diff --git a/remote.log b/remote.log -index 5c72883..2d29c9c 100644 ---- a/remote.log -+++ b/remote.log -@@ -1,3 +1,4 @@ -+1da660c2-fe07-4dcb-aca6-12f2cfdfff52 cipher=***redacted*** cipherkeys==***redacted*** embedcreds=yes encryption=pubkey externaltype=googledrive name=g4 prefix=test root_id==***redacted*** token= type=external timestamp=1591870451.772978233s -***redacted*** - -diff --git a/uuid.log b/uuid.log -index b9196be..4a9fe4d 100644 ---- a/uuid.log -+++ b/uuid.log -@@ -1,3 +1,4 @@ -+1da660c2-fe07-4dcb-aca6-12f2cfdfff52 g4 timestamp=1591870451.772033937s -***redacted*** - -"""]] -I can see that gpg is called after SETCREDS and I would have expected there to be an additional config "credentials": - -[[!format sh """ -+1da660c2-fe07-4dcb-aca6-12f2cfdfff52 cipher=***redacted*** cipherkeys==***redacted*** credentials=*encrypted_creds* embedcreds=yes encryption=pubkey externaltype=googledrive name=g4 prefix=test root_id==***redacted*** token= type=external timestamp=1591870451.772978233s -"""]] - - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -Of course ;) In fact. I've actually never used embedcreds. I just had an [issue](https://github.com/Lykos153/git-annex-remote-googledrive/issues/48) raised about it on github. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/embedcreds_with_external_special_remotes/comment_1_4238c33bcf27e59635f5b03ee97b16ef._comment b/doc/bugs/embedcreds_with_external_special_remotes/comment_1_4238c33bcf27e59635f5b03ee97b16ef._comment deleted file mode 100644 index 0b63591b30..0000000000 --- a/doc/bugs/embedcreds_with_external_special_remotes/comment_1_4238c33bcf27e59635f5b03ee97b16ef._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-06-11T20:11:44Z" - content=""" -[[!commit e63dcbf36c3dd19525641ce5fc3fa797243d07a5]] fixed a similar -reversion in other special remotes, but did not touch the external special -remote. - -But, I think that commit may have also fixed this in passing, -because it also changes `setRemoteCredPair'`, which -is what external is using, and the change IIRC fixed another bug -that caused it to never embed creds. - -Your version predates that fix, so please try a newer version. -"""]] diff --git a/doc/bugs/embedcreds_with_external_special_remotes/comment_2_1b55548cdd10d7e287b2b82615cec4f2._comment b/doc/bugs/embedcreds_with_external_special_remotes/comment_2_1b55548cdd10d7e287b2b82615cec4f2._comment deleted file mode 100644 index 7ecfe6a231..0000000000 --- a/doc/bugs/embedcreds_with_external_special_remotes/comment_2_1b55548cdd10d7e287b2b82615cec4f2._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="lykos@d125a37d89b1cfac20829f12911656c40cb70018" - nickname="lykos" - avatar="http://cdn.libravatar.org/avatar/085df7b04d3408ba23c19f9c49be9ea2" - subject="comment 2" - date="2020-06-11T22:51:09Z" - content=""" -I tried again with version 8.20200522-g01513da127 (which should have this commit, or am I wrong?) but to no avail. The credentials still aren't stored inside the repo. -"""]] diff --git a/doc/bugs/embedcreds_with_external_special_remotes/comment_3_b2b4c8997129328531aa93db828c26e3._comment b/doc/bugs/embedcreds_with_external_special_remotes/comment_3_b2b4c8997129328531aa93db828c26e3._comment deleted file mode 100644 index 2915eadf88..0000000000 --- a/doc/bugs/embedcreds_with_external_special_remotes/comment_3_b2b4c8997129328531aa93db828c26e3._comment +++ /dev/null @@ -1,29 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-06-16T18:36:36Z" - content=""" -Confirmed this bug. This feels like it should trigger a release, as it -could break existing workflows in a surprising way, and even maybe result -in data loss if the user was relying on git-annex embedding the creds and -didn't otherwise have a way to get them. - -It's (yet another) regression caused by 7.20200202.7's sweeping changes to -remote configuration. - -Problem is, there's a externalConfigChanges that SETCONFIG -updates, but SETCREDS does not, instead it swap in a new externalConfig. -But that externalConfig is not examined when extracting the config changes -to store in remote.log, because the types don't match up any longer. - -So, SETCREDS needs to also update externalConfigChanges. - -Related regression: When SETCONFIG is used, followed by GETCONFIG -of the same value, it does not return the value. This doesn't affect -SETCONFIG at init time followed by GETCONFIG later, so it's unlikely to -affect anything, but it's still wrong, and so I've fixed it. - -And they don't stop coming, yet another regression: Not setting embedcreds -was treated as embedcreds=no, because the bool parser defaulted to False. -So the implicit embedcreds when using encryption=pubkey broke. -"""]] diff --git a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression.mdwn b/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression.mdwn deleted file mode 100644 index 88eb825f32..0000000000 --- a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression.mdwn +++ /dev/null @@ -1,21 +0,0 @@ -### Please describe the problem. - -Export-tree exports "small" files even if they don't match the wanted expression for the remote - -### What steps will reproduce the problem? - -Create a special remote with export-tree=yes - -Set wanted expression for remote - -Export-tree to it -> all small files in tree end up on the remote - - -### What version of git-annex are you using? On what operating system? - -8.20200330 on linux i386 - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -I'm very impressed with git-annex overall :P - -> [[wontfix|done]] --[[Joey]] diff --git a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_1_08419003d89dab1f8b1dbfa7941b097d._comment b/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_1_08419003d89dab1f8b1dbfa7941b097d._comment deleted file mode 100644 index ef95028dbf..0000000000 --- a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_1_08419003d89dab1f8b1dbfa7941b097d._comment +++ /dev/null @@ -1,23 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-08-10T16:22:15Z" - content=""" -Preferred content expressions can have a number of terms that require a -git-annex key to evaluate. Files not managed by git-annex do not have such -keys, so it's not possible to evaluate preferred content expressions -against them. - -Another way to look at it is that the export remote is equivilant to a git -remote. Pushing a git tree to a git remote necessarily stores all the files -managed by git onto that remote and so it makes sense for exporting to do -the same. (importing also always gets new versions of these small files w/o -checking the preferred content setting, equivilant to git pull.) - -So, even if it were possible to match some preferred content expressions -against non-annexed files (eg ones that use only the filename or size), -I don't think it would be good for that to be done for exports. - -I have improved the documentation of git-annex export to mention that -it exports all git files. -"""]] diff --git a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_2_1ba87d9d70b59c5304ed3dd63b14b032._comment b/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_2_1ba87d9d70b59c5304ed3dd63b14b032._comment deleted file mode 100644 index fda5228388..0000000000 --- a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_2_1ba87d9d70b59c5304ed3dd63b14b032._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="arnaud@c1d1cc612a3921dc06a417301be08a3e125478c4" - nickname="arnaud" - avatar="http://cdn.libravatar.org/avatar/c0defbf54541c499f6c0464588fc2ad0" - subject="comment 2" - date="2020-08-10T17:25:09Z" - content=""" -May I suggest then that by default, no files be considered \"small\" ? In my beginnings with git-annex, I have been confused several times by small files. If they don't behave the same as other files, I don't think they should happen by default. -"""]] diff --git a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_2_ad22d7a4b5052ff05b39ec5b8c2805ff._comment b/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_2_ad22d7a4b5052ff05b39ec5b8c2805ff._comment deleted file mode 100644 index 898acc9cc3..0000000000 --- a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_2_ad22d7a4b5052ff05b39ec5b8c2805ff._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-08-10T17:49:34Z" - content=""" -Only dotfiles are considered small by default, and there are good reasons -to do so, which are well outside the scope of this bug report. - -If you have any other small files, it's because you've added them to git -and not to git-annex in some way, by setting annex.largefiles or using -command line options or whatever. -"""]] diff --git a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_4_aa029562ec76e83ecc5e6fc76355ad96._comment b/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_4_aa029562ec76e83ecc5e6fc76355ad96._comment deleted file mode 100644 index e964b73b4f..0000000000 --- a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_4_aa029562ec76e83ecc5e6fc76355ad96._comment +++ /dev/null @@ -1,26 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-08-10T19:18:13Z" - content=""" -Understandable mistake. If you might have more such files, -you can `git config annex.dotfiles true` - -> Also, I was surprised to find that include= expressions were relative to -> the part of the tree I was exporting and not the git-annex root ? - -Well, if you run `git annex export somesha --to remote`, all it knows is -the tree for that sha, so it has to match relative to the top of that tree, -and not whatever other tree it might be embedded in on master or wherever. - -When you run `git annex export master:subdir --to remote`, it has enough -information that it could match relative to the top of master, but that -would be inconsistent. - -And the same with a config setting annex-tracking-branch to master:subdir. - -Import does also do the same thing, it has to also for consistency of -course. - -I have mentioned this in the docs now. -"""]] diff --git a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_4_f5d0de491ca51534c229ef7901efb67d._comment b/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_4_f5d0de491ca51534c229ef7901efb67d._comment deleted file mode 100644 index 7bfe7a6444..0000000000 --- a/doc/bugs/export-tree_exports___34__small__34___files_even_if_they_don__39__t_match_the_wanted_expression/comment_4_f5d0de491ca51534c229ef7901efb67d._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="arnaud@c1d1cc612a3921dc06a417301be08a3e125478c4" - nickname="arnaud" - avatar="http://cdn.libravatar.org/avatar/c0defbf54541c499f6c0464588fc2ad0" - subject="comment 4" - date="2020-08-10T18:42:13Z" - content=""" -Oh, indeed, I was confused because this is one of the directories that ended up as a small file : \"Genesis/...And\ Then\ There\ Were\ Three\" ! it's indeed a dotfile, kind of. - -Sorry ! - -Also, I was surprised to find that include= expressions were relative to the part of the tree I was exporting and not the git-annex root ? -"""]] diff --git a/doc/bugs/failing_tests_in_master.mdwn b/doc/bugs/failing_tests_in_master.mdwn deleted file mode 100644 index 05c4e3d9da..0000000000 --- a/doc/bugs/failing_tests_in_master.mdwn +++ /dev/null @@ -1,26 +0,0 @@ -Recent optimisation work has caused some test suite failures, involving -unannex and fsck. --[[Joey]] - -Hmm, the unannex one is because it doesn't check if a file is present -inAnnex anymore, so fails on unannexing a non-present file. - -The check was removed in -[[!commit 75aab72d23ebf8ef0d56d7dd74c121e33d64f1f6]], but that did have -withFilesInGitAnnex doing an inAnnex check and not running the command if -it failed. But later that got changed to only happen when -checkContentPresent is set. - -Everything in that and related commits -needs to be re-examined to make sure no other inAnnex checks were lost. - -> Well, I didn't find any others. - -The fsck failure is that it fails to fail when there are 0 copies of the -file. Same basic cause, but the other direction, it was skipping files -whose content is not present. Also introduced in [[!commit -75aab72d23ebf8ef0d56d7dd74c121e33d64f1f6]]. - -So, also need to audit everything with checkContentPresent = Just True -etc to make sure it's right. Ok, done, there were not many using it yet. - -[[done]] --[[Joey]] diff --git a/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn b/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn deleted file mode 100644 index 601035f63e..0000000000 --- a/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable.mdwn +++ /dev/null @@ -1,50 +0,0 @@ -### Please describe the problem. - - -### What steps will reproduce the problem? - -```touch foo -chmod 755 foo -git-annex add foo -ls -lL foo -``` - -### What version of git-annex are you using? On what operating system? - -``` -git-annex version: 7.20190129 -build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external -operating system: linux x86_64 -supported repository versions: 5 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -local repository version: 5 -``` - -### Please provide any additional information below. - -``` -$ touch foo -$ chmod 755 foo -$ ls -l foo --rwxr-xr-x 1 hans hans 0 sep 2 15:00 foo -$ git-annex add foo -add foo ok -(recording state in git...) -$ ls -lL foo --rw-rw-rw- 1 hans hans 65 jul 24 2015 foo -``` - -NB: the date of the file is now jul 24 2015, but I created foo 2 sep, so what happened here? (This annex is old, so I might have created a file foo in 2015, but that should not intere with this new file). - -I have my private binaries in ~/annex/bin and this has worked well for many years, but recently it stopped working. - - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I've used git-annex since summer 2012, it's great :-) - -> [[notabug|done]] --[[Joey]] diff --git a/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable/comment_1_7e4599871d40c07cc30e0b9487288e3a._comment b/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable/comment_1_7e4599871d40c07cc30e0b9487288e3a._comment deleted file mode 100644 index 5dd046b776..0000000000 --- a/doc/bugs/fails_to_preserve_mode_bits___40__executable_files_becomes_non-executable/comment_1_7e4599871d40c07cc30e0b9487288e3a._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-09-08T16:11:01Z" - content=""" -Appears that you already had an empty file in the repository. Since -git-annex deduplicates content, it used the content of that empty file -for the new file you added. The empty file was created in 2015 is a -giveaway. - -Do note that many git-annex special remotes cannot preserve modes or -other metadata of files anyway, so after moving a file to a remote and -getting it back, the mode can be reset to default. - -The mode is not version controlled when you check a symlink into git, -because symlinks do not have -modes.. If you want to preserve execute bits, the only way to guarantee it -will happen is to add the file unlocked. Then git tracks the execute bit. -"""]] diff --git a/doc/bugs/fatal__58___relative_path_syntax_can__39__t_be_used_outside_working_tree.mdwn b/doc/bugs/fatal__58___relative_path_syntax_can__39__t_be_used_outside_working_tree.mdwn deleted file mode 100644 index 1e30801ca0..0000000000 --- a/doc/bugs/fatal__58___relative_path_syntax_can__39__t_be_used_outside_working_tree.mdwn +++ /dev/null @@ -1,44 +0,0 @@ -### Please describe the problem. -A bare v7 repo fails to upgrade to v8, and displays several times the following error message: "fatal: relative path syntax can't be used outside working tree" - - -### What steps will reproduce the problem? -``` -cd problem_repo.git -git annex upgrade -``` - -### What version of git-annex are you using? On what operating system? -git-annex version: 8.20200617 - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - -upgrade (v7 to v8...) fatal: relative path syntax can't be used outside working tree -fatal: relative path syntax can't be used outside working tree -fatal: relative path syntax can't be used outside working tree -fatal: relative path syntax can't be used outside working tree -fatal: relative path syntax can't be used outside working tree -fatal: relative path syntax can't be used outside working tree -fatal: relative path syntax can't be used outside working tree -fatal: relative path syntax can't be used outside working tree -fatal: relative path syntax can't be used outside working tree -fatal: relative path syntax can't be used outside working tree -fatal: relative path syntax can't be used outside working tree - -git-annex: fd:18: Data.ByteString.hGetLine: end of file -failed -git-annex: user error (git ["--git-dir=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] exited 128) - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Yes! We love it! Thank you so much! Git-annex is a superpower 🦹‍♂️ - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/fatal__58___relative_path_syntax_can__39__t_be_used_outside_working_tree/comment_1_68de6f1477d91e769aae8d565c5f37cb._comment b/doc/bugs/fatal__58___relative_path_syntax_can__39__t_be_used_outside_working_tree/comment_1_68de6f1477d91e769aae8d565c5f37cb._comment deleted file mode 100644 index ae979cb454..0000000000 --- a/doc/bugs/fatal__58___relative_path_syntax_can__39__t_be_used_outside_working_tree/comment_1_68de6f1477d91e769aae8d565c5f37cb._comment +++ /dev/null @@ -1,33 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-09-29T17:34:03Z" - content=""" -I have not been able to reproduce this so far in a usual environment. - -It doesn't even run `git cat-file` in my testing, let alone do -anything with it that fails. - -AFAICS, for `git cat-file` to run, the `git ls-files --cached` that it does -run would have to otput some filenames. Which should be impossible normally -in a bare repository, because there is no index file in a bare repo. - -So, there is something unusual about your repository -or environment. - -Here are two ways to reproduce that failure: - -* Set `GIT_INDEX_FILE` to point to an index file belonging to some other - git repository. -* Copy .git/index from some other repository into the bare repository. - -I guess you, or some past buggy thing, could have done the latter. -Or perhaps you did the former and forgot about the envionment setting? - -Anyway, I'll make git-annex check if it's a bare repo and skip this -part entirely, but really it seems to me that either of those possibilities -is pretty far into foot shooting territory and if git-annex explodes -because you're doing stuff like that, it's hardly a bug. Of course, -if you can somehow rule out both possibilities, that would be a different -story. -"""]] diff --git a/doc/bugs/file_not_correctly_added.mdwn b/doc/bugs/file_not_correctly_added.mdwn deleted file mode 100644 index cb88fa9d8f..0000000000 --- a/doc/bugs/file_not_correctly_added.mdwn +++ /dev/null @@ -1,223 +0,0 @@ -### Please describe the problem. -When I run `git annex add .` the file is no longer symlinked into the directory, but the file is keep as it is and simply added to git. This also happens with a minimalistic test repo as described below: - -### What steps will reproduce the problem? -- `sudo pacman -S git-annex` -- `mkdir ./test` -- `cd test` -- `git init` -- `git annex init` -- `cp ../myLargeFile.CR2 .` -- `git annex add .` - -### What version of git-annex are you using? On what operating system? -``` -$ git annex version -git-annex version: 8.20201103-g2dabd4cc2d -build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV -dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.10.2 http-client-0.7.2.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external -operating system: linux x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 0 1 2 3 4 5 6 7 -local repository version: 8 -``` - -``` -$ uname -a -Linux jcarch 5.4.74-1-lts #1 SMP Sun, 01 Nov 2020 12:58:27 +0000 x86_64 GNU/Linux -``` - -### Please provide any additional information below. - -[[!format sh """ -$ git annex add -d test.CR2 -[2020-11-05 14:05:18.380112289] process [47259] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] -[2020-11-05 14:05:18.380857707] process [47259] done ExitSuccess -[2020-11-05 14:05:18.381159814] process [47260] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] -[2020-11-05 14:05:18.381788658] process [47260] done ExitSuccess -[2020-11-05 14:05:18.382106752] process [47261] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..ae4cf18c6d72434a514230220967195c0b230adb","--pretty=%H","-n1"] -[2020-11-05 14:05:18.382829403] process [47261] done ExitSuccess -[2020-11-05 14:05:18.38340398] process [47262] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"] -[2020-11-05 14:05:18.383651872] process [47263] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] -[2020-11-05 14:05:18.384269396] process [47264] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"] -[2020-11-05 14:05:18.385141135] process [47264] done ExitSuccess -[2020-11-05 14:05:18.385420658] process [47265] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"] -[2020-11-05 14:05:18.386064486] process [47265] done ExitFailure 1 -[2020-11-05 14:05:18.386480384] process [47266] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","-z","--others","--exclude-standard","--","test.CR2"] -[2020-11-05 14:05:18.387250532] process [47267] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"] -add test.CR2 [2020-11-05 14:05:18.389436209] process [47268] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","diff.external=","-c","filter.annex.smudge=","-c","filter.annex.clean=","diff","--cached","--raw","-z","--no-abbrev","-G^/annex/objects/","--diff-filter=AMUT","--no-renames","--ignore-submodules=all","--no-ext-diff"] -[2020-11-05 14:05:18.391345028] process [47268] done ExitSuccess -ok -[2020-11-05 14:05:18.391581576] process [47266] done ExitSuccess -[2020-11-05 14:05:18.39193215] process [47270] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","-z","--modified","--","test.CR2"] -[2020-11-05 14:05:18.392531099] process [47270] done ExitSuccess -[2020-11-05 14:05:18.392820948] process [47271] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff","--name-only","--diff-filter=T","-z","--cached","--","test.CR2"] -[2020-11-05 14:05:18.393490388] process [47271] done ExitSuccess -(recording state in git...) -[2020-11-05 14:05:18.39392609] process [47272] feed: xargs ["-0","git","--git-dir=.git","--work-tree=.","--literal-pathspecs","add","--"] -[2020-11-05 14:05:18.679370104] process [47272] done ExitSuccess -[2020-11-05 14:05:18.680381167] process [47262] done ExitSuccess -[2020-11-05 14:05:18.680659505] process [47263] done ExitSuccess -[2020-11-05 14:05:18.680881174] process [47267] done ExitSuccess -"""]] - -The directory structure of the repo after running `git annex add .`: -[[!format sh """ -$ tree -a -. -├── .git -│   ├── annex -│   │   ├── gitqueue.lck -│   │   ├── index -│   │   ├── index.lck -│   │   ├── journal -│   │   ├── journal.lck -│   │   ├── keysdb -│   │   │   └── db -│   │   ├── keysdb.cache -│   │   ├── keysdb.lck -│   │   ├── othertmp.lck -│   │   ├── sentinal -│   │   └── sentinal.cache -│   ├── branches -│   ├── config -│   ├── description -│   ├── HEAD -│   ├── hooks -│   │   ├── applypatch-msg.sample -│   │   ├── commit-msg.sample -│   │   ├── fsmonitor-watchman.sample -│   │   ├── post-checkout -│   │   ├── post-merge -│   │   ├── post-receive -│   │   ├── post-update.sample -│   │   ├── pre-applypatch.sample -│   │   ├── pre-commit -│   │   ├── pre-commit.sample -│   │   ├── pre-merge-commit.sample -│   │   ├── prepare-commit-msg.sample -│   │   ├── pre-push.sample -│   │   ├── pre-rebase.sample -│   │   ├── pre-receive.sample -│   │   └── update.sample -│   ├── index -│   ├── info -│   │   ├── attributes -│   │   └── exclude -│   ├── logs -│   │   └── refs -│   │   └── heads -│   │   └── git-annex -│   ├── objects -│   │   ├── 0d -│   │   │   └── b47ff2773fdd41f7be54839f6a3450c25da595 -│   │   ├── 4b -│   │   │   └── 825dc642cb6eb9a060e54bf8d69288fbee4904 -│   │   ├── 53 -│   │   │   └── 40ddb9312ba2c09f08cb0308f4f8679b70735e -│   │   ├── 7f -│   │   │   └── 92a6e546f860b696d1ee94694296975761b8f4 -│   │   ├── 9d -│   │   │   └── c023b630e35c131d96c10641e57de7529c3c64 -│   │   ├── ae -│   │   │   └── 4cf18c6d72434a514230220967195c0b230adb -│   │   ├── info -│   │   └── pack -│   └── refs -│   ├── heads -│   │   └── git-annex -│   └── tags -└── test.CR2 -"""]] - -The repo after running `git annex unlock test.CR2 && git annex lock test.CR2`. Now, I do get a symlink, but it is completely broken: -[[!format sh """ -$ tree -a -. -├── .git -│   ├── annex -│   │   ├── gitqueue.lck -│   │   ├── index -│   │   ├── index.lck -│   │   ├── journal -│   │   ├── journal.lck -│   │   ├── keysdb -│   │   │   └── db -│   │   ├── keysdb.cache -│   │   ├── keysdb.lck -│   │   ├── objects -│   │   │   └── Z8 -│   │   │   └── ff -│   │   │   └── SHA256E-s29334740--1ddc219a95f1d1fa37e94667dae44c8a5ec42fff0c5dfd5dc46ac9e6ad43e7fe.CR2 -│   │   ├── othertmp.lck -│   │   ├── sentinal -│   │   └── sentinal.cache -│   ├── branches -│   ├── config -│   ├── description -│   ├── HEAD -│   ├── hooks -│   │   ├── applypatch-msg.sample -│   │   ├── commit-msg.sample -│   │   ├── fsmonitor-watchman.sample -│   │   ├── post-checkout -│   │   ├── post-merge -│   │   ├── post-receive -│   │   ├── post-update.sample -│   │   ├── pre-applypatch.sample -│   │   ├── pre-commit -│   │   ├── pre-commit.sample -│   │   ├── pre-merge-commit.sample -│   │   ├── prepare-commit-msg.sample -│   │   ├── pre-push.sample -│   │   ├── pre-rebase.sample -│   │   ├── pre-receive.sample -│   │   └── update.sample -│   ├── index -│   ├── info -│   │   ├── attributes -│   │   └── exclude -│   ├── logs -│   │   └── refs -│   │   └── heads -│   │   └── git-annex -│   ├── objects -│   │   ├── 0d -│   │   │   └── b47ff2773fdd41f7be54839f6a3450c25da595 -│   │   ├── 3a -│   │   │   └── 23d9e0574e7b37778991638c07005f6c27418f -│   │   ├── 45 -│   │   │   └── 36e1509db6efcf5b3f213e94e139dfffd909b7 -│   │   ├── 4b -│   │   │   └── 825dc642cb6eb9a060e54bf8d69288fbee4904 -│   │   ├── 53 -│   │   │   └── 40ddb9312ba2c09f08cb0308f4f8679b70735e -│   │   ├── 74 -│   │   │   └── 4ae89ab07cb3c592c4e815db44adb1f2b0f679 -│   │   ├── 7f -│   │   │   └── 92a6e546f860b696d1ee94694296975761b8f4 -│   │   ├── 85 -│   │   │   └── f161fd698805ff508cf17712f7297751ea4d51 -│   │   ├── 9c -│   │   │   └── 1989036a57661474b3442062b4fcc3c285681b -│   │   ├── 9d -│   │   │   └── c023b630e35c131d96c10641e57de7529c3c64 -│   │   ├── ae -│   │   │   └── 4cf18c6d72434a514230220967195c0b230adb -│   │   ├── f8 -│   │   │   └── c3a1f4e637bb0cffd25ae8fa61ce38d3b9d2f2 -│   │   ├── info -│   │   └── pack -│   └── refs -│   ├── heads -│   │   └── git-annex -│   └── tags -└── test.CR2 -> .git/annex/objects/Z8/ff/SHA256E-s29334740--1ddc219a95f1d1fa37e94667dae44c8a5ec42fff0c5dfd5dc46ac9e6ad43e7fe.CR2/SHA256E-s29334740--1ddc219a95f1d1fa37e94667dae44c8a5ec42fff0c5dfd5dc46ac9e6ad43e7fe.CR2 -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -It used to work before. It seem to be broken since a system update. I have tried reinstalling `git-annex` as well as all dependencies, bit without any luck. - -> Added warning. [[done]] --[[Joey]] diff --git a/doc/bugs/file_not_correctly_added/comment_10_07583ff86653cb262f84e307c2ffc797._comment b/doc/bugs/file_not_correctly_added/comment_10_07583ff86653cb262f84e307c2ffc797._comment deleted file mode 100644 index 978128448c..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_10_07583ff86653cb262f84e307c2ffc797._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 10""" - date="2020-11-10T15:50:10Z" - content=""" -So was myLargeFile.CR2 in fact an annex pointer file all along? - -Anyway, I think it might be possible to avoid this confusing kind of -situation. If git-annex add sees an annex pointer file or symlink, it can -check if there's a location log file for it. Normally there always is. If -not, that's a situation where some file from another annex is being added, -and it can either warn, or refuse to proceed. - -This will only really be worth doing if git-annex smudge --clean can also -do it, so if git add is used it does the same check. -"""]] diff --git a/doc/bugs/file_not_correctly_added/comment_11_638a67da906d52d77fb6bf06b2e3d9d9._comment b/doc/bugs/file_not_correctly_added/comment_11_638a67da906d52d77fb6bf06b2e3d9d9._comment deleted file mode 100644 index a484850124..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_11_638a67da906d52d77fb6bf06b2e3d9d9._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="jcjgraf" - avatar="http://cdn.libravatar.org/avatar/9dda752f83ac44906fefbadb35e8a6ac" - subject="comment 11" - date="2020-11-10T16:05:57Z" - content=""" -Yes, it was a git-annex pointer. `cat myLargeFile.CR2` gave something like `'/annex/objects/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4.CR2'` - -It would indeed be helpful if there was a warning or something similar. Though, it is an error one is probably only doing once in the course of using git-annex :-) - -Once again, thank you very much and have a good time! -"""]] diff --git a/doc/bugs/file_not_correctly_added/comment_1_3da19ed6975dc562c194175ce956f05a._comment b/doc/bugs/file_not_correctly_added/comment_1_3da19ed6975dc562c194175ce956f05a._comment deleted file mode 100644 index 5b1dbf17f6..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_1_3da19ed6975dc562c194175ce956f05a._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Lukey" - avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" - subject="comment 1" - date="2020-11-06T14:14:18Z" - content=""" -I can't reproduce this with the latest [[standalone build|https://git-annex.branchable.com/install/Linux_standalone/]] (8.20201008). What does `git status` say after `git annex add`? What filesystem is this on? Can you try this again with the standalone build? -"""]] diff --git a/doc/bugs/file_not_correctly_added/comment_2_aadc2c2faa8455614efd6e8427965ece._comment b/doc/bugs/file_not_correctly_added/comment_2_aadc2c2faa8455614efd6e8427965ece._comment deleted file mode 100644 index 9746dcd2b5..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_2_aadc2c2faa8455614efd6e8427965ece._comment +++ /dev/null @@ -1,30 +0,0 @@ -[[!comment format=mdwn - username="jcjgraf" - avatar="http://cdn.libravatar.org/avatar/9dda752f83ac44906fefbadb35e8a6ac" - subject="comment 2" - date="2020-11-06T14:57:52Z" - content=""" -Thanks for the reply! - -- `git status` after running `git annex add` simply reports that there is a new file: - -``` -$ git annex add -add test.CR2 ok -(recording state in git...) - -$ git status -On branch master - -No commits yet - -Changes to be committed: - (use \"git rm --cached ...\" to unstage) - new file: test.CR2 -``` - -- The file system is just plain ext4. - -- The behaviour of the standalone build does not differ. - -"""]] diff --git a/doc/bugs/file_not_correctly_added/comment_3_1423263c8460283c6df90630d03e0a40._comment b/doc/bugs/file_not_correctly_added/comment_3_1423263c8460283c6df90630d03e0a40._comment deleted file mode 100644 index b1b40e106b..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_3_1423263c8460283c6df90630d03e0a40._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Lukey" - avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" - subject="comment 3" - date="2020-11-08T08:25:36Z" - content=""" -Hmm, when testing the standalone build, did you actually point `PATH` to it, so that it overrides your host git and git-annex? I.e. `export PATH=\":$PATH\"` and then `which git-annex` should show you the path to the standalone git-annex. -"""]] diff --git a/doc/bugs/file_not_correctly_added/comment_4_d0eb0c30db384a1dad7516377f5a069e._comment b/doc/bugs/file_not_correctly_added/comment_4_d0eb0c30db384a1dad7516377f5a069e._comment deleted file mode 100644 index 6798e541a9..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_4_d0eb0c30db384a1dad7516377f5a069e._comment +++ /dev/null @@ -1,26 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-11-09T16:16:31Z" - content=""" -So, the test suite contains many tests that would blow up if it was breaking -this way. - -It seems to me you must have something in your git configuration or other -local environment, or in the repo's git-annex config, that is causing this, -since it doesn't happen to anyone else. Or perhaps something else like a -version of git that is somehow problimatic. - -If annex.addunlocked is configured, `git-annex add` will add the file to -the annex in unlocked mode. This must be happening, because a) the command -does not say "non-large file; adding content to git repository" which it -would if annex.largefiles was configured to make that happen and b) -git-annex lock/unlock only operate on annexed files. - -The thing I don't understand, and cannot reproduce, is that -.git/annex/objects should contain an object after `git-annex add`, and you -show it does not. It seems to me that `git annex whereis` should probably -be either complaining that it's lost the content, or tell about some other -repository that the content has somehow moved to. Or if not, at least `git -annex fsck` should have something to say about this situation. -"""]] diff --git a/doc/bugs/file_not_correctly_added/comment_5_33fb965169ce74213684fc29ff807da1._comment b/doc/bugs/file_not_correctly_added/comment_5_33fb965169ce74213684fc29ff807da1._comment deleted file mode 100644 index ea2b990ac5..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_5_33fb965169ce74213684fc29ff807da1._comment +++ /dev/null @@ -1,54 +0,0 @@ -[[!comment format=mdwn - username="jcjgraf" - avatar="http://cdn.libravatar.org/avatar/9dda752f83ac44906fefbadb35e8a6ac" - subject="comment 5" - date="2020-11-09T18:13:47Z" - content=""" - The global git-annex installation was removed when I tried it with the standalone. So there was no possible interference. - -I initiate the repo as described above. There is no local git-annex config as well as no remote. I do have a global `.gitconfig`, but besides the ordinary settings, there is nothing special: - -``` -[user] - name = - email = - signingKey = -[core] - editor = nvim - excludesfile = /home/jeanclaude/.gitignore_global -[merge] - tool = vimdiff -[mergetool] - prompt = true - keepBackup = false -[mergetool \"vimdiff\"] - cmd = nvim -d $BASE $LOCAL $REMOTE $MERGED -c '$wincmd w' -c 'wincmd J' -[commit] - gpgSign = true -[gpg] - program = gpg2 -[pull] - ff = only -``` - -After adding my file, running `git annex whereis` and `git annex fsck` do indeed report was do indeed fail and report that there is no copy of my file. - -``` -$git annex whereis test.CR2 -whereis test.CR2 (0 copies) failed -git-annex: whereis: 1 failed - -# /tmp/test on git:master x C:1 -$git annex fsck -fsck test.CR2 - ** No known copies exist of test.CR2 -failed -(recording state in git...) -git-annex: fsck: 1 failed -``` - -I am running git version 2.29.2. - -Git-annex does not have any \"global\" settings, caches or anything I could have overseen, right? - -"""]] diff --git a/doc/bugs/file_not_correctly_added/comment_6_10c5aadc94c4465b7193cb886b5877f4._comment b/doc/bugs/file_not_correctly_added/comment_6_10c5aadc94c4465b7193cb886b5877f4._comment deleted file mode 100644 index efdc2582b6..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_6_10c5aadc94c4465b7193cb886b5877f4._comment +++ /dev/null @@ -1,51 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 6""" - date="2020-11-09T19:27:12Z" - content=""" -Ok, I have a way to reprouduce this. Let's assume that your -original "myLargeFile.CR2" was from some other git-annex repo. -And it was unlocked in that repo. And you dropped it from that repo. -And then maybe you moved it around or copied it, or whatever and forgot that -this happened, leading you to the "cp ../myLargeFile.CR2 ." in your -original steps to reproduce the problem. - -Then the content of that file would be something like this: - - echo '/annex/objects/SHA256E-s3--98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4.CR2' > myLargeFile.CR2 - -And what happens if I add that? - - # git annex add myLargeFile.CR2 - add myLargeFile.CR2 ok - # git annex fsck - fsck myLargeFile.CR2 - ** No known copies exist of myLargeFile.CR2 - failed - -This happens because I added a git-annex link to the repo so now it contains -this link to an object that never got added. Same would happen if I used -git add, or if I did an ln -s to make a .git/annex/objects symlink and added -that. I don't think it's a bug really. - -Is this what you did? If not, further responses below, but I'll bet we -can stop here. - ----- - -Configuration can be included in the repo (by git-annex config), but you would -have to be cloning from another repo that had those config settings, not -creating a new repo as you seem to show in your original test case. - -Are you sure you didn't miss some place where a git -config could be set? (Eg, /etc/gitconfig) -Running `git config --list | grep annex` would display any git config you -might have missed. - -Do whereis/fsck show that immediately after you run "git-annex add", -without and other commands in between? - -There's a good chance that a full transcript of it happening would have -some other clue. Like if it enters an adjusted unlocked branch for some -reason. -"""]] diff --git a/doc/bugs/file_not_correctly_added/comment_7_9943b430113b98557cb43effbd4480ec._comment b/doc/bugs/file_not_correctly_added/comment_7_9943b430113b98557cb43effbd4480ec._comment deleted file mode 100644 index 770b11cdaa..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_7_9943b430113b98557cb43effbd4480ec._comment +++ /dev/null @@ -1,292 +0,0 @@ -[[!comment format=mdwn - username="jcjgraf" - avatar="http://cdn.libravatar.org/avatar/9dda752f83ac44906fefbadb35e8a6ac" - subject="comment 7" - date="2020-11-09T20:41:00Z" - content=""" -Unfortunately, this is not the cause of my problem. I have tried it with some files which were never part of a git annex repo but without any difference. - -No, there is no other git configuration except the one I have posted and the local one generated during the initialisation of the test repo (see the `git config --list` output at the bottom of this message). - -Yes, I have run these two commands (`whereis` and `fsck`) right after running `git annex add`. - -Here a full transcript of the taken steps. I hope this yield some more insights. If there is anything else which would be helpful, let me know. - -Thanks a lot for all your effort! - - -``` -# ~/Images --> mkdir test - -# ~/Images --> cd test - -# ~/Images/test --> git init -Initialized empty Git repository in /home/jeanclaude/Images/test/.git/ - -# ~/Images/test on git:master o --> git annex init -d -init [2020-11-09 21:18:05.857678458] process [301188] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] -[2020-11-09 21:18:05.858362763] process [301188] done ExitFailure 1 -[2020-11-09 21:18:05.858530918] process [301189] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--verify\",\"-q\",\"origin/git-annex\"] -[2020-11-09 21:18:05.859091599] process [301189] done ExitFailure 1 -[2020-11-09 21:18:05.859778604] process [301190] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"write-tree\"] -[2020-11-09 21:18:05.860578566] process [301190] done ExitSuccess -[2020-11-09 21:18:05.860992266] process [301191] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"commit-tree\",\"4b825dc642cb6eb9a060e54bf8d69288fbee4904\",\"--no-gpg-sign\"] -[2020-11-09 21:18:05.861708745] process [301191] done ExitSuccess -[2020-11-09 21:18:05.86196764] process [301192] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-ref\",\"refs/heads/git-annex\",\"cc29c590c2b3802aad89ba45d82275b378a8c843\"] -[2020-11-09 21:18:05.862691494] process [301192] done ExitSuccess -[2020-11-09 21:18:05.863174225] process [301193] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"annex.uuid\",\"092eb39f-6944-437d-8085-3062827d114f\"] -[2020-11-09 21:18:05.863730795] process [301193] done ExitSuccess -[2020-11-09 21:18:05.863942819] process [301194] read: git [\"config\",\"--null\",\"--list\"] -[2020-11-09 21:18:05.864510469] process [301194] done ExitSuccess -[2020-11-09 21:18:05.8689692] process [301195] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"] -[2020-11-09 21:18:05.869730113] process [301195] done ExitSuccess -[2020-11-09 21:18:05.870051925] process [301196] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] -[2020-11-09 21:18:05.87067267] process [301196] done ExitSuccess -[2020-11-09 21:18:05.871064391] process [301197] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..cc29c590c2b3802aad89ba45d82275b378a8c843\",\"--pretty=%H\",\"-n1\"] -[2020-11-09 21:18:05.871789326] process [301197] done ExitSuccess -[2020-11-09 21:18:05.872219143] process [301198] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"] -[2020-11-09 21:18:05.872450128] process [301199] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"] -[2020-11-09 21:18:05.873070997] process [301200] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"annex.version\",\"8\"] -[2020-11-09 21:18:05.873747221] process [301200] done ExitSuccess -[2020-11-09 21:18:05.873983127] process [301201] read: git [\"config\",\"--null\",\"--list\"] -[2020-11-09 21:18:05.874692554] process [301201] done ExitSuccess -[2020-11-09 21:18:05.875039242] process [301202] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"status\",\"--porcelain\"] -[2020-11-09 21:18:05.875796308] process [301202] done ExitSuccess -[2020-11-09 21:18:05.876081608] process [301203] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"filter.annex.smudge\",\"git-annex smudge -- %f\"] -[2020-11-09 21:18:05.876774707] process [301203] done ExitSuccess -[2020-11-09 21:18:05.877020054] process [301204] read: git [\"config\",\"--null\",\"--list\"] -[2020-11-09 21:18:05.877590492] process [301204] done ExitSuccess -[2020-11-09 21:18:05.877809109] process [301205] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"config\",\"filter.annex.clean\",\"git-annex smudge --clean -- %f\"] -[2020-11-09 21:18:05.878396585] process [301205] done ExitSuccess -[2020-11-09 21:18:05.878568762] process [301206] read: git [\"config\",\"--null\",\"--list\"] -[2020-11-09 21:18:05.879121735] process [301206] done ExitSuccess -(scanning for unlocked files...) -[2020-11-09 21:18:05.879331578] process [301207] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--head\"] -[2020-11-09 21:18:05.879894524] process [301207] done ExitSuccess -[2020-11-09 21:18:05.880285614] process [301208] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"] -[2020-11-09 21:18:05.880860662] process [301208] done ExitSuccess -[2020-11-09 21:18:05.881026905] process [301209] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"] -[2020-11-09 21:18:05.881624482] process [301209] done ExitFailure 1 -[2020-11-09 21:18:05.881911479] process [301210] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"] -[2020-11-09 21:18:05.882544605] process [301210] done ExitSuccess -[2020-11-09 21:18:05.88274148] process [301211] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/master\"] -[2020-11-09 21:18:05.883377364] process [301211] done ExitFailure 1 -[2020-11-09 21:18:05.883636208] process [301212] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"checkout\",\"-q\",\"-B\",\"master\"] -[2020-11-09 21:18:05.884333724] process [301212] done ExitSuccess -[2020-11-09 21:18:05.885138806] process [301213] read: uname [\"-n\"] -[2020-11-09 21:18:05.885414287] process [301213] done ExitSuccess -ok -[2020-11-09 21:18:05.88680324] process [301214] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"] -[2020-11-09 21:18:05.887058838] process [301215] feed: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-index\",\"-z\",\"--index-info\"] -[2020-11-09 21:18:05.88795655] process [301215] done ExitSuccess -[2020-11-09 21:18:05.888423958] process [301216] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] -[2020-11-09 21:18:05.889185379] process [301216] done ExitSuccess -(recording state in git...) -[2020-11-09 21:18:05.889577989] process [301218] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"write-tree\"] -[2020-11-09 21:18:05.890575199] process [301218] done ExitSuccess -[2020-11-09 21:18:05.890901508] process [301219] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"commit-tree\",\"283356f53ffc1158753de3fa8f6ad1fd4ef944a1\",\"--no-gpg-sign\",\"-p\",\"refs/heads/git-annex\"] -[2020-11-09 21:18:05.892116156] process [301219] done ExitSuccess -[2020-11-09 21:18:05.892568401] process [301223] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-ref\",\"refs/heads/git-annex\",\"88db094ee3108764027124ff8528fa9b83886d15\"] -[2020-11-09 21:18:05.893479964] process [301223] done ExitSuccess -[2020-11-09 21:18:05.894556565] process [301198] done ExitSuccess -[2020-11-09 21:18:05.894814565] process [301199] done ExitSuccess -[2020-11-09 21:18:05.895064348] process [301214] done ExitSuccess - -# ~/Images/test on git:master o --> cp ../test.CR2 . - -# ~/Images/test on git:master x --> g annex add -d -[2020-11-09 21:18:37.761856427] process [301520] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"] -[2020-11-09 21:18:37.762517315] process [301520] done ExitSuccess -[2020-11-09 21:18:37.762730127] process [301521] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] -[2020-11-09 21:18:37.76341947] process [301521] done ExitSuccess -[2020-11-09 21:18:37.763862337] process [301522] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..88db094ee3108764027124ff8528fa9b83886d15\",\"--pretty=%H\",\"-n1\"] -[2020-11-09 21:18:37.764700249] process [301522] done ExitSuccess -[2020-11-09 21:18:37.765372445] process [301523] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"] -[2020-11-09 21:18:37.765628934] process [301524] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"] -[2020-11-09 21:18:37.766240345] process [301525] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"] -[2020-11-09 21:18:37.766883313] process [301525] done ExitSuccess -[2020-11-09 21:18:37.767219173] process [301526] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"] -[2020-11-09 21:18:37.767819241] process [301526] done ExitFailure 1 -[2020-11-09 21:18:37.768198125] process [301527] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"-z\",\"--others\",\"--exclude-standard\",\"--\"] -[2020-11-09 21:18:37.768913506] process [301528] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"check-attr\",\"-z\",\"--stdin\",\"annex.backend\",\"annex.numcopies\",\"annex.largefiles\",\"--\"] -add test.CR2 ok -[2020-11-09 21:18:37.800822834] process [301527] done ExitSuccess -[2020-11-09 21:18:37.801541467] process [301530] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"-z\",\"--modified\",\"--\"] -[2020-11-09 21:18:37.804737301] process [301530] done ExitSuccess -[2020-11-09 21:18:37.805576063] process [301531] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"diff\",\"--name-only\",\"--diff-filter=T\",\"-z\",\"--cached\",\"--\",\".\"] -[2020-11-09 21:18:37.809270202] process [301531] done ExitSuccess -(recording state in git...) -[2020-11-09 21:18:37.810761671] process [301532] feed: xargs [\"-0\",\"git\",\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"add\",\"--\"] -[2020-11-09 21:18:38.105427988] process [301532] done ExitSuccess -[2020-11-09 21:18:38.122781502] process [301523] done ExitSuccess -[2020-11-09 21:18:38.123382458] process [301524] done ExitSuccess -[2020-11-09 21:18:38.1240349] process [301528] done ExitSuccess - -# ~/Images/test on git:master x --> tree -a -. -├── .git -│   ├── annex -│   │   ├── gitqueue.lck -│   │   ├── index -│   │   ├── index.lck -│   │   ├── journal -│   │   ├── journal.lck -│   │   ├── keysdb -│   │   │   └── db -│   │   ├── keysdb.lck -│   │   ├── othertmp.lck -│   │   ├── sentinal -│   │   └── sentinal.cache -│   ├── branches -│   ├── config -│   ├── description -│   ├── HEAD -│   ├── hooks -│   │   ├── applypatch-msg.sample -│   │   ├── commit-msg.sample -│   │   ├── fsmonitor-watchman.sample -│   │   ├── post-checkout -│   │   ├── post-merge -│   │   ├── post-receive -│   │   ├── post-update.sample -│   │   ├── pre-applypatch.sample -│   │   ├── pre-commit -│   │   ├── pre-commit.sample -│   │   ├── pre-merge-commit.sample -│   │   ├── prepare-commit-msg.sample -│   │   ├── pre-push.sample -│   │   ├── pre-rebase.sample -│   │   ├── pre-receive.sample -│   │   └── update.sample -│   ├── index -│   ├── info -│   │   ├── attributes -│   │   └── exclude -│   ├── logs -│   │   └── refs -│   │   └── heads -│   │   └── git-annex -│   ├── objects -│   │   ├── 0d -│   │   │   └── b47ff2773fdd41f7be54839f6a3450c25da595 -│   │   ├── 28 -│   │   │   └── 3356f53ffc1158753de3fa8f6ad1fd4ef944a1 -│   │   ├── 4b -│   │   │   └── 825dc642cb6eb9a060e54bf8d69288fbee4904 -│   │   ├── 65 -│   │   │   └── 783ba972653cd323df432c5affe8d62263f81d -│   │   ├── 88 -│   │   │   └── db094ee3108764027124ff8528fa9b83886d15 -│   │   ├── cc -│   │   │   └── 29c590c2b3802aad89ba45d82275b378a8c843 -│   │   ├── info -│   │   └── pack -│   └── refs -│   ├── heads -│   │   └── git-annex -│   └── tags -└── test.CR2 - -22 directories, 40 files - -# ~/Images/test on git:master x --> git annex whereis test.CR2 -d -[2020-11-09 21:19:06.747007494] process [301865] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"] -[2020-11-09 21:19:06.747646855] process [301865] done ExitSuccess -[2020-11-09 21:19:06.747969742] process [301866] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] -[2020-11-09 21:19:06.748661535] process [301866] done ExitSuccess -[2020-11-09 21:19:06.749060672] process [301867] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..88db094ee3108764027124ff8528fa9b83886d15\",\"--pretty=%H\",\"-n1\"] -[2020-11-09 21:19:06.749842614] process [301867] done ExitSuccess -[2020-11-09 21:19:06.75045599] process [301868] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"] -[2020-11-09 21:19:06.750831336] process [301869] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"] -[2020-11-09 21:19:06.751532238] process [301870] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"] -[2020-11-09 21:19:06.752113929] process [301870] done ExitSuccess -[2020-11-09 21:19:06.752381211] process [301871] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"] -[2020-11-09 21:19:06.753053508] process [301871] done ExitFailure 1 -[2020-11-09 21:19:06.753377108] process [301872] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"--stage\",\"-z\",\"--\",\"test.CR2\"] -[2020-11-09 21:19:06.756788587] process [301873] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\",\"--buffer\"] -[2020-11-09 21:19:06.757087393] process [301874] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch=%(objectname) %(objecttype) %(objectsize)\",\"--buffer\"] -[2020-11-09 21:19:06.757493375] process [301875] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch=%(objectname) %(objecttype) %(objectsize)\",\"--buffer\"] -whereis test.CR2 [2020-11-09 21:19:06.758500321] process [301876] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"] -[2020-11-09 21:19:06.75874544] process [301877] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"] -(0 copies) failed -[2020-11-09 21:19:06.759864151] process [301876] done ExitSuccess -[2020-11-09 21:19:06.760118535] process [301877] done ExitSuccess -[2020-11-09 21:19:06.760181101] process [301875] done ExitSuccess -[2020-11-09 21:19:06.760299463] process [301874] done ExitSuccess -[2020-11-09 21:19:06.760395548] process [301873] done ExitSuccess -[2020-11-09 21:19:06.7604857] process [301872] done ExitSuccess -git-annex: whereis: 1 failed - -# ~/Images/test on git:master x C:1 --> git annex fsck -d -[2020-11-09 21:19:16.754635603] process [301990] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"git-annex\"] -[2020-11-09 21:19:16.755282306] process [301990] done ExitSuccess -[2020-11-09 21:19:16.755516429] process [301991] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] -[2020-11-09 21:19:16.756217949] process [301991] done ExitSuccess -[2020-11-09 21:19:16.756624385] process [301992] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"log\",\"refs/heads/git-annex..88db094ee3108764027124ff8528fa9b83886d15\",\"--pretty=%H\",\"-n1\"] -[2020-11-09 21:19:16.757371526] process [301992] done ExitSuccess -[2020-11-09 21:19:16.757964021] process [301993] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"] -[2020-11-09 21:19:16.758225471] process [301994] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"] -[2020-11-09 21:19:16.758986812] process [301995] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"symbolic-ref\",\"-q\",\"HEAD\"] -[2020-11-09 21:19:16.759699011] process [301995] done ExitSuccess -[2020-11-09 21:19:16.759969384] process [301996] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"refs/heads/master\"] -[2020-11-09 21:19:16.760566806] process [301996] done ExitFailure 1 -[2020-11-09 21:19:16.760952487] process [301997] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"ls-files\",\"--stage\",\"-z\",\"--\"] -[2020-11-09 21:19:16.761246486] process [301998] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\",\"--buffer\"] -[2020-11-09 21:19:16.764811773] process [301999] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch=%(objectname) %(objecttype) %(objectsize)\",\"--buffer\"] -[2020-11-09 21:19:16.765130473] process [302000] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch=%(objectname) %(objecttype) %(objectsize)\",\"--buffer\"] -[2020-11-09 21:19:16.76635019] process [302001] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"check-attr\",\"-z\",\"--stdin\",\"annex.backend\",\"annex.numcopies\",\"annex.largefiles\",\"--\"] -[2020-11-09 21:19:16.767203931] process [302002] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"] -[2020-11-09 21:19:16.767473428] process [302003] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"] -fsck test.CR2 [2020-11-09 21:19:16.768325918] process [302004] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"-c\",\"diff.external=\",\"-c\",\"filter.annex.smudge=\",\"-c\",\"filter.annex.clean=\",\"diff\",\"--cached\",\"--raw\",\"-z\",\"--no-abbrev\",\"-G^/annex/objects/\",\"--diff-filter=AMUT\",\"--no-renames\",\"--ignore-submodules=all\",\"--no-ext-diff\"] -[2020-11-09 21:19:16.7705705] process [302006] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"] -[2020-11-09 21:19:16.77093471] process [302007] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"] -[2020-11-09 21:19:16.771967978] process [302004] done ExitSuccess - - ** No known copies exist of test.CR2 -failed -[2020-11-09 21:19:16.774116848] process [302006] done ExitSuccess -[2020-11-09 21:19:16.774269964] process [302007] done ExitSuccess -[2020-11-09 21:19:16.77440891] process [302002] done ExitSuccess -[2020-11-09 21:19:16.774577166] process [302003] done ExitSuccess -[2020-11-09 21:19:16.774745401] process [302001] done ExitSuccess -[2020-11-09 21:19:16.774800806] process [302000] done ExitSuccess -[2020-11-09 21:19:16.77484124] process [301999] done ExitSuccess -[2020-11-09 21:19:16.774873496] process [301998] done ExitSuccess -[2020-11-09 21:19:16.774902674] process [301997] done ExitSuccess -[2020-11-09 21:19:16.775189502] process [302008] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch\"] -[2020-11-09 21:19:16.775473145] process [302009] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"cat-file\",\"--batch-check=%(objectname) %(objecttype) %(objectsize)\"] -[2020-11-09 21:19:16.776696194] process [302010] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"hash-object\",\"-w\",\"--stdin-paths\",\"--no-filters\"] -[2020-11-09 21:19:16.776905151] process [302011] feed: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-index\",\"-z\",\"--index-info\"] -[2020-11-09 21:19:16.777910232] process [302011] done ExitSuccess -[2020-11-09 21:19:16.77823167] process [302012] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] -[2020-11-09 21:19:16.778800722] process [302012] done ExitSuccess -(recording state in git...) -[2020-11-09 21:19:16.779227579] process [302013] read: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"write-tree\"] -[2020-11-09 21:19:16.780331627] process [302013] done ExitSuccess -[2020-11-09 21:19:16.780708844] process [302014] chat: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"commit-tree\",\"89a899c333c8b04da73517700f655a67b93323c6\",\"--no-gpg-sign\",\"-p\",\"refs/heads/git-annex\"] -[2020-11-09 21:19:16.78164598] process [302014] done ExitSuccess -[2020-11-09 21:19:16.782028199] process [302015] call: git [\"--git-dir=.git\",\"--work-tree=.\",\"--literal-pathspecs\",\"update-ref\",\"refs/heads/git-annex\",\"f46e7b12120b14413bcd62d76a20170ae146d9d0\"] -[2020-11-09 21:19:16.782930785] process [302015] done ExitSuccess -[2020-11-09 21:19:16.783900844] process [302008] done ExitSuccess -[2020-11-09 21:19:16.784127486] process [302009] done ExitSuccess -[2020-11-09 21:19:16.784363948] process [302010] done ExitSuccess -git-annex: fsck: 1 failed - -# ~/Images/test on git:master x C:1 --> git config --list | grep annex -annex.uuid=092eb39f-6944-437d-8085-3062827d114f -annex.version=8 -filter.annex.smudge=git-annex smudge -- %f -filter.annex.clean=git-annex smudge --clean -- %f -``` - -"""]] diff --git a/doc/bugs/file_not_correctly_added/comment_8_853b92286502fc5b45e42c9db213a9be._comment b/doc/bugs/file_not_correctly_added/comment_8_853b92286502fc5b45e42c9db213a9be._comment deleted file mode 100644 index 57b7fb0af1..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_8_853b92286502fc5b45e42c9db213a9be._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 8""" - date="2020-11-09T21:29:16Z" - content=""" -Perhaps these CR2 files somehow get incorrectly treated as annex link files -despite not being them? Can you reproduce it with some other file content, -eg - - echo foo > file - git annex add file -"""]] diff --git a/doc/bugs/file_not_correctly_added/comment_9_3b106b9a079e56d2621207432eb41f4e._comment b/doc/bugs/file_not_correctly_added/comment_9_3b106b9a079e56d2621207432eb41f4e._comment deleted file mode 100644 index ddf1ab1446..0000000000 --- a/doc/bugs/file_not_correctly_added/comment_9_3b106b9a079e56d2621207432eb41f4e._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="jcjgraf" - avatar="http://cdn.libravatar.org/avatar/9dda752f83ac44906fefbadb35e8a6ac" - subject="comment 9" - date="2020-11-10T10:17:12Z" - content=""" -I do not know that I did wrong when I tried a different CR2 yesterday, but your suggestion was correct! The `myLargeFile.CR2` is indeed faulty and all other files (CR2s as well as of different types) work flawlessly. - -I should have noticed that myself by trying to open that file in an image viewer. Thank you very much for your efforts and help! -"""]] diff --git a/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported.mdwn b/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported.mdwn deleted file mode 100644 index cba319665d..0000000000 --- a/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported.mdwn +++ /dev/null @@ -1,20 +0,0 @@ -### Please describe the problem. - -The file `Kontoauszug 01.01.2019 - 31.12.2019 - CH1234567890123456789 - 2019-12-31.pdf` can not be exported. The error message is just "failed". - -I added code (pull request underway) to log IOExceptions and saw this exception message: - - /media/thk/thk-sg2/annex-exporttree-sg2/shared/finance/taxes/2019/: openTempFile: invalid argument (Invalid argument) - -After renaming the file to `Kontoauszug 01012019 - 31122019 - CH2480808009412070498 - 2019-12-31.pdf` it could be exported. - -I am not yet sure what exact combinations of dots, spaces and maybe dashes cause the failure. - -### What version of git-annex are you using? On what operating system? - -git-annex version: 8.20200309-05df404212, Debian testing - -[[!meta title="change exception handling of remotes to avoid ever failing -without telling the reason why"]] - -> [[done]] comprehensively --[[Joey]] diff --git a/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_1_b1ac799adea59dad8fe148183060fb22._comment b/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_1_b1ac799adea59dad8fe148183060fb22._comment deleted file mode 100644 index 19da541f02..0000000000 --- a/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_1_b1ac799adea59dad8fe148183060fb22._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-14T19:44:48Z" - content=""" -This seems likely to invole the specific filesystem used for -/media/thk/thk-sg2/. I woud not be surprised if eg FAT had limitations on -filenames with dots. - -Generally I think that if the remote being exported to cannot support the -filename, the export should fail. Mangling the filenames should be up to -the user. - -But there should certianly be a better error message than "failed" in this -case, so that at least needs to be fixed. -"""]] diff --git a/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_2_ebd37f03986ca7e618043f0546260503._comment b/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_2_ebd37f03986ca7e618043f0546260503._comment deleted file mode 100644 index 521450c094..0000000000 --- a/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_2_ebd37f03986ca7e618043f0546260503._comment +++ /dev/null @@ -1,35 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-05-12T17:36:17Z" - content=""" -This is one of a larger pattern of exceptions in remote operations -not being displayed. It should be deal with broadly, not only this case. - -I looked into two approaches.. - -1. Change the remote interface to not return False, but always throw - exceptions on failure. Then each thing using the interface can - catch and display exceptions. - Pro: Catch and display is more centralized, a single remote can't forget - to catch some exception. - Con: What failed might be a command, which displays its own error - message. Then an exception, eg "command foo failed" would still need - to be thrown and displayed. (A workaround would be to add a new type of - exception, to be thrown in this situation, that does not get displayed.) - -2. Add functions like catchBoolIOAndWarn and remote modify code to use them, - so when exceptions are caught, they're displayed. - Pro: Avoids above con. Easy to change a little at a time, easy to - check with grep that a remote is using that rather than catchBoolIO. - Con: Sometimes an exception will be caught for a reason that should - not result in a warning, so a remote will need to keep using catchBoolIO - there. That's ok when the exception is something the remote deals with - itself. OTOH, some actions like renameExport should probably not display - a warning (because when that fails, it falls back to a delete and - re-upload so it's not a hard failure), and which are which may not be - clear. - -I'm leaning toward #1, but without the added type of exception -probably unless that does turn out to be worth doing in some case. -"""]] diff --git a/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_3_d386ca19d803fc6d1c720d0188f00fec._comment b/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_3_d386ca19d803fc6d1c720d0188f00fec._comment deleted file mode 100644 index e344d37e2b..0000000000 --- a/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_3_d386ca19d803fc6d1c720d0188f00fec._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-05-13T17:58:49Z" - content=""" -Converted the storeKey method to throw exceptions. This was a 1000 line -patch, 3 hours of work. Seems likely it will take 24 hours work to finish -converting all the methods.. - -There were quite a few places where it used to return False without -displaying a reason for the failure, so the work seems worth it. -"""]] diff --git a/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_4_6889c0a21026038431dbe874711f0fe5._comment b/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_4_6889c0a21026038431dbe874711f0fe5._comment deleted file mode 100644 index c67ee4767e..0000000000 --- a/doc/bugs/filenames_with_dots_and_spaces_can_not_be_exported/comment_4_6889c0a21026038431dbe874711f0fe5._comment +++ /dev/null @@ -1,7 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-05-15T18:54:23Z" - content=""" -Done with converting all the methods. -"""]] diff --git a/doc/bugs/fsck_tells___39__ok__39___also_if_no_file_present/comment_1_ddde7370b75814987b5dd1bc7d416cf7._comment b/doc/bugs/fsck_tells___39__ok__39___also_if_no_file_present/comment_1_ddde7370b75814987b5dd1bc7d416cf7._comment deleted file mode 100644 index 22ce7a4c23..0000000000 --- a/doc/bugs/fsck_tells___39__ok__39___also_if_no_file_present/comment_1_ddde7370b75814987b5dd1bc7d416cf7._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-04-27T16:02:29Z" - content=""" -fsck finds problems with the repository. - -A file not being present in the local repository is not a problem, if the -file is known to be present in some other repository. - -The ability to have some files not present in the local repository, but -retreivable by `git annex get` when you need them, is one of the main -features of git-annex, since it lets you manage quantities of data larger -than you have space for on your computer. - -If you want to know what files are not present in your local repository, -there is a way to do that: `git annex find --not --in here` -"""]] diff --git a/doc/bugs/gcrypt__58___..but_repository_ID_is_set._Aborting.mdwn b/doc/bugs/gcrypt__58___..but_repository_ID_is_set._Aborting.mdwn deleted file mode 100644 index bd6bc505ad..0000000000 --- a/doc/bugs/gcrypt__58___..but_repository_ID_is_set._Aborting.mdwn +++ /dev/null @@ -1,47 +0,0 @@ -### Please describe the problem. -After month of flawless operations, a gcrypted special (rsync) remote has suddenly stopped working. Below is what since happens on every git annex sync. - - ... - push rsync.net - gcrypt: Repository not found: ssh://usw-a123.rsync.net/data1/home/XXXX/annex/files - gcrypt: ..but repository ID is set. Aborting. - -At the file system level, the remote appears to be intact. - - foo@bar:~$ ssh usw-a123.rsync.net ls -lah annex/files/ - total 175 - drwxr-xr-x 8 XXXX XXXX 11B Sep 8 17:23 . - drwxr-xr-x 3 XXXX XXXX 3B Mar 25 11:42 .. - -rw-r--r-- 1 XXXX XXXX 23B Mar 22 18:25 HEAD - drwxr-xr-x 3 XXXX XXXX 3B Mar 22 18:26 annex - drwxr-xr-x 2 XXXX XXXX 2B Mar 22 18:25 branches - -rw-r--r-- 1 XXXX XXXX 143B Mar 30 10:12 config - -rw-r--r-- 1 XXXX XXXX 73B Mar 22 18:25 description - drwxr-xr-x 2 XXXX XXXX 12B Mar 22 18:25 hooks - drwxr-xr-x 2 XXXX XXXX 3B Mar 22 18:25 info - drwxr-xr-x 255 XXXX XXXX 255B Sep 8 04:56 objects - drwxr-xr-x 4 XXXX XXXX 4B Mar 22 18:25 refs - - -### What steps will reproduce the problem? -Unknown - -### What version of git-annex are you using? On what operating system? -* git-annex-standalone 6.20180807+git230-gaa291acfe-1~ndall+1 amd64 (Neurodebian) -* Ubuntu 18.04 - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I am a vocal advocate of git-annex. :-) - -> [[done]] diff --git a/doc/bugs/gcrypt__58___..but_repository_ID_is_set._Aborting/comment_1_ec28718db57922b7d0c01f3ef672e6d4._comment b/doc/bugs/gcrypt__58___..but_repository_ID_is_set._Aborting/comment_1_ec28718db57922b7d0c01f3ef672e6d4._comment deleted file mode 100644 index de25721ddc..0000000000 --- a/doc/bugs/gcrypt__58___..but_repository_ID_is_set._Aborting/comment_1_ec28718db57922b7d0c01f3ef672e6d4._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="git-annex.branchable.com@79d6855760f61f7fbe0a401b45d8c791ef49b500" - nickname="git-annex.branchable.com" - avatar="http://cdn.libravatar.org/avatar/4bf61f9feda20e8b4fc09d52ee48af39" - subject="comment 1" - date="2018-09-14T16:42:59Z" - content=""" -It turns out that rsync.net silently changed the (absolute) path to my user's home directory and the path in .git/config was thus incorrect. -"""]] diff --git a/doc/bugs/gcrypt__58___..but_repository_ID_is_set._Aborting/comment_2_e4d97ac757e7da09c9c23f74177991ae._comment b/doc/bugs/gcrypt__58___..but_repository_ID_is_set._Aborting/comment_2_e4d97ac757e7da09c9c23f74177991ae._comment deleted file mode 100644 index c103ce3842..0000000000 --- a/doc/bugs/gcrypt__58___..but_repository_ID_is_set._Aborting/comment_2_e4d97ac757e7da09c9c23f74177991ae._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2018-09-14T16:52:56Z" - content=""" -This is gcrypt displaying an error message, not git-annex. - -Do you have any indication it's somehow a git-annex bug? The obvious guess -would be that it's a gcrypt bug. -"""]] diff --git a/doc/bugs/gcrypt_special_remote_not_working.mdwn b/doc/bugs/gcrypt_special_remote_not_working.mdwn deleted file mode 100644 index d8d6d86055..0000000000 --- a/doc/bugs/gcrypt_special_remote_not_working.mdwn +++ /dev/null @@ -1,53 +0,0 @@ -[[!meta title="regression: encryption not used for gcrypt and git-lfs remotes"]] - -### Please describe the problem. - -Adding a special gcrypt remote with encryption set to hybrid does not work as expected. -- files sent to this remote are not encrypted -- git annex info does not report that encryption is activated. - -### What steps will reproduce the problem? - -I first saw this on a special remote that I frequently use. But creating a fresh git annex repo and a fresh gcrypt repository reproduces the problem. - -### What version of git-annex are you using? On what operating system? - -The version of git-annex is 7.20200219-gcd8a208b8 on ArchLinux. I use the packaged version of my distribution. -With version 7.20191230 (recompiled using ArchLinux ABS system) problem is not present. - -### Please provide any additional information below. - -[[!format sh """ -$ git init annex -$ git init --bare crypt -$ cd annex -$ git annex init -$ echo "test file" > test -$ git annex initremote crypt type=gcrypt encryption=hybrid keyid=XXX gitrepo=../crypt -$ git annex add test -$ git annex sync --content -# file is in clear in the annex -$ cat ../crypt/annex/objects/*/*/*/* -test file -# git annex info reports encryption: noneNothing and hybrid is not part of the accepted choices ? -$ git annex info crypt -uuid: 481d596b-489b-53ae-b326-035561b5d915 -description: [crypt] -trust: semitrusted -remote: crypt -cost: 110.0 -type: gcrypt -repository location: /home/XXX/crypt -last synced: 2020-02-26 10:12:32 UTC -encryption: noneNothing[Accepted "autoenable",Accepted "exporttree",Accepted "importtree",Accepted "name",Accepted "type"] -chunking: none -remote annex keys: 1 -remote annex size: 10 bytes - -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I am using git-annex since 2014 for all kinds of situations and I am very grateful for all the work you've put into this software. It even gets better over time ;) - -> [[fixed|done]] in git-annex 7.20200226. --[[Joey]] diff --git a/doc/bugs/gcrypt_special_remote_not_working/comment_1_7eadb5e4cdebed6bffee5182b4bfe87b._comment b/doc/bugs/gcrypt_special_remote_not_working/comment_1_7eadb5e4cdebed6bffee5182b4bfe87b._comment deleted file mode 100644 index 2bd1879e07..0000000000 --- a/doc/bugs/gcrypt_special_remote_not_working/comment_1_7eadb5e4cdebed6bffee5182b4bfe87b._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-02-26T19:04:47Z" - content=""" -Reversion caused by the special -remote config overhaul in version 7.20200202.7. - - -This is really bad. The one good thing is it does not -seem to affect other types of special remotes, including -ones that also use encryption=hybrid. Only gcrypt. -"""]] diff --git a/doc/bugs/gcrypt_special_remote_not_working/comment_2_9f75b01a1283784fb10d74c7f4c7d54e._comment b/doc/bugs/gcrypt_special_remote_not_working/comment_2_9f75b01a1283784fb10d74c7f4c7d54e._comment deleted file mode 100644 index b628473e6f..0000000000 --- a/doc/bugs/gcrypt_special_remote_not_working/comment_2_9f75b01a1283784fb10d74c7f4c7d54e._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-02-26T19:18:09Z" - content=""" -encrypted git-lfs remotes are also affected. - -The cause is that Remote.Git is used for all remotes with an url, which -both of these have, and internally dispatches to generate them. But that -means that the configParser from Remote.Git gets used for them, and it does -not include the encryption fields. -"""]] diff --git a/doc/bugs/gcrypt_special_remote_not_working/comment_3_cc348093cb5c7626930542379a82a729._comment b/doc/bugs/gcrypt_special_remote_not_working/comment_3_cc348093cb5c7626930542379a82a729._comment deleted file mode 100644 index 5b39a6eabe..0000000000 --- a/doc/bugs/gcrypt_special_remote_not_working/comment_3_cc348093cb5c7626930542379a82a729._comment +++ /dev/null @@ -1,35 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-02-26T22:22:38Z" - content=""" -I'm releasing a fix now. - -It didn't seem practical to make git-annex automatically clean up any -encrypted data that got stored by mistake. For one thing, this leakage may -have consequences that git-annex users will need to work through on their -own. (sigh) Also, I just didn't see a good way to make it automatically -delete it. - -So, the release announcement just has this advice for how to recover: - - If your remotes are affected, you will want to make sure to delete - any content that git-annex has stored on them that is not encrypted! - - One way to do so is, before upgrading to this version, - run git-annex move --from the affected remotes. It will move - only the content that was not encrypted. - -I think that will be sufficient for most. It's also possible -that some may need to run `git annex fsck --fast --from` on affected -remotes, if git-annex got confused about encrypted content that was stored -on them not being present, because it wrongly was looking for non-encrypted -content. I think that would only be necessary if a similar fsck was -run earlier using the broken git-annex version. - -If you have encountered this problem and need help with recovery, post -a comment. - -Very sorry about this mess. [[todo/network_test_suite]] is needed to -be sure such problems are caught. -"""]] diff --git a/doc/bugs/gcrypt_special_remote_not_working/comment_4_2fe8b29a06bac1e004f8fa2109a27fab._comment b/doc/bugs/gcrypt_special_remote_not_working/comment_4_2fe8b29a06bac1e004f8fa2109a27fab._comment deleted file mode 100644 index bac6484251..0000000000 --- a/doc/bugs/gcrypt_special_remote_not_working/comment_4_2fe8b29a06bac1e004f8fa2109a27fab._comment +++ /dev/null @@ -1,21 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-02-26T22:34:35Z" - content=""" -Hmm, one idea is that git-annex fsck --from remote, when the remote is -encrypted, could check if the non-encrypted key is present. If so, it -could warn loudly. - -(I don't think it could delete it safely; the user would still have to do -that. Deleting it might violate numcopies and lead to data loss. It -could move it to the local repo, but that risks fsck filling up the local -repo, and also it's not guaranteed that the local repo is a place the user -expects to continue storing data; it could even be a throwaway repo they're -going to nuke after the fsck.) - -This does make a slow fsck path even slower, which is a bit of a heavy -price to pay going forward. Although the belt-and-suspenders nature -of checking that git-annex has not forgotten to encrypt some data may be -worth it. -"""]] diff --git a/doc/bugs/get_behavior_when_location_log_is_out_of_date.mdwn b/doc/bugs/get_behavior_when_location_log_is_out_of_date.mdwn deleted file mode 100644 index 52c2d83b5b..0000000000 --- a/doc/bugs/get_behavior_when_location_log_is_out_of_date.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -When location log says a remote contains a file, -but it is out of date, a get --from that remote will silently fail to do -anything. - -(This happened with a git remote on a drive fwiw). - ---[[Joey]] - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git-annex-copy_--json-error-messages_does_not_write_error_messages_to_stderr_.mdwn b/doc/bugs/git-annex-copy_--json-error-messages_does_not_write_error_messages_to_stderr_.mdwn deleted file mode 100644 index ace80411c7..0000000000 --- a/doc/bugs/git-annex-copy_--json-error-messages_does_not_write_error_messages_to_stderr_.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -### Please describe the problem. - -`git-annex-copy --json --json-error-messages` writes errors messages to stdout, but not to stderr. This can be confusing in that the stderr output is empty, even though there were errors. It would be more intuitive to (also) output error messages as json records to stderr. - -Also, the error messages that *are* written to stderr are not always in json format, e.g. the `git-annex: thread blocked indefinitely in an STM transaction` messages in [[bugs/git-annex-copy_fails_with___34__thread_blocked_indefinitely_in_an_STM_transaction__34__]]. Adding `--debug --verbose` also causes some non-json output. - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -git-annex has been an extremely flexible and versatile tool. So far I've been able to adapt to most usage scenarios I've encountered. - -> [[wontfix|done]] --[[Joey]] diff --git a/doc/bugs/git-annex-copy_--json-error-messages_does_not_write_error_messages_to_stderr_/comment_1_bb4ebeaa11949d2e2d8ede7e5e67f27d._comment b/doc/bugs/git-annex-copy_--json-error-messages_does_not_write_error_messages_to_stderr_/comment_1_bb4ebeaa11949d2e2d8ede7e5e67f27d._comment deleted file mode 100644 index 8d805f3641..0000000000 --- a/doc/bugs/git-annex-copy_--json-error-messages_does_not_write_error_messages_to_stderr_/comment_1_bb4ebeaa11949d2e2d8ede7e5e67f27d._comment +++ /dev/null @@ -1,25 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-09-16T17:27:12Z" - content=""" -The documentation seems to clearly document the current behavior: - -* `--json-error-messages` - - Messages that would normally be output to standard error are included in - the json instead. - -And changing the behavior could break something that relies on the current -behavior. - -This was considered when the interface was designed, and there were -numerous good reasons to choose the current behavior. -[[See thread|todo/include_msg_with_possible_reason_why_command___40__e.g._add__41___failed_into_--json_output]]. - -(Another reason not mentioned there is, it's simply not possible to trap -every possible error message that git-annex could output in any circumstance. -If there's a segfault or a memory error, or libc crashes, or the program fails -to link. So any program parsing stderr for json would somehow have to deal -with this non-json that could be mixed up with it, which seems very hard.) -"""]] diff --git a/doc/bugs/git-annex-fsck_fails___34__thread_blocked_indefinitely_in_an_STM_transaction__34__.mdwn b/doc/bugs/git-annex-fsck_fails___34__thread_blocked_indefinitely_in_an_STM_transaction__34__.mdwn deleted file mode 100644 index d341df040a..0000000000 --- a/doc/bugs/git-annex-fsck_fails___34__thread_blocked_indefinitely_in_an_STM_transaction__34__.mdwn +++ /dev/null @@ -1,50 +0,0 @@ -### Please describe the problem. - -Running git-annex-fsck fails with the error "thread blocked indefinitely in an STM transaction" - -### What steps will reproduce the problem? - - -### What version of git-annex are you using? On what operating system? -git-annex version: 7.20190708-gbc74eb1 -Amazon Linux 2 ( Linux ip-172-31-87-151.ec2.internal 4.14.133-113.105.amzn2.x86_64 #1 SMP Wed Jul 10 16:57:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux ) - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log -git annex fsck --fast -J66 --json --json-error-messages - -... -git-annex: thread blocked indefinitely in an STM transaction - - - -(master_env_v149_py36) 11:56 [viral-ngs-benchmarks] $ git annex version -git-annex version: 7.20190708-gbc74eb1 -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.4.2 http-client-0.5.14 persistent-sql\ -ite-2.9.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_2\ -24 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE\ -2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256\ - BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external -operating system: linux x86_64 -supported repository versions: 5 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -local repository version: 5 -(master_env_v149_py36) 12:00 [viral-ngs-benchmarks] $ uname -a -Linux ip-172-31-87-151.ec2.internal 4.14.133-113.105.amzn2.x86_64 #1 SMP Wed Jul 10 16:57:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux -( - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - - -> This is apparently the same problem as -> [[git-annex-copy_fails_with___34__thread_blocked_indefinitely_in_an_STM_transaction__34__]] -> which was filed by the same person, so merging. [[done]]. --[[Joey]] diff --git a/doc/bugs/git-annex-fsck_fails___34__thread_blocked_indefinitely_in_an_STM_transaction__34__/comment_1_86e93a606e86637f6ebf2c13f19b3151._comment b/doc/bugs/git-annex-fsck_fails___34__thread_blocked_indefinitely_in_an_STM_transaction__34__/comment_1_86e93a606e86637f6ebf2c13f19b3151._comment deleted file mode 100644 index df1e7c6d87..0000000000 --- a/doc/bugs/git-annex-fsck_fails___34__thread_blocked_indefinitely_in_an_STM_transaction__34__/comment_1_86e93a606e86637f6ebf2c13f19b3151._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 1" - date="2019-07-23T20:57:11Z" - content=""" -P.S. Re-running with 20 threads, the error doesn't seem to recur. -"""]] diff --git a/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors.mdwn b/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors.mdwn deleted file mode 100644 index 9ae5f7899e..0000000000 --- a/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors.mdwn +++ /dev/null @@ -1,54 +0,0 @@ -### Please describe the problem. - -[[git-annex-fsck]] flags as an error a key that has zero copies, even after the key has been marked dead with [[git-annex-dead]] . - -### What steps will reproduce the problem? - -see below - -### What version of git-annex are you using? On what operating system? -[[!format sh """ -(master_env_v149_py36) 16:48 [viral-ngs-benchmarks] $ git annex version -git-annex version: 7.20190708-gbc74eb1 -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.4.2 http-client-0.5.14 persistent-sql\ -ite-2.9.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_2\ -24 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE\ -2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256\ - BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external -operating system: linux x86_64 -supported repository versions: 5 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -local repository version: 5 -(master_env_v149_py36) 16:51 [viral-ngs-benchmarks] $ uname -a -Linux ip-172-31-87-151.ec2.internal 4.14.133-113.105.amzn2.x86_64 #1 SMP Wed Jul 10 16:57:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux -(master_env_v149_py36) 16:51 [viral-ngs-benchmarks] $ -"""]] - -### Please provide any additional information below. - -[[!format sh """ - -(master_env_v149_py36) 16:48 [viral-ngs-benchmarks] $ git annex dead --key MD5E-s363253--e5190755e0ded033bff6b5184adc2fb2.pkl.gz -dead MD5E-s363253--e5190755e0ded033bff6b5184adc2fb2.pkl.gz ok -(master_env_v149_py36) 16:48 [viral-ngs-benchmarks] $ echo $? -0 -(master_env_v149_py36) 16:48 [viral-ngs-benchmarks] $ git annex fsck --key MD5E-s363253--e5190755e0ded033bff6b5184adc2fb2.pkl.gz -fsck MD5E-s363253--e5190755e0ded033bff6b5184adc2fb2.pkl.gz - ** No known copies exist of MD5E-s363253--e5190755e0ded033bff6b5184adc2fb2.pkl.gz - These dead repositories used to have copies - fb0aac49-5cdd-41ce-9d3b-101d9a91d623 -- crogit runner crogrun_190718_135857_6024 - - (Avoid this check by running: git annex dead --key ) -failed -(recording state in git...) -git-annex: fsck: 1 failed - -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -git-annex helps me keep my sanity :) - -> [[notabug|done]] per comments --[[Joey]] diff --git a/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_1_cee96c6f5d5d97f6816b9c3524822327._comment b/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_1_cee96c6f5d5d97f6816b9c3524822327._comment deleted file mode 100644 index ac9c790184..0000000000 --- a/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_1_cee96c6f5d5d97f6816b9c3524822327._comment +++ /dev/null @@ -1,43 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="another example of git-annex-fsck reporting dead keys as errors" - date="2020-02-27T21:39:57Z" - content=""" -[[!format sh \"\"\" -(master_env_v174_py36) 16:36 [viral-ngs-benchmarks] $ git annex fsck wdl/quay.io-broadinstitute-viral-ngs-dev_1.25.0-11-g6fe46aa-is-f\ -ix-gatk-mindepth_sha256_20bbb1d96cf58235d325a4ea2def8c3d840d9c2ab5dddf573f9471c57f7414f1/assemble_denovo/tasks_assembly.wdl -fsck wdl/quay.io-broadinstitute-viral-ngs-dev_1.25.0-11-g6fe46aa-is-fix-gatk-mindepth_sha256_20bbb1d96cf58235d325a4ea2def8c3d840d9c2ab\ -5dddf573f9471c57f7414f1/assemble_denovo/tasks_assembly.wdl - ** Required content wdl/quay.io-broadinstitute-viral-ngs-dev_1.25.0-11-g6fe46aa-is-fix-gatk-mindepth_sha256_20bbb1d96cf58235d325a4ea\ -2def8c3d840d9c2ab5dddf573f9471c57f7414f1/assemble_denovo/tasks_assembly.wdl is missing from these repositories: - 449efa47-a0e1-4376-a17f-42c7a1f509d1 -- Benchmarks for viral-ngs, stored on Amazon S3 [viral-ngs-benchmarks-s3] - - ** No known copies exist of wdl/quay.io-broadinstitute-viral-ngs-dev_1.25.0-11-g6fe46aa-is-fix-gatk-mindepth_sha256_20bbb1d96cf58235\ -d325a4ea2def8c3d840d9c2ab5dddf573f9471c57f7414f1/assemble_denovo/tasks_assembly.wdl - These dead repositories used to have copies - 461e3f01-1ff4-43d6-8dfa-71b839a25881 -- crogit runner crogrun_200207_174047__8035____git_9cb32e7230b -failed -(recording state in git...) -git-annex: fsck: 1 failed -(master_env_v174_py36) 16:36 [viral-ngs-benchmarks] $ git annex dead --key `git annex lookupkey wdl/quay.io-broadinstitute-viral-ngs-\ -dev_1.25.0-11-g6fe46aa-is-fix-gatk-mindepth_sha256_20bbb1d96cf58235d325a4ea2def8c3d840d9c2ab5dddf573f9471c57f7414f1/assemble_denovo/ta\ -sks_assembly.wdl` -dead MD5E-s15002--4f40b2c4a2ba81128db649e6096b5fec.wdl ok -(master_env_v174_py36) 16:37 [viral-ngs-benchmarks] $ git annex fsck wdl/quay.io-broadinstitute-viral-ngs-dev_1.25.0-11-g6fe46aa-is-f\ -ix-gatk-mindepth_sha256_20bbb1d96cf58235d325a4ea2def8c3d840d9c2ab5dddf573f9471c57f7414f1/assemble_denovo/tasks_assembly.wdl -fsck wdl/quay.io-broadinstitute-viral-ngs-dev_1.25.0-11-g6fe46aa-is-fix-gatk-mindepth_sha256_20bbb1d96cf58235d325a4ea2def8c3d840d9c2ab\ -5dddf573f9471c57f7414f1/assemble_denovo/tasks_assembly.wdl - ** Required content wdl/quay.io-broadinstitute-viral-ngs-dev_1.25.0-11-g6fe46aa-is-fix-gatk-mindepth_sha256_20bbb1d96cf58235d325a4ea\ -2def8c3d840d9c2ab5dddf573f9471c57f7414f1/assemble_denovo/tasks_assembly.wdl is missing from these repositories: - 449efa47-a0e1-4376-a17f-42c7a1f509d1 -- Benchmarks for viral-ngs, stored on Amazon S3 [viral-ngs-benchmarks-s3] - - ** No known copies exist of wdl/quay.io-broadinstitute-viral-ngs-dev_1.25.0-11-g6fe46aa-is-fix-gatk-mindepth_sha256_20bbb1d96cf58235\ -d325a4ea2def8c3d840d9c2ab5dddf573f9471c57f7414f1/assemble_denovo/tasks_assembly.wdl - These dead repositories used to have copies - 461e3f01-1ff4-43d6-8dfa-71b839a25881 -- crogit runner crogrun_200207_174047__8035____git_9cb32e7230b -failed -(recording state in git...) -git-annex: fsck: 1 failed -\"\"\"]] -"""]] diff --git a/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_2_73125b365e185c8cb814110b9a61f503._comment b/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_2_73125b365e185c8cb814110b9a61f503._comment deleted file mode 100644 index 1c82d707df..0000000000 --- a/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_2_73125b365e185c8cb814110b9a61f503._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-02-28T17:23:52Z" - content=""" -Seems to me that having a file in your working tree whose content is -missing is something that it's reasonable for fsck to mention as a problem. - - When a key is specified, indicates that the content of that - key has been irretrievably lost. This prevents commands like - git annex fsck --all from complaining about it; --all will - not operate on the key anymore. - -That's the intention of marking a key as dead, to avoid a fsck --all -complaining about some long-ago loss that is no longer relevant but left -traces in the repository. - -So I don't see a bug here. -"""]] diff --git a/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_3_99f2e030519d4fd6351fabb20c0c5f10._comment b/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_3_99f2e030519d4fd6351fabb20c0c5f10._comment deleted file mode 100644 index 12d3f64947..0000000000 --- a/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_3_99f2e030519d4fd6351fabb20c0c5f10._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="fsck of dead keys" - date="2020-02-28T18:44:21Z" - content=""" -\"having a file in your working tree whose content is missing is something that it's reasonable for fsck to mention as a problem\" -- makes sense. For [[`git-annex-fsck --key`|git-annex-fsck]], not sure you need to warn about an explicitly [[`git-annex-dead`|git-annex-dead]]'ed key, but I guess that depends on the context in which the command is used. -"""]] diff --git a/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_4_c5f902c1c5fdb1011577b9d5edde6b45._comment b/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_4_c5f902c1c5fdb1011577b9d5edde6b45._comment deleted file mode 100644 index f77fb1447a..0000000000 --- a/doc/bugs/git-annex-fsck_reports_dead_keys_as_errors/comment_4_c5f902c1c5fdb1011577b9d5edde6b45._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-03-02T18:54:05Z" - content=""" -As yes, I missed that you also had an example of fsck --key as well as fsck -of a file in the working tree with a dead key. - -I think the same basic reasoning applies for fsck --key as for fscking a -worktree file. The user appears to be particularly interested in that key, -so fsck should tell them about the obious problem that it has no remaining -content. It's not like --all. -"""]] diff --git a/doc/bugs/git-annex__58___Unexpected_parameters__58___exporttree.mdwn b/doc/bugs/git-annex__58___Unexpected_parameters__58___exporttree.mdwn deleted file mode 100644 index 6c26d4e737..0000000000 --- a/doc/bugs/git-annex__58___Unexpected_parameters__58___exporttree.mdwn +++ /dev/null @@ -1,53 +0,0 @@ -### Please describe the problem. - -I am trying to create a new remote on a S3 server, the exporttree option seems not to work anymore. - - -``` -$ git annex initremote -d s3.bucket type=S3 encryption=none exporttree=no autoenable=true host=hostname.com port=443 protocol=https chunk=1GiB bucket=bucket_xxx requeststyle=path - -[2020-02-28 15:22:26.093] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] -[2020-02-28 15:22:26.186713] process done ExitSuccess -[2020-02-28 15:22:26.187001] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] -[2020-02-28 15:22:26.210825] process done ExitSuccess -[2020-02-28 15:22:26.22362] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..f67678abc873eca16b44dc7f882d3c14fb64fa42","--pretty=%H","-n1"] -[2020-02-28 15:22:26.240175] process done ExitSuccess -[2020-02-28 15:22:26.240441] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..771bb68d6cb2e0a554e2e141979b33684e288a38","--pretty=%H","-n1"] -[2020-02-28 15:22:26.251669] process done ExitSuccess -[2020-02-28 15:22:26.254536] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"] -[2020-02-28 15:22:26.256351] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] -initremote bucket_xxx -git-annex: Unexpected parameters: exporttree -failed -[2020-02-28 15:22:26.307771] process done ExitSuccess -[2020-02-28 15:22:26.308812] process done ExitSuccess -git-annex: initremote: 1 failed - -``` - - -### What steps will reproduce the problem? - -see above. - -### What version of git-annex are you using? On what operating system? - -Ubuntu18.04LTS - -``` -git-annex version: 7.20200219-g5094a3f -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: linux x86_64 -supported repository versions: 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -local repository version: 7 -``` - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Yes! - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git-annex__58___Unexpected_parameters__58___exporttree/comment_1_059d50419385a16fbddc96d85debb439._comment b/doc/bugs/git-annex__58___Unexpected_parameters__58___exporttree/comment_1_059d50419385a16fbddc96d85debb439._comment deleted file mode 100644 index 1c14639154..0000000000 --- a/doc/bugs/git-annex__58___Unexpected_parameters__58___exporttree/comment_1_059d50419385a16fbddc96d85debb439._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-02T17:20:36Z" - content=""" -This was already fixed in git-annex 7.20200226, it turns out. -Some changes required to deal with a related reversion fixed it in passing. - -(Note that exporttree=no is the default, so there should -be no need to specify it.) -"""]] diff --git a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order.mdwn b/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order.mdwn deleted file mode 100644 index 1e484f020d..0000000000 --- a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order.mdwn +++ /dev/null @@ -1,37 +0,0 @@ -### Please describe the problem. -When building the openSUSE package of git-annex 8.20200522 or 8.20200617, every build has a different git-annex binary as result. -See https://reproducible-builds.org/ for why this matters. - -An underlying issue may be that Assistant/WebApp/Types.o varies in its ordering of various symbols (as seen through objdump -d): - - -0000000000008ba8 : - +0000000000008ba8 : - -0000000000008cf8 : - +0000000000008cf8 : - -000000000003a7c8 : - +000000000003a7c8 : - -I found, the build becomes reproducible, when using a filesystem with deterministic readdir order such as disorderfs with sort mode. - -https://github.com/bmwiedemann/openSUSE/blob/master/packages/g/git-annex/git-annex.spec#L173 just calls ./Setup build -v -that in turn calls ghc that calls cc and as with a temporary .s file - -### What steps will reproduce the problem? - - osc checkout openSUSE:Factory/git-annex && cd $_ - osc build --vm-type=kvm --noservice - -### What version of git-annex are you using? On what operating system? -8.20200522 on openSUSE-Tumbleweed-20200628 - -### Please provide any additional information below. - -https://rb.zq1.de/compare.factory/git-annex-compare.out has output from our build-compare tool - might not be very readable here. - -I also saw order variations in Types.o around strings that included "activityicon.gif". - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Sorry, not yet a user. Just looking into the openSUSE packaging. - -> Fixed in yesod-static 1.6.1.0. [[done]] --[[Joey]] diff --git a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_1_33571f940f88ebb3ff9cf08ef58364fb._comment b/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_1_33571f940f88ebb3ff9cf08ef58364fb._comment deleted file mode 100644 index 7dd02cf4ca..0000000000 --- a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_1_33571f940f88ebb3ff9cf08ef58364fb._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-06-30T14:10:23Z" - content=""" -ghc builds with concurrency enabled by default, which is bad for -reproducibility. Passing -j1 may help. - - - -As this is a compiler issue, there's really very little I can do about it. -"""]] diff --git a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_2_a7d3117c18e0ae13d730071a8f2452d4._comment b/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_2_a7d3117c18e0ae13d730071a8f2452d4._comment deleted file mode 100644 index ccc38e8d97..0000000000 --- a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_2_a7d3117c18e0ae13d730071a8f2452d4._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="https://bmwiedemann.zq1.de/" - nickname="bmwiedemann" - avatar="http://cdn.libravatar.org/avatar/96f3cd71c3d677f31ed8f79ffb8fb343a8282c085731f405997ff3ef77a1a71b" - subject="comment 2" - date="2020-06-30T15:26:10Z" - content=""" -We are already building openSUSE haskell packages sequentially since 2017-07-14 for that reason: - - -Here, non-determinism from filesystem readdir order is another independent class of issue. -"""]] diff --git a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_3_1a8277f3db09f49c076f5938da2b4390._comment b/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_3_1a8277f3db09f49c076f5938da2b4390._comment deleted file mode 100644 index b5ff287f67..0000000000 --- a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_3_1a8277f3db09f49c076f5938da2b4390._comment +++ /dev/null @@ -1,33 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="comment 3" - date="2020-06-30T16:26:17Z" - content=""" -[ I don't have a good understanding of the build issues here. I'm - sorry if this isn't relevant. ] - -> I found, the build becomes reproducible, when using a filesystem -> with deterministic readdir order such as disorderfs with sort mode. - -This reminded me of an issue with Guix's git-annex build: -. In that case, -the nondeterminism came from \"package database files that are -generated by ghc-pkg (where readdir is used and the result isn’t -sorted)\". - -The fix on Guix's end, 5de93cdba7 (gnu: ghc-8: Patch ghc-pkg for -reproducibility, 2019-01-17), was at the level of the ghc package. It -replaced the following line in utils/ghc-pkg/Main.hs - -``` -confs = map (path ) $ filter (\".conf\" `isSuffixOf`) fs -``` - -with - -``` -confs = map (path ) $ filter (\".conf\" `isSuffixOf`) (sort fs) -``` - -"""]] diff --git a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_4_a338f3831c6dc6f3f0ba2e8b6657a262._comment b/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_4_a338f3831c6dc6f3f0ba2e8b6657a262._comment deleted file mode 100644 index f45c040885..0000000000 --- a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_4_a338f3831c6dc6f3f0ba2e8b6657a262._comment +++ /dev/null @@ -1,20 +0,0 @@ -[[!comment format=mdwn - username="https://bmwiedemann.zq1.de/" - avatar="http://cdn.libravatar.org/avatar/96f3cd71c3d677f31ed8f79ffb8fb343a8282c085731f405997ff3ef77a1a71b" - subject="comment 4" - date="2020-06-30T19:47:35Z" - content=""" -That is closer, but those diffs in Assistant/WebApp/Types.o seem to still be something different. - -Since strings in Types.o varied around the embedded static/activityicon.gif, I noticed that Assistant/WebApp/Types.hs had lines like - -``` -publicFiles \"static\" -staticRoutes = $(embed \"static\") -``` - -Could it be, that those indirectly iterate over listings of the static/ dir without sorting to generate those lists of filenames embedded in Types.o? - -Maybe via ? -e.g. . That file has more occurrences of fs and none of sort. -"""]] diff --git a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_5_7eed0d8c4801bf0c32af1036c7e42d6f._comment b/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_5_7eed0d8c4801bf0c32af1036c7e42d6f._comment deleted file mode 100644 index 2026afc022..0000000000 --- a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_5_7eed0d8c4801bf0c32af1036c7e42d6f._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2020-06-30T21:17:08Z" - content=""" -Yes, it's using yesod-static. - -I think a bug report on yesod-static to sort it would be the best approach. -"""]] diff --git a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_6_ac6bfd4ef3bf8527c3394b45ad71185d._comment b/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_6_ac6bfd4ef3bf8527c3394b45ad71185d._comment deleted file mode 100644 index e6a3c723e4..0000000000 --- a/doc/bugs/git-annex_does_not_build_reproducibly_from_readdir_order/comment_6_ac6bfd4ef3bf8527c3394b45ad71185d._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="https://bmwiedemann.zq1.de/" - avatar="http://cdn.libravatar.org/avatar/96f3cd71c3d677f31ed8f79ffb8fb343a8282c085731f405997ff3ef77a1a71b" - subject="comment 6" - date="2020-07-01T02:47:05Z" - content=""" -filed -"""]] diff --git a/doc/bugs/git-annex_does_not_operate_on_all_keys_in_tuned_repository_with_annex.tune.branchhash1__61__true.mdwn b/doc/bugs/git-annex_does_not_operate_on_all_keys_in_tuned_repository_with_annex.tune.branchhash1__61__true.mdwn deleted file mode 100644 index 7c10599b43..0000000000 --- a/doc/bugs/git-annex_does_not_operate_on_all_keys_in_tuned_repository_with_annex.tune.branchhash1__61__true.mdwn +++ /dev/null @@ -1,91 +0,0 @@ -### Please describe the problem. - -git annex commands with `--all` option in tuned repository (with `annex.tune.branchhash1=true`) do not do anything. - -### What steps will reproduce the problem? - -1. Initialize a tuned annex repository with `git annex init -c annex.tune.branchhash1=true`. -2. Add some files to annex. -3. Now `git annex whereis --all` and `git annex fsck --all` (and maybe other commands) don't show/do anything. - -### What version of git-annex are you using? On what operating system? - -Version 7.20191230-g985373f8e, compiled from sources, on Debian buster 10.2. - -### Please provide any additional information below. - -[[!format txt """ -~ $ mkdir testdir -~ $ cd testdir -~/testdir $ -~/testdir $ git init -Initialized empty Git repository in /home/test/testdir/.git/ -~/testdir $ -~/testdir $ git annex init -c annex.tune.branchhash1=true testrepo -init testrepo (scanning for unlocked files...) -ok -(recording state in git...) -~/testdir $ -~/testdir $ echo abcabc >file -~/testdir $ -~/testdir $ git annex add file -add file -ok -(recording state in git...) -~/testdir $ -~/testdir $ git commit -m file -[master (root-commit) b910684] file - 1 file changed, 1 insertion(+) - create mode 120000 file -~/testdir $ -~/testdir $ git annex whereis -whereis file (1 copy) - 67d9c35f-e206-404f-a9da-6c94894a4f9f -- testrepo [here] -ok -~/testdir $ -~/testdir $ git annex whereis --all -~/testdir $ -~/testdir $ git annex fsck -fsck file (checksum...) ok -(recording state in git...) -~/testdir $ -~/testdir $ git annex fsck --all -(recording state in git...) -"""]] - -But `--key` option works: - -[[!format txt """ -~/testdir $ git annex lookupkey file -SHA256E-s7--2ed91d820157c0530ffbae54122d998e0de6d958f266b682f7c528942f770470 -~/testdir $ -~/testdir $ git annex whereis --key SHA256E-s7--2ed91d820157c0530ffbae54122d998e0de6d958f266b682f7c528942f770470 -whereis SHA256E-s7--2ed91d820157c0530ffbae54122d998e0de6d958f266b682f7c528942f770470 (1 copy) - 67d9c35f-e206-404f-a9da-6c94894a4f9f -- testrepo [here] -ok -~/testdir $ -~/testdir $ git annex fsck --key SHA256E-s7--2ed91d820157c0530ffbae54122d998e0de6d958f266b682f7c528942f770470 -fsck SHA256E-s7--2ed91d820157c0530ffbae54122d998e0de6d958f266b682f7c528942f770470 (checksum...) ok -(recording state in git...) -"""]] - -Repository status: - -[[!format txt """ -~/testdir $ find .git/annex/objects/ -type f -.git/annex/objects/J3/3f/SHA256E-s7--2ed91d820157c0530ffbae54122d998e0de6d958f266b682f7c528942f770470/SHA256E-s7--2ed91d820157c0530ffbae54122d998e0de6d958f266b682f7c528942f770470 -~/testdir $ -~/testdir $ git ls-tree -r git-annex -100644 blob 20f9faf7ca569d23da5f106a445609d018fa221d activity.log -100644 blob 71f3551b7119daa3c4679d2b790d72b6bc06cbb8 c34/SHA256E-s7--2ed91d820157c0530ffbae54122d998e0de6d958f266b682f7c528942f770470.log -100644 blob d475e423f6fb4863559e8cca981ae8a433f68516 difference.log -100644 blob bf91bd54df30e28f40b49670cf9c9c26ff600a22 uuid.log -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Of course, I love it! Great project, thanks, Joey! - -However, /me always wants more features from it. It's great that git-annex continues to develop. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git-annex_does_not_operate_on_all_keys_in_tuned_repository_with_annex.tune.branchhash1__61__true/comment_1_0e3da1c92dd46e4fc7facd10050fb590._comment b/doc/bugs/git-annex_does_not_operate_on_all_keys_in_tuned_repository_with_annex.tune.branchhash1__61__true/comment_1_0e3da1c92dd46e4fc7facd10050fb590._comment deleted file mode 100644 index 9a9dd2fe56..0000000000 --- a/doc/bugs/git-annex_does_not_operate_on_all_keys_in_tuned_repository_with_annex.tune.branchhash1__61__true/comment_1_0e3da1c92dd46e4fc7facd10050fb590._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-02-14T19:01:50Z" - content=""" -Indeed this is a bug. Easy to see why, note that "3": - - locationLogFileKey path - -- Want only xx/yy/foo.log, not .log files in other places. - | length (splitDirectories (fromRawFilePath path)) /= 3 = Nothing - -So this also affected some other things that use that. Including `git-annex log`, -potentially something to do with v2 upgrade (if a v2 repo could be tuned this way?), -and handling transitions set up by `git annex forget`. - -(I also checked if annex.tune.objecthash1 similarly broke stuff that -enumerated .git/annex/objects, but that was handled ok already.) -"""]] diff --git a/doc/bugs/git-annex_has_no_useful_debug_logging.mdwn b/doc/bugs/git-annex_has_no_useful_debug_logging.mdwn deleted file mode 100644 index aee5025cfb..0000000000 --- a/doc/bugs/git-annex_has_no_useful_debug_logging.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -Exporting to a directory on an USB drive (ext4) fails for exactly one file. All other files exported fine. - -I already renamed the file (it contained an umlaut and a space), I checked permissions, I ran fsck over the file in the annex. - -The metered write goes up to 100% then it stalls for one or two seconds and afterwards git-annex just says "Failed". - -My issue is that git-annex in general is hard to debug. I ran the command with --verbose and --debug but this also did not provide any clue. - -I believe good error handling and logging is a non-trivial problem in Haskell? - -> [[notabug|done]] --[[Joey]] diff --git a/doc/bugs/git-annex_has_no_useful_debug_logging/comment_1_9c9808c6e8363b9c230b160f21115a13._comment b/doc/bugs/git-annex_has_no_useful_debug_logging/comment_1_9c9808c6e8363b9c230b160f21115a13._comment deleted file mode 100644 index e271ddf259..0000000000 --- a/doc/bugs/git-annex_has_no_useful_debug_logging/comment_1_9c9808c6e8363b9c230b160f21115a13._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-16T17:27:59Z" - content=""" -I guess that [[bugs/filenames_with_dots_and_spaces_can_not_be_exported]] -is the bug report about the actual failure, and lack of a useful error -message after the failure, and this bug is left as just a complaint about -lack of debug logging. - -I'm not sure what to do about this bug. I'm not going to go around adding debug -logging statement to git-annex at random until some arbitrary criteria is -reached at which point there is "enough" to close this bug. It kind of -follows that I either leave the bug open forever, which is useless, or -close it immediately. Do you have an alternative suggestion? -"""]] diff --git a/doc/bugs/git-annex_has_no_useful_debug_logging/comment_2_c969e21ef3b85e96e9f8dd19c82f2afd._comment b/doc/bugs/git-annex_has_no_useful_debug_logging/comment_2_c969e21ef3b85e96e9f8dd19c82f2afd._comment deleted file mode 100644 index 6cee5055d3..0000000000 --- a/doc/bugs/git-annex_has_no_useful_debug_logging/comment_2_c969e21ef3b85e96e9f8dd19c82f2afd._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="thk" - avatar="http://cdn.libravatar.org/avatar/bfef10a428769701aeee1db978951461" - subject="comment 2" - date="2020-03-16T20:42:55Z" - content=""" -Now I learned how to add debug messages into git-annex. And I learned that there actually already is logging infrastructure in the code. - -How does one close bugs here in the wiki? - -Actually I would like to assign the bug to myself with the task of finding the right place here in the wiki (or code) to document how to add debug statements. I only discovered accidentally somewhere the debugM method. - - -"""]] diff --git a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames.mdwn b/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames.mdwn deleted file mode 100644 index c77ca745d1..0000000000 --- a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -In the standalone build of git-annex, the runshell script caches locale info in a directory the name of which is based on the full path to the script: - -When the full path is too long, this fails with the error "filename too long". - -Also, if the locale info is specific to the standalone environment, maybe it could be built as part of the process that creates the standalone package, rather than built on-the-fly and cached in the user's home dir? - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_1_f325c3f90b7e910c3c7c5f9609e8b4c7._comment b/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_1_f325c3f90b7e910c3c7c5f9609e8b4c7._comment deleted file mode 100644 index 3f779065b6..0000000000 --- a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_1_f325c3f90b7e910c3c7c5f9609e8b4c7._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-03-18T14:50:40Z" - content=""" -Pre-building it for all locales would add several hundred -megabytes to the download. - -The over-long path is proving difficult to deal with; it seems it would -need to hash the path, but the system may not have a hash program -(this is used on linux systems without coreutils), and -programs from the bundle can't yet be safely run at this point. - -Wouldn't you have had to put it in a directory path that's almost -4096 bytes deep for this to happen? -"""]] diff --git a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_2_71603e2b624c4e07bbf59e260fd83a8f._comment b/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_2_71603e2b624c4e07bbf59e260fd83a8f._comment deleted file mode 100644 index d43684093c..0000000000 --- a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_2_71603e2b624c4e07bbf59e260fd83a8f._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 2" - date="2019-03-18T19:44:07Z" - content=""" -\"Wouldn't you have had to put it in a directory path that's almost 4096 bytes deep for this to happen?\" -- I think it's trying to make a single pathname component corresponding to a directory path, so the relevant limit is 255 chars. - -For my specific use cases I've found a way not to need the standalone version for now, by building git-annex conda package the normal way. -"""]] diff --git a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_3_4fcc1fb89327a7957f28f0bd4575f26e._comment b/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_3_4fcc1fb89327a7957f28f0bd4575f26e._comment deleted file mode 100644 index fa78c29162..0000000000 --- a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_3_4fcc1fb89327a7957f28f0bd4575f26e._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="git-annex standalone and long filenames in locale cache" - date="2020-03-03T22:36:42Z" - content=""" -I'm finding it would be useful to have a conda package variant based on the standalone git-annex version (the normal variant has dependencies that sometimes conflict with other packages). So it would still be useful to fix this issue (another example of it is [here](https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/127349/logs/10)). If `md5sum` is not available, maybe can use `cksum`? -"""]] diff --git a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_4_4cde98f713c147cf357a25e97c01fd3d._comment b/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_4_4cde98f713c147cf357a25e97c01fd3d._comment deleted file mode 100644 index 079127e77a..0000000000 --- a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_4_4cde98f713c147cf357a25e97c01fd3d._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="id for locale cache" - date="2020-03-03T23:31:55Z" - content=""" -Or inode and ctime of the installed script file could be used instead of full pathname. -"""]] diff --git a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_5_4d82ba590dc4875c0b9c11076403832f._comment b/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_5_4d82ba590dc4875c0b9c11076403832f._comment deleted file mode 100644 index e243e717cd..0000000000 --- a/doc/bugs/git-annex_standalone_runshell_locale_cache_fails_on_too-long_filenames/comment_5_4d82ba590dc4875c0b9c11076403832f._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="locale info caching for standalone git-annex" - date="2020-03-04T17:04:42Z" - content=""" -For now I'm using the following [patch](https://github.com/conda-forge/git-annex-feedstock/blob/master/recipe/0001-fix-long-paths-in-locales.patch). -If the dir with the standalone version is writable, may be simplest to just store the locale cache under that dir? - -"""]] diff --git a/doc/bugs/git-annex_upgrade_fails.mdwn b/doc/bugs/git-annex_upgrade_fails.mdwn deleted file mode 100644 index 4067262fa6..0000000000 --- a/doc/bugs/git-annex_upgrade_fails.mdwn +++ /dev/null @@ -1,47 +0,0 @@ -Running git-annex upgrade von a v5 repo fails: ----------------- -$ git-annex upgrade -upgrade (v5 to v6...) (scanning for unlocked files...) -(recording state in git...) - -git status will show some files to be modified, since content availability has changed and git-annex was unable to update the index. This is only a cosmetic problem affecting git status; git add, git commit, etc won't be affected. To fix the git status display, you can run: git update-index -q --refresh -(recording state in git...) - -git status will show some files to be modified, since content availability has changed and git-annex was unable to update the index. This is only a cosmetic problem affecting git status; git add, git commit, etc won't be affected. To fix the git status display, you can run: git update-index -q --refresh - -git-annex: user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","status","--porcelain"] exited -9) -failed -(recording state in git...) - -git status will show some files to be modified, since content availability has changed and git-annex was unable to update the index. This is only a cosmetic problem affecting git status; git add, git commit, etc won't be affected. To fix the git status display, you can run: git update-index -q --refresh -git-annex: upgrade: 1 failed ----------------- - -I have tried to run -`git update-index -q --refresh` -in the working tree. This produced no output and the problem above persists. - -Also -`git-annex repair` -did return without output (after a few hours) and the problem above persists. - -Version info: - apt policy git-annex - git-annex: - Installed: 7.20190129-2~bpo9+1 - Candidate: 7.20190129-2~bpo9+1 - Version table: - *** 7.20190129-2~bpo9+1 100 - 100 http://ftp.debian.org/debian stretch-backports/main amd64 Packages - 100 /var/lib/dpkg/status - 6.20170101-1+deb9u2 990 - 990 http://ftp.de.debian.org/debian stretch/main amd64 Packages - 6.20170101-1+deb9u1 990 - 990 http://security.debian.org stretch/updates/main amd64 Packages - -Would be happy if anybody could share some insight on how to get that repo to v7. - -Regards, -Felix - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git-annex_upgrade_fails/comment_1_244e7e1cc7c1717ecf60591f81eee7e3._comment b/doc/bugs/git-annex_upgrade_fails/comment_1_244e7e1cc7c1717ecf60591f81eee7e3._comment deleted file mode 100644 index 8e80009ca5..0000000000 --- a/doc/bugs/git-annex_upgrade_fails/comment_1_244e7e1cc7c1717ecf60591f81eee7e3._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-08-29T16:24:37Z" - content=""" -git status exiting with -9 means it died of SIGKILL. -The only thing I can think of that could have killed it is the OOM killer. - -Based on your apt config, I guess git was probably older than version 2.23, -so it would have the [[bug that makes git status use a lot of memory|v7_unlock_big_file__58___out_of_memory]]. - -It appears that the v5 repository was a direct mode repository. - -What I suggest you do to recover is, upgrade git to 2.23, and re-run -git-annex upgrade. It should pick up where it left off. -"""]] diff --git a/doc/bugs/git-annex_upgrade_fails/comment_2_1ed5217d2e4997df237256dbc4bd350e._comment b/doc/bugs/git-annex_upgrade_fails/comment_2_1ed5217d2e4997df237256dbc4bd350e._comment deleted file mode 100644 index b689c5ade8..0000000000 --- a/doc/bugs/git-annex_upgrade_fails/comment_2_1ed5217d2e4997df237256dbc4bd350e._comment +++ /dev/null @@ -1,26 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2019-08-29T16:54:35Z" - content=""" -I reproduced the problem using git 2.1.4. The error message was slightly -better: - - $ git annex upgrade - upgrade (v5 to v6...) (scanning for unlocked files...) - (v6 to v7...) ok - (recording state in git...) - fatal: Out of memory? mmap failed: Cannot allocate memory - - git status will show some files to be modified, since content availability has changed and git-annex was unable to update the index. This is only a cosmetic problem affecting git status; git add, git commit, etc won't be affected. To fix the git status display, you can run: git update-index -q --refresh - $ git status - fatal: Out of memory? mmap failed: Cannot allocate memory - -And with the newer git, resuming the failed upgrade did succeed: - - $ git status - On branch adjusted/master(unlocked) - nothing to commit, working tree clean - $ git annex upgrade - upgrade ok -"""]] diff --git a/doc/bugs/git-annex_upgrade_fails/comment_3_acdfcd2bb40820cf69748fe4df956366._comment b/doc/bugs/git-annex_upgrade_fails/comment_3_acdfcd2bb40820cf69748fe4df956366._comment deleted file mode 100644 index 22d42c1f00..0000000000 --- a/doc/bugs/git-annex_upgrade_fails/comment_3_acdfcd2bb40820cf69748fe4df956366._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2019-08-29T17:02:02Z" - content=""" -It doesn't seem practical to get around the old git's OOM problem. -I think that the best thing to do is to not allow direct mode upgrade to v7 -until git has been upgraded. - -The minimum git version is 2.22. Unfortunately, Debian stable shipped 2.20. -So direct mode users there who upgrade git-annex will need to also upgrade -git in order for git-annex to convert their repo -- and since the next -git-annex release drops support for direct mode, they'll have to upgrade -git (or downgrade git-annex) in order to use git-annex in their repo at all. -"""]] diff --git a/doc/bugs/git-annex_upgrade_fails/comment_4_86919d40fe38dc19db443e9c25e411c9._comment b/doc/bugs/git-annex_upgrade_fails/comment_4_86919d40fe38dc19db443e9c25e411c9._comment deleted file mode 100644 index 7466039f09..0000000000 --- a/doc/bugs/git-annex_upgrade_fails/comment_4_86919d40fe38dc19db443e9c25e411c9._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="flixh" - avatar="http://cdn.libravatar.org/avatar/1f7e89860de517a494f35ebfb385288e" - subject="comment 4" - date="2019-09-07T11:44:47Z" - content=""" -You are right, git is at version 2.11.0-3+deb9u4 on this machine. -OOM killer might be, the machine has 2GB RAM. During the upgrade I observed the memory usage increasing steadily but at some point decreasing again. - -Unfortunately I've nuked the repo in the meantime, so I cannot try your suggestion of retrying after upgrading git. Also it looks like I need to go from stretch to testing to get a git >= 2.22. Which will probably not happen too soon. - -Thanks, -Felix -"""]] diff --git a/doc/bugs/git-annex_upgrade_fails/comment_5_ae191f06cc858bef86dc79d44f0aedba._comment b/doc/bugs/git-annex_upgrade_fails/comment_5_ae191f06cc858bef86dc79d44f0aedba._comment deleted file mode 100644 index 876cf07dc9..0000000000 --- a/doc/bugs/git-annex_upgrade_fails/comment_5_ae191f06cc858bef86dc79d44f0aedba._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="lhunath@3b4ff15f4600f3276d1776a490b734fca0f5c245" - nickname="lhunath" - avatar="http://cdn.libravatar.org/avatar/6388e539b56b3875cc9aceb9f404b3ad" - subject="comment 5" - date="2020-03-03T20:33:47Z" - content=""" -Seems like git-annex should report on that. Simply saying \"git-annex: upgrade: 1 failed\" because git is 2.21 is extremely unhelpful. If git-annex requires a certain version of git, it should test for that and be explicit about that requirement in its error output. I needed this comment in order to resolve the issue. - -For me, 2.21 is just what macOS ships with. -"""]] diff --git a/doc/bugs/git-annex_upgrade_fails/comment_6_abadb4c6b9e27a152744a89173436b9f._comment b/doc/bugs/git-annex_upgrade_fails/comment_6_abadb4c6b9e27a152744a89173436b9f._comment deleted file mode 100644 index e6b514df75..0000000000 --- a/doc/bugs/git-annex_upgrade_fails/comment_6_abadb4c6b9e27a152744a89173436b9f._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="minimum git version" - date="2020-03-03T21:15:07Z" - content=""" -\"The minimum git version is 2.22\" (6 months ago) -- is that still the correct minimum git version? -"""]] diff --git a/doc/bugs/git-annex_upgrade_fails/comment_7_24d785c28f1a07f5463ef7bbf0c67e1c._comment b/doc/bugs/git-annex_upgrade_fails/comment_7_24d785c28f1a07f5463ef7bbf0c67e1c._comment deleted file mode 100644 index 1f7eaeef74..0000000000 --- a/doc/bugs/git-annex_upgrade_fails/comment_7_24d785c28f1a07f5463ef7bbf0c67e1c._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 7""" - date="2020-03-04T16:54:32Z" - content=""" -You are responding to a bug that was fixed and closed 6 months ago -with some complaint about a vague error message. - -While it's true that the error message reported in the original bug report -was not very good -- because git was crashing! -- the fix involved making -git-annex detect the problimatic git versions and display a useful message. -See [[!commit 1558e03014f6a5ebf71e04f1498dd3c1acdbad5c]] - -If you are having a problem with current versions of git-annex, you need to -file a new bug report, not followup vaguely to an old one. Thank you. -"""]] diff --git a/doc/bugs/git_annex_can__39__t_see_connected_remote.mdwn b/doc/bugs/git_annex_can__39__t_see_connected_remote.mdwn deleted file mode 100644 index f77f680686..0000000000 --- a/doc/bugs/git_annex_can__39__t_see_connected_remote.mdwn +++ /dev/null @@ -1,48 +0,0 @@ -### Please describe the problem. - -Git annex won't pull from available remote - -### What steps will reproduce the problem? - -See full log demonstrating the issue. The file is accessible from a remote which is recorded, both remotes are synced, but instead it attempts a nonexistent remote download. - -``` -chymera@silenthost ~/data $ git annex get histology/sub-6532/sub-6532_slice-15gfp_zoom-5_scene-5_transmission.tif -get histology/sub-6532/sub-6532_slice-15gfp_zoom-5_scene-5_transmission.tif (not available) - Try making some of these repositories available: - 9f775012-942e-4ea7-96be-3bec8e4fcbf4 -- chymera@localhost.localdomain:~/data -failed -git-annex: get: 1 failed -chymera@silenthost ~/data $ git remote -v -data0 /run/media/chymera/data0/data (fetch) -data0 /run/media/chymera/data0/data (push) -data1 /run/media/chymera/data1/data (fetch) -data1 /run/media/chymera/data1/data (push) -ofmhost ofmhost:data (fetch) -ofmhost ofmhost:data (push) -chymera@silenthost ~/data $ git log | head -1 -commit 6f7b889c3487fb6dacbfd7a3a6fa8c55ea3ed274 -chymera@silenthost ~/data $ pushd /run/media/chymera/data0/data -/run/media/chymera/data0/data ~/data -chymera@silenthost /run/media/chymera/data0/data $ git remote -v -data1 /run/media/chymera/data1/data (fetch) -data1 /run/media/chymera/data1/data (push) -ofmhost ofmhost:data (fetch) -ofmhost ofmhost:data (push) -quiethost /quiethost/home/chymera/data/ (fetch) -quiethost /quiethost/home/chymera/data/ (push) -silenthost /silenthost/home/chymera/data (fetch) -silenthost /silenthost/home/chymera/data (push) -chymera@silenthost /run/media/chymera/data0/data $ ls -lah histology/sub-6532/sub-6532_slice-15gfp_zoom-5_scene-5_transmission.tif --rwxrwxrwx 1 chymera chymera 192 May 12 02:41 histology/sub-6532/sub-6532_slice-15gfp_zoom-5_scene-5_transmission.tif -chymera@silenthost /run/media/chymera/data0/data $ git log | head -1 -commit 6f7b889c3487fb6dacbfd7a3a6fa8c55ea3ed274 -``` - -### What version of git-annex are you using? On what operating system? -=git-annex-6.20170818-r1 on Gentoo Linux - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -I keep coming back to it, it's given me a steady stream of frustration and satisfaction for over 5 years. - -> [[notabug|done]] --[[Joey]] diff --git a/doc/bugs/git_annex_can__39__t_see_connected_remote/comment_1_8eb0b821d4ba63fa0e5bbfbca4488cb9._comment b/doc/bugs/git_annex_can__39__t_see_connected_remote/comment_1_8eb0b821d4ba63fa0e5bbfbca4488cb9._comment deleted file mode 100644 index 160346221b..0000000000 --- a/doc/bugs/git_annex_can__39__t_see_connected_remote/comment_1_8eb0b821d4ba63fa0e5bbfbca4488cb9._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-05-12T16:43:25Z" - content=""" -Does the repository at /run/media/chymera/data0/data -have an annex.uuid of 9f775012-942e-4ea7-96be-3bec8e4fcbf4 ? - -Also, does that repository have the content of the file in it? -Running ls on the file in the working tree will not tell you that. - -Easy way to answer both question is: -cd to the remote and run git-annex whereis on the file - -(If the answer to either question is no, there's no bug.) -"""]] diff --git a/doc/bugs/git_annex_can__39__t_see_connected_remote/comment_2_b96a4539716adf2a27930ad07c2b2b50._comment b/doc/bugs/git_annex_can__39__t_see_connected_remote/comment_2_b96a4539716adf2a27930ad07c2b2b50._comment deleted file mode 100644 index 3182a32ee2..0000000000 --- a/doc/bugs/git_annex_can__39__t_see_connected_remote/comment_2_b96a4539716adf2a27930ad07c2b2b50._comment +++ /dev/null @@ -1,54 +0,0 @@ -[[!comment format=mdwn - username="Chymera" - avatar="http://cdn.libravatar.org/avatar/555d585d6d78c68894ac90fd1e984309" - subject="comment 2" - date="2020-05-13T04:15:53Z" - content=""" -oh my, so yes, this elucidates the issue quite a bit: (1) my repo registry seems to contain a lot of cruft, (2) whereis does not find the file: - -``` -chymera@silenthost ~/data $ git annex whereis histology/sub-6532/sub-6532_slice-15gfp_zoom-5_scene-5_transmission.tif -whereis histology/sub-6532/sub-6532_slice-15gfp_zoom-5_scene-5_transmission.tif (1 copy) - 9f775012-942e-4ea7-96be-3bec8e4fcbf4 -- chymera@localhost.localdomain:~/data -ok -chymera@silenthost ~/data $ git annex info -repository mode: direct -trusted repositories: 0 -semitrusted repositories: 23 - 00000000-0000-0000-0000-000000000001 -- web - 00000000-0000-0000-0000-000000000002 -- bittorrent - 0db2cda2-f637-41d4-a0d2-701b105e734f -- chymera@zenbookhost:/run/media/chymera/data0/NIdata - 1b16e8b5-e8ca-457b-b55f-ecc8b80e8243 -- chymera@zenbookhost:/run/media/chymera/data0/data [data0] - 21038418-d6ae-4115-a64b-2aa48c37f834 -- chymera@darkhost:~/data - 37e50619-7871-4b52-bb0d-dbe68314818d -- chymera@quiethost:~/NIdata - 3e63db63-e218-4b5b-8c27-9ace71133731 -- chymera@quiethost:/mnt/data/data - 410ad0c4-62d4-4820-a3bc-55209e8493b0 -- chymera@zenbookhost:~/data - 4a86a541-33a2-4584-8db3-5e65a164a2bc -- chymera@quiethost:/run/media/chymera/data1/NIdata - 5316c8a8-5321-43f8-8c4c-da938b4b4f2f -- chymera@labhost:~/data - 5f721022-54a8-4833-8d33-7d3e48c929e5 -- data2 - 5fa05d4c-320f-43bf-9b4b-aae548675119 -- chymera@zenbookhost:~/data - 5fec9ad2-83bf-4e7f-abf2-6d981fb17b4a -- chymera@zenbookhost:~///home/chymera/NIdata - 6ccf6850-40f3-4130-a8de-a6a7ded9817e -- chymera@silenthost:~/data [here] - 9a48528e-1595-4a9a-8c47-501fbb6b74bf -- chymera@quiethost:~/data - 9f775012-942e-4ea7-96be-3bec8e4fcbf4 -- chymera@localhost.localdomain:~/data - a88e4011-c0a2-4924-a44d-66b31c5de74f -- chymera@zenhost:~/data - aeb05710-87d5-45d5-8d62-2281973d2834 -- neurohost - dd11ee2f-9b6c-4393-bc9d-df3542be52b4 -- chymera@zenhost:/zenhost/home/chymera/data - de05302b-03e7-4d91-82f5-d58ae66f863d -- chymera@neurohost:/mnt/data9/data - e56c9d76-3483-4c14-8b2e-f1aab168a9a5 -- chymera@quiethost:/run/media/chymera/data1/data - f5044a77-e912-4829-a071-b39ad187804e -- gentoo@ofmhost:~/data [ofmhost] - fd1ddddb-6a4f-49c3-b8a2-bbd3e00b934c -- quiethost -untrusted repositories: 0 -transfers in progress: none -available local disk space: 480.21 gigabytes (+1 megabyte reserved) -local annex keys: 13539 -local annex size: 214.72 gigabytes -annexed files in working tree: 15680 -size of annexed files in working tree: 215.92 gigabytes -bloom filter size: 32 mebibytes (2.7% full) -backend usage: - SHA256E: 15680 -``` - -What can I do to (1) clean up my repo index, and (2) make sure that data0 is correctly tracked? -"""]] diff --git a/doc/bugs/git_annex_can__39__t_see_connected_remote/comment_3_ff2e22dae90a858c46cb39c4af2a6842._comment b/doc/bugs/git_annex_can__39__t_see_connected_remote/comment_3_ff2e22dae90a858c46cb39c4af2a6842._comment deleted file mode 100644 index 429e00ff9e..0000000000 --- a/doc/bugs/git_annex_can__39__t_see_connected_remote/comment_3_ff2e22dae90a858c46cb39c4af2a6842._comment +++ /dev/null @@ -1,21 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-05-21T16:17:55Z" - content=""" -As whereis shows, the content of your file is located only on the repository -chymera@localhost.localdomain:~/data - -When you ran git-annex get in chymera@silenthost:~/data, it did not have any -remote that connected it to that repository, so it had no way to get the content. Its only remote, -/run/media/chymera/data0/data, did not contain the content. - -The data is being correctly tracked. I do not see any bug here. - -You need to track down the chymera@localhost.localdomain:~/data repository, add -a remote pointing to it, and everything will work. Unfortunately, it seems -whatever computer it's located on, git-annex was not able to get a better -hostname for than "localhost", which is kind of unusual. git-annex simply runs -"uname -n" to get the hostname, so whichever of your computers that outputs -"localhost" on is probably the one. -"""]] diff --git a/doc/bugs/git_annex_creds_are_not_embedded_with_newer_git-annex_clients.mdwn b/doc/bugs/git_annex_creds_are_not_embedded_with_newer_git-annex_clients.mdwn deleted file mode 100644 index 9787eb1a7a..0000000000 --- a/doc/bugs/git_annex_creds_are_not_embedded_with_newer_git-annex_clients.mdwn +++ /dev/null @@ -1,58 +0,0 @@ -### Please describe the problem. -With a newer version of git annex (8.20200226 vs 6.20170101) .git/annex/creds folder is not created after `git annex enableremote s3` -With an older version 6.20170101 creds are embedded all-right. - -[[!format sh """ -git annex info s3: - -remote: s3 -type: S3 -creds: embedded in git repository (not encrypted) - -"""]] - -### What steps will reproduce the problem? - -This is a bit tricky, as the remote was setup long ago, god knows with which git-annex version, but after you have an S3 remote, with embedcreds=yes: - -run `git annex enableremote s3` with versions 8.20200226 and 6.20170101 - - -[[!format sh """ -# version 8.20200226 -$ git annex enableremote s3 -$ ls .git/annex/creds -ls: cannot access '.git/annex/creds': No such file or directory - -# version 6.20170101 -$ git annex enableremote s3 -$ ls ../release-archive__/.git/annex/creds - - -$ git annex info s3: -remote: s3 -type: S3 -creds: embedded in git repository (not encrypted) - -"""]] - -### What version of git-annex are you using? On what operating system? - -8.20200226 and 6.20170101 -Linux, Ubuntu 20.04 - - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git_annex_creds_are_not_embedded_with_newer_git-annex_clients/comment_1_48953f4f0eacdba05b200f972204270d._comment b/doc/bugs/git_annex_creds_are_not_embedded_with_newer_git-annex_clients/comment_1_48953f4f0eacdba05b200f972204270d._comment deleted file mode 100644 index 9090c302ae..0000000000 --- a/doc/bugs/git_annex_creds_are_not_embedded_with_newer_git-annex_clients/comment_1_48953f4f0eacdba05b200f972204270d._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-05-21T16:34:34Z" - content=""" -I tried this and the result is that git-annex get from S3 fails. While I do -think it should be writing the creds cache file, the failure to get is the -actual bug symptom. - -Fixed this reversion. -"""]] diff --git a/doc/bugs/git_annex_creds_are_not_embedded_with_newer_git-annex_clients/comment_2_f1de80703cd0f1c15d5cf17f763adc9d._comment b/doc/bugs/git_annex_creds_are_not_embedded_with_newer_git-annex_clients/comment_2_f1de80703cd0f1c15d5cf17f763adc9d._comment deleted file mode 100644 index 51b5bc1caa..0000000000 --- a/doc/bugs/git_annex_creds_are_not_embedded_with_newer_git-annex_clients/comment_2_f1de80703cd0f1c15d5cf17f763adc9d._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="arseni.lapunov@34f437c25a6a8c6d317dce0bb7c5b44d568fa595" - nickname="arseni.lapunov" - avatar="http://cdn.libravatar.org/avatar/ccbdceb84c46bf9af5f9162bbf982f0b" - subject="git-annex get fails" - date="2020-05-22T14:28:23Z" - content=""" -Thank you! Exactly git annex get failure was an actual problem, but I thought that it should not succeed without cached creds. - -Regards -"""]] diff --git a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn b/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn deleted file mode 100644 index 3cf715a1c1..0000000000 --- a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__.mdwn +++ /dev/null @@ -1,130 +0,0 @@ -### Please describe the problem. - -I am trying to copy files from my computer to an external drive, but git annex get in one annex repo on the drive, keeps failing with "Unable to access these remotes: origin" message. It seems to pull the data, compute a checksum and then give me the error message. The same file passes fsck on the computer, and I was able to "git annex get" it from a different computer without issues. - -Comparing the debug log with that of a different annex (on the same external drive), the part that's confusing is that git-annex spins up two rsync processes to copy the one file for some reason, and logs "failed" right before the checksum step? - - -### What steps will reproduce the problem? -- cd into externaldrive/Annex -- git annex get file - - - get 8 Simple Rules - [02x01] - Premiere.mp4 (from origin...) - (from origin...) - - Unable to access these remotes: origin - - Try making some of these repositories available: - 1cca049d-83f7-4a78-ad30-650621c8e6f9 -- amnesiac - 3ae7e0b0-f87a-4f5c-9d37-a507273dde61 -- balrog [origin] - bafcea18-6642-4c43-8ae6-8477092c7d02 -- babadook - failed - git-annex: get: 1 failed - - -- The file then shows up in git annex unused. - -- Other commands like "git annex sync origin" work without issues, git annex copy --to origin also work. - -- I learned about the --debug flag from another post in the forum and here is the output: - - - [2020-06-08 10:02:19.510064296] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","symbolic-ref","-q","HEAD"] - [2020-06-08 10:02:19.511925354] process done ExitSuccess - [2020-06-08 10:02:19.512027344] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","show-ref","refs/heads/master"] - [2020-06-08 10:02:19.513918652] process done ExitSuccess - [2020-06-08 10:02:19.514026372] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","ls-files","--cached","-z","--","8 Simple Rules - [02x01] - Premiere.mp4"] - get 8 Simple Rules - [02x01] - Premiere.mp4 [2020-06-08 10:02:19.520306016] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","show-ref","git-annex"] - [2020-06-08 10:02:19.523513403] process done ExitSuccess - [2020-06-08 10:02:19.523637203] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] - [2020-06-08 10:02:19.525420671] process done ExitSuccess - [2020-06-08 10:02:19.525726091] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","log","refs/heads/git-annex..3faae00aee063e3eafd34c9ad82a25a3455cd699","--pretty=%H","- - n1"] - [2020-06-08 10:02:19.527440579] process done ExitSuccess - [2020-06-08 10:02:19.527526039] read: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","log","refs/heads/git-annex..7a56b81cf0518b9ffefd8e077e15616fc809c0ee","--pretty=%H","- - n1"] - [2020-06-08 10:02:19.531118595] process done ExitSuccess - [2020-06-08 10:02:19.53722095] chat: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","cat-file","--batch"] - [2020-06-08 10:02:19.537616269] chat: git ["--git-dir=../.git","--work-tree=..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] - [2020-06-08 10:02:19.541227956] read: git ["config","--null","--list"] - [2020-06-08 10:02:19.542505395] process done ExitSuccess - (from origin...) - [2020-06-08 10:02:19.559535139] read: cp ["--reflink=always","--preserve=timestamps","../../../../../../../home/kanak/Annex/TV/.git/annex/objects/3P/zP/SHA256-s210775700--f9bd3bcd9d6d3c5b90 - 7605c049d6e00fa40734860e4aadcbc8009a175b37ab49/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c049d6e00fa40734860e4aadcbc8009a175b37ab49","../.git/annex/tmp/SHA256-s210775700--f9bd3bcd9d6d3c5b907 - 605c049d6e00fa40734860e4aadcbc8009a175b37ab49"] - [2020-06-08 10:02:19.561441707] process done ExitFailure 1 - [2020-06-08 10:02:19.561515157] read: rsync ["--progress","--inplace","--perms","../../../../../../../home/kanak/Annex/TV/.git/annex/objects/3P/zP/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c - 049d6e00fa40734860e4aadcbc8009a175b37ab49/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c049d6e00fa40734860e4aadcbc8009a175b37ab49","../.git/annex/tmp/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c0 - 49d6e00fa40734860e4aadcbc8009a175b37ab49"] - 100% 201.01 MiB 428 MiB/s 0s [2020-06-08 10:02:21.820358727] process done ExitSuccess - (from origin...) - [2020-06-08 10:02:21.881519879] read: rsync ["--progress","--inplace","--perms","../../../../../../../home/kanak/Annex/TV/.git/annex/objects/3P/zP/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c - 049d6e00fa40734860e4aadcbc8009a175b37ab49/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c049d6e00fa40734860e4aadcbc8009a175b37ab49","../.git/annex/tmp/SHA256-s210775700--f9bd3bcd9d6d3c5b907605c0 - 49d6e00fa40734860e4aadcbc8009a175b37ab49"] - 100% 201.01 MiB 425 MiB/s 0s [2020-06-08 10:02:22.358355859] process done ExitSuccess - - Unable to access these remotes: origin - - Try making some of these repositories available: - 1cca049d-83f7-4a78-ad30-650621c8e6f9 -- amnesiac - 3ae7e0b0-f87a-4f5c-9d37-a507273dde61 -- balrog [origin] - bafcea18-6642-4c43-8ae6-8477092c7d02 -- babadook - failed - [2020-06-08 10:02:22.388873071] process done ExitSuccess - [2020-06-08 10:02:22.38926145] process done ExitSuccess - git-annex: get: 1 failed - - - -### What version of git-annex are you using? On what operating system? - -git annex version - - - git-annex version: 8.20200330 - build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite - dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.4 feed-1.1.0.0 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.1 - key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external - operating system: linux x86_64 - supported repository versions: 8 - upgrade supported from repository versions: 0 1 2 3 4 5 6 7 - local repository version: 8 - -Operating System: - - - cat /etc/fedora-release - - Fedora release 32 (Thirty Two) - - -### Please provide any additional information below. - -git annex info --fast - - trusted repositories: 7 - 318863a2-d9eb-467f-8094-f6578e26bd9a -- Melange - 33e44b4c-42f1-4383-90b3-5fcbdbbf3baf -- baldwin - 43d346b9-3c60-4036-a654-4c31f2368206 -- thane - bafcea18-6642-4c43-8ae6-8477092c7d02 -- babadook - d0ad767c-9874-4a3e-8ced-5d6917b212b9 -- zuko - d3ba71fb-64fc-4f41-8e76-7de8af14cee1 -- erebus [here] - d6d1d08e-6ee5-4408-a515-adbc9bebf5b1 -- ocho - semitrusted repositories: 1 - 3ae7e0b0-f87a-4f5c-9d37-a507273dde61 -- balrog [origin] - untrusted repositories: 2 - 1cca049d-83f7-4a78-ad30-650621c8e6f9 -- amnesiac - b3407756-8beb-43c3-877d-ff4fd1061fa9 -- chandrian - transfers in progress: none - available local disk space: 977.01 gigabytes (+1 megabyte reserved) - - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -YES. Git annex is one of my favorite software. I am managing around 15 terabytes of data in 5 annexes, with data on 3 computers, and around 8 external drives. This is the first time I've had any kind of issue with it. It has been rock solid and completely amazing. Thank you for your work on the software. - - -[[done]] diff --git a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__/comment_1_c7c28101f2dd05ccd125583d6ff6e8fc._comment b/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__/comment_1_c7c28101f2dd05ccd125583d6ff6e8fc._comment deleted file mode 100644 index 2df349f254..0000000000 --- a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__/comment_1_c7c28101f2dd05ccd125583d6ff6e8fc._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="kanakkshetri@9ea0e7639162bddc7bf9f3bb94cc32e93c793b89" - nickname="kanakkshetri" - avatar="http://cdn.libravatar.org/avatar/3cf305b2854041f8441c8fd8f02e8a2a" - subject="git-annex-repair and git annex fsck: no errors found" - date="2020-06-10T14:39:44Z" - content=""" -I ran git annex repair on both the external drive and the desktop: - - git annex repair - repair Running git fsck ... - No problems found. - ok - -I also fsck'd the files in the repository (on the desktop) and they all passed checksum checks. - -I'm kind of at a loss for what else to try. I also tried copy from desktop to external drive and it fails in the same way (including the two calls to rsync, both of which have ProcessExit Success, but then git annex just writes \"failed\"). -"""]] diff --git a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__/comment_2_85e99158ac826145bd903615299428f4._comment b/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__/comment_2_85e99158ac826145bd903615299428f4._comment deleted file mode 100644 index b9fc43a256..0000000000 --- a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__/comment_2_85e99158ac826145bd903615299428f4._comment +++ /dev/null @@ -1,26 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-06-11T21:03:50Z" - content=""" -It is indeed strange that it runs it twice. That must be a significant -clue. - -Since it also displays "(from origin)" twice, it seems that somehow -git-annex thinks there are two different remotes, both somehow named -"origin", and when the first doesn't work, it moves on to try the second. - -That should not be possible at all. Even if your .git/config -had two origin stanzas, you would not see this behavior. - -Also, after operating on 2 "origin" remotes, it then only lists origin once -in "Unable to access these remotes: origin", but if there are somehow two -origin remotes, it would have to list them both there too. - -So none of this is making any sense, on multiple levels. - -There must be something in the configuration of your repo that explains -this behavior, but I can't guess what it is. If you can show a transcript -of creating new repos in a way that reproduces this behavior, it would -help. -"""]] diff --git a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__/comment_3_274eae36b524b8550970c6ac07cca762._comment b/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__/comment_3_274eae36b524b8550970c6ac07cca762._comment deleted file mode 100644 index 0d7123d7f2..0000000000 --- a/doc/bugs/git_annex_get_fails_with___34__Unable_to_access_these_remotes__34__/comment_3_274eae36b524b8550970c6ac07cca762._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="kanakkshetri@9ea0e7639162bddc7bf9f3bb94cc32e93c793b89" - nickname="kanakkshetri" - avatar="http://cdn.libravatar.org/avatar/3cf305b2854041f8441c8fd8f02e8a2a" - subject="comment 3" - date="2020-06-15T11:42:41Z" - content=""" -Hi Joey, - -Thank you so much for taking a look. Another weird thing I discovered was that only a certain set of files (which were all annexed at around the same time) were affected. - -I ended up using cp manually to put then on external and used fsck --fast to fix location logs. - -I'm not sure how those files ended up in a weird state, but I don't want to waste your time on this. I'll create a new annex if the problem shows up again. -"""]] diff --git a/doc/bugs/git_annex_init_fails.mdwn b/doc/bugs/git_annex_init_fails.mdwn deleted file mode 100644 index 67307064de..0000000000 --- a/doc/bugs/git_annex_init_fails.mdwn +++ /dev/null @@ -1,36 +0,0 @@ -### Please describe the problem. - -git annex fails init step: - -i.e.: -$ mkdir ~/annex -$ cd ~/annex -$ git init -$ git annex init < this step fails: - -mv: cannot move '/home/user/.cache/git-annex/locales.tmp/b03b775f010002f73bc923b1b46a73c7.4328' to '/home/user/.cache/git-annex/locales/b03b775f010002f73bc923b1b46a73c7': No such file or directory - - -### What steps will reproduce the problem? - -install git annex on CentOS8 , try to perform git annex init - -### What version of git-annex are you using? On what operating system? - -CentOS8 / git-annex-standalone-8.20200909-1.x86_64 - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -No, first time install - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git_annex_init_fails/comment_1_2ee620c8ccb2ca411f303edda29bf62a._comment b/doc/bugs/git_annex_init_fails/comment_1_2ee620c8ccb2ca411f303edda29bf62a._comment deleted file mode 100644 index 72bc672184..0000000000 --- a/doc/bugs/git_annex_init_fails/comment_1_2ee620c8ccb2ca411f303edda29bf62a._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="rshalaev@3e2130a1e3cb0aaff7dd80aba7548ad9be0ea2d4" - nickname="rshalaev" - avatar="http://cdn.libravatar.org/avatar/d7f181d338cbcef7418faa01f0441e86" - subject="comment 1" - date="2020-10-23T02:00:08Z" - content=""" -Line endings got removed in the original post, so for clarification, after installing git-annex, I'm following this steps: - -1. `mkdir ~/annex` -2. `cd ~/annex` -3. `git init` -4. `git annex init` - -command that fails is `git annex init` (error message is `mv: cannot move '/home/user/.cache/git-annex/locales.tmp/b03b775f010002f73bc923b1b46a73c7.4662' to '/home/user/.cache/git-annex/locales/b03b775f010002f73bc923b1b46a73c7': No such file or directory`) -"""]] diff --git a/doc/bugs/git_annex_init_fails/comment_2_7c287bc03f8aa7fe40afe71f8f377593._comment b/doc/bugs/git_annex_init_fails/comment_2_7c287bc03f8aa7fe40afe71f8f377593._comment deleted file mode 100644 index 284a1d48f2..0000000000 --- a/doc/bugs/git_annex_init_fails/comment_2_7c287bc03f8aa7fe40afe71f8f377593._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-10-23T16:15:42Z" - content=""" -I've fixed this for the next release. It was simply omitting creating the -/home/user/.cache/git-annex/locales/ directory, so as a workaround you can -create that directory. -"""]] diff --git a/doc/bugs/git_annex_init_fails/comment_3_479898de9a6635d413b9a6b15461ba7c._comment b/doc/bugs/git_annex_init_fails/comment_3_479898de9a6635d413b9a6b15461ba7c._comment deleted file mode 100644 index b56e78e922..0000000000 --- a/doc/bugs/git_annex_init_fails/comment_3_479898de9a6635d413b9a6b15461ba7c._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="rshalaev@3e2130a1e3cb0aaff7dd80aba7548ad9be0ea2d4" - nickname="rshalaev" - avatar="http://cdn.libravatar.org/avatar/d7f181d338cbcef7418faa01f0441e86" - subject="comment 3" - date="2020-10-23T16:51:04Z" - content=""" -Joey, thank you for the quick response! I applied your suggestion to create `locales` directory - and git annex is now working. This is a very exciting project - look forward to diving in to it. Thanks again! ~Ruslan -"""]] diff --git a/doc/bugs/git_annex_sync__58___--fast_and_REMOTE_should_be_exclusive.mdwn b/doc/bugs/git_annex_sync__58___--fast_and_REMOTE_should_be_exclusive.mdwn deleted file mode 100644 index 9b82e0da29..0000000000 --- a/doc/bugs/git_annex_sync__58___--fast_and_REMOTE_should_be_exclusive.mdwn +++ /dev/null @@ -1,26 +0,0 @@ -### Please describe the problem. - -Without checking the manpage, I did: - - cd usb-drive-1/annex - git annex sync --content --fast usb-drive-2 - -I thought, that ´--fast´ would mean the same thing as in [[git-annex-copy]]: - -> When copying content to a remote, avoid a round trip to check if the remote already has content. This can be faster, but might skip copying content to the remote in some cases. - -Only here with [[git-annex-sync]], `--fast` means: - -> Only sync with the remotes with the lowest annex-cost value configured - -This is sub-optimal UI and this ship has sailed. But the command should have failed because it does not make sense to specify both --fast and a REMOTE, does it? - -I only noticed after a while that annex was syncing to remote at ~/annex and filling my laptops hard disk. - -I actually used `--fast` because I was annoyed by annex iterating over all files in the repo during the sync, which for some reason it doesn't do when syncing with an SSH remote. - -### What version of git-annex are you using? On what operating system? - -8.20200330, Debian testing - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/git_annex_sync__58___--fast_and_REMOTE_should_be_exclusive/comment_1_47b3095cd203477cc3e607d9cf840823._comment b/doc/bugs/git_annex_sync__58___--fast_and_REMOTE_should_be_exclusive/comment_1_47b3095cd203477cc3e607d9cf840823._comment deleted file mode 100644 index 3b10626510..0000000000 --- a/doc/bugs/git_annex_sync__58___--fast_and_REMOTE_should_be_exclusive/comment_1_47b3095cd203477cc3e607d9cf840823._comment +++ /dev/null @@ -1,22 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-04-23T19:29:59Z" - content=""" ---fast is a global option that is parsed for all commands and -does varying different things in different commands. -That's mostly a relic from before git-annex's option parser supported -per-command options. It would be hard to untangle at this point. - -What it actually does currently is sync with the remotes you listed and -then the --fast *adds* whatever fastest remote it can find. That must -explain the behavior you saw, although I would expect that it would have -first synced with the remote you listed and only after that, -started filling the fast remote. - -Anyway, it seems that `git annex sync --fast foo bar` -should either pick whichever of foo or bar currently has the lowest cost, -and sync with it, or it should error out as you suggest (necessarily after -option parsing). Adding another remote that has a low cost is certianly too -surprising. -"""]] diff --git a/doc/bugs/git_config_merge.ff__61__only_breaks_sync.mdwn b/doc/bugs/git_config_merge.ff__61__only_breaks_sync.mdwn deleted file mode 100644 index a38c8926c7..0000000000 --- a/doc/bugs/git_config_merge.ff__61__only_breaks_sync.mdwn +++ /dev/null @@ -1,450 +0,0 @@ -### Please describe the problem. - -Having globally configured git-merge to only allow fast forward merges breaks git-annex's sync command. - -### What steps will reproduce the problem? - -Run the following script, beware it will change the global git config for `merge.ff`: - -``` -#!/bin/sh -repro() { - rm -rf repo-a repo-b - mkdir repo-a repo-b - ( cd repo-a; git init; git annex init; git remote add repo-b ../repo-b ) - ( cd repo-b; git init; git annex init; git remote add repo-a ../repo-a ) - - # Setup an initial commit a0 in repo-a - ( cd repo-a - touch a0 - git add a0 - git commit -m a0 - git annex sync - ) - - # Pull a0 into repo-b and create commit 'b' on top of it - ( cd repo-b - git annex sync - touch b - git add b - git commit -m b - ) - - # Back in repo-a create a diverging commit 'a1' and try to sync - ( cd repo-a - touch a1 - git add a1 - git commit -m a1 - git annex sync -d - ) -} - -# First try without merge.ff=only -git config --global --unset merge.ff -repro; rv=$? - -# Now with -git config --global merge.ff only -repro || echo "===== Breaks with merge.ff=only =====" -[ $rv -eq 0 ] && echo "===== Works without merge.ff=only =====" -``` - -### What version of git-annex are you using? On what operating system? - -git-annex version 7.20190129-3 from Debian buster - -``` -$ git annex version -git-annex version: 7.20190129 -build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.0.0 ghc-8.4.4 http-client-0.5.13.1 persistent-sqlite-2.8.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external -operating system: linux x86_64 -supported repository versions: 5 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -``` - -### Please provide any additional information below. - -A run of the repro script above with `sh -x`: - -[[!format sh """ -+ git config --global --unset merge.ff -+ repro -+ rm -rf repo-a repo-b -+ mkdir repo-a repo-b -+ cd repo-a -+ git init -Initialized empty Git repository in /tmp/repo-a/.git/ -+ git annex init -init ok -(recording state in git...) -+ git remote add repo-b ../repo-b -+ cd repo-b -+ git init -Initialized empty Git repository in /tmp/repo-b/.git/ -+ git annex init -init ok -(recording state in git...) -+ git remote add repo-a ../repo-a -+ cd repo-a -+ touch a0 -+ git add a0 -+ git commit -m a0 -[master (root-commit) 6c82d47] a0 - 1 file changed, 0 insertions(+), 0 deletions(-) - create mode 100644 a0 -+ git annex sync -commit -On branch master -nothing to commit, working tree clean -ok -pull repo-b -remote: Enumerating objects: 4, done. -remote: Counting objects: 100% (4/4), done. -remote: Compressing objects: 100% (2/2), done. -remote: Total 3 (delta 0), reused 0 (delta 0) -Unpacking objects: 100% (3/3), done. -From ../repo-b - * [new branch] git-annex -> repo-b/git-annex -ok -(merging repo-b/git-annex into git-annex...) -(recording state in git...) -push repo-b -Enumerating objects: 12, done. -Counting objects: 100% (12/12), done. -Delta compression using up to 8 threads -Compressing objects: 100% (5/5), done. -Writing objects: 100% (9/9), 805 bytes | 805.00 KiB/s, done. -Total 9 (delta 1), reused 0 (delta 0) -To ../repo-b - * [new branch] git-annex -> synced/git-annex - * [new branch] master -> synced/master -ok -+ cd repo-b -+ git annex sync -commit -On branch master - -Initial commit - -nothing to commit -ok -fatal: ambiguous argument 'refs/heads/master..refs/heads/synced/master': unknown revision or path not in the working tree. -Use '--' to separate paths from revisions, like this: -'git [...] -- [...]' -pull repo-a -From ../repo-a - * [new branch] git-annex -> repo-a/git-annex - * [new branch] master -> repo-a/master - * [new branch] synced/master -> repo-a/synced/master - - -Already up to date. -ok -+ touch b -+ git add b -+ git commit -m b -[master d3ecbbf] b - 1 file changed, 0 insertions(+), 0 deletions(-) - create mode 100644 b -+ cd repo-a -+ touch a1 -+ git add a1 -+ git commit -m a1 -[master 7c4054b] a1 - 1 file changed, 0 insertions(+), 0 deletions(-) - create mode 100644 a1 -+ git annex sync -d -[2020-01-21 12:41:12.808810342] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] -[2020-01-21 12:41:12.810537111] process done ExitSuccess -[2020-01-21 12:41:12.810623305] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] -[2020-01-21 12:41:12.812254921] process done ExitSuccess -[2020-01-21 12:41:12.812457308] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..a2ceb59a5879cdf88bd33104c0ee8ac390b3f62e","--pretty=%H","-n1"] -[2020-01-21 12:41:12.813939135] process done ExitSuccess -[2020-01-21 12:41:12.814207234] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"] -[2020-01-21 12:41:12.814533549] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] -[2020-01-21 12:41:12.820975511] read: git ["config","--null","--list"] -[2020-01-21 12:41:12.822586611] process done ExitSuccess -commit -[2020-01-21 12:41:12.823337737] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit","-a","-m","git-annex in dxld@Eli:/tmp/repo-a"] -On branch master -nothing to commit, working tree clean -[2020-01-21 12:41:12.851712964] process done ExitFailure 1 -ok -[2020-01-21 12:41:12.851809058] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"] -[2020-01-21 12:41:12.852844432] process done ExitSuccess -[2020-01-21 12:41:12.85289997] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"] -[2020-01-21 12:41:12.853966619] process done ExitSuccess -[2020-01-21 12:41:12.854022004] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/heads/synced/master"] -[2020-01-21 12:41:12.85488393] process done ExitSuccess -[2020-01-21 12:41:12.854935336] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/heads/synced/master","--pretty=%H","-n1"] -[2020-01-21 12:41:12.85636602] process done ExitSuccess -pull repo-b -[2020-01-21 12:41:12.85658479] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","fetch","repo-b"] -remote: Enumerating objects: 3, done. -remote: Counting objects: 100% (3/3), done. -remote: Compressing objects: 100% (2/2), done. -remote: Total 2 (delta 0), reused 0 (delta 0) -Unpacking objects: 100% (2/2), done. -From ../repo-b - * [new branch] master -> repo-b/master -[2020-01-21 12:41:12.869425035] process done ExitSuccess -[2020-01-21 12:41:12.86951948] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"] -[2020-01-21 12:41:12.870907097] process done ExitSuccess -[2020-01-21 12:41:12.870950213] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/repo-b/master"] -[2020-01-21 12:41:12.871908557] process done ExitSuccess -[2020-01-21 12:41:12.871965108] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/remotes/repo-b/master","--pretty=%H","-n1"] -[2020-01-21 12:41:12.873517012] process done ExitSuccess -[2020-01-21 12:41:12.873627395] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/repo-b/synced/master"] -[2020-01-21 12:41:12.874965525] process done ExitSuccess -[2020-01-21 12:41:12.875039117] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/synced/master..refs/remotes/repo-b/synced/master","--pretty=%H","-n1"] -[2020-01-21 12:41:12.876601605] process done ExitSuccess - -[2020-01-21 12:41:12.876699748] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/master"] -[2020-01-21 12:41:12.878506453] process done ExitSuccess -[2020-01-21 12:41:12.878622705] read: git ["--version"] -[2020-01-21 12:41:12.879681841] process done ExitSuccess -[2020-01-21 12:41:12.879787682] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","merge","--no-edit","refs/remotes/repo-b/master","--allow-unrelated-histories"] -Merge made by the 'recursive' strategy. - b | 0 - 1 file changed, 0 insertions(+), 0 deletions(-) - create mode 100644 b -[2020-01-21 12:41:12.884914215] process done ExitSuccess -ok -[2020-01-21 12:41:12.885012011] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] -[2020-01-21 12:41:12.886558493] process done ExitSuccess -[2020-01-21 12:41:12.88678805] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] -[2020-01-21 12:41:12.888312105] process done ExitSuccess -[2020-01-21 12:41:12.888486534] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..a2ceb59a5879cdf88bd33104c0ee8ac390b3f62e","--pretty=%H","-n1"] -[2020-01-21 12:41:12.890100289] process done ExitSuccess -[2020-01-21 12:41:12.890310425] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"] -[2020-01-21 12:41:12.892087109] process done ExitSuccess -[2020-01-21 12:41:12.892162993] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/repo-b/synced/master"] -[2020-01-21 12:41:12.893271619] process done ExitSuccess -[2020-01-21 12:41:12.893323578] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/remotes/repo-b/synced/master..refs/heads/synced/master","--pretty=%H","-n1"] -[2020-01-21 12:41:12.894429364] process done ExitSuccess -push repo-b -[2020-01-21 12:41:12.894494807] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","push","repo-b","+git-annex:synced/git-annex","master:synced/master"] -Enumerating objects: 6, done. -Counting objects: 100% (6/6), done. -Delta compression using up to 8 threads -Compressing objects: 100% (4/4), done. -Writing objects: 100% (4/4), 472 bytes | 472.00 KiB/s, done. -Total 4 (delta 1), reused 0 (delta 0) -To ../repo-b - 6c82d47..40d7fc6 master -> synced/master -[2020-01-21 12:41:12.939565851] process done ExitSuccess -[2020-01-21 12:41:12.939790389] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","push","repo-b","git-annex"] -[2020-01-21 12:41:12.945667784] process done ExitSuccess -[2020-01-21 12:41:12.945804791] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","push","repo-b","master"] -[2020-01-21 12:41:12.960382215] process done ExitFailure 1 -ok -[2020-01-21 12:41:12.961199639] process done ExitSuccess -[2020-01-21 12:41:12.961575886] process done ExitSuccess -+ rv=0 -+ git config --global merge.ff only -+ repro -+ rm -rf repo-a repo-b -+ mkdir repo-a repo-b -+ cd repo-a -+ git init -Initialized empty Git repository in /tmp/repo-a/.git/ -+ git annex init -init ok -(recording state in git...) -+ git remote add repo-b ../repo-b -+ cd repo-b -+ git init -Initialized empty Git repository in /tmp/repo-b/.git/ -+ git annex init -init ok -(recording state in git...) -+ git remote add repo-a ../repo-a -+ cd repo-a -+ touch a0 -+ git add a0 -+ git commit -m a0 -[master (root-commit) 250bdc3] a0 - 1 file changed, 0 insertions(+), 0 deletions(-) - create mode 100644 a0 -+ git annex sync -commit -On branch master -nothing to commit, working tree clean -ok -pull repo-b -warning: no common commits -remote: Enumerating objects: 5, done. -remote: Counting objects: 100% (5/5), done. -remote: Compressing objects: 100% (3/3), done. -remote: Total 5 (delta 1), reused 0 (delta 0) -Unpacking objects: 100% (5/5), done. -From ../repo-b - * [new branch] git-annex -> repo-b/git-annex -ok -(merging repo-b/git-annex into git-annex...) -(recording state in git...) -push repo-b -Enumerating objects: 13, done. -Counting objects: 100% (13/13), done. -Delta compression using up to 8 threads -Compressing objects: 100% (6/6), done. -Writing objects: 100% (11/11), 1023 bytes | 1023.00 KiB/s, done. -Total 11 (delta 0), reused 0 (delta 0) -To ../repo-b - * [new branch] git-annex -> synced/git-annex - * [new branch] master -> synced/master -ok -+ cd repo-b -+ git annex sync -commit -On branch master - -Initial commit - -nothing to commit -ok -fatal: ambiguous argument 'refs/heads/master..refs/heads/synced/master': unknown revision or path not in the working tree. -Use '--' to separate paths from revisions, like this: -'git [...] -- [...]' -pull repo-a -From ../repo-a - * [new branch] git-annex -> repo-a/git-annex - * [new branch] master -> repo-a/master - * [new branch] synced/master -> repo-a/synced/master - - -Already up to date. -ok -+ touch b -+ git add b -+ git commit -m b -[master af18b51] b - 1 file changed, 0 insertions(+), 0 deletions(-) - create mode 100644 b -+ cd repo-a -+ touch a1 -+ git add a1 -+ git commit -m a1 -[master 91e5f6f] a1 - 1 file changed, 0 insertions(+), 0 deletions(-) - create mode 100644 a1 -+ git annex sync -d -[2020-01-21 12:41:13.526664743] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] -[2020-01-21 12:41:13.52932842] process done ExitSuccess -[2020-01-21 12:41:13.529400247] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] -[2020-01-21 12:41:13.530793109] process done ExitSuccess -[2020-01-21 12:41:13.530950751] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..4b03666eac77ff7f709a83289267cc1f28f80391","--pretty=%H","-n1"] -[2020-01-21 12:41:13.532177334] process done ExitSuccess -[2020-01-21 12:41:13.532400895] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"] -[2020-01-21 12:41:13.532647616] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"] -[2020-01-21 12:41:13.538048182] read: git ["config","--null","--list"] -[2020-01-21 12:41:13.539367779] process done ExitSuccess -commit -[2020-01-21 12:41:13.540002284] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit","-a","-m","git-annex in dxld@Eli:/tmp/repo-a"] -On branch master -nothing to commit, working tree clean -[2020-01-21 12:41:13.568174475] process done ExitFailure 1 -ok -[2020-01-21 12:41:13.568319083] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"] -[2020-01-21 12:41:13.57005936] process done ExitSuccess -[2020-01-21 12:41:13.5701439] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"] -[2020-01-21 12:41:13.571885769] process done ExitSuccess -[2020-01-21 12:41:13.571979878] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/heads/synced/master"] -[2020-01-21 12:41:13.573389866] process done ExitSuccess -[2020-01-21 12:41:13.573486391] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/heads/synced/master","--pretty=%H","-n1"] -[2020-01-21 12:41:13.575824245] process done ExitSuccess -pull repo-b -[2020-01-21 12:41:13.575950332] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","fetch","repo-b"] -remote: Enumerating objects: 3, done. -remote: Counting objects: 100% (3/3), done. -remote: Compressing objects: 100% (2/2), done. -remote: Total 2 (delta 0), reused 0 (delta 0) -Unpacking objects: 100% (2/2), done. -From ../repo-b - * [new branch] master -> repo-b/master -[2020-01-21 12:41:13.586295016] process done ExitSuccess -[2020-01-21 12:41:13.586392073] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"] -[2020-01-21 12:41:13.587730993] process done ExitSuccess -[2020-01-21 12:41:13.5877987] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/repo-b/master"] -[2020-01-21 12:41:13.588901362] process done ExitSuccess -[2020-01-21 12:41:13.588969352] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/remotes/repo-b/master","--pretty=%H","-n1"] -[2020-01-21 12:41:13.5904904] process done ExitSuccess -[2020-01-21 12:41:13.590563378] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/repo-b/synced/master"] -[2020-01-21 12:41:13.591686652] process done ExitSuccess -[2020-01-21 12:41:13.591740872] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/synced/master..refs/remotes/repo-b/synced/master","--pretty=%H","-n1"] -[2020-01-21 12:41:13.593110565] process done ExitSuccess - -[2020-01-21 12:41:13.593182929] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/master"] -[2020-01-21 12:41:13.594562053] process done ExitSuccess -[2020-01-21 12:41:13.594660841] read: git ["--version"] -[2020-01-21 12:41:13.595595012] process done ExitSuccess -[2020-01-21 12:41:13.59568225] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","merge","--no-edit","refs/remotes/repo-b/master","--allow-unrelated-histories"] -fatal: Not possible to fast-forward, aborting. -[2020-01-21 12:41:13.597312517] process done ExitFailure 128 -[2020-01-21 12:41:13.597552001] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--unmerged","-z","--","."] -[2020-01-21 12:41:13.598903873] process done ExitSuccess -[2020-01-21 12:41:13.598967865] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--deleted","-z","--","."] -[2020-01-21 12:41:13.600107434] process done ExitSuccess -failed -[2020-01-21 12:41:13.600185848] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] -[2020-01-21 12:41:13.601485474] process done ExitSuccess -[2020-01-21 12:41:13.601548875] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] -[2020-01-21 12:41:13.602954053] process done ExitSuccess -[2020-01-21 12:41:13.603111747] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..4b03666eac77ff7f709a83289267cc1f28f80391","--pretty=%H","-n1"] -[2020-01-21 12:41:13.604578405] process done ExitSuccess -[2020-01-21 12:41:13.60470379] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","branch","-f","synced/master","refs/heads/master"] -[2020-01-21 12:41:13.606371207] process done ExitSuccess -[2020-01-21 12:41:13.606457892] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--verify","-q","refs/remotes/repo-b/synced/master"] -[2020-01-21 12:41:13.607658403] process done ExitSuccess -[2020-01-21 12:41:13.607726481] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/remotes/repo-b/synced/master..refs/heads/synced/master","--pretty=%H","-n1"] -[2020-01-21 12:41:13.609097845] process done ExitSuccess -push repo-b -[2020-01-21 12:41:13.60918811] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","push","repo-b","+git-annex:synced/git-annex","master:synced/master"] -Enumerating objects: 3, done. -Counting objects: 100% (3/3), done. -Delta compression using up to 8 threads -Compressing objects: 100% (2/2), done. -Writing objects: 100% (2/2), 231 bytes | 231.00 KiB/s, done. -Total 2 (delta 0), reused 0 (delta 0) -To ../repo-b - 250bdc3..91e5f6f master -> synced/master -[2020-01-21 12:41:13.651508284] process done ExitSuccess -[2020-01-21 12:41:13.651669133] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","push","repo-b","git-annex"] -[2020-01-21 12:41:13.657342122] process done ExitSuccess -[2020-01-21 12:41:13.657603861] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","push","repo-b","master"] -[2020-01-21 12:41:13.663212647] process done ExitFailure 1 -To ../repo-b - ! [rejected] master -> master (non-fast-forward) -error: failed to push some refs to '../repo-b' -hint: Updates were rejected because the tip of your current branch is behind -hint: its remote counterpart. Integrate the remote changes (e.g. -hint: 'git pull ...') before pushing again. -hint: See the 'Note about fast-forwards' in 'git push --help' for details. -ok -[2020-01-21 12:41:13.663856234] process done ExitSuccess -[2020-01-21 12:41:13.664249322] process done ExitSuccess -git-annex: sync: 1 failed -+ echo ===== Breaks with merge.ff=only ===== -===== Breaks with merge.ff=only ===== -+ [ 0 -eq 0 ] -+ echo ===== Works without merge.ff=only ===== -===== Works without merge.ff=only ===== -# End of transcript or log. -"""]] - -Notice the call to git-merge, failing: - -``` -[2020-01-21 12:41:13.59568225] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","merge","--no-edit","refs/remotes/repo-b/master","--allow-unrelated-histories"] -fatal: Not possible to fast-forward, aborting. -``` - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Sure! Using git-annex to keep artifacts in development repos works great usually :) - -> Closing as I don't think this behavior is actually a bug. [[done]] -> --[[Joey]] diff --git a/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_1_53fd805e1afcd958134b5774550e7644._comment b/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_1_53fd805e1afcd958134b5774550e7644._comment deleted file mode 100644 index a5c94ec3a6..0000000000 --- a/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_1_53fd805e1afcd958134b5774550e7644._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-01-21T18:31:22Z" - content=""" -I feel it's right for git-annex sync to honor git configs, so it's right -for it to not merge origin/master. And, without that merge, it's right for -it to fail to push master to origin. Since it does push synced/master, this -does not prevent other clones of the repo, where git-annex sync is later -ran, from getting the changes made by this sync. - -That leaves only this ugly thing: - -fatal: ambiguous argument 'refs/heads/master..refs/heads/synced/master': unknown revision or path not in the working tree. - -Which comes from Git.Branch.changed, but I'm not clear how the fast forward -configuration would prevent either of those refs from existing. -"""]] diff --git a/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_2_db82f6c4a27ac8ee862b5aaa45670085._comment b/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_2_db82f6c4a27ac8ee862b5aaa45670085._comment deleted file mode 100644 index 337e648c09..0000000000 --- a/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_2_db82f6c4a27ac8ee862b5aaa45670085._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="dxld" - avatar="http://cdn.libravatar.org/avatar/742547a848e15c9f7fb381191c239141" - subject="comment 2" - date="2020-01-21T19:28:29Z" - content=""" -Honestly I feel like the (perceived) semantics of sync are broken by this behaviour. I would expect git-annex to do what it has to to make what I asked for happen. - -I agree that in general it's a good thing not to needlessly override git settings but for the sync command I really don't see any way that not merging can be considered sensible behaviour. To me as a user it just feels like I changed a setting completely unrelated to git-annex-sync and suddenly sync broke. - -Consider this: the git-annex-sync(1) man page never actually mentions that it will run git-merge. On the other hand git-pull(1) is very forthcoming with the fact that it's just a shorthand for `git fetch; git merge` so it's obvious to me that settings affecting merge will affect git-pull, not so for sync. - -I've been unable to sync my git-annex repos for a couple of months now because of this issue so firmly believe this is a serious usabiliy issue. - -At the very least we have a documentation issue here. Though I would still argue the behaviour is bonkers :) - -"""]] diff --git a/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_3_3f78dd62e99f55388e415a2e2fb86a3f._comment b/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_3_3f78dd62e99f55388e415a2e2fb86a3f._comment deleted file mode 100644 index 39976671df..0000000000 --- a/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_3_3f78dd62e99f55388e415a2e2fb86a3f._comment +++ /dev/null @@ -1,35 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-05-05T17:05:32Z" - content=""" -The sync man page does mention fetching and merging: - - The sync process involves first committing any local changes to files - that have previously been added to the repository, - then fetching and merging the current branch - -Not that I really buy the argument that not mentioning use of some specific git -command means that it should bypass git configurations that apply to that -command. git commands often don't document other git commands that they use -as plumbing, for example. - -The output with this configuration is essentially: - - pull repo-b - fatal: Not possible to fast-forward, aborting. - failed - -So the user is told right there that sync tried to pull. -And the pull failed. Why? Something to do with fast-forward. While this would -be clearer if git mentioned the config that is making it require a fast-forward, -it does not seem an indecipherable behavior. - -And also, you have to have *globally* configured an option that prevents fast forwarding. -That is a major, massive change to git's behavior. git is going to be falling over on -pull all the time in many circumstances with the same error message. - -Similarly, if you had configured merge.verifySignatures globally, git-annex -sync would be likely to fail. All your arguments would seem to require I also -override that option. But that could be a big security hole. -"""]] diff --git a/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_4_1b7133c5915baabb3076e95f0ac9807a._comment b/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_4_1b7133c5915baabb3076e95f0ac9807a._comment deleted file mode 100644 index a3ad330a46..0000000000 --- a/doc/bugs/git_config_merge.ff__61__only_breaks_sync/comment_4_1b7133c5915baabb3076e95f0ac9807a._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-05-05T17:27:31Z" - content=""" -As to the "fatal: ambiguous argument" message, it happens in the same test -case w/o that global git config. - - [2020-05-05 13:28:27.204706608] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/master..refs/heads/synced/master","--pretty=%H","-n1"] - fatal: ambiguous argument 'refs/heads/master..refs/heads/synced/master': unknown revision or path not in the working tree. - -It's caused by sync being run with no master branch yet, and nothing to commit -to create one. refs/heads/synced/master does exist; it was synced from the -other repo. Fixed that message. -"""]] diff --git a/doc/bugs/hooks_permissions.mdwn b/doc/bugs/hooks_permissions.mdwn deleted file mode 100644 index 5c85bb6cd9..0000000000 --- a/doc/bugs/hooks_permissions.mdwn +++ /dev/null @@ -1,32 +0,0 @@ -### Please describe the problem. - -I'm using accessing to a repo with sshfs, but I noticed hooks do not get executed. I noticed the permissions of hooks created by git-annex were all set to 700 (while the default .sample hooks seem to be 755). Is it intended? - -### What steps will reproduce the problem? - -(example) -[[!format sh """ -$ git checkout 'adjusted/master(unlocked)' -Updating files: 100% (3499/3499), done. -Switched to branch 'adjusted/master(unlocked)' -hint: The '.git/hooks/post-checkout' hook was ignored because it's not set as executable. -hint: You can disable this warning with `git config advice.ignoredHook false`. -"""]] - -### What version of git-annex are you using? On what operating system? - -git-annex version: 8.20201103, on Debian sid - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -git-annex is super great :-) - -> That's a bug indeed! I've fixed it. -> -> I decided to not make re-running `git-annex init` fix up the file mode -> if the hook already existed with the wrong permissions. -> That seemed like probably a bad idea, because it's at least possible -> the user might intend to unset one of the x bits and git-annex should not -> get in the way of that. So, you should `chmod a+x` the hooks yourself. -> -> [[done]] --[[Joey]] diff --git a/doc/bugs/hooks_permissions/comment_1_8734faef6ef61c070c13881056f7490a._comment b/doc/bugs/hooks_permissions/comment_1_8734faef6ef61c070c13881056f7490a._comment deleted file mode 100644 index fe393e83f6..0000000000 --- a/doc/bugs/hooks_permissions/comment_1_8734faef6ef61c070c13881056f7490a._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="gueux" - avatar="http://cdn.libravatar.org/avatar/47e44a21505727b2d6bb5d88f0468f34" - subject="comment 1" - date="2020-11-18T10:54:56Z" - content=""" -Thanks for the fix! I've manually updated the permissions of the hooks to 755 but unfortunately they are still not executed in a repo mounted via sshfs. There seems to be a kind of protection feature (which I can understand, as we're not sure of what's inside a remote executable). - - - $ ls -l .git/hooks/post-checkout - .rwxr-xr-x 76 gueux 27 Feb 20:33 .git/hooks/post-checkout - $ .git/hooks/post-checkout - bash: .git/hooks/post-checkout: Permission denied -"""]] diff --git a/doc/bugs/httpalso_mode_expects_encryption.mdwn b/doc/bugs/httpalso_mode_expects_encryption.mdwn deleted file mode 100644 index 25557c434d..0000000000 --- a/doc/bugs/httpalso_mode_expects_encryption.mdwn +++ /dev/null @@ -1,44 +0,0 @@ -### Please describe the problem. - -Initializing a httpalso remote fails by requesting an encryption mode (but then rejecting it). - -### What steps will reproduce the problem? - -[[!format sh """ -git init test1 -cd test1 -dd if=/dev/urandom count=1000 of=testfile -git annex init -git annex add testfile -git commit -m "initial checkin" -cd .. -git clone test1 test2 -cd test2 -git annex init -git annex initremote --sameas=origin origin-http type=httpalso url=http://example.com/foo -"""]] - -That is the minimal example as I understand the [httpalso documentation](special_remotes/httpalso/). - -This gives an error message: - -``` -initremote origin-http -git-annex: Specify encryption=hybrid or encryption=none or encryption=pubkey or encryption=shared or encryption=sharedpubkey. -failed -git-annex: initremote: 1 failed -``` - -Adding `encryption=none` (or any other) gives `Unexpected parameters: encryption`. - -### What version of git-annex are you using? On what operating system? - -git annex 8.20200908 on current Debian sid (as packaged) - -### Have you had any luck using git-annex before? - -My first git-annex repository dates back 2010-12-21 ... so: yes. - -The introduction of httpalso simplifies the setup in the [annex-to-web](https://gitlab.com/chrysn/annex-to-web) server, which on one hand consumes them in order to pass out redirects, and on the other hand . - -> [[fixed|done]] to the extent this was a bug at all. --[[Joey]] diff --git a/doc/bugs/httpalso_mode_expects_encryption/comment_1_8efe8d6276a16ee339aaf5782b45c239._comment b/doc/bugs/httpalso_mode_expects_encryption/comment_1_8efe8d6276a16ee339aaf5782b45c239._comment deleted file mode 100644 index 320956094e..0000000000 --- a/doc/bugs/httpalso_mode_expects_encryption/comment_1_8efe8d6276a16ee339aaf5782b45c239._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-09-17T18:20:49Z" - content=""" -This only affects using with regular git remotes. -All special remotes have an encryption config, which it inherits, but -regular git remotes do not have that config. - -Note that, regular git remotes can be accessed via git-annex over http just -fine without using httpalso. - -It might make sense for httpalso to reject trying to be used with a regular -git remote. - -Or it could force the equivilant of encryption=none in that -case. I would expect it to work then, if the url pointed to the -.git/annex/objects directory. -"""]] diff --git a/doc/bugs/httpalso_mode_expects_encryption/comment_2_5f31ee4052f7b0203aa287b5ec9abcab._comment b/doc/bugs/httpalso_mode_expects_encryption/comment_2_5f31ee4052f7b0203aa287b5ec9abcab._comment deleted file mode 100644 index c534108aa0..0000000000 --- a/doc/bugs/httpalso_mode_expects_encryption/comment_2_5f31ee4052f7b0203aa287b5ec9abcab._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="https://christian.amsuess.com/chrysn" - nickname="chrysn" - avatar="http://christian.amsuess.com/avatar/c6c0d57d63ac88f3541522c4b21198c3c7169a665a2f2d733b4f78670322ffdc" - subject="Re: comment 1" - date="2020-09-17T20:42:10Z" - content=""" -In my use case, I'd actually declare access the regular remotes with exporttree=true -- because the HTTP server serves only the checkout, and does not even give access to the .git repository. For this case, I don't see how declaring a regular git HTTP remote would help. - -Apart from that, a declared httpalso remote on exporttree=no would also make the HTTP server available using enableremote. That is especially beneficial because git-annex on its own seems to be unable to derive the git-annex UUID for an HTTP git remote on a dumb server. (Serving only the files and running `git update-server-info`, I could do a `git fetch`, but an annex sync would always give \"Remote origin-via-http not usable by git-annex; setting annex-ignore\" until I set remote.origin-via-http.annex-uuid manually, and then fetching did work.) -"""]] diff --git a/doc/bugs/httpalso_mode_expects_encryption/comment_3_26bd32dcaae0fd2c5899a65f6c437e08._comment b/doc/bugs/httpalso_mode_expects_encryption/comment_3_26bd32dcaae0fd2c5899a65f6c437e08._comment deleted file mode 100644 index 52200db195..0000000000 --- a/doc/bugs/httpalso_mode_expects_encryption/comment_3_26bd32dcaae0fd2c5899a65f6c437e08._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-09-17T20:54:02Z" - content=""" -httpalso automatically enables exporttree=yes when the --sameas remote is -such a remote. The working tree of a regular git repository is not such -a remote, and that cannot possibly work. Use a directory special remote -with exporttree=yes and then httpalso will be able to access it, as is -documented. - -To access a git remote over http, git-annex needs to be able to download and -parse its .git/config to determine its annex.uuid. When the http server -and url are configured properly, that does work. - -"""]] diff --git a/doc/bugs/httpalso_mode_expects_encryption/comment_4_50edcdec6ba08fc739f5c2f4ba826191._comment b/doc/bugs/httpalso_mode_expects_encryption/comment_4_50edcdec6ba08fc739f5c2f4ba826191._comment deleted file mode 100644 index 80c1407eed..0000000000 --- a/doc/bugs/httpalso_mode_expects_encryption/comment_4_50edcdec6ba08fc739f5c2f4ba826191._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-09-29T17:52:03Z" - content=""" -Made it check if the inherited config has encryption and skip setting it -up. So if there is a special remote that does not support encryption -config, it won't fail this way. - -Come to think, at least tahoe does not support encryption config. -Although it seems very unlikely it would be useful combined with tahoe. - -Of course, it still won't, and cannot possibly work when combined with a -http export of a git working tree. But then the docs are clear it won't -work with everything. -"""]] diff --git a/doc/bugs/initremote_with_export_and_import_allowed_with_encryption.mdwn b/doc/bugs/initremote_with_export_and_import_allowed_with_encryption.mdwn deleted file mode 100644 index 8f1a6b0cd0..0000000000 --- a/doc/bugs/initremote_with_export_and_import_allowed_with_encryption.mdwn +++ /dev/null @@ -1,8 +0,0 @@ -This should be rejected, but currently succeeds: - - git annex initremote d type=directory directory=../d exporttree=yes importtree=yes encryption=shared - -There is code in adjustExportImportRemoteType, and I remember it used to -work. --[[Joey]] - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/initremote_with_export_and_import_allowed_with_encryption/comment_1_8d60ad477356f0661d8bf0a0945e7ed1._comment b/doc/bugs/initremote_with_export_and_import_allowed_with_encryption/comment_1_8d60ad477356f0661d8bf0a0945e7ed1._comment deleted file mode 100644 index 942a870fc8..0000000000 --- a/doc/bugs/initremote_with_export_and_import_allowed_with_encryption/comment_1_8d60ad477356f0661d8bf0a0945e7ed1._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="encryption=onlycreds" - date="2020-12-17T21:01:33Z" - content=""" -If [[todo/encrypt_only_the_credentials]] gets implemented, then one form of `encryption=` (`encryption=onlycreds`) would be ok for export remotes. -"""]] diff --git a/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected.mdwn b/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected.mdwn deleted file mode 100644 index 538de2fad0..0000000000 --- a/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected.mdwn +++ /dev/null @@ -1,38 +0,0 @@ -### Please describe the problem. - -*As of filing:* - -* [downloads.kitenet.net/git-annex/OSX/current/10.11_El_Capitan/git-annex.dmg.info](https://downloads.kitenet.net/git-annex/OSX/current/10.11_El_Capitan/git-annex.dmg.info) contains: `distributionVersion = "8.20200618"` -* whereas: git-annex 8.20200720.1 (and 8.20200720) are the only versions published since 8.20200617. (There was no release matching 8.20200618) - -*Expected behavior:* - -* `distributionVersion` and included `git-annex`'s version matches an announced released version - -### What steps will reproduce the problem? - -1. On macOS, download and deploy the current [git-annex.dmg](https://downloads.kitenet.net/git-annex/OSX/current/10.11_El_Capitan/git-annex.dmg) -2. Open a terminal session and enter `git-annex version` - -### What version of git-annex are you using? On what operating system? - -[[!format sh """ -git-annex version: 8.20200617-gd46490c17 - -$/usr/bin/sw_vers - ProductName: Mac OS X - ProductVersion: 10.12.6 - BuildVersion: 16G2136 -"""]] - -### Please provide any additional information below. - -* Assuming this is the same problem with the autobuilder described in [kite net OSX/current/ distribution is 7.20190913 but 7.20191009 expected](https://git-annex.branchable.com/bugs/kite_net_OSX__47__current__47___distribution_is_7.20190913_but_7.20191009_expected/) -* Please give it a look and let us know when we have a distributionVersion matching release notes. Would like to pickup the fixes described in 8.20200720(.1) news. - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Still rock solid maintaining the .dmgs and .pkgs for a 67GB [Munki](https://www.munki.org) repository. - -> [[done]] --[[Joey]] diff --git a/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected/comment_1_adbceacb2f27d3527273686fae1b68d2._comment b/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected/comment_1_adbceacb2f27d3527273686fae1b68d2._comment deleted file mode 100644 index bcb5201866..0000000000 --- a/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected/comment_1_adbceacb2f27d3527273686fae1b68d2._comment +++ /dev/null @@ -1,20 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-07-25T16:09:17Z" - content=""" -This is a near duplicate of a bug you filed before. I'm dubious about the -value of these bug reports. As I said before: - -The build version being a little behind is not uncommon when the -autobuilder has not been keeping up for whatever reason. I generally don't -block releases on autobuilds if it's the fault of the autobuilder and not -a bug in git-annex's build process or test suite, and just ship the latest -available build. - -But maybe I'm missing something about how problimatic the build being -slightly behind until the next release is for you? - -The autobuilder has since caught up again, so the daily -build is current. -"""]] diff --git a/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected/comment_2_e976cda6c4cc2103b2a83ce185a71f06._comment b/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected/comment_2_e976cda6c4cc2103b2a83ce185a71f06._comment deleted file mode 100644 index 9819a3040a..0000000000 --- a/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected/comment_2_e976cda6c4cc2103b2a83ce185a71f06._comment +++ /dev/null @@ -1,32 +0,0 @@ -[[!comment format=mdwn - username="leej" - avatar="http://cdn.libravatar.org/avatar/eb1c6bd57680f694fb4658388e6de4ed" - subject="comment 2" - date="2020-07-28T14:57:58Z" - content=""" -Yes, I've, too, had the feeling that I'm not getting something about your process. Thank you for your continued patience. Here’s a bit more about the assumptions under which I’ve been operating: - -So far, in 2020, my Autopkg instance has pulled down these builds and imported them into Munki for testing and internal distribution to clients. - -* 7.20191231 -> (build) -* 7.20200205 -> (build) -* 8.20200226 -> (documented git-annex release) -* 8.20200309 -> (documented git-annex release) -* 8.20200310 -> (build) -* 8.20200331 -> (build) -* 8.20200502 -> (build) -* 8.20200617 -> (documented git-annex release) -* 8.20200618 -> (build) - -If your intermediate builds that appear on kitenet (e.g., 20200618 between the announced releases 20200617 and 20200720) are, in fact, ship quality. And, there are no behavioral surprises introduced in these builds between public releases, then I would be happy to just accept the undocumented, intermediate build that Autopkg is finding whenever the version is incremented on the dmg on kitenet. - -However, if they are more works-in-progress, as I have been assuming, and have the potential for introducing yet undocumented changes, then out of caution, I would (continue to) wait for a build matching one of your public releases that you document with a news entry. - -Let me know your thoughts as to the quality of these intermediate builds. - -And, also, if there's a better way of letting you, or someone else, know simply that the macOS auto builder needs a nudge, other than cutting a bug, please let me know. - -Note finally: I’m still seeing distributionVersion = \"8.20200618\", distributionReleasedate = 2020-07-20 18:49:32.302161119 UTC up on downloads.kitenet - -As always, thank you for your help and the value you’ve put into this incredibly useful program. -"""]] diff --git a/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected/comment_3_afbc3f35d1d428f7acd45cf12cd15dbd._comment b/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected/comment_3_afbc3f35d1d428f7acd45cf12cd15dbd._comment deleted file mode 100644 index 1d0e7b45af..0000000000 --- a/doc/bugs/kite_net_OSX__47__current__47___distribution_is_8.20200618_but_8.20200720.1_expected/comment_3_afbc3f35d1d428f7acd45cf12cd15dbd._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-08-10T17:01:31Z" - content=""" -Autobuilds may contain changes that are not release quality yet. - -Anyway, the release OSX build will be caught up by the release today. -"""]] diff --git a/doc/bugs/make_debianstandalone_-_FTBFS.mdwn b/doc/bugs/make_debianstandalone_-_FTBFS.mdwn deleted file mode 100644 index c202f2046e..0000000000 --- a/doc/bugs/make_debianstandalone_-_FTBFS.mdwn +++ /dev/null @@ -1,105 +0,0 @@ -### Please describe the problem. - -``` -$> make debianstandalone -... -[644 of 645] Compiling CmdLine.GitAnnex ( CmdLine/GitAnnex.hs, dist/build/git-annex/git-annex-tmp/CmdLine/GitAnnex.o ) -[645 of 645] Compiling Main ( git-annex.hs, dist/build/git-annex/git-annex-tmp/Main.o ) -Linking dist/build/git-annex/git-annex ... -ghc --make Build/Standalone -Wall -fno-warn-tabs -[18 of 30] Compiling BuildInfo ( BuildInfo.hs, BuildInfo.o ) -[19 of 30] Compiling Build.BundledPrograms ( Build/BundledPrograms.hs, Build/BundledPrograms.o ) -[20 of 30] Compiling Utility.DebugLocks ( Utility/DebugLocks.hs, Utility/DebugLocks.o ) -[21 of 30] Compiling Utility.Env ( Utility/Env.hs, Utility/Env.o ) -[22 of 30] Compiling Utility.FileSize ( Utility/FileSize.hs, Utility/FileSize.o ) -[23 of 30] Compiling Utility.PartialPrelude ( Utility/PartialPrelude.hs, Utility/PartialPrelude.o ) -[24 of 30] Compiling Utility.Network ( Utility/Network.hs, Utility/Network.o ) -[25 of 30] Compiling Utility.Path.AbsRel ( Utility/Path/AbsRel.hs, Utility/Path/AbsRel.o ) -[26 of 30] Compiling Utility.LinuxMkLibs ( Utility/LinuxMkLibs.hs, Utility/LinuxMkLibs.o ) -[27 of 30] Compiling Common ( Common.hs, Common.o ) -[28 of 30] Compiling Utility.CopyFile ( Utility/CopyFile.hs, Utility/CopyFile.o ) -[29 of 30] Compiling Build.LinuxMkLibs ( Build/LinuxMkLibs.hs, Build/LinuxMkLibs.o ) - -Build/LinuxMkLibs.hs:45:41: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: String - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the second argument of ‘writeFile’, namely - ‘(parentDir $ head gconvlibs)’ - In a stmt of a 'do' block: - writeFile (top "gconvdir") (parentDir $ head gconvlibs) - In the expression: - do fs <- dirContentsRecursive top - exes <- filterM checkExe fs - libs <- parseLdd <$> readProcess "ldd" exes - glibclibs <- glibcLibs - .... - | -45 | writeFile (top "gconvdir") (parentDir $ Prelude.head gconvlibs) - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Build/LinuxMkLibs.hs:45:53: error: - • Couldn't match type ‘[Char]’ - with ‘Data.ByteString.Internal.ByteString’ - Expected type: System.Posix.ByteString.FilePath.RawFilePath - Actual type: FilePath - • In the second argument of ‘($)’, namely ‘head gconvlibs’ - In the second argument of ‘writeFile’, namely - ‘(parentDir $ head gconvlibs)’ - In a stmt of a 'do' block: - writeFile (top "gconvdir") (parentDir $ head gconvlibs) - | -45 | writeFile (top "gconvdir") (parentDir $ Prelude.head gconvlibs) - | ^^^^^^^^^^^^^^^^^^^^^^ - -Build/LinuxMkLibs.hs:110:17: error: - • Variable not in scope: - relPathDirToFile :: FilePath -> [Char] -> IO FilePath - • Perhaps you meant ‘relPathDirToFileAbs’ (imported from Utility.Path) - | -110 | link <- relPathDirToFile (top exedir) (top ++ linker) - | ^^^^^^^^^^^^^^^^ - -Build/LinuxMkLibs.hs:132:31: error: - • Couldn't match type ‘Data.ByteString.Internal.ByteString’ - with ‘[Char]’ - Expected type: FilePath - Actual type: System.Posix.ByteString.FilePath.RawFilePath - • In the second argument of ‘($)’, namely ‘parentDir f’ - In the expression: inTop top $ parentDir f - In an equation for ‘destdir’: destdir = inTop top $ parentDir f - | -132 | destdir = inTop top $ parentDir f - | ^^^^^^^^^^^ - -Build/LinuxMkLibs.hs:132:41: error: - • Couldn't match type ‘[Char]’ - with ‘Data.ByteString.Internal.ByteString’ - Expected type: System.Posix.ByteString.FilePath.RawFilePath - Actual type: FilePath - • In the first argument of ‘parentDir’, namely ‘f’ - In the second argument of ‘($)’, namely ‘parentDir f’ - In the expression: inTop top $ parentDir f - | -132 | destdir = inTop top $ parentDir f - | ^ -make[4]: *** [Makefile:165: Build/Standalone] Error 1 -make[4]: Leaving directory '/home/yoh/proj/git-annex' -make[3]: *** [Makefile:173: linuxstandalone] Error 2 -make[3]: Leaving directory '/home/yoh/proj/git-annex' -make[2]: *** [debian/rules:24: override_dh_auto_build] Error 2 -make[2]: Leaving directory '/home/yoh/proj/git-annex' -make[1]: *** [debian/rules:17: build] Error 2 -make[1]: Leaving directory '/home/yoh/proj/git-annex' -dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2 -make: *** [Makefile:205: dpkg-buildpackage-F] Error 2 -make debianstandalone 289.29s user 9.60s system 99% cpu 5:00.49 total - - -$> git describe -8.20201103-34-gd8e8d145e -``` -``` - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/make_debianstandalone_-_FTBFS/comment_1_a746750dd3dbca7c7ca087f5cf0f2a7e._comment b/doc/bugs/make_debianstandalone_-_FTBFS/comment_1_a746750dd3dbca7c7ca087f5cf0f2a7e._comment deleted file mode 100644 index abf82dd98c..0000000000 --- a/doc/bugs/make_debianstandalone_-_FTBFS/comment_1_a746750dd3dbca7c7ca087f5cf0f2a7e._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 1" - date="2020-11-07T23:59:51Z" - content=""" -didn't look in detail but entire dashboard is largely red: https://git-annex.branchable.com/builds/ (only Linux armel building some older I guess commitish is Warning only). -"""]] diff --git a/doc/bugs/move_violates_numcopies___40__regression__41__.mdwn b/doc/bugs/move_violates_numcopies___40__regression__41__.mdwn deleted file mode 100644 index 9ab92e08de..0000000000 --- a/doc/bugs/move_violates_numcopies___40__regression__41__.mdwn +++ /dev/null @@ -1,58 +0,0 @@ -### Please describe the problem. - -Using `git annex move` will reduce the number of copies below `numcopies`. - -Relevant links: - -- [original bug report](https://git-annex.branchable.com/bugs/move_violates_numcopies/) -- [devblog day 496: move numcopies safety revisited](https://git-annex.branchable.com/devblog/day_496__move_numcopies_safety_revisited/) - -### What steps will reproduce the problem? - - ╭─@ringo ~/backup/annex/test-drop ‹python-2.7.16› ‹master•1b16944› - ╰─ git annex whereis - whereis 1403415.jpg (2 copies) - 766c9221-1404-4d68-bfa6-46d064bf4798 -- ting@ringo:/Volumes/apple/backup/annex/test-drop [apple] - 9cfda232-8504-4c3b-b694-4a6c85dc6c9c -- ting@ringo:~/backup/annex/test-drop [ringo] [here] - ok - ╭─@ringo ~/backup/annex/test-drop ‹python-2.7.16› ‹master•1b16944› - ╰─ g annex numcopies - 2 - ╭─@ringo ~/backup/annex/test-drop ‹python-2.7.16› ‹master•1b16944› - ╰─ g annex move 1403415.jpg --to=apple - move 1403415.jpg ok - (recording state in git...) - ╭─@ringo ~/backup/annex/test-drop ‹python-2.7.16› ‹master•1b16944› - ╰─ git annex whereis - whereis 1403415.jpg (1 copy) - 766c9221-1404-4d68-bfa6-46d064bf4798 -- ting@ringo:/Volumes/apple/backup/annex/test-drop [apple] - ok - - - -### What version of git-annex are you using? On what operating system? - -`git-annex` installed via Homebrew on OS X. Version: 8.20201103 - - ╭─@ringo ~/backup/annex/test-drop ‹python-2.7.16› ‹master•1b16944› - ╰─ git annex version - git-annex version: 8.20201103 - build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Feeds Testsuite S3 WebDAV - dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.2.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0 - key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* - remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external - operating system: darwin x86_64 - supported repository versions: 8 - upgrade supported from repository versions: 0 1 2 3 4 5 6 7 - local repository version: 8 - -### Please provide any additional information below. - - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I'm really excited about using git-annex, and have spent 3-4 days reading documentation, past bug reports, setting up repos, transferring data, and manually testing failure cases (e.g. this one :p). - -Thanks for all your hard work! - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/move_violates_numcopies___40__regression__41__/comment_1_c2857172eba97c0ae6c829c634aadd15._comment b/doc/bugs/move_violates_numcopies___40__regression__41__/comment_1_c2857172eba97c0ae6c829c634aadd15._comment deleted file mode 100644 index 7281ad07bb..0000000000 --- a/doc/bugs/move_violates_numcopies___40__regression__41__/comment_1_c2857172eba97c0ae6c829c634aadd15._comment +++ /dev/null @@ -1,40 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="comment 1" - date="2020-11-13T16:36:45Z" - content=""" -I can trigger this on my end. - -With the script below, it bisects [*] to 0133b7e5a (move: Improve -resuming a move that was interrupted after the object was transferred, -2020-10-21), which was included in the 8.20201103 release. - - [*] Well, almost. I couldn't build the two commits right before - 0133b7e5a (363acfb55 and 62d630272). - -[[!format sh \"\"\" -cd \"$(mktemp -d \"${TMPDIR:-/tmp}\"/gx-XXXXXXX)\" || exit 125 -git annex version -git init a -( - cd a - echo one >one - git annex init a - git annex add one - git commit -mone -) -git clone -o a a b -( - cd b - git annex init b - git annex get one - git annex numcopies 2 - git annex numcopies - git annex whereis -) -git -C b annex move --to=a one || exit 0 -exit 1 -\"\"\"]] - -"""]] diff --git a/doc/bugs/move_violates_numcopies___40__regression__41__/comment_2_98a0c583ed094fec0e8c666e5839c2f7._comment b/doc/bugs/move_violates_numcopies___40__regression__41__/comment_2_98a0c583ed094fec0e8c666e5839c2f7._comment deleted file mode 100644 index a34630e89b..0000000000 --- a/doc/bugs/move_violates_numcopies___40__regression__41__/comment_2_98a0c583ed094fec0e8c666e5839c2f7._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-11-13T18:01:29Z" - content=""" -Oh what a dumb mistake, literally a copied "False" that should be "True". - -I'm going to move the next release up due to this regression, but -thankfully it doesn't result in data loss, so am not going to do a crash -priority release. Probably Monday. - -The test suite also has a test case for move and numcopies now. It used to be -ok for it to only test drop and numcopies, but that changed a while back -and it should have had one already. -"""]] diff --git a/doc/bugs/move_violates_numcopies___40__regression__41__/comment_3_b8b263650c3580763101c741978c81a4._comment b/doc/bugs/move_violates_numcopies___40__regression__41__/comment_3_b8b263650c3580763101c741978c81a4._comment deleted file mode 100644 index fb734520a0..0000000000 --- a/doc/bugs/move_violates_numcopies___40__regression__41__/comment_3_b8b263650c3580763101c741978c81a4._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="wting" - avatar="http://cdn.libravatar.org/avatar/ff65dc4f3eb58490220b89a7984c4d02" - subject="comment 3" - date="2020-11-13T23:08:26Z" - content=""" -Thanks everyone for the quick response and turnaround on this! -"""]] diff --git a/doc/bugs/networkbsd_flag_default_differs_in_stack.yaml_and_git-annex.cabal.mdwn b/doc/bugs/networkbsd_flag_default_differs_in_stack.yaml_and_git-annex.cabal.mdwn deleted file mode 100644 index 20e1d33c6d..0000000000 --- a/doc/bugs/networkbsd_flag_default_differs_in_stack.yaml_and_git-annex.cabal.mdwn +++ /dev/null @@ -1,3 +0,0 @@ -Not sure it's a bug, but the networkbsd flag defaults to true in git-annex.cabal and to false in stack.yaml -- is that intended? - -> not a bug, [[done]] --[[Joey]] diff --git a/doc/bugs/networkbsd_flag_default_differs_in_stack.yaml_and_git-annex.cabal/comment_1_73e6442cc842f21d6f26e20041d7f88d._comment b/doc/bugs/networkbsd_flag_default_differs_in_stack.yaml_and_git-annex.cabal/comment_1_73e6442cc842f21d6f26e20041d7f88d._comment deleted file mode 100644 index 38dedf53ce..0000000000 --- a/doc/bugs/networkbsd_flag_default_differs_in_stack.yaml_and_git-annex.cabal/comment_1_73e6442cc842f21d6f26e20041d7f88d._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-16T17:14:20Z" - content=""" -Yes. When cabal is used, it will unset the flag as -necessary if an older version of network is available. Meanwhile, -stack uses lts-14.27, which includes network-2.x, and somehow stack -forces the networkbsd flag to not be unset in that case. (I don't know -how/why.) -"""]] diff --git a/doc/bugs/new_merge.directoryRenames_behavior_breaks_sync_merge_conflict_resolution.mdwn b/doc/bugs/new_merge.directoryRenames_behavior_breaks_sync_merge_conflict_resolution.mdwn deleted file mode 100644 index afc2eae8f3..0000000000 --- a/doc/bugs/new_merge.directoryRenames_behavior_breaks_sync_merge_conflict_resolution.mdwn +++ /dev/null @@ -1,50 +0,0 @@ -git has a new feature: - - CONFLICT (file location): x/foo/bar.ext added in refs/remotes/whatever/master inside a directory that was renamed in HEAD, suggesting it should perhaps be moved to y/bar.ext. - -When this happens during a git-annex sync, the result is: - - y/bar.variant-nnnn.ext - y/bar.ext - -With the .variant file checked in, but the other one not checked in. - -Which is problimatic both because the file does not get checked in, and -because if this is a merge conflict at all (I disagree with git that it -should be, didn't used to be), it's a conflict between x/foo/bar.ext and -y/bar.ext and so git-annex sync should presumably make those two into -.variant files, not 2 files in the same directory. Although, it might be -better for git-annex not to make variant files at all in this situation, -since both files have the same content. - -The worse problem, perhaps, is that git-annex sync succeeds and if the user -was not expecting this behavior, it can seem like their file has been lost, -because it was moved to a directory they did not move it to! -git merge is relying on this being a merge conflict for the user to -see the error message and know where it renamed their file to, but this is -counter to sync's desire to avoid merge conflicts for users who are not -able to deal with that complication. - -This new git feature does have a config. Set merge.directoryRenames=false to -disable it. Or, merge.directoryRenames=true will avoid it being treated as -a merge conflict. (The config was added in 2.22, while git started doing -this in 2.18. Annoyingly, Debian stable shipped git 2.20 so produces conflicts -with no way to turn it off.) One option would be for sync to set one -of the two if it's not set (probably =false), and if it is set to =conflict -(or the git version lacks the config), to deal with the conflict. - -Based on git commit 8c8e5bd6eb331d055aa7fa6345f6dcdadd658979, -it uses a higher than usual stage level in the index for the conflict -produced by case, so it might be possible for git-annex to use that -to detect it and handle it specially. - ---[[Joey]] - -> Fixed the file not being checked in. In fact, it needed to be cleaned up. -> -> That avoids part of the problem. To avoid the surprising rename, -> git-annex sync should set merge.directoryRenames=false when -> merge.directoryRenames is not configured, unless automatic merge conflict -> resolution is disabled. --[[Joey]] -> -> > [[done]] --[[Joey]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs.mdwn b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs.mdwn deleted file mode 100644 index 07ecbb9f69..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs.mdwn +++ /dev/null @@ -1,58 +0,0 @@ -### Please describe the problem. - -See "test-annex (crippled-home)" run on [datalad-extensions github actions](https://github.com/datalad/datalad-extensions/pull/15/checks?check_run_id=758524896). yes -- I really hate all those scrollbars etc, but you should be able also to click on "..." -> "View raw logs" which would lead to one [large log](https://pipelines.actions.githubusercontent.com/2UPlDxaVvvbkeFX4btxWorCjpJvj40zvWY5ogH2yZibhOMcU7O/_apis/pipelines/1/runs/745/signedlogcontent/14?urlExpires=2020-06-11T14%3A42%3A25.6501110Z&urlSigningMethod=HMACV1&urlSignature=Kmm%2BTBYZt5jzojQgrDTOgSrVjYq8VgUHLd3sUtFJd0c%3D) - -in which you can find - -[[!format sh """ -2020-06-10T16:02:40.4507392Z Detected a filesystem without fifo support. -2020-06-10T16:02:40.4508127Z Disabling ssh connection caching. -2020-06-10T16:02:40.4579131Z Detected a crippled filesystem. -2020-06-10T16:02:40.6831961Z export_import: [adjusted/master(unlocked) 4c3ac42] empty -2020-06-10T16:02:40.7515700Z adjust ok -2020-06-10T16:02:40.8085540Z initremote foo ok -2020-06-10T16:02:40.8152161Z (recording state in git...) -2020-06-10T16:02:40.8811878Z get foo (from origin...) -2020-06-10T16:02:40.9178237Z -2020-06-10T16:02:40.9190085Z 100% 20 B 548 B/s 0s -2020-06-10T16:02:40.9201375Z -2020-06-10T16:02:40.9240907Z (checksum...) ok -2020-06-10T16:02:40.9261455Z get sha1foo (from origin...) -2020-06-10T16:02:40.9325841Z -2020-06-10T16:02:40.9334418Z 100% 25 B 4 KiB/s 0s -2020-06-10T16:02:40.9336072Z -2020-06-10T16:02:40.9405494Z (checksum...) ok -2020-06-10T16:02:40.9406396Z (recording state in git...) -2020-06-10T16:02:41.1056415Z commit -2020-06-10T16:02:41.2164981Z On branch adjusted/master(unlocked) -2020-06-10T16:02:41.2165985Z Your branch is ahead of 'origin/adjusted/master(unlocked)' by 1 commit. -2020-06-10T16:02:41.2166158Z (use "git push" to publish your local commits) -2020-06-10T16:02:41.2166244Z -2020-06-10T16:02:41.2166353Z nothing to commit, working tree clean -2020-06-10T16:02:41.2166474Z ok -2020-06-10T16:02:41.3970703Z export foo bar.c ok -2020-06-10T16:02:41.4080765Z export foo foo ok -2020-06-10T16:02:41.4123344Z export foo sha1foo ok -2020-06-10T16:02:41.4301472Z (recording state in git...) -2020-06-10T16:02:41.4804732Z list foo ok -2020-06-10T16:02:41.4913408Z import foo import -2020-06-10T16:02:41.4915106Z -2020-06-10T16:02:41.4916599Z .git/annex/tmp/CID-s16--24892 16 1591804960 : openBinaryFile: invalid argument (Invalid argument) -2020-06-10T16:02:41.4916699Z -2020-06-10T16:02:41.4917067Z -2020-06-10T16:02:41.4917254Z ok -2020-06-10T16:02:41.4924772Z -2020-06-10T16:02:41.4925784Z Failed to import some files from foo. Re-run command to resume import. -2020-06-10T16:02:41.5156287Z merge foo/master ok -2020-06-10T16:02:41.5276712Z FAIL -"""]] - -overall it seems to be only this test failing: 1 out of 702 tests failed (198.43s) - -git annex is 8.20200522+git142-g9102d3172-1~ndall and .deb available within the .zip within "Artifacts" drop down on the top. - -details of the setup are [in that PR](https://github.com/datalad/datalad-extensions/pull/15/files#diff-8364c688b76bfaf5df947cfd4d74eef7R76) - -PS determining the boundaries and names of the tests git annex had ran is a tricky business on its own -- I wondered if tests output formatting and annotation could have been improved as well. E.g. unlikely there is a point to print all output if test passes. With `nose` in Python / datalad we get a summary of all failed tests (and what was output when they were ran) at the end of the full sweep. That helps to avoid needing to search the entire long list - -> [[done]] --[[Joey]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_10_0b41fe04a5d820dbb17412eaa2ea8760._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_10_0b41fe04a5d820dbb17412eaa2ea8760._comment deleted file mode 100644 index 4c0caa0b33..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_10_0b41fe04a5d820dbb17412eaa2ea8760._comment +++ /dev/null @@ -1,33 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 10""" - date="2020-06-17T19:41:29Z" - content=""" -Hmm, implemented all that, it did fix my shell test case. - -But, the test suite is still hanging in the same place when run on vfat. - -stracing the git-annex smudge command that is hung: - - fcntl(0, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0} - -fd 0 is .git/annex/gitqueue.lck - -So hmm, this seems like a different problem than the one I fixed. -It's not involving pid locking. - -The parent git-annex also has the same lock file open, and presumably -locked exclusively. So it makes sense it would deadlock, both the parent -and child trying to lock the same lock. It's kind of odd that this only -happens on vfat. I don't think this is because of vfat's limitations -though. It's a legitimate deadlock. - -I get the feeling that gitAnnexGitQueueLock needs to be taken in a more -fine-grained way. Or Annex.Link should not be using the git queue -for what it does, which is not really a queue in the same way Git.Queue is -usually used. If Annex.Link instead deferred the action until cleanup in -some other way, it would not take gitAnnexGitQueueLock and so when it runs -smudge it would not matter if that did take the lock. - -Unfortunately I'm out of time to fix it before today's planned release. -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_11_1147b988f7b770b4ce4d077bb10a9760._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_11_1147b988f7b770b4ce4d077bb10a9760._comment deleted file mode 100644 index ffc9f57ee9..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_11_1147b988f7b770b4ce4d077bb10a9760._comment +++ /dev/null @@ -1,20 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 11""" - date="2020-06-18T15:52:10Z" - content=""" -git-annex smudge is locking the git queue in order to run -restagePointerFile. - -And of course, restagePointerFile has run git update-index, which has run -git-annex smudge. - -Normally git-annex smudge avoids this loop. In this case, -Database.Keys.reconcileStaged is what calls restagePointerFile -(for reasons explained in [[!commit 50fa17aee64ae778eb071cc21d5be85ce969bd9b]]). -So, there needs to be something to prevent restagePointerFile from running -in git-annex smudge, no matter how it's called. - -Also, I suspect that vfat's timestamp granularity is exposing the problem. -On other filesystems, I don't see git-annex smudge calling restagePointerFile. -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_12_9a0d303dee589c28139374fe3047b849._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_12_9a0d303dee589c28139374fe3047b849._comment deleted file mode 100644 index 07aab1c52d..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_12_9a0d303dee589c28139374fe3047b849._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 12""" - date="2020-06-18T17:04:50Z" - content=""" -I've fixed the deadlock. - -Test suite succeeds on vfat. -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_13_622060abe39191491f9f7b0bde497775._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_13_622060abe39191491f9f7b0bde497775._comment deleted file mode 100644 index 473fa72430..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_13_622060abe39191491f9f7b0bde497775._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 13" - date="2020-06-18T19:17:26Z" - content=""" -awesome -- thanks! I am confirming that all \"crippledfs\" tests pass on https://github.com/datalad/datalad-extensions/pull/15/checks?check_run_id=785585888 ! (datalad tests are still running ATM but I guess they will hang in datalad \"master\" as before anyways - unrelated to this issue) -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_1_5e8f095d89689e3dacc3376b19be207c._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_1_5e8f095d89689e3dacc3376b19be207c._comment deleted file mode 100644 index 4befe2fed7..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_1_5e8f095d89689e3dacc3376b19be207c._comment +++ /dev/null @@ -1,23 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-06-11T19:46:25Z" - content=""" -Is this about HOME being on the FS, or about git-annex test being run -inside the FS? - -It looks to me to be the latter, since the file it fails -on is in .git/annex/tmp - - 2020-06-10T16:02:41.4916599Z .git/annex/tmp/CID-s16--24892 16 1591804960 : openBinaryFile: invalid argument (Invalid argument) - -Might this filesystem have trouble with filenames with spaces, -particularly trailing spaces? That's kind of weird, and I've changed it so -it won't contain those spaces. - -Also this line just after the FAIL seems relevant. (I suspect -the log might not have every line in the right order, like maybe stdout and -stderr are being read independantly and written to the log.) - - 2020-06-10T16:02:41.5278208Z Exception: import: getSymbolicLinkStatus: does not exist (No such file or directory) -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_2_40a890ce20c7f58a912acbdf3e483292._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_2_40a890ce20c7f58a912acbdf3e483292._comment deleted file mode 100644 index 4868931ef3..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_2_40a890ce20c7f58a912acbdf3e483292._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 2" - date="2020-06-11T21:13:38Z" - content=""" -> Is this about HOME being on the FS, or about git-annex test being run inside the FS? - -Both. In [that script](https://github.com/datalad/datalad-extensions/pull/15/files#diff-8364c688b76bfaf5df947cfd4d74eef7R76) I do - -[[!format sh \"\"\" -export HOME=/crippledfs -cd $HOME -git annex test -\"\"\"]] - -while TMPDIR was **not** set to point to `/crippledfs` -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_3_f71db95be71186fe41f3ffb34b25595f._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_3_f71db95be71186fe41f3ffb34b25595f._comment deleted file mode 100644 index 846e196532..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_3_f71db95be71186fe41f3ffb34b25595f._comment +++ /dev/null @@ -1,28 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="stalls" - date="2020-06-12T03:40:54Z" - content=""" -Testing 8.20200522+git156-gc8ff3e082-1~ndall+1 leads to a stall with HOME on crippled fs. Unfortunately I didn't store that log, restarted -- passed - -Downloaded and installed .deb from the artifacts of https://github.com/datalad/datalad-extensions/pull/15/checks?check_run_id=763826515, set crippled fs the same way as on github actions -- passed on the first try, and stalled on the next one: - -[[!format sh \"\"\" -pull origin -ok -push origin -To /mnt/crippledfs/.t/main2 - b9a7a37..4c439c8 git-annex -> synced/git-annex -ok -OK (4.24s) - concurrent get of dup key regression: Detected a filesystem without fifo support. - Disabling ssh connection caching. - Detected a crippled filesystem. -[adjusted/master(unlocked) c8e63f5] empty -adjust ok -get foo (from origin...) (checksum...) ok -(recording state in git...) - -\"\"\"]] -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_4_6d191ed804ff61de3a1756233fce3f45._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_4_6d191ed804ff61de3a1756233fce3f45._comment deleted file mode 100644 index 0f9bdd0a8d..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_4_6d191ed804ff61de3a1756233fce3f45._comment +++ /dev/null @@ -1,37 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-06-16T18:05:31Z" - content=""" -By the time this stall happens, the test suite has gotten past the failing -`export_import` test. It would be good to know if that test now succeeds. - -Did you wait on the stall long enough for annex.pidlocktimeout to expire? -(Default 300 seconds.) - -I tried this to reproduce it, which did not here. Would be interesting to -know if it does on the filesystem in question. - - git init a - cd a - git annex init - git config annex.pidlock true - date > foo - git annex add - cp -a foo bar - git add bar - git commit -m add - cd .. - git clone a b - cd b - git annex init - git config annex.pidlock true - git annex get -J1 foo bar - git annex drop -J1 - cp -a foo baz - git add baz - git annex get -J2 foo bar baz - git annex drop -J2 - -That should be identical to what the test case does. -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_5_dae646dea279e9f60d7e853a155a3828._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_5_dae646dea279e9f60d7e853a155a3828._comment deleted file mode 100644 index 69fc33aeaa..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_5_dae646dea279e9f60d7e853a155a3828._comment +++ /dev/null @@ -1,69 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 5" - date="2020-06-16T18:30:07Z" - content=""" -
-Example does hang for me when ran in that crippled fs, even without setting $HOME - -```shell -> git init a -Initialized empty Git repository in /mnt/crippledfs/a/.git/ -> cd a -> git annex init -init - Detected a filesystem without fifo support. - - Disabling ssh connection caching. - - Detected a crippled filesystem. -(scanning for unlocked files...) - - Entering an adjusted branch where files are unlocked as this filesystem does not support locked files. - -Switched to branch 'adjusted/master(unlocked)' -ok -(recording state in git...) -> git config annex.pidlock true -> date -> git annex add -add foo -ok -(recording state in git...) -> cp -a foo bar -> git add bar -> git commit -m add -[adjusted/master(unlocked) 8dc937d] add - 2 files changed, 2 insertions(+) - create mode 100644 bar - create mode 100755 foo -> cd .. -> git clone a b -Cloning into 'b'... -done. -> cd b -> git annex init -init - Detected a filesystem without fifo support. - - Disabling ssh connection caching. - - Detected a crippled filesystem. -(merging origin/git-annex into git-annex...) -(recording state in git...) -(scanning for unlocked files...) -ok -(recording state in git...) -> git config annex.pidlock true -> git annex get -J1 foo bar -get foo (from origin...) (checksum...) ok -(recording state in git...) - -``` -
- -> I tried this to reproduce it, which did not here. - -have you used the [script I use](https://github.com/datalad/datalad-extensions/pull/15/files#diff-0785403f62dbda964724769c173d0dc5)? -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_6_c2f4201da9c58740f46d87f8145255ed._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_6_c2f4201da9c58740f46d87f8145255ed._comment deleted file mode 100644 index 8531969fc9..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_6_c2f4201da9c58740f46d87f8145255ed._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 6""" - date="2020-06-16T18:40:38Z" - content=""" -Ah, I missed that the script for this crippled FS is just making a vfat. - -git-annex test has definitely succeeded on vfat in the past. -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_7_e96712bc3e65006d2f0778d23daf0beb._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_7_e96712bc3e65006d2f0778d23daf0beb._comment deleted file mode 100644 index 307b482b7f..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_7_e96712bc3e65006d2f0778d23daf0beb._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 7""" - date="2020-06-17T16:11:37Z" - content=""" -Confirmed that, on vfat, "foo " is not a legal filename. News to me. - -So, the original failing test case was indeed caused by that, and has -been fixed. -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_8_1be9990d0cab0e69936de356072ea890._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_8_1be9990d0cab0e69936de356072ea890._comment deleted file mode 100644 index 65ddb75f24..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_8_1be9990d0cab0e69936de356072ea890._comment +++ /dev/null @@ -1,50 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 8""" - date="2020-06-17T16:16:19Z" - content=""" -Minimal test case for the hang: - - git init a - cd a - git annex init - git annex adjust --unlock - git config annex.pidlock true - date > foo - git annex add --force - git commit -m add - cd .. - git clone a b - cd b - git annex init - git annex adjust --unlock - git config annex.pidlock true - git annex get foo --force - -That does not need vfat to hang. - - 364479 pts/2 Sl+ 0:00 | \_ /home/joey/bin/git-annex get foo - 364504 pts/2 Sl+ 0:00 | \_ git --git-dir=.git --work-tree=. --literal-pathspecs -c core.safecrlf=false update-index -q --refresh -z --stdin - 364506 pts/2 S+ 0:00 | \_ /bin/sh -c git-annex smudge --clean -- 'foo' git-annex smudge --clean -- 'foo' - 364507 pts/2 Sl+ 0:00 | \_ git-annex smudge --clean -- foo - -So is git-annex smudge waiting on the pidlock its parent has? - -Yes: After setting annex.pidlocktimeout 2: - - 2 second timeout exceeded while waiting for pid lock file .git/annex/pidlock - git-annex: Gave up waiting for possibly stale pid lock file .git/annex/pidlock - error: external filter 'git-annex smudge --clean -- %f' failed 1 - -What I'm not sure about: annex.pidlock is not set by default on vfat, -so why would the test suite have failed there, and intermittently? -Maybe annex.pidlock does get set in some circumstances? - -Anyway, there's a clear problem that annex.pidlock prevents more than 1 git-annex -process that uses locking from running, and here we have a parent git-annex -that necessarily runs a child git-annex that does locking. - -Could the child process check if a parent/grandparent has the pid lock held -and so safely skip taking it? Or do all places git-annex runs itself -have to shut down pid locking? -"""]] diff --git a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_9_cf253f0ad3f9857b9f0746e678d8dbd8._comment b/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_9_cf253f0ad3f9857b9f0746e678d8dbd8._comment deleted file mode 100644 index f29b477fc7..0000000000 --- a/doc/bugs/one_annex_test_FAILs_when_HOME_is_a_crippled_fs/comment_9_cf253f0ad3f9857b9f0746e678d8dbd8._comment +++ /dev/null @@ -1,50 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 9""" - date="2020-06-17T17:04:57Z" - content=""" -Oh this is tricky.. git-annex is taking the gitAnnexGitQueueLock -while running the queued git update-index command. Which is the command -that then runs git-annex. - -So dropping the pid lock before running the command won't work. - -If pid locks were fine grained, this would not be a problem because the -child process is really locking a different resource than its grandparent. - -But, I think the reasons for not making them fine grained do make sense: -Since git-annex sometimes takes a posix lock on a content file, it would -need to use some other lock file for the pid lock. So every place that -uses a lock file would have to specifiy the filename to use for pid -locking. Which makes pid locking complicate the rest of the code base -quite a lot, and every code path involving locking would have to be tested -twice, in order to check that the pid lock file used by that lock works. -Doubling the complexity of your file locking is a good thing to avoid. - -Hmm.. I said "the child process is really locking a different resource than -its grandparent". And that generally has to be the case, or people using -git-annex w/o pid locking would notice that hey, these child processes -fail to take a lock and crash. - -So.. If we assume that is the case, and that there are plenty of git-annex -users not using pid locking, then there's no need for a child process -to take the pid lock, if its parent currently has the pid lock held, -and will keep it held. - -And this could be communicated via an env var. When the pid lock is taken -set `ANNEX_PIDLOCKED`, and unset when it's dropped. Then, so long -as childen inherit that env variable, they can skip taking the pid lock when -it's set. - -To make sure that's safe, any time git-annex runs a child process -(or a git command that runs git-annex), it ought to hold the pid lock -while doing it. Holding any lock will do. The risk is, if one thread -has some lock held for whatever reason, and another thread runs the child -process, then the child process will rely on the unrelated thread keeping -the lock held. Explicitly holding some lock avoids such a scenario. - -So, let's make it more explicit, add a runsGitAnnex that, when pid locking -is enabled, holds the pid lock while running the action. Then that has to -be wrapped around any places where a git-annex child process is run, -which can be done broadly, or just as these issues come up. -"""]] diff --git a/doc/bugs/p2p_auth_token_can_only_be_retrieved_at_generation_time.mdwn b/doc/bugs/p2p_auth_token_can_only_be_retrieved_at_generation_time.mdwn deleted file mode 100644 index 5d0eeecd46..0000000000 --- a/doc/bugs/p2p_auth_token_can_only_be_retrieved_at_generation_time.mdwn +++ /dev/null @@ -1,12 +0,0 @@ -### Please describe the problem. - -`git annex p2p --gen-address` creates an auth token and than immediately prints it out so that it can be used to pair with another machine. - -But what am I supposed to do if I want to pair a third machine later? I do not want to call `--gen-address` again since it probably creates another token or even worse overwrites the token already used for one pairing. - -So we lack a command like `git annex p2p --show-address` that just prints the same as `--gen-address`. This is trivial since we just need to concatenate `.git/annex/creds/p2paddrs`, a colon and `.git/annex/creds/p2pauth`? - -Actually this would be a fun starter project for a new contributor to git-annex? (me?) - -> I added a mention to the docs that --gen-address can be run more than -> once, so [[done]] I suppose --[[Joey]] diff --git a/doc/bugs/p2p_auth_token_can_only_be_retrieved_at_generation_time/comment_1_53132295395cb3e504b55c90bc705518._comment b/doc/bugs/p2p_auth_token_can_only_be_retrieved_at_generation_time/comment_1_53132295395cb3e504b55c90bc705518._comment deleted file mode 100644 index 1878278469..0000000000 --- a/doc/bugs/p2p_auth_token_can_only_be_retrieved_at_generation_time/comment_1_53132295395cb3e504b55c90bc705518._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-16T17:20:10Z" - content=""" -You can run git-annex p2p --gen-addresses repeatedly. Each address -can be given to one entity you want to grant access to the repository. - -That way, if you choose to revoke access later, you can revoke just one. - -Not that there's currently a command to revoke, but it just deletes -the address from .git/annex/creds/p2pauth. - -If you'd like to contribute, you could maybe improve the documentation -so it's clearer that --gen-addresses can be used repeatedly, -or add a --remove-address or something like that. -"""]] diff --git a/doc/bugs/p2p_protocol_misbehavior_when_location_log_out_of_date.mdwn b/doc/bugs/p2p_protocol_misbehavior_when_location_log_out_of_date.mdwn deleted file mode 100644 index 5e5aa74721..0000000000 --- a/doc/bugs/p2p_protocol_misbehavior_when_location_log_out_of_date.mdwn +++ /dev/null @@ -1,50 +0,0 @@ -With a ssh remote using the p2p protocol, git-annex move of 2 files, -with the first file no longer present in the remote (having moved -elsewhere) and the second file still, fails like this: - - move file1 (from remote...) - verification of content failed - failed - move file2 (from remote..) - Lost connection (fd:15: hGetChar: illegal operation (handle is closed)) - transfer failed - -The problem is on the first move, the protocol does not handle a file -that's not present well, so it's not clear why it failed. And since that -closes the connection, the next move fails when it should not need to. ---[[Joey]] - -> Protocol dump: -> -> P2P > VERSION 1 -> P2P < VERSION 1 -> P2P > GET 0 foo SHA256E-s30--f8f8766a836fb6120abf4d5328ce8761404e437529e997aaa0363bdd4fecd7bb -> P2P < GET 0 foo SHA256E-s30--f8f8766a836fb6120abf4d5328ce8761404e437529e997aaa0363bdd4fecd7bb -> P2P > DATA 0 -> P2P > VALID -> P2P < DATA 0 -> P2P > SUCCESS -> P2P < SUCCESS -> -> So it's sending an empty object and claiming it's valid when in -> fact it does not have the object. That was done because the sender -> does not have any other way in the protocol to indicate that. -> The receiver is what sends SUCCESS/FAILURE, not the sender.a -> -> Seems like, it could send DATA 0 followed by INVALID, -> to avoid needing to add to the protocol. That should avoid -> the spurious "verification of content failed". -> -> > Done and it did. -> -> But what causes the connection to get closed? It seems that -> while the server sends VALID, the client never debugs that it received -> it. Indeeed, the receiveMessage call that should receive it -> fails because the handle is closed at that point. Seems that -> this is caused by trying to receive 0 bytes as indicated by DATA -> ending up closing the handle. Another case of it involved getting -> an empty file followed by a second file. -> -> > This bug is fixed. - -[[done]] --[[Joey]] diff --git a/doc/bugs/parallel_copy_fails.mdwn b/doc/bugs/parallel_copy_fails.mdwn deleted file mode 100644 index a47f567dba..0000000000 --- a/doc/bugs/parallel_copy_fails.mdwn +++ /dev/null @@ -1,82 +0,0 @@ -### Please describe the problem. - -I keep encountering situations where git-annex fails on parallel operations. Retrying the operation several times sometimes resolves the issue -- it would be better if git-annex could be configured to do the retrying. I have set `annex.retry` and `annex.retry-delay`, but these don't seem to be enough. Maybe, these config vars could apply to all idempotent operations? - -I have also seen a parallel copy operation, copying to an S3 remote, hang indefinitely. Killing and retrying fixed the problem; it would be better if git-annex could have a timeout for operations, and kill and retry them automatically. This especially matters when building scripts on top of git-annex. - -### What steps will reproduce the problem? - -I have included the log below. I think repeatedly trying to copy from a local repo to an S3 bucket (non-export-tree) should eventually reproduce the problem. - -### What version of git-annex are you using? On what operating system? - -[[!format sh """ - -(master_env_v142_py36) 17:29 [v1.23.0-rc29] $ git annex version -git-annex version: 7.20190626-g85db42d -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.4.2 http-client-0.5.14 persistent-sql\ -ite-2.9.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_2\ -24 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE\ -2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256\ - BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external -operating system: linux x86_64 -supported repository versions: 5 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -local repository version: 5 -(master_env_v142_py36) 17:32 [v1.23.0-rc29] $ uname -a -Linux ip-172-31-81-187.ec2.internal 4.14.128-112.105.amzn2.x86_64 #1 SMP Wed Jun 19 16:53:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux -(master_env_v142_py36) 17:32 [v1.23.0-rc29] $ - -# End of transcript or log. -"""]] - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - -2019-07-03 17:30:22,365 - workflow_utils:289:_run - INFO - running command: git annex copy . --to=viral-ngs-benchmarks-s3 cwd=/dev/shm\ -/crogrun_190703_162305_36571/viral-ngs-benchmarks kw={'cwd': '/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks/benchmarks/rab\ -ies200/analysis-FK7B7p00761bfq2516zQb58F/benchmark_variants/v1.23.0-rc29'} -copy assemble_denovo.wdl (checking viral-ngs-benchmarks-s3...) ok -copy input-spec.json (checking viral-ngs-benchmarks-s3...) ok -copy analysis_labels.json (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok -copy inputs-git-links.json (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok -copy inputs-local.json (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok -copy inputs-orig.json (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok -copy imports.zip (checking viral-ngs-benchmarks-s3...) (to viral-ngs-benchmarks-s3...) ok -(recording state in git...) -error: git-annex died of signal 11 -2019-07-03 17:30:23,429 - workflow_utils:303:_run - INFO - RETRY 2 sleep 4 command (cwd=/dev/shm/crogrun_190703_162305_36571/viral-ngs\ --benchmarks, kw={'cwd': '/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks/benchmarks/rabies200/analysis-FK7B7p00761bfq2516zQb\ -58F/benchmark_variants/v1.23.0-rc29'}) in 1.063608169555664s: git annex copy . --to=viral-ngs-benchmarks-s3 exception Command 'git ann\ -ex copy . --to=viral-ngs-benchmarks-s3' returned non-zero exit status 139. -2019-07-03 17:30:27,434 - workflow_utils:312:_run - INFO - command (cwd=/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks, kw=\ -{'cwd': '/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks/benchmarks/rabies200/analysis-FK7B7p00761bfq2516zQb58F/benchmark_va\ -riants/v1.23.0-rc29'}) FAILED in 5.06814980506897s: git annex copy . --to=viral-ngs-benchmarks-s3 -copy assemble_denovo.wdl (checking viral-ngs-benchmarks-s3...) ok -copy analysis_labels.json (checking viral-ngs-benchmarks-s3...) ok -copy inputs-local.json (checking viral-ngs-benchmarks-s3...) ok -copy inputs-orig.json (checking viral-ngs-benchmarks-s3...) ok -copy input-spec.json (checking viral-ngs-benchmarks-s3...) ok -copy imports.zip (checking viral-ngs-benchmarks-s3...) ok -copy inputs-git-links.json (checking viral-ngs-benchmarks-s3...) ok -2019-07-03 17:30:27,952 - workflow_utils:312:_run - INFO - command (cwd=/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks, kw=\ -{'cwd': '/dev/shm/crogrun_190703_162305_36571/viral-ngs-benchmarks/benchmarks/rabies200/analysis-FK7B7p00761bfq2516zQb58F/benchmark_va\ -riants/v1.23.0-rc29'}) SUCCEEDED in 5.586166620254517s: git annex copy . --to=viral-ngs-benchmarks-s3 -2 - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - - -> Assuming this is a [[dup|done]] of -> [[concurrent_git-annex-copy_to_s3_special_remote_fails]]. --[[Joey]] - diff --git a/doc/bugs/parallel_copy_fails/comment_1_a8ffe32e04cbcbf9551e9d3a1ecf716d._comment b/doc/bugs/parallel_copy_fails/comment_1_a8ffe32e04cbcbf9551e9d3a1ecf716d._comment deleted file mode 100644 index 33c3d8340f..0000000000 --- a/doc/bugs/parallel_copy_fails/comment_1_a8ffe32e04cbcbf9551e9d3a1ecf716d._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="similar bug report" - date="2019-08-05T17:55:21Z" - content=""" -[[bugs/concurrent_git-annex-copy_to_s3_special_remote_fails]] -"""]] diff --git a/doc/bugs/parallel_copy_fails/comment_2_6a0e3514a111d48662bb50ca6a15b01f._comment b/doc/bugs/parallel_copy_fails/comment_2_6a0e3514a111d48662bb50ca6a15b01f._comment deleted file mode 100644 index 140dc91de0..0000000000 --- a/doc/bugs/parallel_copy_fails/comment_2_6a0e3514a111d48662bb50ca6a15b01f._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2019-11-13T19:37:16Z" - content=""" -The signal 11 is very significant. It points to a problem in a lower-level -library (or ghc runtime), or perhaps a bad memory problem. git-annex does -not itself contain any code that can segfault, afaik. - -Almost certianly the same as the other bug. -"""]] diff --git a/doc/bugs/parallel_copy_to_S3_fails.mdwn b/doc/bugs/parallel_copy_to_S3_fails.mdwn deleted file mode 100644 index c5cec17d96..0000000000 --- a/doc/bugs/parallel_copy_to_S3_fails.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -I'm seeing various failures with the latest git-annex when doing multi-threaded copy --to S3 . Using -J1 fixes the problem. - -> Assuming this is a [[dup|done]] of -> [[concurrent_git-annex-copy_to_s3_special_remote_fails]]. --[[Joey]] diff --git a/doc/bugs/parallel_copy_to_S3_fails/comment_1_1cef076780dc0bf4bb299be51c9497ca._comment b/doc/bugs/parallel_copy_to_S3_fails/comment_1_1cef076780dc0bf4bb299be51c9497ca._comment deleted file mode 100644 index 7bf3de4cc9..0000000000 --- a/doc/bugs/parallel_copy_to_S3_fails/comment_1_1cef076780dc0bf4bb299be51c9497ca._comment +++ /dev/null @@ -1,7 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-05-23T17:37:33Z" - content=""" -Can you be more specific please? -"""]] diff --git a/doc/bugs/possible_data_loss_when_unsized_key_stored_chunked.mdwn b/doc/bugs/possible_data_loss_when_unsized_key_stored_chunked.mdwn deleted file mode 100644 index 677dd49043..0000000000 --- a/doc/bugs/possible_data_loss_when_unsized_key_stored_chunked.mdwn +++ /dev/null @@ -1,48 +0,0 @@ -When a key has no known size (from addurl --relaxed eg), I think data loss -could occur in this situation: - -* repo A has an object for the key with size X -* repo B has an object for the same key with size Y (!= X) -* repo A transfers to the special remote -* then B transfers to the special remote -* B transfers one more chunk than A, because of the different size -* B actually "resumes" after the last chunk A uploaded. So now the remote - contains A's chunks, followed by B's extra chunk. -* A and B sync up, which merges the chunk logs. Since that log - uses "key:chunksize" as the log key, and the two logs have two different - ones, one will win or come first in the merged log. Suppose it's - the entry for B. So, the log then will be interpreted as the number of - chunks being B's. -* Now when the object is retrieved from the special remote, it will - retrieve and concacenate A's chunks, followed by B's extra chunk. - -So this is corruption at least, it can be recovered from, but to do so -you have to know the original length of A's object. Note that most keys -with unknown size also have no checksum to use to verify them, so it would -be easy for this to happen and not be caught. - -(Alternatively, after B transfers, it can sync with A, drop, and get -the content back from the special remote. Same result by another route, -and without needing any particular git-annex branch merge behavior to -happen so easier to reproduce. (I have not tried either yet.)) - -A simulantaneous upload by A and B might cause unrecoverable data loss -if they eg alternate chunks. Unsure if that can really happen. - -If A starts to transfer, sends some chunks, but is interrupted, and B -then transfers, resuming after the last chunk A stored, that would be data -loss. - -It might be best to just disable storing in chunks for keys of unknown size, -since it can fail so badly with them, and they're kind of a side thing? - -(Could continue retrieving, for whatever is stored hopefully w/o being -corrupted already.) ---[[Joey]] - -> This would also affect any key that is not stable. -> And oh, it stopped using chunks to store non-stable keys in 2014. -> -> So, this can't really happen with url keys, because they're not stable. -> Ok, not a bug then, because if the key is stable, there can only be one -> object for it, by definition. Whew! [[done]] --[[Joey]] diff --git a/doc/bugs/release_8.20201116_doesn__39__t_build_on_windows.mdwn b/doc/bugs/release_8.20201116_doesn__39__t_build_on_windows.mdwn deleted file mode 100644 index a344855665..0000000000 --- a/doc/bugs/release_8.20201116_doesn__39__t_build_on_windows.mdwn +++ /dev/null @@ -1,212 +0,0 @@ -### Please describe the problem. - -Lately I have been building my own git-annex binary for Windows from source ([[install/fromsource]]) -just to keep myself on edge, so to speak, and usually I don't have a problem doing that with -the `stack setup` & `stack build` method under PowerShell (v7.1.0) if I choose a tagged commit -from Git Annex's git repo. However, this latest release doesn't seem to build without additional minor -coaxing/editing of the Haskell source due to type mismatches. I don't really know Haskell but I have -some rudimentary understanding of the ML family of languages, so I went ahead and peppered the troublesome -points of code with some invocations of `fromRawFilePath` and `toRawFilePath` to please the GHC -gods. Additionally, I cherry-picked commit `bce8865` on top of my own corrections to make things work. -This indeed allowed the code to compile but then the part of the testsuite which I exercised, -namely the "Unit Tests v8" set failed completely due to not being able to init the annex. The gist of -the matter is that git-annex.exe complains thusly: -"git-annex.exe: System.PosixCompat.Files.removeLink: not supported: illegal operation". - -### What steps will reproduce the problem? - -Checking out tag `8.20201116`, making the corrections in the diff below, and then building with `stack setup` -and `stack build` (with no extras enabled, not even libmagic/MagicMime); finally testing the resulting binary -within *Git Bash* (in a folder like `c:\annx`) in the following fashion: - -[[!format sh """ -$ pwd -/c/annx -$ ./git-annex.exe test -p 'adjusted unlocked branch' 2>&1 | tee git-annex.test--p-adjusted_unlocked_branch.LOG~101 -Init Tests - init: Tests - Unit Tests v8 adjusted unlocked branch - add dup: init test repo - Detected a filesystem without fifo support. - - Disabling ssh connection caching. - - Detected a crippled filesystem. -(scanning for unlocked files...) - - Entering an adjusted branch where files are unlocked as this filesystem does not support locked files. - -Switched to branch 'adjusted/master(unlocked)' -ok -(recording state in git...) -git-annex.exe: System.PosixCompat.Files.removeLink: not supported: illegal operation -FAIL (2.92s) - Test.hs:371: - init failed - add: add foo -^M100% 20 B 555 B/s 0s^M ^Mok -(recording state in git...) -git-annex.exe: System.PosixCompat.Files.removeLink: not supported: illegal operation -FAIL (1.06s) - Test.hs:381: - add failed - -2 out of 2 tests failed (3.98s) -[...] - addurl: FAIL - Exception: init tests failed! cannot continue - CallStack (from HasCallStack): - error, called at .\\Test\\Framework.hs:432:49 in main:Test.Framework - -71 out of 71 tests failed (3.99s) - (Failures above could be due to a bug in git-annex, or an incompatibility - with utilities, such as git, installed on this system.) -# end of transcript -"""]] - -### What version of git-annex are you using? On what operating system? - -Version 8.20201116, commit 864af53a2, with some corrections that are in the diff below. This is on -Windows 10 version 20H2 (build 19042.630), 64 bit. - -[[!format sh """ -git-annex version: 8.20201116-g864af53a2 -build flags: Assistant Webapp Pairing TorrentParser Feeds Testsuite S3 WebDAV -dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external -operating system: mingw32 x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 2 3 4 5 6 7 -"""]] - -### Please provide any additional information below. - -Here's the diff to my changes on top of commit 864af53a2: - -[[!format diff """ -diff --git a/Annex/Content.hs b/Annex/Content.hs -index a8bcd1666..af6d95435 100644 ---- a/Annex/Content.hs -+++ b/Annex/Content.hs -@@ -170,7 +170,7 @@ inAnnexSafe key = inAnnex' (fromMaybe True) (Just False) go key - Nothing -> return is_locked - Just lockhandle -> do - dropLock lockhandle -- void $ tryIO $ removeWhenExistsWith removeLink lockfile -+ void $ tryIO $ removeWhenExistsWith removeLink (fromRawFilePath lockfile) - return is_unlocked - , return is_missing - ) -@@ -249,7 +249,7 @@ winLocker :: (LockFile -> IO (Maybe LockHandle)) -> ContentLocker - winLocker takelock _ (Just lockfile) = do - modifyContent lockfile $ - void $ liftIO $ tryIO $ -- writeFile lockfile "" -+ writeFile (fromRawFilePath lockfile) "" - liftIO $ takelock lockfile - -- never reached; windows always uses a separate lock file - winLocker _ _ Nothing = return Nothing -diff --git a/Assistant.hs b/Assistant.hs -index a0bbd3704..d7e75e23d 100644 ---- a/Assistant.hs -+++ b/Assistant.hs -@@ -102,7 +102,7 @@ startDaemon assistant foreground startdelay cannotrun listenhost startbrowser = - createAnnexDirectory (parentDir logfile) - ifM (liftIO $ isNothing <$> getEnv flag) - ( liftIO $ withNullHandle $ \nullh -> do -- loghandle <- openLog logfile -+ loghandle <- openLog (fromRawFilePath logfile) - e <- getEnvironment - cmd <- programPath - ps <- getArgs -@@ -115,7 +115,7 @@ startDaemon assistant foreground startdelay cannotrun listenhost startbrowser = - exitcode <- withCreateProcess p $ \_ _ _ pid -> - waitForProcess pid - exitWith exitcode -- , start (Utility.Daemon.foreground (Just pidfile)) $ -+ , start (Utility.Daemon.foreground (Just (fromRawFilePath pidfile))) $ - case startbrowser of - Nothing -> Nothing - Just a -> Just $ a Nothing Nothing -diff --git a/Logs/Transfer.hs b/Logs/Transfer.hs -index 2722702d2..369c6a630 100644 ---- a/Logs/Transfer.hs -+++ b/Logs/Transfer.hs -@@ -131,7 +131,7 @@ checkTransfer t = debugLocks $ do - v <- liftIO $ lockShared lck - liftIO $ case v of - Nothing -> catchDefaultIO Nothing $ -- readTransferInfoFile Nothing tfile -+ readTransferInfoFile Nothing (fromRawFilePath tfile) - Just lockhandle -> do - dropLock lockhandle - cleanstale -diff --git a/Remote/Directory.hs b/Remote/Directory.hs -index 5c20894ee..0a8b6da71 100644 ---- a/Remote/Directory.hs -+++ b/Remote/Directory.hs -@@ -258,7 +258,7 @@ removeDirGeneric topdir dir = do - #ifdef mingw32_HOST_OS - {- Windows needs the files inside the directory to be writable - - before it can delete them. -} -- void $ tryIO $ mapM_ allowWrite =<< dirContents dir -+ void $ tryIO $ mapM_ (allowWrite . toRawFilePath) =<< dirContents dir - #endif - tryNonAsync (removeDirectoryRecursive dir) >>= \case - Right () -> return () -@@ -446,7 +446,7 @@ retrieveExportWithContentIdentifierM dir loc cid dest mkkey p = - #ifndef mingw32_HOST_OS - =<< getFdStatus fd - #else -- =<< getFileStatus f -+ =<< getFileStatus (fromRawFilePath f) - #endif - guardSameContentIdentifiers cont cid currcid - -diff --git a/Utility/Daemon.hs b/Utility/Daemon.hs -index 2e1a9baa8..9a3425589 100644 ---- a/Utility/Daemon.hs -+++ b/Utility/Daemon.hs -@@ -112,7 +112,7 @@ lockPidFile pidfile = do - pid <- getPID - writeFile pidfile (show pid) - lckfile <- winLockFile pid pidfile -- writeFile lckfile "" -+ writeFile (fromRawFilePath lckfile) "" - void $ lockExclusive lckfile - #endif - -@@ -176,14 +176,14 @@ stopDaemon pidfile = go =<< checkDaemon pidfile - - when eg, restarting the daemon. - -} - #ifdef mingw32_HOST_OS --winLockFile :: PID -> FilePath -> IO FilePath -+winLockFile :: PID -> FilePath -> IO RawFilePath - winLockFile pid pidfile = do - cleanstale -- return $ prefix ++ show pid ++ suffix -+ return $ toRawFilePath $ prefix ++ show pid ++ suffix - where - prefix = pidfile ++ "." - suffix = ".lck" - cleanstale = mapM_ (void . tryIO . removeFile) =<< -- (filter iswinlockfile <$> dirContents (parentDir pidfile)) -+ (filter iswinlockfile <$> dirContents (fromRawFilePath (parentDir (toRawFilePath pidfile)))) - iswinlockfile f = suffix `isSuffixOf` f && prefix `isPrefixOf` f - #endif -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I've been trying out Git Annex for a number of years but only for a year now I've had a nice use case for it -with archiving/mirroring my Macrium Reflect backup files (multi-gigabyte disk images with .mrimg extension) -to a secondary drive. Obviously because I'm on Windows I use adjusted branches a lot (even -`git-annex adjust --hide-missing --unlock`) with the `annex.thin` set to true so I rely on basic v8 features -to just work. I haven't tried special remotes yet, so I'm not at the moment concerned about that part of Git -Annex -- my use case is more like local archiving and stuff. Git Annex does give me some rigor to how I move -my backup files around and it's always nice to have an external checksum (I use MD5E w/Annex) recorded -"in the system" so that you can be sure your files are not altered due to bit rot, etc. In any case, -big thanks go to Joey (et al.) for this wonderful tool you made! - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/release_8.20201116_doesn__39__t_build_on_windows/comment_1_8252525da4c289e071b18d1df940868f._comment b/doc/bugs/release_8.20201116_doesn__39__t_build_on_windows/comment_1_8252525da4c289e071b18d1df940868f._comment deleted file mode 100644 index 7c0f5be165..0000000000 --- a/doc/bugs/release_8.20201116_doesn__39__t_build_on_windows/comment_1_8252525da4c289e071b18d1df940868f._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-11-19T16:31:21Z" - content=""" -Thanks for that patch, I had been fixing the windows build one line per -day, so that sped it up considerably. - -I've fixed the issue with the test suite I think, although I will have to -wait on tomorrow's windows build to know for sure. -"""]] diff --git a/doc/bugs/removeLink_failed_when_initializing_a_repo_in_a_VirtualBox_shared_folder.mdwn b/doc/bugs/removeLink_failed_when_initializing_a_repo_in_a_VirtualBox_shared_folder.mdwn deleted file mode 100644 index f9132677d5..0000000000 --- a/doc/bugs/removeLink_failed_when_initializing_a_repo_in_a_VirtualBox_shared_folder.mdwn +++ /dev/null @@ -1,58 +0,0 @@ -### Please describe the problem. -Got an error message trying to initialize a git-annex repo on a VirtualBox shared folder (Linux guest, Windows host). The shared folder is on an external USB drive. - -### What steps will reproduce the problem? -See log - -### What version of git-annex are you using? On what operating system? - -[[!format sh """ -(newer3-gdrive-remote-env) [ilya@cg-router1 WPAE9-305]$ git annex version -git-annex version: 8.20200617-g02765b8 -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.4 feed-1.2.0.1 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.1 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: linux x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 0 1 2 3 4 5 6 7 -local repository version: 8 -(newer3-gdrive-remote-env) [ilya@cg-router1 WPAE9-305]$ uname -a -Linux cg-router1.broadinstitute.org 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux -(newer3-gdrive-remote-env) [ilya@cg-router1 WPAE9-305]$ -"""]] - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log -(newer3-gdrive-remote-env) [ilya@cg-router1 WPAE9-305]$ git init -Initialized empty Git repository in /mnt/shared/d/WindowsImageBackup/WPAE9-305/.git/ -(newer3-gdrive-remote-env) [ilya@cg-router1 WPAE9-305]$ git annex init 'blue wd passport ultra' -init blue wd passport ultra - Detected a filesystem without fifo support. - - Disabling ssh connection caching. - - Detected a crippled filesystem. -(scanning for unlocked files...) - - Entering an adjusted branch where files are unlocked as this filesystem does not support locked files. - -Switched to branch 'adjusted/master(unlocked)' -ok -(recording state in git...) -git-annex: .git/annex/othertmp/jlog12400-5: removeLink: resource busy (Text file busy) -(newer3-gdrive-remote-env) [ilya@cg-router1 WPAE9-305]$ - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Certainly. Right now trying to organize a new set of backups from multiple places, which without git-annex would be an organizational nightmare. - -> Closing this because the FS seems too buggy to try to support. -> [[done]] --[[Joey]] diff --git a/doc/bugs/removeLink_failed_when_initializing_a_repo_in_a_VirtualBox_shared_folder/comment_1_5f84d7d8b0647ae9161e54d6266464a7._comment b/doc/bugs/removeLink_failed_when_initializing_a_repo_in_a_VirtualBox_shared_folder/comment_1_5f84d7d8b0647ae9161e54d6266464a7._comment deleted file mode 100644 index 755c2c146d..0000000000 --- a/doc/bugs/removeLink_failed_when_initializing_a_repo_in_a_VirtualBox_shared_folder/comment_1_5f84d7d8b0647ae9161e54d6266464a7._comment +++ /dev/null @@ -1,42 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="adding files also fails" - date="2020-07-01T19:51:34Z" - content=""" -Trying to add a file also gives errors; posting here because cause is likely the same: - -[[!format sh \"\"\" -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log -(newer3-gdrive-remote-env) [ilya@cg-router1 Backup 2020-06-30 043950]$ git annex add Esp.vhdx -add Esp.vhdx -100% 46 MiB 94 MiB/s 0s -git-annex: failed to commit changes to sqlite database: Just user error (SQLite3 returned ErrorReadOnly while attempting to perform step: attempt to write a readonly database(after successful open)) -CallStack (from HasCallStack): - error, called at ./Database/Handle.hs:115:26 in main:Database.Handle -failed -git-annex: thread blocked indefinitely in an MVar operation -(newer3-gdrive-remote-env) [ilya@cg-router1 Backup 2020-06-30 043950]$ echo $? - -(newer3-gdrive-remote-env) [ilya@cg-router1 WPAE9-305]$ git config -l -core.repositoryformatversion=0 -core.filemode=false -core.bare=false -core.logallrefupdates=true -core.symlinks=false -core.ignorecase=true -annex.uuid=9f2b6628-5921-47d2-a712-ac902a662d04 -annex.sshcaching=false -annex.crippledfilesystem=true -annex.version=8 -annex.addunlocked=true -annex.thin=true -filter.annex.smudge=git-annex smudge -- %f -filter.annex.clean=git-annex smudge --clean -- %f -(newer3-gdrive-remote-env) [ilya@cg-router1 WPAE9-305]$ - - -# End of transcript or log. -\"\"\"]] -"""]] diff --git a/doc/bugs/removeLink_failed_when_initializing_a_repo_in_a_VirtualBox_shared_folder/comment_2_eeacb58430a809e6f2e7ca1823d7e8b1._comment b/doc/bugs/removeLink_failed_when_initializing_a_repo_in_a_VirtualBox_shared_folder/comment_2_eeacb58430a809e6f2e7ca1823d7e8b1._comment deleted file mode 100644 index 3e3bf79619..0000000000 --- a/doc/bugs/removeLink_failed_when_initializing_a_repo_in_a_VirtualBox_shared_folder/comment_2_eeacb58430a809e6f2e7ca1823d7e8b1._comment +++ /dev/null @@ -1,33 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-07-02T16:02:25Z" - content=""" -The code around the removeLink in question is this: - - hClose jlogh - nukeFile jlogf - -So, git-annex actually closes the file handle before unlinking it. -And the file is a temp file that only that process can access, and -it's closed the only handle to it, so how can it possibly be busy after -being closed? - -(Not that closing a handle before removing a file is necessary on unix in -the first place. According to unlink(2), EBUSY only -happens in situations involving mount points or some NFS hacks.) - -I googled "attempt to write a readonly database(after successful open)" -and -suggests its something to do with file deletion as well. Other hits -suggest it's permissions related. Perhaps sqlite is looking at the -permissions of the database file and they seem to not allow it to write so -it opens it readonly, but of course if the same user has created the -database that would not usually happen. - -All of this points squarly at the filesystem being broken, and probably too -broken to even thing about supporting. You might find that the directory -special remote with exporttree=yes and importtree=yes works on -this filesystem, just because it's simpler and so has less probability of -tickling its posix violations. -"""]] diff --git a/doc/bugs/resource_exhausted_at_end_of_100k_files_copy.mdwn b/doc/bugs/resource_exhausted_at_end_of_100k_files_copy.mdwn deleted file mode 100644 index 5b60e89783..0000000000 --- a/doc/bugs/resource_exhausted_at_end_of_100k_files_copy.mdwn +++ /dev/null @@ -1,40 +0,0 @@ -Ran git-annex copy --to origin -J4 in a repo with 100k files. origin was a -local git repo. After all the files were transferred, it died: - - copy bar (to origin...) (checksum...) ok - git-annex: git: createProcess: fork: resource exhausted (Resource temporarily unavailable) - -exit 1 - -.git/annex/journal/ had 76k files in it. git-annex merge dealt with it -without a problem. --[[Joey]] - -> This is reproducible, and the second time was with slightly fewer files, -> probably around 60k. Some tries with around 10-20k files did not -> show any problem, never more than 5-10 processes, let alone a full fork -> bomb worth. -> -> I did not see any evidence of processes building up shortly before it crashed, -> while it was still copying files. -> --[[Joey]] -> -> > Saw it again, after copying 10k files. --[[Joey]] -> > -> > --debug shows a 8000+ git hash-object -w --stdin-paths --no-filters -> > just before. Each process is logged as having exited successfully though, -> > before the next starts, so how could this be a problem? However, it is -> > surprising, it's supposed to write all the changes to just 1 process. -> > -> > Even getting only 1000 files shows these extra processes, -> > though not enough to crash it. Utility.CoProcess -> > is restarting git hash-object due to an IO exception: -> > "fd:12: hPutStr: illegal operation (handle is closed) -> > I can't see anything that would close the handle early. -> > -> > Reverted [[!commit 7013798df5a161f00962985ffaea613a87cc4fe4]] -> > on a hunch, and that seems to have fixed it. Which is very weird, -> > because AFAICS it was not getting an async exception. Even -> > removing the `catchAsync forcerestart` is enough to avoid the problem -> > though. -> > -> > [[fixed|done]] though without full understanding of what that commit -> > did that caused such strange behavior. --[[Joey]] diff --git a/doc/bugs/running_git_annex_commands_in_worktree_of_a_non-git-annex_repo_changes_the_.git.mdwn b/doc/bugs/running_git_annex_commands_in_worktree_of_a_non-git-annex_repo_changes_the_.git.mdwn deleted file mode 100644 index a5cb961747..0000000000 --- a/doc/bugs/running_git_annex_commands_in_worktree_of_a_non-git-annex_repo_changes_the_.git.mdwn +++ /dev/null @@ -1,68 +0,0 @@ -### Please describe the problem. -If `git annex init` has not been run on a repo, running git-annex commands on the linked worktrees should not change them, but seems to. - -### What steps will reproduce the problem? - -See log below -### What version of git-annex are you using? On what operating system? -8.20200226 on Amazon Linux 2 - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log -(master_env_v178_py36) 17:07 [a] $ ls -alt -total 4 -drwxrwxr-x 8 ilya ilya 166 Mar 6 17:07 .git -drwxrwxr-x 3 ilya ilya 32 Mar 6 17:07 . --rw-rw-r-- 1 ilya ilya 0 Mar 6 17:07 myfile -drwxrwxrwt 15 root root 4096 Mar 6 17:07 .. -(master_env_v178_py36) 17:07 [a] $ git worktree add ../b -Preparing worktree (new branch 'b') -HEAD is now at 9a5d353 first commit -(master_env_v178_py36) 17:08 [a] $ pushd ../b -/tmp/b /tmp/a /data/branches/is-add-asm-improvability-metrics -(master_env_v178_py36) 17:08 [b] $ ls -alt -total 8 -drwxrwxr-x 2 ilya ilya 32 Mar 6 17:08 . --rw-rw-r-- 1 ilya ilya 0 Mar 6 17:08 myfile -drwxrwxrwt 16 root root 4096 Mar 6 17:08 .. --rw-rw-r-- 1 ilya ilya 32 Mar 6 17:08 .git -(master_env_v178_py36) 17:08 [b] $ git annex get -git-annex: First run: git-annex init -(master_env_v178_py36) 17:08 [b] $ ls -alt -total 4 -drwxrwxr-x 2 ilya ilya 32 Mar 6 17:08 . -lrwxrwxrwx 1 ilya ilya 21 Mar 6 17:08 .git -> ../a/.git/worktrees/b --rw-rw-r-- 1 ilya ilya 0 Mar 6 17:08 myfile -drwxrwxrwt 16 root root 4096 Mar 6 17:08 .. -(master_env_v178_py36) 17:08 [b] $ ls -alt /tmp/b/../a/.git/worktrees/b/annex -lrwxrwxrwx 1 ilya ilya 11 Mar 6 17:08 /tmp/b/../a/.git/worktrees/b/annex -> ../../annex - - -(master_env_v178_py36) 17:12 [b] $ git annex version -git-annex version: 8.20200226-g2d3ef2c07 -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlit\ -e-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_2\ -24 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE\ -2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224\ - BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: linux x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 0 1 2 3 4 5 6 7 - -(master_env_v178_py36) 17:14 [b] $ uname -a -Linux ip-172-31-80-211.ec2.internal 4.14.171-136.231.amzn2.x86_64 #1 SMP Thu Feb 27 20:22:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/running_git_annex_commands_in_worktree_of_a_non-git-annex_repo_changes_the_.git/comment_1_d5d8754af32ac45cd6303805bfe99911._comment b/doc/bugs/running_git_annex_commands_in_worktree_of_a_non-git-annex_repo_changes_the_.git/comment_1_d5d8754af32ac45cd6303805bfe99911._comment deleted file mode 100644 index bcf902a17c..0000000000 --- a/doc/bugs/running_git_annex_commands_in_worktree_of_a_non-git-annex_repo_changes_the_.git/comment_1_d5d8754af32ac45cd6303805bfe99911._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-09T18:31:46Z" - content=""" -Hmm, fixupUnusualRepos was earlier changed to check for .noannex -files, and avoid doing anything. Didn't think big enough I suppose. - -I agree, git-annex should not be hacking on git repos that have not had -git-annex initialized in them yet. Luckily all the hacks are about making -.git files into symlinks so links to annexed files work, so it will be ok -for the .git file to remain unconverted until the end of git-annex init, -or even by a subsequent git-annex command. -"""]] diff --git a/doc/bugs/separate-git-dir_OK_under_Linux_FAIL_under_Windows.mdwn b/doc/bugs/separate-git-dir_OK_under_Linux_FAIL_under_Windows.mdwn deleted file mode 100644 index acf9391139..0000000000 --- a/doc/bugs/separate-git-dir_OK_under_Linux_FAIL_under_Windows.mdwn +++ /dev/null @@ -1,182 +0,0 @@ -### Please describe the problem. - -When a git-annex is cloned from one where git init was run with --separate-git-dir, it works fine under Linux, but fails under Windows - -[[!format sh """ -shaddy@JAZZ2-W10 C:\slaveproj -> git annex get services -get services - Unable to access these remotes: origin - - Try making some of these repositories available: - 615bdcd5-6a5e-4330-84a2-b6c30cb3e5e4 -- shaddy@jazz2-w10:C:\masterproj [origin] -failed -git-annex: get: 1 failed - -"""]] - -### What steps will reproduce the problem? - -[[!format sh """ -shaddy@JAZZ2-W10 C:\ -> git init --separate-git-dir masterproj.git masterproj -Initialized empty Git repository in C:/masterproj.git/ - -shaddy@JAZZ2-W10 C:\ -> cd masterproj - -shaddy@JAZZ2-W10 C:\masterproj -> git annex init -init - Detected a filesystem without fifo support. - - Disabling ssh connection caching. - - Detected a crippled filesystem. -(scanning for unlocked files...) - - Entering an adjusted branch where files are unlocked as this filesystem does not support locked files. - -Switched to branch 'adjusted/master(unlocked)' -ok -(recording state in git...) - -shaddy@JAZZ2-W10 C:\masterproj -> copy c:\Windows\System32\drivers\etc\services . - 1 file(s) copied. - -shaddy@JAZZ2-W10 C:\masterproj -> git annex add services -add services -ok -(recording state in git...) - -shaddy@JAZZ2-W10 C:\masterproj -> git commit -m services -[adjusted/master(unlocked) e9d5e15] services - 1 file changed, 1 insertion(+) - create mode 100644 services - -shaddy@JAZZ2-W10 C:\masterproj -> cd \ - -shaddy@JAZZ2-W10 C:\ -> git clone masterproj slaveproj -Cloning into 'slaveproj'... -done. - -shaddy@JAZZ2-W10 C:\ -> cd slaveproj - -shaddy@JAZZ2-W10 C:\slaveproj -> git annex init -init - Detected a filesystem without fifo support. - - Disabling ssh connection caching. - - Detected a crippled filesystem. -(merging origin/git-annex into git-annex...) -(recording state in git...) -(scanning for unlocked files...) -ok -(recording state in git...) - -shaddy@JAZZ2-W10 C:\slaveproj -> dir - Volume in drive C has no label. - Volume Serial Number is DE6E-211A - - Directory of C:\slaveproj - -22/05/2020 08:57 PM . -22/05/2020 08:57 PM .. -22/05/2020 08:57 PM 97 services - 1 File(s) 97 bytes - 2 Dir(s) 24,926,748,672 bytes free - -shaddy@JAZZ2-W10 C:\slaveproj -> type services -/annex/objects/SHA256E-s17463--b26309dfd89a9cc94481536b4d662941429df79873bb59620f53db939ff5ec29 - -shaddy@JAZZ2-W10 C:\slaveproj -> git annex get services -get services - Unable to access these remotes: origin - - Try making some of these repositories available: - 615bdcd5-6a5e-4330-84a2-b6c30cb3e5e4 -- shaddy@jazz2-w10:C:\masterproj [origin] -failed -git-annex: get: 1 failed - -shaddy@JAZZ2-W10 C:\slaveproj -> -"""]] - -### What version of git-annex are you using? On what operating system? - -[[!format sh """ -shaddy@JAZZ2-W10 C:\slaveproj -> git annex version -git-annex version: 8.20200330-g971791563 -build flags: Assistant Webapp Pairing S3 WebDAV TorrentParser Feeds Testsuite -dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.4 feed-1.2.0.1 ghc-8.6.5 http-client-0.6.4 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.1 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: mingw32 x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 2 3 4 5 6 7 -local repository version: 8 -"""]] - -### Please provide any additional information below. - -Works fine under Linux: - -[[!format sh """ -shaddy@jazz-debian:~$ git init --separate-git-dir /var/tmp/masterproj.git masterproj -Initialized empty Git repository in /var/tmp/masterproj.git/ -shaddy@jazz-debian:~$ for d in masterproj/ /var/tmp/masterproj.git/; do stat -f $d | head -2; done - File: "masterproj/" - ID: 4efe0be616b5a688 Namelen: 255 Type: ext2/ext3 - File: "/var/tmp/masterproj.git/" - ID: 6736efe4caec433 Namelen: 255 Type: ext2/ext3 -shaddy@jazz-debian:~$ cd masterproj/ -shaddy@jazz-debian:~/masterproj$ git annex init -init ok -(recording state in git...) -shaddy@jazz-debian:~/masterproj$ cp /etc/services . -shaddy@jazz-debian:~/masterproj$ git annex add services -add services ok -(recording state in git...) -shaddy@jazz-debian:~/masterproj$ git commit -m services -[master (root-commit) a42f63a] services - 1 file changed, 1 insertion(+) - create mode 120000 services -shaddy@jazz-debian:~/masterproj$ cd ~/ -shaddy@jazz-debian:~$ git clone masterproj/ slaveproj/ -Cloning into 'slaveproj'... -done. -shaddy@jazz-debian:~$ cd slaveproj/ -shaddy@jazz-debian:~/slaveproj$ git annex init -init (merging origin/git-annex into git-annex...) -(recording state in git...) -ok -(recording state in git...) -shaddy@jazz-debian:~/slaveproj$ ls -lL services -ls: cannot access 'services': No such file or directory -shaddy@jazz-debian:~/slaveproj$ git annex get services -get services (from origin...) -SHA256E-s18774--60ef85f14cd630692bf91756e2e81351c54e2eace0310188eb4f6af4d9858c91 - 18,774 100% 0.00kB/s (checksum...) 0:00:00 (xfr#1, to-chk=0/1) -ok -(recording state in git...) -shaddy@jazz-debian:~/slaveproj$ ls -lL services --r--r--r-- 1 shaddy shaddy 18774 May 22 20:53 services -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -This issue is not a show stopper. I otherwise love the concepts behind git-annex very much. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/separate-git-dir_OK_under_Linux_FAIL_under_Windows/comment_1_97ab8d245680e23a9e4b471796db2119._comment b/doc/bugs/separate-git-dir_OK_under_Linux_FAIL_under_Windows/comment_1_97ab8d245680e23a9e4b471796db2119._comment deleted file mode 100644 index 1c24fbe3f0..0000000000 --- a/doc/bugs/separate-git-dir_OK_under_Linux_FAIL_under_Windows/comment_1_97ab8d245680e23a9e4b471796db2119._comment +++ /dev/null @@ -1,35 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-08-28T18:03:46Z" - content=""" ---separate-git-dir makes eg, masterproj/.git be a file that contains the -path to the git dir. - -git-annex init converts such a .git file to a symlink. That is necessary -for git-annex symlinks to be resolved, since they always link to files -inside .git, and if it's a file, the symlinks won't point to the object -files. - -On windows, since core.symlinks is false, it does not make that change. - -So, I think the problem might be that, when .git is a file rather -than a symlink, git-annex in the clone doesn't read the .git file to determine -the real git repo location, and so can't find annexed files in its remote. - -Seem to have found a way to produce the same problem on linux: - -* git init --separate-git-dir masterproj.git masterproj -* cd masterproj; git config core.symlinks false -* git-annex init -* add file, commit, make clone - -git-annex get in the clone then fails - - openat(AT_FDCWD, "../masterproj/.git/annex/objects/3f/3K/SHA256E-s30--32a7ec5a2905bbeda54a699ed797c96c1fea7b5bc31a6cc104b2ddefe65a95bb/SHA256E-s30--32a7ec5a2905bbeda54a699ed797c96c1fea7b5bc31a6cc104b2ddefe65a95bb", O_RDONLY) = -1 ENOTDIR (Not a directory) - -So, I think it would make sense for git-annex to notice when -a remote's .git is a file rather than a directory, and adjust the remote -accordingly, so it uses its actual git directory. That will solve -this situation. -"""]] diff --git a/doc/bugs/signal_weirdness.mdwn b/doc/bugs/signal_weirdness.mdwn deleted file mode 100644 index 890767386c..0000000000 --- a/doc/bugs/signal_weirdness.mdwn +++ /dev/null @@ -1,54 +0,0 @@ -For the record, there is a slight weirdness with how git-annex -handles a signal like ctrl-c. - -For example: - - joey@gnu:~/tmp/b>git annex copy a b --to origin - copy a (checking origin...) (to origin...) - SHA256-s104857600--20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e - 3272 0% 0.00kB/s 0:00:00 ^C - zsh: interrupt git annex copy a --to origin - joey@gnu:~/tmp/b> - rsync error: unexplained error (code 130) at rsync.c(549) [sender=3.0.9] - -Here git-annex exits before rsync has fully exited. Not a large problem -but sorta weird. - -The culprit is `CmdLine.startup` in Utility.SafeCommand, which installs -a default signal handler for SIGINT, which causes it to immediatly -terminate git-annex. rsync, in turn, has its own SIGINT handler, which -prints the message, typically later. - -(Why it prints that message and not its more usual message about having -received a signal, I'm not sure?) - -It's more usual for a `system` like thing to block SIGINT, letting the child -catch it and exit, and then detecting the child's exit status and terminating. -However, since rsync *is* trapping SIGINT, and exiting nonzero explicitly, -git-annex can't tell that rsync failed due to a SIGINT by examining the -`waitpid` result. -And, git-annex typically doesn't stop when a single child fails. In the -example above, it would go on to copy `b` after a ctrl-c! - -A further complication is that git-annex is itself a child process -of git, which does not block SIGINT either. So if git-annex blocks SIGINT, -it will be left running in the background after git exits, and continuing -with further actions too. (Perhaps its SIGINT handling is a bug in git.) - -Now, rsync does have a documented exit code it uses after a SIGINT. -But other programs git-annex runs generally do not. So it would be possible -to special case in support for rsync, blocking SIGINT while running it, -noticing it exited with 20, and git-annex then stopping. But this is -ugly and failure prone if rsync's code 20 changes. And it only -would fix the rsync case, not helping with other commands like wget, unless -it assumes they never trap SIGINT on their own. - -Which is why the current behavior of not blocking SIGINT was chosen, -as a less bad alternative. Still, I'd like to find a better one. ---[[Joey]] - -[[!tag confirmed]] - -> [[done]], I think this was fixed long ago, git-annex no longer installs a -> sigint handler and I interrupt it all the time and it behaves as I would -> expect and not as shown here --[[Joey]] diff --git a/doc/bugs/smudge_filter_drastically_slows_down_git_diff.mdwn b/doc/bugs/smudge_filter_drastically_slows_down_git_diff.mdwn deleted file mode 100644 index 3899d18fd8..0000000000 --- a/doc/bugs/smudge_filter_drastically_slows_down_git_diff.mdwn +++ /dev/null @@ -1,118 +0,0 @@ -### Please describe the problem. - -Running `git diff` on a small (v8) repo is extremely slow. - -I see [[todo/Long_Running_Filter_Process]] so this is maybe a known issue, but I thought I'd report just in case, and hopefully the extra data is useful. - -### What steps will reproduce the problem? - -Just run `git diff` with some unstaged changes: - -``` -$ diff --git a/README.md b/README.md -index 6532edf..2edc620 100644 ---- a/README.md -+++ b/README.md -@@ -42,3 +42,4 @@ General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program. If not, see . -+foo -git diff 0.17s user 0.07s system 115% cpu 0.204 total - -``` - -We can see that execve(2) is called 77 times, and there's a lot of I/O: - -``` -$ ( strace -c -w -f git diff 3>&1 >/dev/null 2>&3- ) | grep -v strace | head -n 15 -% time seconds usecs/call calls errors syscall ------- ----------- ----------- --------- --------- ---------------- - 42.18 3.777959 1397 2704 27 read - 19.27 1.725834 13697 126 50 wait4 - 16.30 1.460262 3023 483 64 futex - 8.24 0.737923 92240 8 poll - 8.04 0.720108 11430 63 epoll_wait - 1.36 0.122156 22 5387 3519 openat - 0.87 0.078340 20 3812 2858 stat - 0.60 0.053536 23 2243 mmap - 0.52 0.046553 20 2319 16 close - 0.42 0.037637 19 1890 fstat - 0.41 0.037030 480 77 execve - 0.26 0.023351 19 1207 4 rt_sigaction - 0.17 0.014840 19 760 rt_sigprocmask -``` - -If I comment out the smudge filter in `.git/config`, it becomes instant: - -``` -git diff 0.00s user 0.00s system 108% cpu 0.005 total -``` - -and we see only one `execve(2)` and a lot less I/O: - -``` -% time seconds usecs/call calls errors syscall ------- ----------- ----------- --------- --------- ---------------- - 20.42 0.001402 18 75 lstat - 15.06 0.001034 19 53 15 openat - 11.25 0.000773 16 48 fstat - 11.16 0.000766 20 38 read - 9.49 0.000652 16 39 close - 8.40 0.000577 16 35 mmap - 4.95 0.000340 18 18 11 access - 4.49 0.000308 308 1 execve - 2.70 0.000185 23 8 getdents64 - 2.62 0.000180 20 9 mprotect - 2.62 0.000180 17 10 stat - 1.11 0.000076 15 5 brk - 1.08 0.000074 24 3 munmap -``` - -0.17s might not sound like a lot, but when there are many unstaged changes this becomes painful. For example, it changes a git status operation in one of the popular git GUIs (magit) from being instant to taking 6 seconds just for a few one-line changes in a small repository. - -It's worth noting that this repo doesn't even have any annexed files - `annex init` was only run recently: - -``` -$ git annex info -trusted repositories: 0 -semitrusted repositories: 7 - 00000000-0000-0000-0000-000000000001 -- web - 00000000-0000-0000-0000-000000000002 -- bittorrent - 31167abf-8d32-4b40-96cf-4a3310fd1b10 -- arabian - 6371770a-303e-474c-9209-2c1565b25608 -- adam@pacific:~/.config/mr [pacific] - da95efd3-09f9-432b-ae28-bc4c5891c062 -- adam@aegean:~/.config/mr [aegean] - e0112d70-c2b3-44df-9057-887eefe5d225 -- adam@ionian:~/.config/mr [here] - ec80d559-508f-445c-a1fd-48f2805315b0 -- my.personal.domain.org -untrusted repositories: 0 -transfers in progress: none -available local disk space: 1 terabyte (+1 megabyte reserved) -local annex keys: 0 -local annex size: 0 bytes -annexed files in working tree: 0 -size of annexed files in working tree: 0 bytes -bloom filter size: 32 mebibytes (0% full) -backend usage: -``` - -### What version of git-annex are you using? On what operating system? - -``` -git-annex version: 8.20200618-g4c44deb2c -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: linux x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 0 1 2 3 4 5 6 7 -local repository version: 8 -``` - -Running on openSUSE Tumbleweed. - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -Yes, mostly a happy user ;-) - -> [[dup|done]] diff --git a/doc/bugs/smudge_filter_drastically_slows_down_git_diff/comment_1_249cf95263d05cb6ed8ca4d895d58c54._comment b/doc/bugs/smudge_filter_drastically_slows_down_git_diff/comment_1_249cf95263d05cb6ed8ca4d895d58c54._comment deleted file mode 100644 index 56b08e612a..0000000000 --- a/doc/bugs/smudge_filter_drastically_slows_down_git_diff/comment_1_249cf95263d05cb6ed8ca4d895d58c54._comment +++ /dev/null @@ -1,36 +0,0 @@ -[[!comment format=mdwn - username="branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e" - nickname="branchable" - avatar="http://cdn.libravatar.org/avatar/ae41dba34ee6000056f00793c695be75" - subject="GIT_TRACE=1 debug shows smudge --clean is run twice per file" - date="2020-07-16T17:06:12Z" - content=""" -I just realised that `GIT_TRACE=1 git diff` gives much more useful debug, and reveals that `smudge --clean` is run twice per file, as shown below. - -Based on [[todo/Long_Running_Filter_Process]], running it once per file would be expected (although somewhat unfortunate), but twice seems more like a bug, especially given that it causes the exact same set of git commands to be run twice. Or is there a good reason for that? - -``` -18:00:20.218359 git.c:442 trace: built-in: git diff -18:00:20.219470 run-command.c:663 trace: run_command: unset GIT_PAGER_IN_USE; LV=-c 'less -h100 -i -M -q -R -W -y100 -X -F -S' -18:00:20.221554 run-command.c:663 trace: run_command: 'git-annex smudge --clean -- '\''README.md'\''' -18:00:20.358194 git.c:442 trace: built-in: git config --null --list -18:00:20.376580 git.c:442 trace: built-in: git cat-file --batch -18:00:20.377090 git.c:442 trace: built-in: git cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' -18:00:20.395429 git.c:442 trace: built-in: git show-ref git-annex -18:00:20.413485 git.c:442 trace: built-in: git show-ref --hash refs/heads/git-annex -18:00:20.430842 git.c:442 trace: built-in: git log refs/heads/git-annex..f8cdef27e7fae6b3cc312130b86aa3c553d2ff3b '--pretty=%H' -n1 -18:00:20.451166 git.c:442 trace: built-in: git cat-file --batch -18:00:20.456687 git.c:442 trace: built-in: git cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' -18:00:20.467209 git.c:442 trace: built-in: git check-attr -z --stdin annex.backend annex.numcopies annex.largefiles -- -18:00:20.474348 run-command.c:663 trace: run_command: 'git-annex smudge --clean -- '\''README.md'\''' -18:00:20.607471 git.c:442 trace: built-in: git config --null --list -18:00:20.620552 git.c:442 trace: built-in: git cat-file --batch -18:00:20.626814 git.c:442 trace: built-in: git cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' -18:00:20.639210 git.c:442 trace: built-in: git show-ref git-annex -18:00:20.656755 git.c:442 trace: built-in: git show-ref --hash refs/heads/git-annex -18:00:20.675204 git.c:442 trace: built-in: git log refs/heads/git-annex..f8cdef27e7fae6b3cc312130b86aa3c553d2ff3b '--pretty=%H' -n1 -18:00:20.692108 git.c:442 trace: built-in: git cat-file --batch -18:00:20.706338 git.c:442 trace: built-in: git cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' -18:00:20.708876 git.c:442 trace: built-in: git check-attr -z --stdin annex.backend annex.numcopies annex.largefiles -- -``` -"""]] diff --git a/doc/bugs/smudge_filter_drastically_slows_down_git_diff/comment_2_98cdd89081d2f31a45b693a04b20f345._comment b/doc/bugs/smudge_filter_drastically_slows_down_git_diff/comment_2_98cdd89081d2f31a45b693a04b20f345._comment deleted file mode 100644 index 9f35e654f5..0000000000 --- a/doc/bugs/smudge_filter_drastically_slows_down_git_diff/comment_2_98cdd89081d2f31a45b693a04b20f345._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-07-21T14:00:05Z" - content=""" -This is exactly what that todo is about. I'm going to close this as a dup. - -I suppose git diff runs it twice because it's smudging both sides of the diff. - -Note that git status does not re-run the smudge filter on the same file -repeatedly (at least most of the time), so the overhead there is only one -run per modification of a file. Unfortunately git diff does not benefit from -such caching. Still, it only runs once per modified file in the repo, -not on unmodified files. -"""]] diff --git a/doc/bugs/stack_build_failure_with_recent_update.mdwn b/doc/bugs/stack_build_failure_with_recent_update.mdwn deleted file mode 100644 index 79e906b00c..0000000000 --- a/doc/bugs/stack_build_failure_with_recent_update.mdwn +++ /dev/null @@ -1,65 +0,0 @@ -On 10abf964f (add new deps, 2020-06-30), `stack build` fails for me -with - -``` -/home/kyle/src/haskell/git-annex/git-lfs 1.1.0/: getDirectoryContents:openDirStream: does not exist (No such file or directory) -``` - -I think this is due to a hyphen missing before the version. I've -added those locally to stack.yaml (assuming that's correct, -stack-lts-12.14.yaml should of course get the same update). - -[[!format diff """ -diff --git a/stack.yaml b/stack.yaml -index 8724904f8..0a65a66b6 100644 ---- a/stack.yaml -+++ b/stack.yaml -@@ -24,8 +24,8 @@ extra-deps: - - sandi-0.5 - - tasty-rerun-1.1.17 - - torrent-10000.1.1 -- - git-lfs 1.1.0 -- - http-client-restricted 0.0.2 -+ - git-lfs-1.1.0 -+ - http-client-restricted-0.0.2 - explicit-setup-deps: - git-annex: true - resolver: lts-14.27 -"""]] - - -After doing that, it looks like there is a dependency mismatch that -can't be resolved: - -``` -Error: While constructing the build plan, the following exceptions were encountered: - -In the dependencies for http-client-restricted-0.0.2: - http-client-0.6.4 from stack configuration does not match >=0.5 && <0.6 (latest matching version - is 0.5.14) -needed due to git-annex-8.20200617 -> http-client-restricted-0.0.2 - -Some different approaches to resolving this: - - * Set 'allow-newer: true' - in /home/kyle/.stack/config.yaml to ignore all version constraints and build anyway. - - * Recommended action: try adding the following to your extra-deps - in /home/kyle/src/haskell/git-annex/stack.yaml: - -- http-client-0.5.14@sha256:4880b27d6741e331454a1d4c887d96ce3d7d625322c8433983a4b1cd08538577,5348 - -Plan construction failed. -``` - -Here's my stack version, which is running on top of Debian Buster: - -``` -$ stack --version -Version 2.3.1, Git revision de2a7b694f07de7e6cf17f8c92338c16286b2878 (8103 commits) x86_64 hpack-0.33.0 -``` - -Locally, I've set `httpclientrestricted` and `gitlfs` flags to false -to use the vendored version for now. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/stack_build_failure_with_recent_update/comment_1_5844bc527037a213f2fd7f1a14726142._comment b/doc/bugs/stack_build_failure_with_recent_update/comment_1_5844bc527037a213f2fd7f1a14726142._comment deleted file mode 100644 index a800ca2dca..0000000000 --- a/doc/bugs/stack_build_failure_with_recent_update/comment_1_5844bc527037a213f2fd7f1a14726142._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="comment 1" - date="2020-06-30T21:24:56Z" - content=""" -Confirmed on my end. Thanks for the quick fix! -"""]] diff --git a/doc/bugs/surprising_behavior_operating_on_file_behind_symlink.mdwn b/doc/bugs/surprising_behavior_operating_on_file_behind_symlink.mdwn deleted file mode 100644 index b7893cc569..0000000000 --- a/doc/bugs/surprising_behavior_operating_on_file_behind_symlink.mdwn +++ /dev/null @@ -1,12 +0,0 @@ -If foo is a symlink to bar, and foo/file exists, then `git annex add bar/file` -silently does nothing. So do other commands if the file is already annexed. - -git's behavior on these is to complain that it's "beyond a symbolic link" -and fail. - -This happens because git ls-files doesn't list the file, but then I think -the code that handles erroring if a file that does not exist is specified -doesn't catch it because the file does exist, it's just behind the symlink. ---[[Joey]] - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/sync_content_copies_unwanted_removed_files.mdwn b/doc/bugs/sync_content_copies_unwanted_removed_files.mdwn deleted file mode 100644 index 88ff237753..0000000000 --- a/doc/bugs/sync_content_copies_unwanted_removed_files.mdwn +++ /dev/null @@ -1,42 +0,0 @@ -A couple of times I have seen a git-annex sync -C . upload files to the -remote that are not part of the remote's preferred content. In the most -recent case, I had moved the file to another directory while the sync was -downloading a previous file. I suspect that the file being removed causes -preferred content checks to mess up. --[[Joey]] - -> Reproduced reliably as follows: Have a bigfile in the remote -> and a smallfile in the local repo. Have the remote's preferred content -> be "not (copies=1)". Have the local repo's preferred content -> `include=*`. Run `git-annex sync -C.` while that's running, `git rm -> smallfile`. (bigfile has to be big enough to give time to run that -> command) -> -> smallfile gets sent to the remote unexpectedly. If it's not deleted -> first, that does not happen. -> -> --[[Joey]] - -> > Hmm, so limitCopies uses checkKey, which for MatchingFile, uses -> > lookupKey. And with a deleted file, lookupKey falls -> > into a case where it uses catKeyFile, but since the file has been -> > removed from the index, that also fails. And when it fails, -> > that means it assumes it does not have 1 copy, and so the -> > "not (copies=1)" evalulates to true, so it thinks it's matched as -> > preferred content. -> > -> > The preferred content is being checked via wantSend, which already knows -> > the key in this case. -> > -> > It knows the key already because sync uses seekFilteredKeys and so it's -> > already streamed the file though and looked up the key before -> > it's deleted. If the file got deleted before that could look up the -> > key, it would skip it. It may be that recent changes to add this -> > streaming for performance led to this bug. -> > -> > So one fix might be to change it to use MatchingKey, -> > and so avoid the later lookup? Investigating the git history -> > and the code I see no reason not to do this. It didn't used to be that -> > MatchingKey included an AssociatedFile, which is probably why it was -> > not used in this case originally. - -[[fixed|done]] diff --git a/doc/bugs/sync_to_export_with-J_STM_error.mdwn b/doc/bugs/sync_to_export_with-J_STM_error.mdwn deleted file mode 100644 index 5777048dcc..0000000000 --- a/doc/bugs/sync_to_export_with-J_STM_error.mdwn +++ /dev/null @@ -1,40 +0,0 @@ - git annex sync exportremote -J2 --content - ... - git-annex: thread blocked indefinitely in an MVar operation - failed - git-annex: thread blocked indefinitely in an STM transaction - -Also, git-annex export -J2 crashes the same way. I discovered this bug -when adding -J to export, but then found sync had the same bug. - -To reproduce this, there may need there to be a tree of several annexed -files whose content is not locally available. In my case, -there were 338 of them. It seems to act on almost all before -crashing. --[[Joey]] - ----- - -It's crashing in finishCommandActions. DebugLocks does not show a backtrace. - -Dumping the worker pool inside the crashing STM action, it looks like this: - -WorkerPool UsedStages {initialStage = PerformStage, stageSet = fromList [PerformStage,CleanupStage]} [IdleWorker StartStage,ActiveWorker PerformStage,IdleWorker PerformStage,IdleWorker StartStage,IdleWorker PerformStage,IdleWorker StartStage,IdleWorker CleanupStage,IdleWorker CleanupStage,IdleWorker CleanupStage] 8 - -Always ends with an ActiveWorker PerformStage. So a worker thread is -apparently still running, but the retry blocks indefinitely, so -somehow the worker thread never transitions back to idle. - -Also, the MVar crash is not from this code, so maybe the MVar crash is -the real culprit and it just also leads to the STM crash. - ---- - -Added debugLocks to the MVar uses in Command.Export, and it's -the one in failedsend that is causing the MVar deadlock. So that must be -the root cause. Looks like fillExport is starting threads with -commandActions, but then assumes they'll all be done, so takes a MVar, -before all the threads *are* done, so a thread tries to modify the MVar and -deadlocks. - -Ok, [[fixed|done]] by using includeCommandAction instead, although that -does reduce the actual concurrency. --[[Joey]] diff --git a/doc/bugs/unlock_should_warn_if_file_isn__39__t_in_repo.mdwn b/doc/bugs/unlock_should_warn_if_file_isn__39__t_in_repo.mdwn deleted file mode 100644 index c131e226c4..0000000000 --- a/doc/bugs/unlock_should_warn_if_file_isn__39__t_in_repo.mdwn +++ /dev/null @@ -1,57 +0,0 @@ -### Please describe the problem. - -I'm using git-annex with the [calibre e-book library](https://calibre-ebook.com/) software. Sometimes calibre will rename a directory. After a directory rename git annex unlock no longer works. It works once I've committed the changes to git. - -It would be nice if git-annex could give an error to explain why the unlock fails. I'd even be happy with an extra flag to unlock, like --debug or --verbose to show the message. - -### What steps will reproduce the problem? - -1. create a directory inside a repository -2. save a file in the directory -3. add the file to git annex and commit -4. rename the directory -5. try unlocking the file - -### What version of git-annex are you using? On what operating system? - -6.20161012 on Debian - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - - ~/scratch$ mkdir annex - ~/scratch$ cd annex/ - ~/scratch/annex$ git init - Initialized empty Git repository in /home/edward/scratch/annex/.git/ - ~/scratch/annex (master)$ git annex init --version=6 - init ok - (recording state in git...) - ~/scratch/annex (master)$ mkdir foo - ~/scratch/annex (master)$ cd foo - ~/scratch/annex/foo (master)$ echo test > test - ~/scratch/annex/foo (master)$ git annex add test - add test ok - (recording state in git...) - ~/scratch/annex/foo (master)$ git commit -m 'test' - [master (root-commit) 6368036] test - 1 file changed, 1 insertion(+) - create mode 120000 foo/test - ~/scratch/annex/foo (master)$ cd .. - ~/scratch/annex (master)$ mv foo bar - ~/scratch/annex (master)$ cd bar - ~/scratch/annex/bar (master)$ git annex unlock test - ~/scratch/annex/bar (master)$ - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -git-annex is amazing, I use it all the time. Thanks! - -> annex.skipunknown false will make git-annex error out in this situation. -> That will become the default in a couple of years, but can be set already -> by those who don't like the behavior of skipping. [[done]] --[[Joey]] diff --git a/doc/bugs/unlock_should_warn_if_file_isn__39__t_in_repo/comment_1_4fb6a9aa7cd3b141b2d2f1b4acd2092f._comment b/doc/bugs/unlock_should_warn_if_file_isn__39__t_in_repo/comment_1_4fb6a9aa7cd3b141b2d2f1b4acd2092f._comment deleted file mode 100644 index 7fe2096045..0000000000 --- a/doc/bugs/unlock_should_warn_if_file_isn__39__t_in_repo/comment_1_4fb6a9aa7cd3b141b2d2f1b4acd2092f._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2016-11-07T17:25:10Z" - content=""" -In general, git-annex silently skips files that are not known to git, -or are not annexed files. That's what's happening here. - -This skipping behavior can sometimes be confusing, if you are not -familiar with it. But it's also very convenient. For example `git annex -unlock *` will unlock only annexed files and not complain about any other -files; `git annex unlock .` will recursively unlock only annexed files and not -complain about non-annexed files or files that are already unlocked. - -In your example, git-annex has no way to tell that the file has been -renamed; a file that has never been added to git would look the same -to it as that renamed file. -"""]] diff --git a/doc/bugs/unlock_should_warn_if_file_isn__39__t_in_repo/comment_2_2e878c5ae1892f16f5b4e515fbe8875b._comment b/doc/bugs/unlock_should_warn_if_file_isn__39__t_in_repo/comment_2_2e878c5ae1892f16f5b4e515fbe8875b._comment deleted file mode 100644 index 8f34197b6c..0000000000 --- a/doc/bugs/unlock_should_warn_if_file_isn__39__t_in_repo/comment_2_2e878c5ae1892f16f5b4e515fbe8875b._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="edward" - avatar="http://cdn.libravatar.org/avatar/1e64ab07e0faced09520a5c589deb70b" - subject="comment 2" - date="2016-11-07T17:43:31Z" - content=""" -I see your point, feel free to close this bug. - -There is a way for git-annex to tell that the file is probably meant to be part of the repo because it is a symlink, it doesn't look the same as a new file. - - ~/scratch/annex/bar (master)$ ls -l - total 4 - lrwxrwxrwx 1 edward edward 181 Nov 4 18:02 test -> ../.git/annex/objects/w8/pv/SHA256E-s5--f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2/SHA256E-s5--f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2 - ~/scratch/annex/bar (master)$ -"""]] diff --git a/doc/bugs/upgrade_to_V8_fails.mdwn b/doc/bugs/upgrade_to_V8_fails.mdwn deleted file mode 100644 index 5f9069affa..0000000000 --- a/doc/bugs/upgrade_to_V8_fails.mdwn +++ /dev/null @@ -1,37 +0,0 @@ -### Please describe the problem. - -Automatic upgrade fails with error messages like - -git-annex: Repository /data/images is at unsupported version 7. Automatic upgrade exception! roebi.jpg: getSymbolicLinkStatus: does not exist (No such file or directory) - -This is the second repository where this happens. - -The problem occurs when issuing any git-annex command from the command line, like git-annex add, sync etc. - -### What steps will reproduce the problem? - -not clear when this happens. Both, the file and the symlink exist. -The file is symlinked from two directories within git-annex. - -### What version of git-annex are you using? On what operating system? - -Arch Linux -git-annex version: 8.20200226-g8bc393a63 -the error first apperard in 8.20200226-5 - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - - -everything worked like a charm until this recent change - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/upgrade_to_V8_fails/comment_1_ef44b98740b33d6a7cab0cc5caf4be15._comment b/doc/bugs/upgrade_to_V8_fails/comment_1_ef44b98740b33d6a7cab0cc5caf4be15._comment deleted file mode 100644 index 895a649d77..0000000000 --- a/doc/bugs/upgrade_to_V8_fails/comment_1_ef44b98740b33d6a7cab0cc5caf4be15._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="atomic/transactional upgrades" - date="2020-03-05T19:10:16Z" - content=""" -Related question: is [[git-annex-upgrade]] atomic, in the sense of restoring old state if an upgrade fails? -"""]] diff --git a/doc/bugs/upgrade_to_V8_fails/comment_2_907374c4bc0a9abe35fbe984016a7fd5._comment b/doc/bugs/upgrade_to_V8_fails/comment_2_907374c4bc0a9abe35fbe984016a7fd5._comment deleted file mode 100644 index 2831632277..0000000000 --- a/doc/bugs/upgrade_to_V8_fails/comment_2_907374c4bc0a9abe35fbe984016a7fd5._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-03-09T17:46:31Z" - content=""" -You need to tell me a sequence of commands that can reproduce this problem. -At the moment, I don't know how to, and so don't know how to fix it. - -If you don't know how to reporoduce the problem, you need to at least -more clearly explain it. I don't know what it could mean for a file to be -"symlinked from two directories". What was the full filename of the -file that you deleted? What was the path in which you ran git-annex? -"""]] diff --git a/doc/bugs/upgrade_to_V8_fails/comment_2_93dcb2d4da07eba998e88c62d0ea38d3._comment b/doc/bugs/upgrade_to_V8_fails/comment_2_93dcb2d4da07eba998e88c62d0ea38d3._comment deleted file mode 100644 index 02cf165d85..0000000000 --- a/doc/bugs/upgrade_to_V8_fails/comment_2_93dcb2d4da07eba998e88c62d0ea38d3._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="scinu" - avatar="http://cdn.libravatar.org/avatar/c5a190c5c0ce61a5be141609dff37fe1" - subject="Problem solved (partially)" - date="2020-03-09T17:00:52Z" - content=""" -If I \"git rm\" the files causing problems, the upgrade to V8 runs through. - -So, I'm not sure what's causing the problem, but at least I can work again. - -Best, Scinu -"""]] diff --git a/doc/bugs/upgrade_to_V8_fails/comment_4_98739a9ae1bad44add6245fcded692c5._comment b/doc/bugs/upgrade_to_V8_fails/comment_4_98739a9ae1bad44add6245fcded692c5._comment deleted file mode 100644 index b4d03f2093..0000000000 --- a/doc/bugs/upgrade_to_V8_fails/comment_4_98739a9ae1bad44add6245fcded692c5._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-03-09T20:53:45Z" - content=""" -I was able to reverse engineer the error message by deleting one of the -file in the git repository, but not staging the deletion. - -That does not seem to match what you were saying about the symlink -existing, but perhaps it was upgrading some other repository where -the file was deleted, or very likely I am failing to understand what you -were trying to explain. - -Since what I've fixed is the only way it could throw that exception, I'm -going to assume this bug is fixed, but if you try the next version of -git-annex and it still has the problem, do follow up. -"""]] diff --git a/doc/bugs/upgrade_to_V8_fails/comment_5_e6de2eb142feb3f44a9eeda75b5bb109._comment b/doc/bugs/upgrade_to_V8_fails/comment_5_e6de2eb142feb3f44a9eeda75b5bb109._comment deleted file mode 100644 index 0bbc51459c..0000000000 --- a/doc/bugs/upgrade_to_V8_fails/comment_5_e6de2eb142feb3f44a9eeda75b5bb109._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="scinu" - avatar="http://cdn.libravatar.org/avatar/c5a190c5c0ce61a5be141609dff37fe1" - subject="more specifics" - date="2020-03-10T03:49:04Z" - content=""" -Thanks a lot! - -The double symlinks exist, since I often have collections of files living in several subdirectories. This is useful for my scientific paper collection which is one big pile, but I also copy some git-annex symlink to several topical subdiretories. Since its just a symlink this does not take up space. The big advantage is to be able to get the contents of specific topics I'm working on, and dropping them if I don't need them. - -I use the same structure for data files that I only use occasionally. (I know about tags, but this is much more convenient.) - -As for the error, it's difficult to debug for me. Possibly its the one you found (symlink to non-existing content). Can I go back to the V7 state of the repository by checking out an older git version? If not, I might try on some harddrives that I haven't synced yet since the version 7->8 upgrade. -"""]] diff --git a/doc/bugs/upgrade_to_V8_fails/comment_6_8a4dd651d5123e9d6ae15275a730261e._comment b/doc/bugs/upgrade_to_V8_fails/comment_6_8a4dd651d5123e9d6ae15275a730261e._comment deleted file mode 100644 index a3457b7cf9..0000000000 --- a/doc/bugs/upgrade_to_V8_fails/comment_6_8a4dd651d5123e9d6ae15275a730261e._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 6""" - date="2020-03-10T17:28:24Z" - content=""" -What do you mean when you talk about "double symlinks"? - -That could meany any of several things, possibly dozens of things. I am not -a mind reader. -"""]] diff --git a/doc/bugs/upgrade_to_V8_fails/comment_8_2f106656bb3cee79a41eb086af45f0d2._comment b/doc/bugs/upgrade_to_V8_fails/comment_8_2f106656bb3cee79a41eb086af45f0d2._comment deleted file mode 100644 index 47997978de..0000000000 --- a/doc/bugs/upgrade_to_V8_fails/comment_8_2f106656bb3cee79a41eb086af45f0d2._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 8" - date="2020-03-10T22:22:45Z" - content=""" -Wouldn't it be `/papers/topic1/a.pdf -> ../.git/annex/objects/pm/XX/SHA256E-s107393... `? -"""]] diff --git a/doc/bugs/upgrade_to_V8_fails/comment_9_eea7e7aaccded131b201acabfe8ac699._comment b/doc/bugs/upgrade_to_V8_fails/comment_9_eea7e7aaccded131b201acabfe8ac699._comment deleted file mode 100644 index 6700a6affc..0000000000 --- a/doc/bugs/upgrade_to_V8_fails/comment_9_eea7e7aaccded131b201acabfe8ac699._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="scinu" - avatar="http://cdn.libravatar.org/avatar/c5a190c5c0ce61a5be141609dff37fe1" - subject="comment 9" - date="2020-03-11T04:51:17Z" - content=""" -yes, you're right -"""]] diff --git a/doc/bugs/upgrade_to_V8_fails/comment_9_f7d7b8a5defbc5f546545ffd9475c9df._comment b/doc/bugs/upgrade_to_V8_fails/comment_9_f7d7b8a5defbc5f546545ffd9475c9df._comment deleted file mode 100644 index dd8a32d7bc..0000000000 --- a/doc/bugs/upgrade_to_V8_fails/comment_9_f7d7b8a5defbc5f546545ffd9475c9df._comment +++ /dev/null @@ -1,27 +0,0 @@ -[[!comment format=sh - username="scinu" - avatar="http://cdn.libravatar.org/avatar/c5a190c5c0ce61a5be141609dff37fe1" - subject="comment 9" - date="2020-03-10T22:17:20Z" - content=""" -Sorry for not being clear - -for example /papers is a directory under git annex, - -a.pdf is a symlink to a file in the annex (annex filename abbreviated) as created by \"git-annex add\" - -topic1/a.pdf is another symlink to the same file in the annex --> this is a copy of a.pdf, registered with \"git-annex add\" -topic2/a.pdf is yet another symlink to the same file in the annex --> this is another copy of a.pdf, registered with \"git-annex add\" - -so the directory listing would look like this: - -/papers/.git/annex/.... -/papers/a.pdf -> .git/annex/objects/pm/XX/SHA256E-s107393... - -/papers/topic1/a.pdf -> .git/annex/objects/pm/XX/SHA256E-s107393... -/papers/topic2/a.pdf -> .git/annex/objects/pm/XX/SHA256E-s107393... - -All of these symlinks point to the same file on disk under /papers/.git/annex/... - - -"""]] diff --git a/doc/bugs/v5_direct_upgrade_fail_with_annex.alwayscommit=false.mdwn b/doc/bugs/v5_direct_upgrade_fail_with_annex.alwayscommit=false.mdwn deleted file mode 100644 index 65810f5918..0000000000 --- a/doc/bugs/v5_direct_upgrade_fail_with_annex.alwayscommit=false.mdwn +++ /dev/null @@ -1,39 +0,0 @@ -`upgrade` will fail and leave the repo broken when run in a -v5 direct repo with annex.alwayscommit=false and not synced since the last -change. - -> Forwarded from a private email to me, have not reproduced yet. --[[Joey]] - -Update: Got a repo tarball sent to me that reproduces the problem. It looks like -this: - - git annex upgrade - upgrade (v5 to v6...) - caught exception: Somehow no branch is checked out - - failed - (recording state in git...) - git-annex: upgrade: 1 failed - -.git/HEAD points to refs/heads/annex/direct/master, -but that ref does not actually exist yet. - -The repo was created with git-annex 7.20190731-gbb16a2610 -and this sequence of commands: - - git init . - - git config user.email "email@example.com" - git config user.name "Name Name" - git config annex.alwayscommit false - git config annex.autoupgraderepository false - git annex init foobar - git annex direct - echo "aaaa" > a && echo "bbbb" > b - git annex add a b - git annex drop --force a - -Note that lack of git-annex sync, which is why the branch -never got created. The index file has the adds staged of course. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/v7_upgrade_of_local_clone_bug.mdwn b/doc/bugs/v7_upgrade_of_local_clone_bug.mdwn deleted file mode 100644 index 1192f9bedf..0000000000 --- a/doc/bugs/v7_upgrade_of_local_clone_bug.mdwn +++ /dev/null @@ -1,39 +0,0 @@ -When a local clone was at v7 and gets upgraded to v8 by a command run in -a repo that has it as a remote, this is displayed: - -"fatal: ../path/to/clone is outside repository" - -This happens because git ls-files is run to list the files of the clone. -But, it has some strange behavior when relative paths are used. Result is -it always fails. - -This also causes the keys database of the clone to not get -repopulated after being deleted for the upgrade. -That's not a fatal problem because git-annex is always prepared for the -keys database being out of date, but it could result in considerably more -work being done later. - -Also, the associated files does not get repopulated when it fails like -that. That could cause worse problems. I have not tried to see what happens -when the repo that fails to be upgraded has unlocked files. - -I also found some v1 upgrade code that does the same thing, and presumably -also has the problem, although there are probably no v1 repos left. -This seems like it could be a larger problem that only upgrades, but -luckily upgrades seem like the only time that git-annex, running in one -repo, needs to do anything involving listing the files in the working tree -of a remote. - -Hmm, Upgrade.V5 also has a LsFiles.stagedDetails -that looks like it would have then problem. That would affect upgrading -from a V5 direct mode repo to V6. - ---[[Joey] - -> Fixed all such upgrade bugs by changing how local remotes are upgraded, -> running git-annex upgrade inside the remote. -> -> Also, guarded all git ls-files calls with a check that it's not being -> run on a local remote, just in case there are other such bugs. -> -> [[done]] --[[Joey]] diff --git a/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__.mdwn b/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__.mdwn deleted file mode 100644 index a7d87953aa..0000000000 --- a/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__.mdwn +++ /dev/null @@ -1,75 +0,0 @@ -### Please describe the problem. - -I have a special directory remote with exporttree=yes (encryption=none) on an USB hard drive. Both `git annex sync --content` and `git annex export` only write around 400 KiB/s. Thus an export of a 9GB DVD iso takes a whole night. - -The drive is not blazing fast, but: - -- `sync; dd if=/dev/zero of=tempfile bs=1M count=10; sync` gives something around 10MB/s (don't recall the exact number) -- rsync (with --progress turned on) copies files with 2.35MB/s - -`mount` for this drive shows: - -> /dev/sdc1 on /media/thk/thk-sg1 type ext4 (rw,nosuid,nodev,relatime,sync,stripe=8191,uhelper=udisks2) - -I tried to mount the drive without sync but failed. Even with the usdisks2 service stopped I could not manually mount the drive without sync (or with async). It always ended up being mounted with sync. - -### What steps will reproduce the problem? - -TODO(thk): try other drive and other laptop once the current transfer finishes... - -Update 2020-03-07: - -- export to a different USB drive (both seagate, same size, similar age) from the same machine with the exact same setup (but NTFS filesystem) runs with ~80 MiB/s. So this is perfect. This time there is also no problem with a lost exporttree=yes config. - -Update 2020-03-10: - -- Export from a different laptop, same drive goes blazingly fast with >200MB/s. - -### What version of git-annex are you using? On what operating system? - -- git-annex version: 8.20200227-gf56dfe791 -- Debian testing with Kernel 5.2.17 - -### Please provide any additional information below. - -I now learned that there is no Linux kernel primitive to copy a file but that this is actually a high art: - - - -I was surprised to see the implementation of `meteredWrite` in *Utility/Metered.hs*. I hoped that there would be some haskell standard library for efficient file copying? I wonder how rsync implements its progress meter? And whether the progress meter is the reason why rsync had slower write speed than dd. - -Maybe it would make sense to call out to the *cp* command and just issue a *stat()* every few seconds for the progress meter? This is what I do to monitor cp progress manually. - -I have no clue, but maybe these could help for fast file copying in Haskell? - -- -- -- reddit: [What is your take on conduits, pipes, and streams?](https://www.reddit.com/r/haskell/comments/7w79q1/what_is_your_take_on_conduits_pipes_and_streams/) - -### Have you had any luck using git-annex before? - -Well, I'm coming back to git-annex after several years. So far it is better than I remembered: - -- tor support is great and solves the need for a central server -- I hope that the sqlite integration will now make large collections of files managable -- Finally we have exporttree, yeah! - - -## 2020-03-07 update - -Turns out, the problem is more complex. I wanted to be clever. When I set up the two synced annex repos I made the mistake of not specifying exporttree=yes at the beginning. But I wanted to re-use the initial name. So I tried hard to remove all evidence of the previous existence of a special remote with that name from git-annex. - -I checked out the git-annex branch in a separate worktree (see **man git worktree**) in both repositories, deleted the lines for that remote from remote.log and pushed to the other repo (not git annex sync). I even made the changes in parallel in both repos before pushing in both directions so that the special merge does not bring the lines back. I actually was sure there was nothing left of the old remotes. Of course I also deleted them from .git/config. - -Somehow, there is again a line in remote.log for that remote without exporttree=yes. So now, after the last git annex sync --content, I have a mixture of an exported tree and an exported annex object store in the same special remote dir. - -I also noticed that the repo that was so slow did not have the `remote.$REMOTE.annex-tracking-branch` config. But I could still run `sync --content` somehow. After adding this config, the last sync actually ran with 2 MB/s but it still wrote in object store format, not as an exported tree. - -Some questions: - -- Is there any other place where git-annex stores information about remotes then remote.log? -- The object store files in the remote were stored in format AAA/BBB/$HASH with three character directory names. While in .git/annex/objects the folders have two characters. What are those characters? I believe the 3 characters format is for remotes that potentially do not distinguish letter case? -- Is there a command to get the full path of a file in the object store (two or three letters) from the hash? -- Maybe there is still a bug. Is there a possibility that git-annex could forget that a remote is configured with exporttree=yes? Especially if I export to the same directory on the same usb drive from two synced repos? - -[[done]] diff --git a/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__/comment_1_921af0fe20bfe690c008f0173394b765._comment b/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__/comment_1_921af0fe20bfe690c008f0173394b765._comment deleted file mode 100644 index 86b5a9e0ac..0000000000 --- a/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__/comment_1_921af0fe20bfe690c008f0173394b765._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-16T17:37:27Z" - content=""" -The directory special remote uses haskell's native ByteString library, -which tends to be easily capable of fully saturating IO on most systems. -The imposition of a meter by Utility.Metered does not change that -appreciably, it just uses a small amount of CPU time to update a counter -and occasionally display progress. - -ByteString uses a default 32kb chunk size, so if having the drive mounted sync -means it flushes the cache after every 32kb write, well there's your problem. - -All your other questions have been discussed elsewhere on this website -AFAIK, and I'm not going to try to discuss all that here. Bug reports -should be about a single bug. -"""]] diff --git a/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__/comment_2_a485eecc4e2541d1f56f90ec3ea9296b._comment b/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__/comment_2_a485eecc4e2541d1f56f90ec3ea9296b._comment deleted file mode 100644 index 94f0f9d313..0000000000 --- a/doc/bugs/very_slow_export_to_directory_special_remote___40__on_USB_drive__41__/comment_2_a485eecc4e2541d1f56f90ec3ea9296b._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="thk" - avatar="http://cdn.libravatar.org/avatar/bfef10a428769701aeee1db978951461" - subject="Please close" - date="2020-03-16T20:46:26Z" - content=""" -Yes. all issues are solved now. Please close this, I don't know how to close bugs here. -"""]] diff --git a/doc/bugs/view.mdwn b/doc/bugs/view.mdwn deleted file mode 100644 index e1e44370cb..0000000000 --- a/doc/bugs/view.mdwn +++ /dev/null @@ -1,71 +0,0 @@ -### Please describe the problem. - -git-annex-view fails with an error message that `viewindex.lock` exists and another git process is running in the repository. -The problem appears even with newly initialized repositories. - -### What steps will reproduce the problem? - -The steps I followed to reproduce the problem: - -[[!format sh """ -cd /tmp -mkdir testgit -cd testgit -git init -git annex init -echo 123 > test1 -echo 456 > test2 -git annex add test1 test2 -git commit -m 'initial commit' -git annex metadata -s test=test1 test1 -git annex metadata -s test=test2 test2 -git annex view test=test1 -"""]] - -Note: The problem does not appear with git-annex version 7.20190819+git2-g908476a9b-1~ndall+1 but it appears with 8.20201007-g903b2f1. - - -### What version of git-annex are you using? On what operating system? - -I could reproduce the problem at two different machines, both with Ubuntu 18.04 LTS. -`git-annex` was installed via conda using the command `conda install -c conda-forge git-annex` - -``` -git-annex version: 8.20201007-g903b2f1 -build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV -dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X* -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external -operating system: linux x86_64 -supported repository versions: 8 -upgrade supported from repository versions: 0 1 2 3 4 5 6 7 -local repository version: 8 -``` - - -### Please provide any additional information below. - -Here is the complete error message: - -[[!format sh """ -view (searching...) fatal: Unable to create '/tmp/mytmp/.git/annex/viewindex.lock': File exists. - -Another git process seems to be running in this repository, e.g. -an editor opened by 'git commit'. Please make sure all processes -are terminated then try again. If it still fails, a git process -may have crashed in this repository earlier: -remove the file manually to continue. - -git-annex: failed to read sha from git write-tree -CallStack (from HasCallStack): - error, called at ./Git/Sha.hs:23:15 in main:Git.Sha -failed -git-annex: view: 1 failed -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -git-annex is an awesome tool! Really appreciate your work and everything else works great! - -> Thank you for reporting, I've fixed this reversion and am also going to -> add something like that to the test suite, which lacked any testing of -> views. [[done]] --[[Joey]] diff --git a/doc/bugs/warning_about_ssh_caching_keeps_showing.mdwn b/doc/bugs/warning_about_ssh_caching_keeps_showing.mdwn deleted file mode 100644 index 987a853cd1..0000000000 --- a/doc/bugs/warning_about_ssh_caching_keeps_showing.mdwn +++ /dev/null @@ -1,59 +0,0 @@ -### Please describe the problem. -I keep getting the warning about ssh caching being disabled, even when I explicitly enable it. - -### What steps will reproduce the problem? -See log below - -### What version of git-annex are you using? On what operating system? -7.20200204 on Amazon Linux 2 - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - -(just-git-annex-env) 13:00 [viral-ngs-benchmarks] $ git annex sync -c annex.sshcaching=true -On branch is-devel -Your branch is up to date with 'origin/is-devel'. - - -It took 8.50 seconds to enumerate untracked files. 'status -uno' -may speed it up, but you have to be careful not to forget to add -new files yourself (see 'git help status'). -nothing to commit, working tree clean -commit ok -pull origin - You have enabled concurrency, but ssh connection caching is not enabled. This may result in multiple ssh processes prompting for pas\ -swords at the same time. -ok - -(just-git-annex-env) 13:00 [viral-ngs-benchmarks] $ uname -a -Linux ip-172-31-86-201.ec2.internal 4.14.165-131.185.amzn2.x86_64 #1 SMP Wed Jan 15 14:19:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux -(just-git-annex-env) 13:02 [viral-ngs-benchmarks] $ git annex version -git-annex version: 7.20200204-g4db801d -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.21.1 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sql\ -ite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_2\ -24 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE\ -2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224\ - BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: linux x86_64 -supported repository versions: 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -local repository version: 7 - - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - -I've been using git-annex for 1.5 years to manage bioinformatics analyses. It's a very versatile and well-designed tool. I've been able to adapt it to many use cases; -the ability to easily write your own external backends has been especially helpful for that. The amount of work and thought that has gone into designing/building git-annex is -enormous, and very much appreciated. - -> [[done]]; see comment --[[Joey]] diff --git a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_1_711a8e784ba8fde0164469fa08176687._comment b/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_1_711a8e784ba8fde0164469fa08176687._comment deleted file mode 100644 index b09d81c481..0000000000 --- a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_1_711a8e784ba8fde0164469fa08176687._comment +++ /dev/null @@ -1,25 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-02-14T17:48:41Z" - content=""" -It seems you must have annex.jobs set to something, since concurrency -is enabled without any -J option, so the easy fix is just to unset that. - -It kind of looks like your build of git-annex may have been made without -ssh connection caching support, which would happen if its configure program -detected at build time that ssh doesn't support it. - -That would be unusual if so, all the builds of git-annex that I'm aware of -are made with ssh that does support it. - -There are a couple of even less likely scenarios, like -`GIT_ANNEX_SSH_SOCKET_DIR` being set to a directory you can't write to. - -I've changed the code to always say explicitly why ssh caching can't be -enabled. I also let annex.sshcaching override the build-time detection. - -I guess that's enough to close this, unless it turns out its -reasons for not enabling it are not one of those I mentioned above, but -something entirely bogus. -"""]] diff --git a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_2_43b651b09b91986d5cb4ae14c83017c7._comment b/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_2_43b651b09b91986d5cb4ae14c83017c7._comment deleted file mode 100644 index c2335f2b14..0000000000 --- a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_2_43b651b09b91986d5cb4ae14c83017c7._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="ssh caching" - date="2020-02-14T20:53:27Z" - content=""" -\"your build of git-annex may have been made without ssh connection caching support, which would happen if its configure program detected at build time that ssh doesn't support it\" -- yes, according to the [build log](https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/117561/logs/8). I can add an ssh dependency to the conda-forge git-annex recipe. It would be more flexible to not have that dependency and instead to have git-annex's behavior depend on the ssh available at runtime; but, I guess there's a reason it's a compile-time option? - -Also, I don't have ssh prompting for passwords since I use ssh-agent, and having the warning shown every time is distracting. Maybe, a config option could be added to disable the warning? -"""]] diff --git a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_3_49182db51bd86f0c60fadde5e4643f77._comment b/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_3_49182db51bd86f0c60fadde5e4643f77._comment deleted file mode 100644 index 0b8e66f301..0000000000 --- a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_3_49182db51bd86f0c60fadde5e4643f77._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 3" - date="2020-02-15T04:45:01Z" - content=""" -Ilya, you wrote \"the ability to easily write your own external **backends** has been especially helpful\". Did you mean \"external **remotes**\"? since \"external backends\" are yet [TODO AFAIK](https://git-annex.branchable.com/todo/external_backends/) -"""]] diff --git a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_4_ab823204f02dd26ebfe0613f711e0f9d._comment b/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_4_ab823204f02dd26ebfe0613f711e0f9d._comment deleted file mode 100644 index 9a029fefbf..0000000000 --- a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_4_ab823204f02dd26ebfe0613f711e0f9d._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 4" - date="2020-02-16T04:05:11Z" - content=""" -Yes, I meant external remotes. -"""]] diff --git a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_5_24f7717029ef520f932123130088d76d._comment b/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_5_24f7717029ef520f932123130088d76d._comment deleted file mode 100644 index 5dfc6d355f..0000000000 --- a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_5_24f7717029ef520f932123130088d76d._comment +++ /dev/null @@ -1,21 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2020-02-17T16:29:58Z" - content=""" -I think the idea with detecting at build time is that if git-annex is being -built on a platform where ssh doesn't support it, eg because it's not -openssh but some other ssh implementation, it might as well compile out -support rather than fail obscurely when it tries to use it. And it's -uncommon for the systems where a program is built and used to have -different ssh implementations, so runtime probing would only slow it -down. (git-annex makes similar assumptions about eg, `cp --reflink` being -supported or not, and I don't think it's very unusual to probe OS features -at compile time.) - -The warning seems useful, because here we've discovered that you have been -building git-annex without support for ssh caching all along! - -The way to disable the warning is to set annex.sshcaching=true -(after [[!commit a4909470688287fc0009eaf82dab2e108bd214f1]]). -"""]] diff --git a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_6_0b316eb8b0436350ce7fbec5093975d1._comment b/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_6_0b316eb8b0436350ce7fbec5093975d1._comment deleted file mode 100644 index ea6b1c9082..0000000000 --- a/doc/bugs/warning_about_ssh_caching_keeps_showing/comment_6_0b316eb8b0436350ce7fbec5093975d1._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="git-annex build-time dependencies" - date="2020-02-18T19:33:32Z" - content=""" -\"git-annex makes similar assumptions about eg, `cp --reflink` being supported or not, and I don't think it's very unusual to probe OS features at compile time\" -- this works well for package managers tied to specific distros. But consider something like [[install/conda]] that creates packages meant to be installed on a variety of systems. I can add a run-time dependency on `coreutils` to ensure that `cp --reflink` works, but I'm a bit wary about requiring git-annex users to replace all core utils with conda-forge ones. For one, these may be slower, being compiled for a generic architecture. For two, if they're not fully backwards-compatible, they make break some assumptions relied on by other parts of the distro. -"""]] diff --git a/doc/bugs/webdav_export_slow__44___does_not_reuse_connections.mdwn b/doc/bugs/webdav_export_slow__44___does_not_reuse_connections.mdwn deleted file mode 100644 index 0330909193..0000000000 --- a/doc/bugs/webdav_export_slow__44___does_not_reuse_connections.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -### Please describe the problem. - -I am running an initial export to my NAS over WebDAV. The repo contains many small files and it already runs for a day. Even the smallest file takes around a second to upload. - -Looking at strace and the code, it seems that for each file, git annex not only establishes a complete new TCP connection, it even also reads the credentials from .git/annex/creds for each file. - -Are there ways to make WebDAV faster? -Could http connections be reused? -Could multiple files be uploaded in parallel? - -Apparently files are also upload to a temporary location and renamed after successful upload. This adds additional latency and thus parallel uploads could provide a speed up? - -[[!tag confirmed]] - -> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/webdav_export_slow__44___does_not_reuse_connections/comment_1_878309c705dd9ff06c89b9559b54bd40._comment b/doc/bugs/webdav_export_slow__44___does_not_reuse_connections/comment_1_878309c705dd9ff06c89b9559b54bd40._comment deleted file mode 100644 index 14cbeb90a9..0000000000 --- a/doc/bugs/webdav_export_slow__44___does_not_reuse_connections/comment_1_878309c705dd9ff06c89b9559b54bd40._comment +++ /dev/null @@ -1,20 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-20T15:48:35Z" - content=""" -Indeed, regular webdav special remote uses prepareDav -which sets up a single DAV context that is used for all stores, -but export does not and so creates a new context each time. - -S3 gets around this using an MVar that contains the S3 handle. -Webdav should be able to do the same. - -(The upload to a temp location is necessary, otherwise resuming an -interrupted upload would not be able to check which files had been fully -uploaded yet in some situations. Or something like that. I forget the exact -circumstances, but it's documented in a comment where storeExport is defined -in Types.Remote.) - -(Opened [[todo/support_concurrency_for_export]].) -"""]] diff --git a/doc/bugs/windows_dosn__39__t_build_again__63____33__.mdwn b/doc/bugs/windows_dosn__39__t_build_again__63____33__.mdwn deleted file mode 100644 index c46fffd3ff..0000000000 --- a/doc/bugs/windows_dosn__39__t_build_again__63____33__.mdwn +++ /dev/null @@ -1,26 +0,0 @@ -### Please describe the problem. -Seems like the Windows version is not updated? - -1. https://downloads.kitenet.net/git-annex/windows/current/git-annex-installer.exe.info ==> distributionVersion = "8.20200815" -2. https://git-annex.branchable.com/news/version_8.20201007/#news-version-8.20201007.default ==> 8.20201007 ==> "Fix a build failure on Windows" ==> so where is that version / installer? - -### What version of git-annex are you using? On what operating system? -- git-annex version: 8.20200815-g335aae266 - -### Please provide any additional information below. - -[[!format sh """ -# If you can, paste a complete transcript of the problem occurring here. -# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log - - -# End of transcript or log. -"""]] - -### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) -- use it to ensure some of my private data / files are save on 2x different media's :) - -> The windows autobuilder is currenntly broken and has to be periodically -> rebooted or something to be "fixed". So updated builds are sporadic. -> This is not due to a bug in git-annex, so I don't really want to track -> it here. [[done]] --[[Joey]] diff --git a/doc/bugs/windows_dosn__39__t_build_again__63____33__/comment_1_df3359fee007094877286161e192d8af._comment b/doc/bugs/windows_dosn__39__t_build_again__63____33__/comment_1_df3359fee007094877286161e192d8af._comment deleted file mode 100644 index 94b0e99e05..0000000000 --- a/doc/bugs/windows_dosn__39__t_build_again__63____33__/comment_1_df3359fee007094877286161e192d8af._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="nix.zahlen@1211ac6c964ba2d68b70655f747bef1383032e77" - nickname="nix.zahlen" - avatar="http://cdn.libravatar.org/avatar/66cc45a05749fe8d4ca36d8c6071da51" - subject="where to trace the Windows build errors?" - date="2020-10-25T15:23:01Z" - content=""" -Thanks for you comment / fix... - -So where would you like to trace the Windows build errors if not here? -"""]] diff --git a/doc/todo/Alternative_mode_control_for_import.mdwn b/doc/todo/Alternative_mode_control_for_import.mdwn deleted file mode 100644 index 41a9b8bcbd..0000000000 --- a/doc/todo/Alternative_mode_control_for_import.mdwn +++ /dev/null @@ -1,20 +0,0 @@ -This suggestion has come from being surprised at the behaviour of "import --skip-duplicates" which copies files instead of moving them and leaves the source directory untouched (description implies it will just leave duplicates alone). - -Apologies for the brevity, I've already typed this out once.. - -"import" has several behaviours which can be controlled through some options, but they don't cover all wanted behaviours. This suggestion is for an alternative interface to control these behaviours, totally stolen from rsync :P - - # create symlinks (s), inject content (i) and delete from source (d) - # duplicate (D) and new (N) files - git annex import --mode=Dsid,Nsid $src # (default behaviour) - git annex import --mode=Dsi,Nsi $src # --duplicate - git annex import --mode=Dd,Nsid $src # --deduplicate - git annex import --mode=Nsi $src # --skip-duplicates - git annex import --mode=Dd $src # --clean-duplicates - git annex import --mode=Did,Nsid $src # (import new, reinject duplicate.. really want this!) - git annex import --mode=Ns $src # (just creates symlinks for new) - git annex import --mode=Nsd $src # (invalid mode due to data loss) - git annex import --mode=Nid $src # (invalid or require --force) - -> Current thinking is in [[remove_legacy_import_directory_interface]]. -> This old todo is redundant, so [[wontfix|done]] --[[Joey]] diff --git a/doc/todo/Alternative_mode_control_for_import/comment_1_77dc4b03e243d46c227f43cb3f573ec0._comment b/doc/todo/Alternative_mode_control_for_import/comment_1_77dc4b03e243d46c227f43cb3f573ec0._comment deleted file mode 100644 index 7b94f92815..0000000000 --- a/doc/todo/Alternative_mode_control_for_import/comment_1_77dc4b03e243d46c227f43cb3f573ec0._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="CandyAngel" - avatar="http://cdn.libravatar.org/avatar/15c0aade8bec5bf004f939dd73cf9ed8" - subject="comment 1" - date="2017-01-16T10:30:55Z" - content=""" -This [[TODO|todo/import_--reinject/]] (and \"reinject --known\") would then be: - - git annex import --mode=Did - -"""]] diff --git a/doc/todo/Alternative_mode_control_for_import/comment_2_21540a75c78c3bb7745f76b8e92a2ec5._comment b/doc/todo/Alternative_mode_control_for_import/comment_2_21540a75c78c3bb7745f76b8e92a2ec5._comment deleted file mode 100644 index 278616359e..0000000000 --- a/doc/todo/Alternative_mode_control_for_import/comment_2_21540a75c78c3bb7745f76b8e92a2ec5._comment +++ /dev/null @@ -1,33 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2017-02-07T20:24:29Z" - content=""" -Bearing in mind that I would have to *support* all of the resulting -combinatorial explosion, and that several combinations don't make sense, -or are unsafe, or seem useless, I think I'd rather keep it limited to -well-selected points from the space. - -I've fixed the description of --skip-duplicates to match its behavior. -I don't know if there's a good motivation for it not deleting the files it -does import. I'd almost rather have thought that was a bug in the -implementation, but the implementation explicitly copies rather than moves -files for --skip-duplicates, so that does seem to have been done -intentionally. In any case, `--clean-duplicates` can be run after it to -delete dups, I suppose. - -An implementation of --mode=Did,Nsid seemed worth adding at first, perhaps -as --reinject-duplicates. But thinking about it some more, -that would be the same as: - - git annex reinject --known /path/* - git annex import /path/* - -The first command moves all known files into the annex, which leaves -only non-duplicate files for the second command to import. - -The only time I can think of that this might not be suitable is if `/path` is -getting new files added to it while the commands run... But in that case -you can `mkdir /path/toimport; mv /path/* /path/toimport` and then -run the 2 commands on `/path/toimport/*` -"""]] diff --git a/doc/todo/Alternative_mode_control_for_import/comment_3_cd4b0f6752457a70f2ab908d23913d76._comment b/doc/todo/Alternative_mode_control_for_import/comment_3_cd4b0f6752457a70f2ab908d23913d76._comment deleted file mode 100644 index 09ce568ee3..0000000000 --- a/doc/todo/Alternative_mode_control_for_import/comment_3_cd4b0f6752457a70f2ab908d23913d76._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="CandyAngel" - avatar="http://cdn.libravatar.org/avatar/15c0aade8bec5bf004f939dd73cf9ed8" - subject="comment 3" - date="2017-02-07T22:51:15Z" - content=""" - An implementation of --mode=Did,Nsid seemed worth adding at first, perhaps as --reinject-duplicates. But thinking about it some more, that would be the same as - - git annex reinject --known /path/* - git annex import /path/* - - ---mode=Did,Nsid would be quite a bit faster because it wouldn't hash the files twice, which is an advantage this suggestion has over any multiple command alternative. - -If you want to keep it to certain points in space rather than deal with all combinations, you could whitelist which ones are acceptable and people can request more to be whitelisted as they discover use cases for those modes. The current commands would alias to the modes (which would also make their behaviour obvious if this alias is mentioned in the documentation). -"""]] diff --git a/doc/todo/Alternative_mode_control_for_import/comment_4_767dfbaf72de52bd5fbe4c37add5bd91._comment b/doc/todo/Alternative_mode_control_for_import/comment_4_767dfbaf72de52bd5fbe4c37add5bd91._comment deleted file mode 100644 index 70c49cf794..0000000000 --- a/doc/todo/Alternative_mode_control_for_import/comment_4_767dfbaf72de52bd5fbe4c37add5bd91._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2017-02-09T19:33:46Z" - content=""" -Actually, import --deduplicate, --skip-duplicates, --clean-duplicates -are implemeted naively and do hash files twice. So it's -the same efficiency.. - -But, I just finished a more complicated implementation that avoids -the second hashing. - -That does make the combined action worth adding, I suppose. Done so as ---reinject-duplicates. -"""]] diff --git a/doc/todo/Alternative_mode_control_for_import/comment_5_ded83da89e70cadfa4076de9ddc36b55._comment b/doc/todo/Alternative_mode_control_for_import/comment_5_ded83da89e70cadfa4076de9ddc36b55._comment deleted file mode 100644 index 8e5471c310..0000000000 --- a/doc/todo/Alternative_mode_control_for_import/comment_5_ded83da89e70cadfa4076de9ddc36b55._comment +++ /dev/null @@ -1,27 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2017-02-09T19:45:26Z" - content=""" -I feel that the problem with this idea is that the suggested -actions "create symlinks (s), inject content (i) and delete from source (d)" -are only an approximation of how import is implemented. If they perfectly -matched the implementation, then import could treat them as a DSL and -simply evaluate the expression to do its work. But it's not that simple. -For one thing, --deduplicate and --clean-duplicates don't simply "delete from source" the -duplicates; they first check that numcopies can be satisfied. The default -import behavior doesn't "sid", in fact it moves from source to the work tree -(thus implicitly deleting from source first), then injects, and then creates -the symlink. Everything has dependencies and interrelationships, and the best -way I've found to express that so far is as the Haskell code in -Command/Import.hs. - -Even exposing that interface and using the current implementation for -particular canned expressions seems risky; exposing imperfect abstractions -can shoot you in the foot later when something under the abstraction needs -to change. - -So I'd rather improve the documentation for git-annex import if it is -unclear. Not opposed to finding a way to work in these "Dsid,Nsid" -summaries to the the documentation. -"""]] diff --git a/doc/todo/add_a_--branch_to_applicable_git-annex_commands.mdwn b/doc/todo/add_a_--branch_to_applicable_git-annex_commands.mdwn deleted file mode 100644 index 9852e671d0..0000000000 --- a/doc/todo/add_a_--branch_to_applicable_git-annex_commands.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -My original use case was for using git-annex find from scripts, where I didn't want to depend on the branch -checked out at the time, but rather write something like "git annex find --branch=master $searchterms" - -> this was [[done]] some years leter and this todo forgotten about until I -> noticed it now, so closing belatedly. --[[Joey]] diff --git a/doc/todo/add_a_--branch_to_applicable_git-annex_commands/comment_1_3e0a1d1c41f317514dfc496f2274ad1c._comment b/doc/todo/add_a_--branch_to_applicable_git-annex_commands/comment_1_3e0a1d1c41f317514dfc496f2274ad1c._comment deleted file mode 100644 index 6d5320d41a..0000000000 --- a/doc/todo/add_a_--branch_to_applicable_git-annex_commands/comment_1_3e0a1d1c41f317514dfc496f2274ad1c._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="http://joeyh.name/" - ip="209.250.56.154" - subject="comment 1" - date="2014-03-17T19:48:57Z" - content=""" -The difficulty with adding a --branch is that if it causes git-annex to operate on a list of (file, key) from the branch, then commands that actually modify the working tree would modify it, instead of the branch. So the options seem to be only generating a list of keys, and so only letting commands that operate on keys work (which rules out the `git annex find` example), or carefully arranging for commands that actually affect the work tree to not be usable with this option. - -I'm not sure how many commands are affected. The ones I can immediately think of are sync, lock, unlock. (Commands like get obviously affect the work tree in direct mode, but it's fine to have getting a file from a branch also update files in the work tree, if they pointed at the same key.) -"""]] diff --git a/doc/todo/addunlocked_config_setting.mdwn b/doc/todo/addunlocked_config_setting.mdwn deleted file mode 100644 index e8163d8a75..0000000000 --- a/doc/todo/addunlocked_config_setting.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -Can the `annex.addunlocked` be extended to have the same syntax as `annex.largefiles`? Also, can there be separate settings affecting `git add` and `git annex add`, e.g. `annex.git-add.addunlocked` and `annex.git-annex-add.addunlocked`, with both defaulting to the value of `annex.addunlocked` if not set? - -Basically, I want a reliable way to prevent inadvertently adding files as annexed unlocked files. - -Related: [[forum/lets_discuss_git_add_behavior]] - -> [[done]] --[[Joey]] diff --git a/doc/todo/addunlocked_config_setting/comment_1_3815d409a57e85025d46bf6349ae472c._comment b/doc/todo/addunlocked_config_setting/comment_1_3815d409a57e85025d46bf6349ae472c._comment deleted file mode 100644 index 6402e16a4e..0000000000 --- a/doc/todo/addunlocked_config_setting/comment_1_3815d409a57e85025d46bf6349ae472c._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-10-08T18:35:06Z" - content=""" -It is not possible for `git add` to add files in locked form. git's -interface simply does not allow that. -"""]] diff --git a/doc/todo/addunlocked_config_setting/comment_2_0989d05a27ba8a56787bc5dbe581d1ae._comment b/doc/todo/addunlocked_config_setting/comment_2_0989d05a27ba8a56787bc5dbe581d1ae._comment deleted file mode 100644 index e5e8e030ff..0000000000 --- a/doc/todo/addunlocked_config_setting/comment_2_0989d05a27ba8a56787bc5dbe581d1ae._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="preventing inadvertently adding annexed files in unlocked form" - date="2019-10-11T16:38:07Z" - content=""" -> It is not possible for git add to add files in locked form. git's interface simply does not allow that. - -Makes sense. Then, [[separate annex.largefiles.git-add and annex.largefiles.git-annex-add settings]] seems like the way to prevent inadvertently adding files to annex in unlocked form. - -Related: [[todo/auto-lock_files_after_one_edit]] -"""]] diff --git a/doc/todo/addunlocked_config_setting/comment_3_6909726735abb7945930ba45632e4769._comment b/doc/todo/addunlocked_config_setting/comment_3_6909726735abb7945930ba45632e4769._comment deleted file mode 100644 index e16430a758..0000000000 --- a/doc/todo/addunlocked_config_setting/comment_3_6909726735abb7945930ba45632e4769._comment +++ /dev/null @@ -1,32 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2019-12-19T15:29:40Z" - content=""" -Retargeting this todo at something useful post-git-add-kerfluffle, -annex.addunlocked could usefully be a pagespec to allow adding some files -unlocked and others locked (by git-annex add only, not git add). -"true" would be the same as "anything" and false as "nothing". - ---- - -It may also then make sense to let it be configured in .gitattributes. -Although, the ugliness of setting a pagespec in .gitattributes, -as was done for annex.largefiles, coupled with the overhead of needing to -query that from git-check-attr for every file, makes me wary. - -(Surprising amount of `git-annex add` time is in querying the -annex.largefiles and annex.backend attributes. Setting the former in -gitconfig avoids the attribute query and speeds up add of smaller files by -2%. Granted I've sped up add (except hashing) by probably 20% this month, -and with large files the hashing dominates.) - -The query overhead could maybe be finessed: Since adding a file -already queries gitattributes for two other things, a single query could be -done for a file and the result cached. - -Letting it be globally configured via `git-annex config` is an alternative -that I'm leaning toward. -(That would also need some caching, easier to implement and faster -since it is not a per-file value as the gitattribute would be.) -"""]] diff --git a/doc/todo/addunlocked_config_setting/comment_4_e9c9552c4f558ada76fe1e3eecfa1627._comment b/doc/todo/addunlocked_config_setting/comment_4_e9c9552c4f558ada76fe1e3eecfa1627._comment deleted file mode 100644 index 5a8864bf32..0000000000 --- a/doc/todo/addunlocked_config_setting/comment_4_e9c9552c4f558ada76fe1e3eecfa1627._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2019-12-20T19:45:21Z" - content=""" -Made annex.addunlocked support expressions like annex.largefiles. - -And both of them can be set globally with `git annex config`. I did not -make annex.addunlocked be settable by git attribute, because my sense is -that `git annex config` covers that use case, or mostly so. -"""]] diff --git a/doc/todo/addunlocked_config_setting/comment_5_ef936e0beb9d3446a4894885f065eecf._comment b/doc/todo/addunlocked_config_setting/comment_5_ef936e0beb9d3446a4894885f065eecf._comment deleted file mode 100644 index 8892334d34..0000000000 --- a/doc/todo/addunlocked_config_setting/comment_5_ef936e0beb9d3446a4894885f065eecf._comment +++ /dev/null @@ -1,83 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="what am I doing wrong?" - date="2020-01-13T20:05:38Z" - content=""" -I have tried to use this but I do not see it in effect: - -[[!format sh \"\"\" -$> mkdir repo && cd repo && git init && git annex init && git annex config --set addunlocked anything && git show git-annex:config.log && touch 1 2 && git add 1 && git annex add 2 && git commit -m 'committing' && ls -l && git show -Initialized empty Git repository in /tmp/repo/.git/ -init (scanning for unlocked files...) -ok -(recording state in git...) -addunlocked anything ok -(recording state in git...) -1578945668.466039639s addunlocked anything -add 2 -ok -(recording state in git...) -[master (root-commit) e428211] committing - 2 files changed, 1 insertion(+) - create mode 100644 1 - create mode 120000 2 -total 4 --rw------- 1 yoh yoh 0 Jan 13 15:01 1 -lrwxrwxrwx 1 yoh yoh 178 Jan 13 15:01 2 -> .git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -commit e428211fe0c64e67cf45d8c92165c866db5ba75f (HEAD -> master) -Author: Yaroslav Halchenko -Date: Mon Jan 13 15:01:08 2020 -0500 - - committing - -diff --git a/1 b/1 -new file mode 100644 -index 0000000..e69de29 -diff --git a/2 b/2 -new file mode 120000 -index 0000000..ea46194 ---- /dev/null -+++ b/2 -@@ -0,0 +1 @@ -+.git/annex/objects/pX/ZJ/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - -\"\"\"]] - -so I have tried to say that \"anything\" (all files) should be added unlocked. But it seems that neither file (`1` added via `git add` and `2` added via `git annex add`) were added unlocked. - -
-Here is some info on version/config: (click to expand) - - -[[!format sh \"\"\" -(git-annex)lena:/tmp/repo[master] -$> cat .git/config -[core] - repositoryformatversion = 0 - filemode = true - bare = false - logallrefupdates = true -[annex] - uuid = f220cc03-1510-4e23-acb5-b95723ecf9fc - version = 7 -[filter \"annex\"] - smudge = git-annex smudge -- %f - clean = git-annex smudge --clean -- %f -(dev3) 1 17256.....................................:Mon 13 Jan 2020 03:03:30 PM EST:. -(git-annex)lena:/tmp/repo[master] -$> git annex version -git-annex version: 7.20191230+git2-g2b9172e98-1~ndall+1 -build flags: Assistant Webapp Pairing S3 WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite -dependency versions: aws-0.20 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.3 feed-1.0.1.0 ghc-8.6.5 http-client-0.5.14 persistent-sqlite-2.9.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0 -key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL -remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external -operating system: linux x86_64 -supported repository versions: 7 -upgrade supported from repository versions: 0 1 2 3 4 5 6 -local repository version: 7 - -\"\"\"]] - -
-"""]] diff --git a/doc/todo/addunlocked_config_setting/comment_6_838a093daae08837f37ec00482839d3c._comment b/doc/todo/addunlocked_config_setting/comment_6_838a093daae08837f37ec00482839d3c._comment deleted file mode 100644 index 163fbc662d..0000000000 --- a/doc/todo/addunlocked_config_setting/comment_6_838a093daae08837f37ec00482839d3c._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="re: what am I doing wrong?" - date="2020-01-14T03:19:19Z" - content=""" -I believe that should be `git annex config --set annex.addunlocked anything` (i.e. an \"annex.\" in front of the name). -"""]] diff --git a/doc/todo/addurl_improvements.mdwn b/doc/todo/addurl_improvements.mdwn deleted file mode 100644 index 35129f3d57..0000000000 --- a/doc/todo/addurl_improvements.mdwn +++ /dev/null @@ -1,37 +0,0 @@ -When an external special remote tells git-annex a fuller URL for a given file, git-annex-addurl does not use that information: - - [2018-10-28 16:12:39.933464] git-annex-remote-dnanexus[1] <-- CLAIMURL dx://file-FJZjVx001pB2BQPVKY4zX8kk/ - [2018-10-28 16:12:39.933515] git-annex-remote-dnanexus[1] --> CLAIMURL-SUCCESS - [2018-10-28 16:12:39.933568] git-annex-remote-dnanexus[1] <-- CHECKURL dx://file-FJZjVx001pB2BQPVKY4zX8kk/ - [2018-10-28 16:12:40.469292] git-annex-remote-dnanexus[1] --> CHECKURL-MULTI dx://file-FJZjVx001pB2BQPVKY4zX8kk/A4.assembly1-trinity.fasta 11086 A4.assembly1-trinity.fasta - addurl dx://file-FJZjVx001pB2BQPVKY4zX8kk/ (from mydx) (to A4.assembly1_trinity.fasta) [2018-10-28 16:12:40.469503] read: git ["--version"] - -It would be better if, in the above log, the URL key was based on dx://file-FJZjVx001pB2BQPVKY4zX8kk/A4.assembly1-trinity.fasta , which would preserve the .fasta extension in the key and therefore in the symlink target. - -> [[fixed|done]] --[[Joey]] - -Also, it would be good if the external special remote could return an etag -for the URL, which would be a value guaranteed to change if the URL's -contents changes; and if git-annex would then compute the URL key based on -the combination of URL and etag. - -> This might be a good idea if sufficiently elaborated on, but I am a one -> idea, one bug, one page kind of guy. I dislike reading over a long detailed -> discussion of something, like the problem above and my analysis of it, -> only to find a second, unrelated discussion of something else. -> Suddenly the mental state is polluted with -> different distinct things, some fixed, other still open. The bug tracking -> system has then failed because it's not tracking state in any useful way. -> Which is why I've closed this todo item with my fix of -> a single item from it. --[[Joey]] - -It'd also be good if there was a option to automatically migrate URL keys -to the default backend whenever a file from a URL key is downloaded. Also, -to record the checksummed key (e.g. MD5E) as metadata of the URL key (in a -field named e.g. alternateKeys), and if addurl --fast is later done on a -URL key for which a checksummed key is recorded in the metadata, to add the -checksummed key instead of the URL key . - -> Again, mixing discussion of several things in one place is a good way to -> muddy the waters. I think this idea has several problems, but don't want -> to discuss them here. --[[Joey]] diff --git a/doc/todo/addurl_improvements/comment_1_eedc1d967b871cf8282a0758127c6545._comment b/doc/todo/addurl_improvements/comment_1_eedc1d967b871cf8282a0758127c6545._comment deleted file mode 100644 index 15e20a8009..0000000000 --- a/doc/todo/addurl_improvements/comment_1_eedc1d967b871cf8282a0758127c6545._comment +++ /dev/null @@ -1,45 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2018-10-29T17:51:27Z" - content=""" -Looking at the code, it addurl clearly *does* use the urls returned by -CHECKURL. In Command/AddUrl.hs: - - go deffile (Right (UrlMulti l)) - | isNothing (fileOption (downloadOptions o)) = - forM_ l $ \(u', sz, f) -> do - let f' = adjustFile o (deffile fromSafeFilePath f) - void $ commandAction $ - startRemote r o f' u' sz - -`l` is the list of values it returns, and `u'` is individual urls from that list, -as opposed to `u` which is the url the user provided. -`u'` is passed to `startRemote`, and `u` is not. - -Hmm, but in Remote/External.hs there is a special case: - - -- Treat a single item multi response specially to - -- simplify the external remote implementation. - CHECKURL_MULTI ((_, sz, f):[]) -> - result $ UrlContents sz $ Just $ mkSafeFilePath f - CHECKURL_MULTI l -> result $ UrlMulti $ map mkmulti l - -That does not have any kind of rationalle in [[!commit 8a17bcb0be91c345a52d78c08009285b0fcd6e3a]], -but the next commit added `doc/special_remotes/external/git-annex-remote-torrent` -and I think I can see why I felt it simplified things. That script always -replies with CHECKURL-MULTI, but a torrent often contains a single file, and -it would be perhaps bettter to use the original url provided to the user for such a -file from a torrent, rather than an url that asks for file "#1" from the torrent. -Although AFAICS either would work, and Remote/BitTorrent.hs contains just the kind -of special case for a single item torrent that I was wanting to avoid external -special remotes needing to worry about. - -The other benefit to special casing UrlContents is that lets addurl --file specify -where to put the file, which the fileOption check in the first code block -above prevents for UrlMulti. But, that could just as well be handled by -adding a single file special case to the code in AddUrl. - -I suppose changing this won't break anything, or if it does it was relying -on this undocumented behavior. -"""]] diff --git a/doc/todo/addurl_video_from_a_Twitter_post.mdwn b/doc/todo/addurl_video_from_a_Twitter_post.mdwn deleted file mode 100644 index 58d5803355..0000000000 --- a/doc/todo/addurl_video_from_a_Twitter_post.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -Is there already a way to addurl video from a Twitter post. Question came up while proposing git annex as a tech for archival in https://github.com/2020PB/police-brutality/issues/315#issuecomment-640163911 - -> I don't think there's anything for git-annex to do here, so -> [[closing|done]] --[[Joey]] diff --git a/doc/todo/addurl_video_from_a_Twitter_post/comment_1_f53373e62af38a139fcf2d15433ca65d._comment b/doc/todo/addurl_video_from_a_Twitter_post/comment_1_f53373e62af38a139fcf2d15433ca65d._comment deleted file mode 100644 index 63a3effeed..0000000000 --- a/doc/todo/addurl_video_from_a_Twitter_post/comment_1_f53373e62af38a139fcf2d15433ca65d._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-06-22T18:38:33Z" - content=""" -If youtube-dl supports the web site, `git annex addurl` will automatically -use it to download the video. - -Looks like youtube-dl does support twitter, so it should just work. - -If it didn't though, I'd punt it over to youtube-dl. - -(If you also wanted to archive the twitter -page itself, you could use `git annex addurl --raw` to archive the html. -Although there's a good chance the html alone is not enough, and so you -might want to use other tools to archive javascript and other assets; -this is beyond the scope of git-annex, although of course you can `git -annex add` whatever files you end up downloading.) -"""]] diff --git a/doc/todo/allow_overriding_untrust_of_import_remotes.mdwn b/doc/todo/allow_overriding_untrust_of_import_remotes.mdwn deleted file mode 100644 index f183ddd365..0000000000 --- a/doc/todo/allow_overriding_untrust_of_import_remotes.mdwn +++ /dev/null @@ -1,28 +0,0 @@ -importtree=yes remotes are untrusted, because something is modifying that -remote other than git-annex, and it could change a file at any time, so -git-annex can't rely on the file being there. However, it's possible the user -has a policy of not letting files on the remote be modified. It may even be -that some remotes use storage that avoids such problems. So, there should be -some way to override the default trust level for such remotes. - -Currently: - - joey@darkstar:/tmp/y8>git annex semitrust borg - semitrust borg - This remote's trust level is overridden to untrusted. - -The borg special remote is one example of one where it's easy for the user to -decide they're going to not delete old archives from it, and so want git-annex -to trust it. - -Below is some docs I wrote for the borg special remote page, should be -moved there when this gets fixed. --[[Joey]] - -> There is Remote.appendonly, which prevents making import remotes -> untrusted. So if there were a way to set that for borg, it could -> be configured at initremote/enableremote time. But, -> Remote.Helper.ExportImport also assumes appendonly means that content can -> be accessed by Key, rather than by ImportLocation, which does not work -> for borg. - ->> [[done]] via Remote.untrustworthy --[[Joey]] diff --git a/doc/todo/arm64_autobuilder.mdwn b/doc/todo/arm64_autobuilder.mdwn deleted file mode 100644 index 4d2566575f..0000000000 --- a/doc/todo/arm64_autobuilder.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -Add an arm64 autobuilder (linux). - -This is needed to run in termux on some android devices. -And of course there are arm64 servers, although the armel build probably -also works on them. - -Status: Builds fine on arm64, but needs an autobuilder. Building under -emulation could be done, or a scaleway arm64 server, which would be a -$5/month expense. Or, perhaps someone has an arm64 that could host the -autobuilder? --[[Joey]] - -Currently running release builds for arm64 on my phone, but it's not -practical to run an autobuilder there. --[[Joey]] - ->> [[done]]; the current qemu based autobuilder is not ideal, often gets ->> stuck, but there's no point leaving this todo open. --[[Joey]] diff --git a/doc/todo/arm64_autobuilder/comment_1_381c675c33aa4bdedb823700f40dd3a6._comment b/doc/todo/arm64_autobuilder/comment_1_381c675c33aa4bdedb823700f40dd3a6._comment deleted file mode 100644 index cc599e31e5..0000000000 --- a/doc/todo/arm64_autobuilder/comment_1_381c675c33aa4bdedb823700f40dd3a6._comment +++ /dev/null @@ -1,20 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="arm64 possible CIs etc" - date="2018-10-11T22:22:50Z" - content=""" -According to the great details @mmarmm on github provided in request to support [arm64 for neurodebian](https://github.com/neurodebian/dockerfiles/issues/10#issuecomment-406644418): - -Shippable supports free Arm64 CI/CD and I believe Codefresh does too (both 64-bit and 32-bit for both providers): - -https://blog.shippable.com/shippable-arm-packet-deliver-native-ci-cd-for-arm-architecture -http://docs.shippable.com/platform/tutorial/workflow/run-ci-builds-on-arm/ - -CodeFresh Arm Beta signup: https://goo.gl/forms/aDhlk56jZcblYokj1 - -If you need raw infrastructure the WorksOnArm project will supply full servers if you want to deal with metal: https://github.com/worksonarm/cluster/ - - -I personally haven't looked into any of them yet -"""]] diff --git a/doc/todo/avoid_autoinit_when_git-annex_branch_already_exists.mdwn b/doc/todo/avoid_autoinit_when_git-annex_branch_already_exists.mdwn deleted file mode 100644 index f94a128bf1..0000000000 --- a/doc/todo/avoid_autoinit_when_git-annex_branch_already_exists.mdwn +++ /dev/null @@ -1,28 +0,0 @@ -Somehow one of my usb removable drives got a new annex uuid assigned to a -repo in it than the one it had before. Since the drive is now frequently -falling off the USB bus with lots of IO errors, I hypothesize what might -have happened is that when git-annex read the git config, it somehow got -a corrupted version where annex.uuid was not set. So, it autoinitialized with -a new uuid. - -(Arguing against this theory is that when git config then wrote to the -file, it would normally use the same cached value so would have written the -corrupted version. Which did not happen.) - -I have checked, and if git config exits nonzero, git-annex does not -continue with autoinitialization. So it seems it was not as simple as a -read failure. - -To avoid any kind of problem like this leading the a new uuid being -generated, which can be pretty annoying to recover from especially if you -don't notice it for a long time, maybe git-annex should avoid autoinit when -there's a git-annex branch already, or if .git/annex/index already exists. -After all, that implies the repo should have already been initialized, and -now it isn't, so something unusual is going on. - -A bare repo that was just cloned will have a git-annex branch -before it gets initialized. So for bare repos, would need to not consider -that, but looking if annex/index exists would still do. Or may be better -not to special case it, and only look for the annex/index file? --[[Joey]] - -> [[done]] --[[Joey]] diff --git a/doc/todo/borg_special_remote.mdwn b/doc/todo/borg_special_remote.mdwn deleted file mode 100644 index 935e656196..0000000000 --- a/doc/todo/borg_special_remote.mdwn +++ /dev/null @@ -1,19 +0,0 @@ -borg backup is pretty cool, and could be a great special remote backend. -In particular it does delta compression and stuff. - -There seem to be two ways it could work. Probably there are borg commands -that allow storing a given blob in it, and retrieving a given blob. And -that could be used for a traditional special remote. - -But also, if a whole git-annex repository has been backed up with borg, -then git-annex could look inside such a backup, and see if -.git/annex/object/ contains an object. It could then mark it as -present in the borg special remote. This way you'd use borg to take -backups, and git-annex would then be aware of what was backed up in borg, -and could do things like count that as a copy. - ---[[Joey]] - -[[!tag needsthought]] - -> [[done]]! --[[Joey]] diff --git a/doc/todo/borg_special_remote/comment_1_f8543e05bea3a88769d9c54fdfce3723._comment b/doc/todo/borg_special_remote/comment_1_f8543e05bea3a88769d9c54fdfce3723._comment deleted file mode 100644 index 0a8caf6d9d..0000000000 --- a/doc/todo/borg_special_remote/comment_1_f8543e05bea3a88769d9c54fdfce3723._comment +++ /dev/null @@ -1,20 +0,0 @@ -[[!comment format=mdwn - username="RonnyPfannschmidt" - avatar="http://cdn.libravatar.org/avatar/c5379a3fe2188b7571858c49f9db63c6" - subject="the remote im working on" - date="2018-06-04T07:51:57Z" - content=""" -Hi Joey, - -i am currently working on a remote to use borg as a tree import source and a content souce - -the work is started in https://github.com/RonnyPfannschmidt/git-annex-borg - -note that borg does **not** do delta storage - it does content informed dynamic chunk sizes (which helps deduplication) - -freestanding borg will not be a good remote for putting things out, -so i will be pulling things out mostly (but i hope to hit a point where its viable to generate a borg archive from the tree of expected contents thats viable for putting things in) - --- Ronny - -"""]] diff --git a/doc/todo/borg_special_remote/comment_2_cec11f364d0c1e62d05c4205b3bf2a2d._comment b/doc/todo/borg_special_remote/comment_2_cec11f364d0c1e62d05c4205b3bf2a2d._comment deleted file mode 100644 index 890540f0b5..0000000000 --- a/doc/todo/borg_special_remote/comment_2_cec11f364d0c1e62d05c4205b3bf2a2d._comment +++ /dev/null @@ -1,65 +0,0 @@ -[[!comment format=mdwn - username="anarcat" - avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7" - subject="progress?" - date="2018-11-27T06:47:26Z" - content=""" -How's that remote going, RonnyPfannschmidt? :) I can't tell from the [homepage](https://github.com/RonnyPfannschmidt/git-annex-borg/) but from the source code, it looks like initremote is supported so far, but not much else... - -From what I remember, borg supports storing arbitrary blobs with the `borg debug-put-obj` function, and retrieve one with `borg debug-get-obj`. Here's an example of how this could work: - - [1145]anarcat@angela:test$ sha256sum /etc/motd - a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 /etc/motd - [1146]anarcat@angela:test$ borg init -e none repo - [1147]anarcat@angela:test$ borg debug-put-obj repo /etc/motd - object a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 put. - [1148]anarcat@angela:test$ borg debug-get-obj repo a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 tmp - object a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 fetched. - [1149]anarcat@angela:test$ sha256sum tmp - a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 tmp - -This assumes the underlying blob ID in borg is a SHA256 hash, but that -seems like a fair assumption to make. Naturally, this could cause -problems with git-annex, which supports multiple hashing algorithms -thanks to the multiple [[backends]] support. But maybe this can just -work this out by refusing to store non-matchin backends. - -That is, if borg actually worked that way. Unfortunately, while the -above actually works, the resulting repository is not quite right: - - $ borg debug dump-repo-objs . - Dumping 000000_0000000000000000000000000000000000000000000000000000000000000000.obj - Data integrity error: Chunk a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921: Invalid encryption envelope - -So borg does not like the repository at all... I'm not sure why, but -it sure looks like borg \"objects\" are not as transparent as I -hoped and that this low-level interface will not be suitable for -git-annex. - -The higher level interface is \"archives\", which have (more or less) a -CRUD interface (without the U, really) through the -\"create/list/extract/prune\" interface. It's far from what we need: -items are deplicated across archives so it means it is impossible to -reliably delete a key unless we walk (and modify!) the entire archive list, which is -slow and impractical. But it *could* definitely be used to add keys to -a repository, using: - - $ time borg create --stdin-name SHA256-a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 .::'{utcnow}' - < /etc/motd - 1.30user 0.10system 0:01.62elapsed 86%CPU (0avgtext+0avgdata 81464maxresident)k - 72inputs+1496outputs (0major+31135minor)pagefaults 0swaps - -As you can see, however, that is *slow* (although arguably not slower -than `debug-put-obj` which is surprising). - -But even worse, that blob is now hidden behind that archive - you'd -need to list all archives (which is also expensive) to find it. - -So I hit a dead end so I'm curious to hear how you were planning to -implement this, Ronny. :) Presumably there should be a way to generate -an object compatible with `debug-put-obj`, but that interface seems -very brittle and has all sorts of warnings all around it... And on the -other hand, the archive interface is clunky and slow... I wish there -was a better way, and suspect it might be worth talking with upstream -(which I'm not anymore) to see if there's a better way to work this -problem. -- [[anarcat]] -"""]] diff --git a/doc/todo/borg_special_remote/comment_3_f6bd07bfbb76a3cd20f009276676b45b._comment b/doc/todo/borg_special_remote/comment_3_f6bd07bfbb76a3cd20f009276676b45b._comment deleted file mode 100644 index c628444817..0000000000 --- a/doc/todo/borg_special_remote/comment_3_f6bd07bfbb76a3cd20f009276676b45b._comment +++ /dev/null @@ -1,55 +0,0 @@ -[[!comment format=mdwn - username="anarcat" - avatar="http://cdn.libravatar.org/avatar/4ad594c1e13211c1ad9edb81ce5110b7" - subject="restic" - date="2018-11-27T07:13:29Z" - content=""" -and for what it's worth, borg's main rival, restic, handles this much better and faster: - - [1331]anarcat@angela:test$ RESTIC_PASSWORD=test restic init -r repo4 - created restic repository 2c75411732 at repo4 - - Please note that knowledge of your password is required to access - the repository. Losing your password means that your data is - irrecoverably lost. - [1334]anarcat@angela:test1$ RESTIC_PASSWORD=test time restic -r repo4 backup --stdin --stdin-filename SHA256-a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 < /etc/motd - repository 2c754117 opened successfully, password is correct - created new cache in /home/anarcat/.cache/restic - - Files: 1 new, 0 changed, 0 unmodified - Dirs: 0 new, 0 changed, 0 unmodified - Added to the repo: 656 B - - processed 1 files, 0 B in 0:00 - snapshot 87c0db00 saved - 0.55user 0.04system 0:00.80elapsed 73%CPU (0avgtext+0avgdata 48384maxresident)k - 0inputs+88outputs (0major+9665minor)pagefaults 0swaps - [1337]anarcat@angela:test$ RESTIC_PASSWORD=test time restic -r repo4 backup --stdin --stdin-filename SHA256-a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921 < /etc/motd - repository 2c754117 opened successfully, password is correct - - Files: 0 new, 1 changed, 0 unmodified - Dirs: 0 new, 0 changed, 0 unmodified - Added to the repo: 370 B - - processed 1 files, 0 B in 0:00 - snapshot 5b3af830 saved - 0.55user 0.04system 0:00.80elapsed 73%CPU (0avgtext+0avgdata 48568maxresident)k - 0inputs+64outputs (0major+9691minor)pagefaults 0swaps - [1348]anarcat@angela:test$ RESTIC_PASSWORD=test time restic -r repo4 backup --stdin --stdin-filename SHA256-533128ceb96cb2a6d8039453c3ecf202586c0e001dce312ecbd6a7a356b201dc < ~/folipon.jpg - repository 2c754117 opened successfully, password is correct - - Files: 1 new, 0 changed, 0 unmodified - Dirs: 0 new, 0 changed, 0 unmodified - Added to the repo: 372 B - - processed 1 files, 0 B in 0:00 - snapshot 18879aa4 saved - 0.54user 0.03system 0:00.78elapsed 73%CPU (0avgtext+0avgdata 48504maxresident)k - 0inputs+64outputs (0major+9700minor)pagefaults 0swaps - [1349]anarcat@angela:test$ RESTIC_PASSWORD=test time restic -r repo4 dump latest SHA256-533128ceb96cb2a6d8039453c3ecf202586c0e001dce312ecbd6a7a356b201dc | sha256sum - - 0.50user 0.02system 0:00.73elapsed 72%CPU (0avgtext+0avgdata 47848maxresident)k - 0inputs+8outputs (0major+9513minor)pagefaults 0swaps - 533128ceb96cb2a6d8039453c3ecf202586c0e001dce312ecbd6a7a356b201dc - - -Of course it doesn't validate those checksums, and might freak out with the number of snapshots we would create, but it's a much better start than borg. ;) -"""]] diff --git a/doc/todo/borg_special_remote/comment_4_75611482f67e5d52f50d52fdb8c68e8b._comment b/doc/todo/borg_special_remote/comment_4_75611482f67e5d52f50d52fdb8c68e8b._comment deleted file mode 100644 index 96f9b8ddd9..0000000000 --- a/doc/todo/borg_special_remote/comment_4_75611482f67e5d52f50d52fdb8c68e8b._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="michael@ff03af62c7fd492c75066bda2fbf02370f5431f4" - nickname="michael" - avatar="http://cdn.libravatar.org/avatar/125bdfa8a2b91432c072615364bc3fa1" - subject="Borg vs. restic, some design considerations" - date="2018-12-05T14:36:45Z" - content=""" -As I have been looking for a new, de-duplicating, reliable backup system I read through the design documentations of [borg](https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#archives) and [restic](https://restic.readthedocs.io/en/latest/100_references.html#design). While the design of restic seems to be much simpler and actually quite straightforward, I decided for borg in the end due to its support for compression and the more efficient removal of single backups. Further, it [seems](https://blog.stickleback.dk/borg-or-restic/) the RAM usage is lower for borg. - -Here are some comments on both concerning the usability as git annex storage backend. Note that they are all based on my understanding of the design documents that describe how the data is stored in restic and borg. It is well possible that I have misunderstood something or some parts are just impossible due to implementation details. Further, I am quite sure that what I propose is not possible with the current external APIs of git annex and borg. - -For none of them, it seems to be a good idea to store individual archives (borg) or snapshots (restic) per file as both of them assume that the list of archives/snapshots is reasonably small, can be presented to the user as a single list and can be pruned based on certain rules about how many to keep per timespan (though that is per group of archives/snapshots). borg stores descriptions of all archives in a single item, the manifest (which means that when an archive is added, the whole list needs to be rewritten), restic stores each archive as a json document in a directory which might scale better but is probably still not a good idea. I think instead of storing individual files, git annex should store the whole set of exported files in a single archive/snapshot, i.e., store some kind of (virtual) directory structure in borg or restic that represents all items that shall be stored. Then, whenever git annex syncs with the borg/restic remote, a new archive/snapshot would be added. The user could then use the time-based pruning rules to remove old snapshots. This would also integrate well with using the same borg/restic repository for other backups, too. It might seem this would make the retrieval of a single file quite inefficient. Both borg and restic split a file into a list of chunks and store information where these chunks can be found. Therefore, it should be possible for a borg/restic special remote to just store this list of chunks for every annexed file. Then, to get a file, git annex would only need to ask for these chunks if it wants to get a single file. For restoring a lot of files, in particular with a non-local restic repository, this might be very inefficient though as restic might need to download a lot of data just to get these chunks - there just getting the whole last archive/snapshot might be more efficient (as far as I understood, then restic downloads each pack of chunks only once and directly writes all of them to the files that want them). Restic stores separate objects for every directory and this directory contains a list of subdirectories and files, where files contain a list of chunks. To add or remove files from a snapshot in restic, git annex would just need to execute the chunker for files not already present in the previous snapshot and could use the already stored chunk ids for the already present files. However, each snapshot would create a completely new directory. Without subdirectories, this would basically mean that the list of all files needs to be re-written for every snapshot. Subdirectories would help with that, but only if few subdirectories are modified. Due to the nature of hashing, this seems unlikely in the case of a git annex special remote (but of course this makes backups of unchanged directories very efficient). Borg doesn't have this directory structure but instead just stores the metadata of every file in one large stream. This stream is chunked in parts consisting of around 128KiB and therefore, only parts where changes occurred need to be stored again. The list of these metadata chunks needs to be stored, nevertheless, but is much smaller. Again, everything that is needed for storing a file could be generated without having the actual source file if the chunk ids are present. In fact, this is what borg does with a file cache that stores for every file of the previous backup both properties like size, timestamp and inode id to identify modifications and a list of chunks. If borg finds the same file again, it just uses the stored chunk list. If the git annex borg special remote could also keep the order of all previously present files the same, this would result in re-using basically all metadata chunks - however, I don't know if borg assumes any order on the files. Note that borg needs to know which chunks are referenced in an archive as borg stores reference counts for all chunks to determine if a chunk is still needed, so just re-using the metadata chunks without reading their content is definitely not possible. Restic has no such reference counts, it needs to iterate over all trees to determine if a chunk can be deleted (which [seems](https://blog.stickleback.dk/borg-or-restic/) to be terribly slow). Nevertheless, both implementations of cleaning up chunks require that chunks are referenced in some file that is contained in some archive/snapshot. -"""]] diff --git a/doc/todo/borg_special_remote/comment_5_5b39e3fbf480e0e558e7fb4466798124._comment b/doc/todo/borg_special_remote/comment_5_5b39e3fbf480e0e558e7fb4466798124._comment deleted file mode 100644 index 49a7c7e14e..0000000000 --- a/doc/todo/borg_special_remote/comment_5_5b39e3fbf480e0e558e7fb4466798124._comment +++ /dev/null @@ -1,38 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2019-08-01T16:02:06Z" - content=""" -Half a second to store a single annex object with restic is pretty slow, -and that's before the snapshots directory gets bloated with a hundred -thousand files. - -I wonder if my original idea up top was not a better approach: Let these -backup tools back up a whole annex repo (or at least .git/annex/objects), -and then make git-annex interoperate with the backups by peering inside -them and learning what has been backed up. - -In the meantime, git-annex has gotten tree import facilities, -which is a similar concept, of listing content in a data store -and so learning what's stored in there, and then being able to -retrieve objects out of that data store on demand. - -Importing annex objects from a backup is not quite the same as a tree -import, because it wouldn't result in any kind of file tree that -you'd want to merge back into your git repo. Also tree importing has -to download files in order to hash them, while in this case the -object's annex key can be seen in the backup. - -But from a user perspective it could be quite similar, something like: - - git annex initremote restic type=restic repolocation=... - git annex import --from restic - git annex get - -That would use `restic list snapshots` and then `restic ls` each -snapshot and find filenames that look like annex keys -(perhaps looking for part of the annex directory structure to avoid -false positives). Keys it found would be marked as present in -the remote, and the snapshot(s) that contain them recorded in -the git-annex branch for use by git-annex get. -"""]] diff --git a/doc/todo/borg_special_remote/comment_6_a6945345a752b166dc683484d80dfcc2._comment b/doc/todo/borg_special_remote/comment_6_a6945345a752b166dc683484d80dfcc2._comment deleted file mode 100644 index 80bed89d70..0000000000 --- a/doc/todo/borg_special_remote/comment_6_a6945345a752b166dc683484d80dfcc2._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 6""" - date="2019-08-01T16:34:22Z" - content=""" -I made a restic repo with 2000 single-file snapshots. -Adding the first snapshot took 0.55s. Adding the 2000th -snapshot took 1.10s. - -So that's a very big scalability problem with using restic with single-file -snapshots. - -2000 files in a directory is not going to cause that kind of slowdown; -my guess is restic needs to load all past snapshots, or something like -that. -"""]] diff --git a/doc/todo/borg_special_remote/comment_7_fc336d8da2a449cb7c74cc2e48a43ba6._comment b/doc/todo/borg_special_remote/comment_7_fc336d8da2a449cb7c74cc2e48a43ba6._comment deleted file mode 100644 index 7e24f83d81..0000000000 --- a/doc/todo/borg_special_remote/comment_7_fc336d8da2a449cb7c74cc2e48a43ba6._comment +++ /dev/null @@ -1,22 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 7""" - date="2020-12-04T01:17:01Z" - content=""" -The Remote interface recently got importKey, which gets us -unexpectedly a *lot* closer to making `git-annex import --from borg` a reality! - -The Remote would need a listImportableContents that finds all annex objects -in all (new) snapshots, and generates a ContentIdentifier that is just the -snapshot plus object path. Then importKey can simply generate a Key from -that ContentIdentifier without doing any more work. (And, so getting an -object from the remote will also work, because it will have the -ContentIdentifier recorded and so will know what snapshot and path in the -borg repo.) - -Seems that all that would be needed is a way to skip generating the git tree -for the imported files, since it would be useless. -And a way to force --no-content, since importing from a borg backup should not -get all the backed up annex objects. It may be best to make this a new -command, that just happens to use the ImportActions interface. -"""]] diff --git a/doc/todo/borg_special_remote_add_subdir_config.mdwn b/doc/todo/borg_special_remote_add_subdir_config.mdwn deleted file mode 100644 index dba92c739b..0000000000 --- a/doc/todo/borg_special_remote_add_subdir_config.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -Sometimes a borg backup contains several git-annex repos. Then pointing -git-annex at the whole thing will find objects not belonging to the current -repo. To avoid this, add subdir= config. - -[[done]] --[[Joey]] diff --git a/doc/todo/borg_sync_tree_not_grafted.mdwn b/doc/todo/borg_sync_tree_not_grafted.mdwn deleted file mode 100644 index ecab6def75..0000000000 --- a/doc/todo/borg_sync_tree_not_grafted.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -The tree generated by git-annex sync with a borg remote -does not seem to get grafted into the git-annex branch, so -would be subject to being lost to GC. - -Is this a general problem affecting importtree too? - -> Yes, it was. It would have only caused a problem if the user -> kept doing imports from a remote, but never exporting to it. -> Then, in a clone of the repo that was importing, they would not be able -> to get files. [[fixed|done]] --[[Joey]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones.mdwn b/doc/todo/clarify_that_v7_applies_to_all_clones.mdwn deleted file mode 100644 index 021d545128..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones.mdwn +++ /dev/null @@ -1,46 +0,0 @@ -I am not sure this is the case, but from first-hand experience, it -sure looks like you can't turn on v7 (or really v6, actually) on a -single git worktree. For example, if I have my `pictures` repository -on `curie` and turn on v7, `angela` will *also* need to run `git annex -upgrade` on their worktree otherwise git-annex -(e.g. 6.20180913-1~bpo9+1 on Debian stretch) will be really confused: - - anarcat@angela:calendes$ less calendrier/calendes.pdf - /annex/objects/SHA256E-s117451415--8d7d8366094a63c54bef99b5cd2e2b5187092f834d8bf7002e1d5fdceb38a710.pdf - anarcat@angela:calendes$ git annex get calendrier/calendes.pdf - anarcat@angela:calendes$ git annex whereis calendrier/calendes.pdf - anarcat@angela:calendes$ # OMG WHERE ARE MY FILES! /me flails wildly - -:) - -It seems to me there should be a warning in the [[upgrades]] page -about this. I would have done so myself, but I'm not sure (like in my -last bug report) if I am doing things right. - -In this case, this repository was already present (v5, indirect mode) -on both machines. I upgraded (using `git annex upgrade`) the -repository on curie (7.20181121 Debian buster) which went well. - -(Then I messed around with that thumb drive, which led to -[[bugs/v7_fails_to_fetch_files_on_FAT_filesystem]], but probably -unrelated here.) - -Then i powered on my laptop (`angela`) and saw the above. I would have -expected it to either upgrade automatically or warn me about the -repository inconsistency. Of failing that, the upgrades page should at -least warn us this is a "system-wide" (how do we call that?) change... - -The workaround is to run `git annex upgrade` on that other repo, of -course, but if the source repo was also upgraded, it might be -difficult to sync files, as you will see that warning: - - $ git annex get - get calendrier/calendes.pdf (from sneakernet...) - Repository version 7 is not supported. Upgrade git-annex. - -Considering there's no backport of 7.x in Debian stretch, it makes the -upgrade path rather delicate... Is there a way to "downgrade" that -sneakernet repo? :) (Thankfully, the main server still runs v5 so the -files are still accessible from stretch....) -- [[anarcat]] - -Updated the [[upgrades]] page, [[done]]. diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_10_094362fc0257e3c15bf2059520fc22cf._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_10_094362fc0257e3c15bf2059520fc22cf._comment deleted file mode 100644 index 5a589959d6..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_10_094362fc0257e3c15bf2059520fc22cf._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 10""" - date="2020-10-05T18:40:32Z" - content=""" -Unless it entered an adjusted unlocked branch, this upgrade cannot have -changed locked files to unlocked files itself. So if you were not using -unlocked files in this repo before, and didn't make any changes after the -upgrade that would add any, you don't need to worry about them. - -The only risk if it was downgraded to v5 with an unlocked files -is that a command like `git commit -a` would commit the -large content to git. Easy enough to notice that with `git status` after -the downgrade too. - -(But do checkout master if the currently checked out branch is -"adjusted/master(unlocked)") -"""]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_11_9554e07b916b8d7ca1caf72b522fefd4._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_11_9554e07b916b8d7ca1caf72b522fefd4._comment deleted file mode 100644 index 0dca147893..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_11_9554e07b916b8d7ca1caf72b522fefd4._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 11" - date="2020-10-05T23:17:07Z" - content=""" -> .. after the upgrade that would add any, you don't need to worry about them. - -With the datalad pixie dust on top of git-annex, I am never 100% sure ;) I would better worry and do some basic check before proceeding... -- will do later today/tomorrow, God bless BTRFS and its snapshots - I can get a \"sandbox\" clone of the entire filesystem to play with safely. -"""]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_1_62ef170f11f0b700a99c2749018fa234._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_1_62ef170f11f0b700a99c2749018fa234._comment deleted file mode 100644 index 68b62235ae..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_1_62ef170f11f0b700a99c2749018fa234._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2018-12-04T20:53:49Z" - content=""" -You only need to upgrade to v7 when the repository has unlocked files -committed to it. If a file contains a pointer to an annex object, it won't -work with v5. There is not a good way for git-annex to detect when that is -the case; such a file could be committed any time. Committing unlocked -files and upgrading has to be coordinated amoung the users of the repository. -"""]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_2_9d40aba1b10ca98c491e5f6480a41185._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_2_9d40aba1b10ca98c491e5f6480a41185._comment deleted file mode 100644 index 2a13d9b529..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_2_9d40aba1b10ca98c491e5f6480a41185._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 2" - date="2020-10-01T14:06:32Z" - content=""" -Is there a sensible way (could be a helper script) to safely (checks for git links to be used etc) to downgrade version from 8 to 5? - -Rationale: On the original host (smaug) of the monstrous http://datasets.datalad.org (on falkor) I have managed to invoke our cron update script while using a newer annex and I had no `annex.autoupgraderepository` set, so annex upgraded a number of clones (originally version 5) locally to version 8. As I am still by default use older annex, I would like to downgrade those clones on smaug back to 5. -"""]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_3_e0d7aa2360b97fd5428560db60715075._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_3_e0d7aa2360b97fd5428560db60715075._comment deleted file mode 100644 index 08fb593f67..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_3_e0d7aa2360b97fd5428560db60715075._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-10-01T16:54:21Z" - content=""" -If the repository did not get switched from direct mode to adjusted -unlocked branch, and does not use any unlocked files, you can: - -* remove the filter.annex.smudge and filter.annex.clean from .git/config -* remove .git/info/attributes (or at least the filter=annex line) -* remove .git/hooks/post-checkout and .git/hooks/post-merge -* remove sqlite databases (all of .git/annex/keysdb* .git/annex/fsck/ .git/annex/export/ .git/annex/cidsdb*) -* change annex.version - -To get back from adjusted unlocked branch to direct mode, you'd first want -to check out the master branch, and then do all of the above, then `git -annex direct` to get back into direct mode. -"""]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_4_a78152614c1912b5603bdb456098e0aa._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_4_a78152614c1912b5603bdb456098e0aa._comment deleted file mode 100644 index 3d9c377f38..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_4_a78152614c1912b5603bdb456098e0aa._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 4" - date="2020-10-01T17:26:09Z" - content=""" -THANK YOU! What is the most efficient way to identify if there are unlocked files in the tree (or full repository)? I know that annex scans for unlocked files after a clone, so I guess you might have considered different options and already chose the most efficient ;) -"""]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_5_e0fc732a977dafe2644635d47e153495._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_5_e0fc732a977dafe2644635d47e153495._comment deleted file mode 100644 index a854c52519..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_5_e0fc732a977dafe2644635d47e153495._comment +++ /dev/null @@ -1,24 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="comment 5" - date="2020-10-01T19:07:45Z" - content=""" -> What is the most efficient way to identify if there are unlocked -> files in the tree (or full repository)? - -I can't say anything about efficiency, but FWIW with git-annex -7.20191009 or later there's an `--unlocked` matching item, so you can -say `git annex find --unlocked`. Since you're working in the context -of repos that have already been upgraded, I think you could use that -to find unlocked files in the working tree. - -As for outside of the working tree, `find` takes a `--branch` -argument, but, as far as I can tell, that doesn't match anything when -combined with `--unlocked` (tried with 8.20200908). However, I'm not -sure you'd need to consider anything other than the working tree. If -all of these repos were v5 before, then an unlocked file could have -only been in an uncommitted state, so I don't see how it'd end on -another ref without committing/switching branches afterwards. - -"""]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_6_d08112cda6dc6e6f292bc1caf50f3c03._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_6_d08112cda6dc6e6f292bc1caf50f3c03._comment deleted file mode 100644 index 79af795bd8..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_6_d08112cda6dc6e6f292bc1caf50f3c03._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 6" - date="2020-10-05T15:26:08Z" - content=""" -THANK YOU Kyle! `find --unlocked` works! - -But the tricky part is that I wanted to use some \"single\" instance of git-annex which would support `find --unlocked` and also v5 so I could fsck and do some other tests after I do the evil downgrade. But older versions, such as 7.20191114, which support v5 do not support v8, so cannot do `find --unlocked` on v8. So I need to either - -- find another later version which would support both v5 and v8 -- make script use multiple versions of git-annex from different locations (one for initial `find --unlocked` and then another one for subsequent checks etc) -- find a way for `find --unlocked` without invoking `git-annex`. -"""]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_7_55995a092e64f1f6d7454d4f9b75a553._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_7_55995a092e64f1f6d7454d4f9b75a553._comment deleted file mode 100644 index a65e8cafac..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_7_55995a092e64f1f6d7454d4f9b75a553._comment +++ /dev/null @@ -1,37 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="comment 7" - date="2020-10-05T17:42:23Z" - content=""" -> - find a way for `find --unlocked` without invoking `git-annex`. - -Assuming you're interested in finding just the v6+ pointer files, -instead of also finding the uncommitted type changes for v5 unlocked -files, perhaps you could use something like this - -[[!format python \"\"\" -import subprocess as sp - -p_ls = sp.Popen([\"git\", \"ls-files\", \"--stage\"], stdout=sp.PIPE) -p_cat = sp.Popen([\"git\", \"cat-file\", \"--batch\"], stdin=sp.PIPE, stdout=sp.PIPE) -with p_ls: - with p_cat: - for line in p_ls.stdout: - info, fname = line.strip().split(b\"\t\") - mode, objid = info.split(b\" \")[:2] - if mode != b\"100644\": - continue - p_cat.stdin.write(objid + b\"\n\") - p_cat.stdin.flush() - out = p_cat.stdout.readline() - _, objtype, size = out.split() - size = int(size) - if size > 0: - content = p_cat.stdout.read(size) - if content.startswith(b\"/annex/objects/\"): - print(fname.decode()) - p_cat.stdout.readline() -\"\"\"]] - -"""]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_8_842f3b0356ee1844d400a8fe564abab5._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_8_842f3b0356ee1844d400a8fe564abab5._comment deleted file mode 100644 index c0ddf2b44d..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_8_842f3b0356ee1844d400a8fe564abab5._comment +++ /dev/null @@ -1,20 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 8" - date="2020-10-05T18:02:48Z" - content=""" -Thank you Kyle! I came up with - -```shell -unlocked=( `git grep -l -a --no-textconv --cached '^/annex/objects/' || :` ) -if [ \"${#unlocked[*]}\" -ge 1 ]; then - error \"Found ${#unlocked[*]} unlocked files. Cannot do: ${unlocked[*]}\" 2 -fi -``` - -do you think it would miss something? - -Here is my complete script ATM (didn't try in \"production\" yet, switched to other tasks for now but it is ready, also does some testing of operation at the end, so must not be applied as is to existing repos without commenting that out): http://www.onerussian.com/tmp/downgrade-annex - -"""]] diff --git a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_9_600a47e0a1ed8fa43130308e0f9dfb6a._comment b/doc/todo/clarify_that_v7_applies_to_all_clones/comment_9_600a47e0a1ed8fa43130308e0f9dfb6a._comment deleted file mode 100644 index 4e904e9cd0..0000000000 --- a/doc/todo/clarify_that_v7_applies_to_all_clones/comment_9_600a47e0a1ed8fa43130308e0f9dfb6a._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="comment 9" - date="2020-10-05T18:09:03Z" - content=""" -Ah, I didn't think of using `git grep` for this. I think that's much -better than my suggestion. -"""]] diff --git a/doc/todo/consider_meow_backend.mdwn b/doc/todo/consider_meow_backend.mdwn deleted file mode 100644 index f826b387a1..0000000000 --- a/doc/todo/consider_meow_backend.mdwn +++ /dev/null @@ -1,38 +0,0 @@ -I recently discovered (thanks to Paul Wise) the [Meow hash][]. The -TL;DR: is that it's a fast non-crypto hash which might be useful for -git-annex. Here's their intro, quoted from the website: - -[Meow hash]: https://mollyrocket.com/meowhash - -> The Meow hash is a high-speed hash function named after the character -> Meow in [Meow the Infinite][]. We developed the hash function at -> [Molly Rocket][] for use in the asset pipeline of [1935][]. -> -> Because we have to process hundreds of gigabytes of art assets to build -> game packages, we wanted a fast, non-cryptographic hash for use in -> change detection and deduplication. We had been using a cryptographic -> hash ([SHA-1][]), but it was -> unnecessarily slowing things down. -> -> To our surprise, we found a lack of published, well-optimized, -> large-data hash functions. Most hash work seems to focus on small input -> sizes (for things like dictionary lookup) or on cryptographic quality. -> We wanted the fastest possible hash that would be collision-free in -> practice (like SHA-1 was), and we didn't need any cryptograhic security. -> -> We ended up creating Meow to fill this niche. - - [1935]: https://molly1935.com/ - [Molly Rocket]: https://mollyrocket.com/ - [Meow the Infinite]: https://meowtheinfinite.com/ - [SHA-1]: https://en.m.wikipedia.org/wiki/SHA-1 - -I don't an immediate use case for this right now, but I think it could -be useful to speed up checks on larger files. The license is a -*little* weird but seems close enough to a BSD to be acceptable. - -I know it might sound like a conflict of interest, but I *swear* I am -not bringing this up only as a oblique feline reference. ;) -- [[anarcat]] - -> Let's concentrate on [[xxhash|todo/add_xxHash_backend]] or other new hashes that are getting general -> adoption, not niche hashes like meow. [[done]] --[[Joey]] diff --git a/doc/todo/consider_meow_backend/comment_1_33266c088bbf1c7fa93917ff86a3cfcd._comment b/doc/todo/consider_meow_backend/comment_1_33266c088bbf1c7fa93917ff86a3cfcd._comment deleted file mode 100644 index c12a24cd92..0000000000 --- a/doc/todo/consider_meow_backend/comment_1_33266c088bbf1c7fa93917ff86a3cfcd._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-01-06T19:36:32Z" - content=""" -xxhash seems to fill a similar niche and is getting a lot more use from -what I can see. - -Meow seems to claim a faster gb/s rate than xxhash does, but -it's hard to tell if the benchmarks are really equivilant. -"""]] diff --git a/doc/todo/ctrl_c_handling.mdwn b/doc/todo/ctrl_c_handling.mdwn deleted file mode 100644 index 48b0426a7a..0000000000 --- a/doc/todo/ctrl_c_handling.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -Sometimes I start off a large file transfer to a new remote (a la "git-annex copy . --to glacier"). - -I believe all of the special remotes transfer the files one at a time, which is good, and provides a sensible place to interrupt a copy/move operation. - -Wish: When I press ctrl+c in the terminal, git-annex will catch that and finish it's current transfer and then exit cleanly (ie: no odd backtraces in the special remote code). For the case where the file currently being transfered also needs to be killed (ie: it's a big .iso) then subsequent ctrl+c's can do that. - -> I'm going to close this, because 6 years later, I just don't think it's a -> good idea. I think that blocking ctrl-c from interrupting the program -> violates least surprise. [[done]] --[[Joey]] diff --git a/doc/todo/ctrl_c_handling/comment_1_3addbe33817db5de836c014287b14c07._comment b/doc/todo/ctrl_c_handling/comment_1_3addbe33817db5de836c014287b14c07._comment deleted file mode 100644 index 16139c78da..0000000000 --- a/doc/todo/ctrl_c_handling/comment_1_3addbe33817db5de836c014287b14c07._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="http://joeyh.name/" - ip="209.250.56.172" - subject="comment 1" - date="2014-02-21T21:36:14Z" - content=""" -This really depends on the remote, some can resume where they were interrupted, such as rsync, and some cannot, such as glacier (and, er, encrypted rsync). -"""]] diff --git a/doc/todo/ctrl_c_handling/comment_2_cc2776dc4805421180edcdf96a89fcaa._comment b/doc/todo/ctrl_c_handling/comment_2_cc2776dc4805421180edcdf96a89fcaa._comment deleted file mode 100644 index 827b99afa1..0000000000 --- a/doc/todo/ctrl_c_handling/comment_2_cc2776dc4805421180edcdf96a89fcaa._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="http://grossmeier.net/" - nickname="greg" - subject="very remote specific" - date="2014-02-21T22:11:16Z" - content=""" -Yeah, this is very remote specific and probably means adding the functionality there as well (eg: in the glacier.py code, not only in git-annex haskell). Maybe I should file bugs there accordingly :) -"""]] diff --git a/doc/todo/ctrl_c_handling/comment_3_8d7d357368987f5d5d59b4d8d99a0e06._comment b/doc/todo/ctrl_c_handling/comment_3_8d7d357368987f5d5d59b4d8d99a0e06._comment deleted file mode 100644 index ed7e4d3b62..0000000000 --- a/doc/todo/ctrl_c_handling/comment_3_8d7d357368987f5d5d59b4d8d99a0e06._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="http://joeyh.name/" - ip="209.250.56.172" - subject="comment 3" - date="2014-02-21T22:34:14Z" - content=""" -Hmm, I forget if it's possible for git-annex to mask SIGINT when it runs glacier or rsync, so that the child process does not receive it, but the parent git-annex does. -"""]] diff --git a/doc/todo/ctrl_c_propagation_to_transferrer_process.mdwn b/doc/todo/ctrl_c_propagation_to_transferrer_process.mdwn deleted file mode 100644 index e0c41ce031..0000000000 --- a/doc/todo/ctrl_c_propagation_to_transferrer_process.mdwn +++ /dev/null @@ -1,28 +0,0 @@ -When annex.stalldetection is set, and git-annex transferrer is used, -a ctrl-c does not propagate to the transferrer process. - -The result is that, the next time the process sends a message to its output -handle (eg a progress update), it gets a SIGINT, and so an ugly message is -output to the console, after the user was returned to the prompt. - -The SIGINT is not propagated because a child process group is used for -git-annex transferrer, in order to let child processes of it be killed -along with it when a stall is detected. - -Maybe what's needed is a SIGINT handler in the main git-annex that -signals all the transferrer processes with SIGINT and waits on them -exiting. And other signals, eg SIGTSTP for ctrl-z. - -> Implemented this, but not for windows (yet). But not gonna leave open -> for something that on windows in my experience does not work very -> reliably in general. (I've many times hit ctrl-c in a windows terminal and -> had the whole terminal lock up.) So, [[done]] --[[Joey]] - -Or, note that it would suffice to remove the child process group stuff, -if we assume that all child processes started by git-annex transferrer are -talking to a pipe, and will output something, eg a progress update, -and so receive a SIGPIPE once the transferrer process has caught the -SIGINT and exited. -[[todo/stalldetection_does_not_work_for_rsync_and_gcrypt]] would be a -prereq for this approach. But, might there be long-running child processes -that are not on a pipe, and that need to be shutdown on a stall, too? diff --git a/doc/todo/faster_key_lookup_for_limits.mdwn b/doc/todo/faster_key_lookup_for_limits.mdwn deleted file mode 100644 index f6f9e46778..0000000000 --- a/doc/todo/faster_key_lookup_for_limits.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -As part of the work in [[precache_logs_for_speed_with_cat-file_--buffer]], -key lookups are now done twice as fast as before. - -But, limits that look up keys still do a key lookup, before the key -is looked up efficiently. Avoiding that would speed up --in etc, probably -another 1.5x-2x speedup when such limits are used. What that optimisation -needs is a way to tell if the current limit needs the key or not. If it -does, then match on it after getting the key (and precaching the location -log for limits that need that), otherwise before getting the key. - -> So this needs a way to introspect a limit to see if the terms used in it -> match some criteria. Another todo that also needs that is -> [[sync_fast_import]] --[[Joey]] - -[[done]] --[[Joey]] diff --git a/doc/todo/generic_readonly_http_remote.mdwn b/doc/todo/generic_readonly_http_remote.mdwn deleted file mode 100644 index baaa8f442a..0000000000 --- a/doc/todo/generic_readonly_http_remote.mdwn +++ /dev/null @@ -1,19 +0,0 @@ -Many special remotes can potentially end up exposed in public http. There -is not currently a way to access them over http, without adding per-remote -support (like S3 has). - -But generally the filenames used are the same, eg rsync and directory and -webdav and S3. Or if there are differences, they are generally small and -trying a couple of different urls is doable. - -And sameas allows for - -now. - -So, there could be a new special remote type, that allows generic readonly -access of other special remotes whose data stores are exposed via http. - -Call it "http" maybe. (There may be some confusion between this and the web -special remote by users looking for such a thing.) --[[Joey]] - -> httpalso special remote implemented, [[done]] --[[Joey]] diff --git a/doc/todo/git-annex-init_should_configure_git_diff_driver.mdwn b/doc/todo/git-annex-init_should_configure_git_diff_driver.mdwn deleted file mode 100644 index 846ae0c4ee..0000000000 --- a/doc/todo/git-annex-init_should_configure_git_diff_driver.mdwn +++ /dev/null @@ -1,3 +0,0 @@ -`git diff` for annexed files, especially unlocked annexed files, is currently uninformative. It would help if [[`git-annex-init`|git-annex-init]] configured a [git diff driver](https://git-scm.com/docs/gitattributes#_generating_diff_text) to diff the contents of the annexed files, rather than the pointer files. - -> [[wontfix|done]], see comment diff --git a/doc/todo/git-annex-init_should_configure_git_diff_driver/comment_1_e1161efe2fa7c87c6d62d4a2624737f4._comment b/doc/todo/git-annex-init_should_configure_git_diff_driver/comment_1_e1161efe2fa7c87c6d62d4a2624737f4._comment deleted file mode 100644 index 70c2d59ea0..0000000000 --- a/doc/todo/git-annex-init_should_configure_git_diff_driver/comment_1_e1161efe2fa7c87c6d62d4a2624737f4._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-01-06T18:39:27Z" - content=""" -Normally annexed files are huge binary files. Line-by-line diff of such -files is unlikely to be useful. - -So you would need some domain-specific diff for the kind of binary files -you are storing in git-annex. If you have one, you can use -[[git-annex-diffdriver]] to make git use it when diffing annexed files. - -Not seeing anything more I can do here, so I'm going to close this todo. -"""]] diff --git a/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different.mdwn b/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different.mdwn deleted file mode 100644 index e0fbed78d0..0000000000 --- a/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -I noticed that with the default SHA256E backend, `git annex reinject --known FILE` will fail if FILE has a different extension than it has in the annex. Presumably this is because `git annex calckey FILE` does not generate the same key, even though the file has the same checksum. - -I think it would be better if `git annex reinject --known` would ignore the file extension when deciding whether a file is known. A case where that would be much better is caused by the fact that git-annex has changed how it determines a file's extension over time. E.g. if foo.bar.baz was added to the annex a long time ago, it might have a key like `SHA256E-s12--37833383383.baz`. Modern git-annex would calculate a key like `SHA256E-s12--37833383383.bar.baz` and so the reinject of the file using modern git-annex would fail. - -This problem does not affect `git annex reinject` without `--known`. - ---spwhitton - -> mentioned this on the git-annex reinject man page; [[done]] --[[Joey]] diff --git a/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_1_366978d6fda3bc00ccbfc2f9548a1492._comment b/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_1_366978d6fda3bc00ccbfc2f9548a1492._comment deleted file mode 100644 index 3e25f3cad4..0000000000 --- a/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_1_366978d6fda3bc00ccbfc2f9548a1492._comment +++ /dev/null @@ -1,20 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-01-06T17:11:58Z" - content=""" -I can't think of a reasonable way to implement this. - -It would need to hash and then look for a known SHA256E key that uses the -hash. But the layout of the git-annex branch doesn't provide any way to do -that, except for iterating over every filename in the branch. Which -would be prohibitively slow when reinjecting many files. (N times git -ls-tree -r) So it would need to build a data structure to map from SHA256 -to known SHA256E key. That can't be stored in memory, git-annex doesn't -let the content of the repo cause it to use arbitrary amounts of memory -(hopefully). - -All I can think of is to traverse the git-annex branch and build a sqlite -database and then query that, but that would add quite a lot of setup -overhead to the command. -"""]] diff --git a/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_2_3df0601d0d999bdcbeb0f462e01f8863._comment b/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_2_3df0601d0d999bdcbeb0f462e01f8863._comment deleted file mode 100644 index fb54f48029..0000000000 --- a/doc/todo/git-annex-reinject_--known_should_not_fail_when_the_file_extension_is_different/comment_2_3df0601d0d999bdcbeb0f462e01f8863._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="spwhitton" - avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb" - subject="comment 2" - date="2020-01-07T12:29:47Z" - content=""" -Thank you for your reply. Makes sense. If that's the only way to do it then it might as well be a helper script rather than part of git-annex. - -Leaving this bug open because it would be good to have the limitation documented in git-annex-reinject(1). -"""]] diff --git a/doc/todo/git-annex-reinject_does_not_work_in_a_bare_repo.mdwn b/doc/todo/git-annex-reinject_does_not_work_in_a_bare_repo.mdwn deleted file mode 100644 index d4813403fc..0000000000 --- a/doc/todo/git-annex-reinject_does_not_work_in_a_bare_repo.mdwn +++ /dev/null @@ -1,30 +0,0 @@ -`git annex reinject --known` doesn't work in a bare repo. - - spwhitton@iris:~/tmp>echo foo >bar - spwhitton@iris:~/tmp>mkdir baz - spwhitton@iris:~/tmp>cd baz - spwhitton@iris:~/tmp/baz>git init --bare - Initialized empty Git repository in /home/spwhitton/tmp/baz/ - spwhitton@iris:~/tmp/baz>git annex init - init (scanning for unlocked files...) - ok - (recording state in git...) - spwhitton@iris:~/tmp/baz>git annex reinject --known ../bar - fatal: relative path syntax can't be used outside working tree. - fatal: relative path syntax can't be used outside working tree. - fatal: relative path syntax can't be used outside working tree. - fatal: relative path syntax can't be used outside working tree. - fatal: relative path syntax can't be used outside working tree. - fatal: relative path syntax can't be used outside working tree. - fatal: relative path syntax can't be used outside working tree. - fatal: relative path syntax can't be used outside working tree. - fatal: relative path syntax can't be used outside working tree. - fatal: relative path syntax can't be used outside working tree. - fatal: relative path syntax can't be used outside working tree. - git-annex: fd:15: hGetLine: end of file - -Obviously this wasn't actually a file known to git-annex. But I get the same error in a non-dummy bare repo I am trying to reinject. - -A workaround is to use `git worktree add` and run `git annex reinject` from there. - -> [[fixed|done]] --[[Joey]] diff --git a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths.mdwn b/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths.mdwn deleted file mode 100644 index 9c8d2b13f5..0000000000 --- a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -`git annex find --batch` will not accept absolute paths to files in the repo, but `git annex find /abs/path` works. - -I tested `git annex lookupkey --batch` which does not have this problem. - ---spwhitton - -> [[fixed|done]] --[[Joey]] diff --git a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_1_5340007d4d849db6057b3138ba1ba37c._comment b/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_1_5340007d4d849db6057b3138ba1ba37c._comment deleted file mode 100644 index 019b84cdf5..0000000000 --- a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_1_5340007d4d849db6057b3138ba1ba37c._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-16T18:06:47Z" - content=""" -Hmm, I am not reproducing this problem here. - -Were you passing other options besides --batch, to eg match some files? - -And what version? -"""]] diff --git a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_2_fb15c2a1ae1c38ece951cc5e0d07f3db._comment b/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_2_fb15c2a1ae1c38ece951cc5e0d07f3db._comment deleted file mode 100644 index e3ec903ec4..0000000000 --- a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_2_fb15c2a1ae1c38ece951cc5e0d07f3db._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="spwhitton" - avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb" - subject="comment 2" - date="2020-03-25T16:31:13Z" - content=""" -Hello Joey, - -I was passing `--unlocked` only. - -Version 8.20200226 installed from buster-backports. - -Thanks! -"""]] diff --git a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_3_dee1bdb5f3d05b904e664f1f0fc2fc7e._comment b/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_3_dee1bdb5f3d05b904e664f1f0fc2fc7e._comment deleted file mode 100644 index c6c816c898..0000000000 --- a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_3_dee1bdb5f3d05b904e664f1f0fc2fc7e._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-04-15T19:05:59Z" - content=""" -Reproduced it, the problem only happens when the files are unlocked, -not the locked files I was trying. The --unlocked option is not the -problem. -"""]] diff --git a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_4_bdb19362d1aeec443f7f1a7b66edc5ae._comment b/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_4_bdb19362d1aeec443f7f1a7b66edc5ae._comment deleted file mode 100644 index fe0294f62a..0000000000 --- a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_4_bdb19362d1aeec443f7f1a7b66edc5ae._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-04-15T19:13:39Z" - content=""" -Other commands like whereis --batch also behave the same. - -Looks like what's going on is, when an absolute path is passed -as a parameter, it feeds thru git ls-files, producing a relative file. -But with --batch, it stays absolute. This causes things that try to eg, -look up the file in the tree to not find it. - -So, --batch needs to make filepaths relative too.. -"""]] diff --git a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_5_b215be5cea338326e25057a4b93ea85f._comment b/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_5_b215be5cea338326e25057a4b93ea85f._comment deleted file mode 100644 index ee530068b7..0000000000 --- a/doc/todo/git-annex_find_--batch_will_not_accept_absolute_paths/comment_5_b215be5cea338326e25057a4b93ea85f._comment +++ /dev/null @@ -1,23 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2020-04-15T19:22:12Z" - content=""" -Most of it can be fixed by making batchStart make -files relative. - -Other affected commands that do custom parsing of -batch input, so will need to make the file from it -relative themselves: fromkey metadata rekey rmurl - -Also, `git annex info /path/to/file` fails for unlocked -files and works for locked files, because it does not pass -filenames through git ls-files. I think it's the only -command that does not, when not in batch mode. - -(I suppose alternatively, lookupKey could make the filename relative, -but I don't know if that is the only thing that fails on absolute -filenames, so prefer to make them all relative on input.) - -Ok, all done.. -"""]] diff --git a/doc/todo/git-annex_transferrer_does_not_propigate_-c.mdwn b/doc/todo/git-annex_transferrer_does_not_propigate_-c.mdwn deleted file mode 100644 index 23e05c4041..0000000000 --- a/doc/todo/git-annex_transferrer_does_not_propigate_-c.mdwn +++ /dev/null @@ -1,12 +0,0 @@ -When `git annex -c foo.bar` runs git-annex transferrer, -it does not pass along the settings from -c. - -(Note that, `git -c foo.bar annex` does propagate the -c. Git does it by -setting an environment variable, which causes git config to reflect the -override. The environment variable propagates to child processes.) - -There are a lot of config settings that impact transfers, -and some of them might be commonly used at the command line, so something -needs to be done about this. --[[Joey]] - -> [[done]] diff --git a/doc/todo/git_annex_add_option_to_control_to_where.mdwn b/doc/todo/git_annex_add_option_to_control_to_where.mdwn deleted file mode 100644 index ea06f44b38..0000000000 --- a/doc/todo/git_annex_add_option_to_control_to_where.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -Make `git-annex add --force-large` and `git-annex add --force-small` -add a specific file to annex or git, bypassing annex.largefiles -and all other configuration and state. - -One reason to want this is that it avoids users doing stuff like this: - - git -c annex.largefiles=anything annex add foo.c - -Such a temporary setting of annex.largefiles can be problimatic, as explored in - - -Also, this could also be used to easily switch a file from one storage to -the other. I suppose the file would have to be touched first to make git-annex -add process it? - -> [[done]] --[[Joey]] diff --git a/doc/todo/idea__58___git-annex_cat.mdwn b/doc/todo/idea__58___git-annex_cat.mdwn deleted file mode 100644 index 9dae267f8d..0000000000 --- a/doc/todo/idea__58___git-annex_cat.mdwn +++ /dev/null @@ -1,22 +0,0 @@ -I wanted to share some thoughts for an idea I had. - -There are times when I want to stream data from a remote -- I want to start processing it immediately, and do not want to keep it in my annex when I am done with it. - -I can give some examples: - -* I have several projects which have a large number of similar text files, and they compress really well with borg or bup. For example, I have a repo with many [ncdu](https://dev.yorhel.nl/ncdu) json index files. They total 60G, but in a bup special remote, they are ~3G. In another repo, I have large highly differential tsv files. -* I have an annex with 5-10G video files that are stored in a variety of network special remotes. Most of them are in my Google Drive. I would like to be able to immediately start playing them with VLC rather than downloading and verifying them in their entirety. - -It would look like this: - -``` -git annex cat "someindex.ncdu" | ncdu -f - - -diff <(git annex cat "huge-data-dump1.tsv" -f mybupremote ) <(git annex cat "huge-data-dump2.tsv" -f mybupremote ) - -git annex cat "myvideo.mp4" -f googledrive | vlc - -``` - -I imagine that there might be issues with verification. But I really am ok with not verifying a video file I am streaming. - -> [[dup|done]] --[[Joey]] diff --git a/doc/todo/idea__58___git-annex_cat/comment_1_cc0b67a0bbf93841d5755a9b1dd74d36._comment b/doc/todo/idea__58___git-annex_cat/comment_1_cc0b67a0bbf93841d5755a9b1dd74d36._comment deleted file mode 100644 index 2ecf59bcc7..0000000000 --- a/doc/todo/idea__58___git-annex_cat/comment_1_cc0b67a0bbf93841d5755a9b1dd74d36._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="git-annex-cat" - date="2020-07-09T00:21:02Z" - content=""" -Related: [[todo/git-annex-cat]] -"""]] diff --git a/doc/todo/import_--no-content_largefiles_conflict.mdwn b/doc/todo/import_--no-content_largefiles_conflict.mdwn deleted file mode 100644 index d09500d4d0..0000000000 --- a/doc/todo/import_--no-content_largefiles_conflict.mdwn +++ /dev/null @@ -1,35 +0,0 @@ -git-annex import --no-content means annex.largefiles is not checked, so -non-large files get added as annexed files. That's done because -annex.largefiles can contain expressions that need to examine the content -of the file. In particular for mimetype and mimeencoding. - -So, if someone uses import --no-content in one repo, and in another clone -it's used with --content, importing the same files both times, a merge -conflict can result. - -May be worth removing support for matching annex.largefiles when the -expression needs the file content, when importing from a special remote. - -Or could detect when those are used, and only allow -importing with --content in that case. - -> So this needs a way to introspect a preferred content expression -> to see if the terms used in it -> match some criteria. Another todo that also needs that is -> [[faster_key_lookup_for_limits]] --[[Joey]] - -> > That introspection is implemented now. - -Which is better? The repo may have annex.largefiles set in gitattributes -for good workflow reasons, so it would be very annoying to have importing -error out. And if importing ignores the configuration, the user is likely -to see that as a bug. If importing with --no-content looks at the config -and say "sorry, I can't, need the file content", the user can then choose -between changing largefiles or using --content, and it's clear how they're -asking for contradictory things. - -Hmm, if largefiles does not match, it would have to download the file -content to add it to git, even though --no-content is used. A little weird, -but it's a small file, presumably. - -[[done]] --[[Joey]] diff --git a/doc/todo/import_tree.mdwn b/doc/todo/import_tree.mdwn deleted file mode 100644 index 6c728db64b..0000000000 --- a/doc/todo/import_tree.mdwn +++ /dev/null @@ -1,211 +0,0 @@ -This todo is about `git-annex import branch --from remote`, which is -implemented now. - -> [[done]] --[[Joey]] - -## race conditions - -(Some thoughts about races that the design should cover now, but kept here -for reference.) - -A file could be modified on the remote while -it's being exported, and if the remote then uses the mtime of the modified -file in the content identifier, the modification would never be noticed by -imports. - -To fix this race, we need an atomic move operation on the remote. Upload -the file to a temp file, then get its content identifier, and then move it -from the temp file to its final location. Alternatively, upload a file and -get the content identifier atomically, which eg S3 with versioning enabled -provides. It would make sense to have the storeExport operation always return -a content identifier and document that it needs to get it atomically by -either using a temp file or something specific to the remote. - ----- - -There's also a race where a file gets changed on the remote after an -import tree, and an export then overwrites it with something else. - -One solution would be to only allow one of importtree or exporttree -to a given remote. This reduces the use cases a lot though, and perhaps -so far that the import tree feature is not worth building. The adb -special remote needs both. Also, such a limitation seems like one that -users might try to work around by initializing two remotes using the same -data and trying to use one for import and the other for export. - -Really fixing this race needs locking or an atomic operation. Locking seems -unlikely to be a portable enough solution. - -An atomic rename operation could at least narrow the race significantly, eg: - -1. get content identifier of $file, check if it's what was expected else - abort (optional but would catch most problems) -2. upload new version of $file to $tmp1 -3. rename current $file to $tmp2 -4. Get content identifier of $tmp2, check if it's what was expected to - be. If not, $file was modified after the last import tree, and that - conflict has to be resolved. Otherwise, delete $tmp2 -5. rename $tmp1 to $file - -That leaves a race if the file gets overwritten after it's moved out -of the way. If the rename refuses to overwrite existing files, that race -would be detected by it failing. renameat(2) with `RENAME_NOREPLACE` can do that, -but probably many special remote interfaces don't provide a way to do that. - -S3 lacks a rename operation, can only copy and then delete. Which is not -good enough; it risks the file being replaced with new content before -the delete and the new content being deleted. - -Is this race really a significant problem? One way to look at it is -analagous to a git merge overwriting a locally modified file. -Git can certianly use similar techniques to entirely detect and recover -from such races (but not the similar race described in the next section). -But, git does not actually do that! I modified git's -merge.c to sleep for 10 seconds after `refresh_index()`, and verified -that changes made to the work tree in that window were silently overwritten -by git merge. In git's case, the race window is normally quite narrow -and this is very unlikely to happen (the similar race described in the next -section is more likely). - -If git-annex could get the race window similarly small out would perhaps be -ok. Eg: - -1. upload new version of $file to $tmp -2. get content identifier of $file, check if it's what was expected else - abort -3. rename (or copy and delete) $tmp to $file - -The race window between #2 and #3 could be quite narrow for some remotes. -But S3, lacking a rename, does a copy that can be very slow for large files. - -S3, with versioning, could detect the race after the fact, by listing -the versions of the file, and checking if any of the versions is one -that git-annex did not know the file already had. -[Using this api](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETVersion.html), -with version-id-marker set to the previous version of the file, -should list only the previous and current versions; if there's an -intermediate version then the race occurred and it could roll the change -back, or otherwise recover the overwritten version. This could be done at -import time, to detect a previous race, and recover from it; importing -a tree with the file(s) that were overwritten due to the race, leading to a -tree import conflict that the user can resolve. This likely generalizes -to importing a sequence of trees, so each version written to S3 gets -imported. - ----- - -A remaining race is that, if the file is open for write at the same -time it's renamed, the write might happen after the content identifer -is checked, and then whatever is written to it will be lost. - -But: Git worktree update has the same race condition. Verified with -this perl oneliner, run in a worktree and a second later -followed by a git pull. The lines that it appended to the -file got lost: - - perl -e 'open (OUT, ">>foo") || die "$!"; sleep(10); while (<>) { print OUT $_ }' - -Since this is acceptable in git, I suppose we can accept it here too.. - -## S3 versioning and import - -Listing a versioned S3 bucket with past versions results in S3 sending -a list that's effectively: - - foo current-version - foo past-version - bar deleted - bar past-version - bar even-older-version - -Each item on the list also has a LastModified date, and IsLatest -is set for the current version of each file. - -This needs to be converted into a ImportableContents tree of file trees. - -Getting the current file tree is easy, just filter on IsLatest. - -Getting the past file trees seems hard. Two things are in tension: - -* Want to generate the same file tree in this import that was used in past - imports. Since the file tree is converted to a git tree, this avoids - a proliferation of git trees. - -* Want the past file trees to reflect what was actually in the - S3 bucket at different past points in time. - -So while it would work fine to just make one past file tree for each -file, that contains only that single file, the user would not like -the resulting history when they explored it with git. - -With the example above, the user expects something like this: - - ImportableContents [(foo, current-version)] - [ ImportableContents [(foo, past-version), (bar, past-version)] - [ ImportableContents [(bar, even-older-version)] - [] - ] - ] - -And the user would like for the inner-most list to also include -(foo, past-version) if it were in the S3 bucket at the same time -(bar, even-older-version) was added. So depending on the past -modificatio times of foo vs bar, they may really expect: - - let l = ImportableContents [(foo, current-version)] - [ ImportableContents [(foo, past-version), (bar, past-version)] - [ ImportableContents [(foo, past-version), (bar, even-older-version)] - [ ImportableContents [(foo, past-version)] - [] - ] - ] - ] - -Now, suppose that foo is deleted and subsequently bar is added back, -so S3 now sends this list: - - bar new-version - bar deleted - bar past-version - bar even-older-version - foo deleted - foo current-version - foo past-version - -The user would expect this to result in: - - ImportableContents [(bar, new-version)] - [ ImportableContents [] - l - ] - -But l needs to be the same as the l above to avoid git trees proliferation. - -What is the algorythm here? - -1. Build a list of files with historical versions ([[a]]). -2. Extract a snapshot from the list -3. Remove too new versions from the list -4. Recurse with the new list. - -Extracting a snapshot: - -Map over the list, taking the head version of each item and tracking -the most recent modification time. Add the filenames to a snapshot list -(unless the item is a deletion). - -Removing too new versions: - -Map over the list, and when the head version of a file matches the most -recent modification time, pop it off. - -This results in a list that is only versions before the snapshot. - -Overall this is perhaps a bit better than O(n^2) because the size of the list -decreases as it goes? - ---- - -See also, [[adb_special_remote]] - -[[!tag confirmed]] diff --git a/doc/todo/import_tree_should_honor_annex.largefiles.mdwn b/doc/todo/import_tree_should_honor_annex.largefiles.mdwn deleted file mode 100644 index ba0fb4acf9..0000000000 --- a/doc/todo/import_tree_should_honor_annex.largefiles.mdwn +++ /dev/null @@ -1,73 +0,0 @@ -Need to support annex.largefiles when importing a tree from a special -remote. - -Note that the legacy `git annex import` from a directory does honor -annex.largefiles. - -> annex.largefiles will either need to be matched by downloadImport -> (changing to return `Either Sha Key`, or by buildImportTrees). -> -> If it's done in downloadImport, to avoid re-download of non-large files, -> the content identifier will -> need to be recorded as using the git sha1. This needs a way to encode -> a git sha as a key, that's a bijective mapping (so distinct from annex -> sha1 keys). -> -> Problem: In downloadImport, startdownload checks getcidkey -> to see if the ContentIdentifier is already known, and if so, returns the -> key used for it before. But, with annex.largefiles, the same content -> might be annexed given one filename, and not annexed with another. -> So, the key from getcidkey might not be the right one (or there could be -> more than one, an annex key and a translated git key). -> -> That argues against making downloadImport match annex.largefiles. - -> But, if instead buildImportTrees matches annex.largefiles, -> then downloadImport has already run moveAnnex on the download, -> so the content is in the annex. Moving it back out of the annex is -> difficult (there may be other files in the repo using the same key). -> So, downloadImport would then need to not moveAnnex, but move it to -> somewhere temporary. Like the gitAnnexTmpObjectLocation, but using -> that would be a problem if there was a file in the repo -> and git-annex get was run on it at the same time. So an equivilant -> but separate location. -> -> Further problem: downloadImport might skip a download of a CID -> that's already been seen. That CID might have generated a key -> before. The key's content may not still be present in the local -> repo. Then, if buildImportTrees checks annex.largefiles and wants -> to add it directly to git, it won't have the content available to add to -> git. (Conversely, the CID may have been added to git before, but -> annex.largefiles matches now, and so it would need to extract -> the content from git only to store it in the annex, which is doable but -> seems pointless as it's not going to save any space.) -> -> Would it be acceptable for annex.largefiles to be ignored if the same -> content was already imported from a remote earlier? I think maybe so. -> -> Then all these problems are not a concern, and back to downloadImport -> checking annex.largefiles being the simplest approach, since it avoids -> needing the separate temp file location. -> -> From the user's perspective, the special remote contained a file, -> it was already imported in the past, and the file has been renamed. -> It makes no more sense for importing it again to change how it's -> stored between git and annex than it makes sense for git mv of a file -> to change how it's stored. -> -> However... If two people can access the special remote, and import -> from it at different times, and get different trees as a result, -> that might break some assumptions, might also lead to merge -> conflicts. --[[Joey]] -> -> > Importing updates export.log, to indicate the state of the remote -> > (the log file could have been named better). So an annex.largefiles -> > change would result in an export/import conflict. Such a conflict -> > can be resolved by using git-annex export, but this could be a -> > surprising situation for users to encounter, since there is no real -> > conflict. -> > -> > Still, this doesn't feel like a reason not to implement the feature, -> > necessarily. - -[[done]] diff --git a/doc/todo/importfeed__58___allow___36____123__itemdate__125___with_--template.mdwn b/doc/todo/importfeed__58___allow___36____123__itemdate__125___with_--template.mdwn deleted file mode 100644 index 3949f3372d..0000000000 --- a/doc/todo/importfeed__58___allow___36____123__itemdate__125___with_--template.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -It would be great to be able to use the pubDate of the entries with the --template option of importfeed. - -Text.Feed.Query has a getItemPublishDate (and a getFeedPubDate, if we want some kind of ${feeddate}). - -The best would be to allow a reformating of the date(s) with (for example) %Y-%m-%D - -> itempubdate was added years ago and I forgot to close this, -> but I've now also added itempubmonth, itempubday, etc. [[done]] -> --[[Joey]] diff --git a/doc/todo/importfeed__58___allow___36____123__itemdate__125___with_--template/comment_1_62752c760fc12eca0c34d67d58753d00._comment b/doc/todo/importfeed__58___allow___36____123__itemdate__125___with_--template/comment_1_62752c760fc12eca0c34d67d58753d00._comment deleted file mode 100644 index cc3d85faf0..0000000000 --- a/doc/todo/importfeed__58___allow___36____123__itemdate__125___with_--template/comment_1_62752c760fc12eca0c34d67d58753d00._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="gueux" - ip="2a01:240:fe6d:0:7986:3659:a8bd:64f1" - subject="syntax" - date="2013-09-12T14:05:16Z" - content=""" -use \"itemdate\" and \"feeddate\" as names? - -use ${itemdate=%Y-%m-%D} syntax option? -"""]] diff --git a/doc/todo/importfeed__58___allow___36____123__itemdate__125___with_--template/comment_2_21672360060f48bc2eacfa535ff4c94d._comment b/doc/todo/importfeed__58___allow___36____123__itemdate__125___with_--template/comment_2_21672360060f48bc2eacfa535ff4c94d._comment deleted file mode 100644 index c8770ec6ed..0000000000 --- a/doc/todo/importfeed__58___allow___36____123__itemdate__125___with_--template/comment_2_21672360060f48bc2eacfa535ff4c94d._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="http://joeyh.name/" - ip="4.154.2.134" - subject="comment 2" - date="2013-09-13T19:53:52Z" - content=""" -getItemPublishDate returns a String, which can contain any of several date formats. Deferred until the feed library has something more sane. -Upstream bug: - -As for how to format the date in the feed, I would be ok with having itemdate (YYYYMMDD), itemyear (YYYY), itemmonth (MM) and itemday (DD). Full date formatting seems like overkill here. -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading.mdwn b/doc/todo/importing_from_special_remote_without_downloading.mdwn deleted file mode 100644 index 7caea81e88..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -The documentation for the new import remote command says, "Importing from a special remote first downloads all new content from it". For many special remotes -- such as Google Cloud Storage or DNAnexus -- checksums and sizes of files can be determined without downloading the files. For other special remotes, data files might have associated checksum files (e.g. md5) stored next to them in the remote. In such cases, it would help to be able to import the files without downloading (which can be costly, especially from cloud provider egress charges), similar to addurl --fast . - -[[!tag confirmed]] - -> [[done]] (only implemented for directory for now) --[[Joey]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_10_255a74838d9d88fe75095c4824cdbc9b._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_10_255a74838d9d88fe75095c4824cdbc9b._comment deleted file mode 100644 index 44752cc46c..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_10_255a74838d9d88fe75095c4824cdbc9b._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 10" - date="2020-07-03T19:55:36Z" - content=""" -\"the key generated by import --fast is probably not be the same one generated by a regular import\" -- but that happens already with addurl; is the problem worse here? -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_11_06cd77a9265289decceb5c3b4024db4b._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_11_06cd77a9265289decceb5c3b4024db4b._comment deleted file mode 100644 index 93b1bff1c7..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_11_06cd77a9265289decceb5c3b4024db4b._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 11""" - date="2020-07-24T17:50:03Z" - content=""" -Yes, it can also happen with addurl, but I think it's less likely that two -users add the same url with and without --fast or --relaxed than that two -users sync with the same remote with and without --content. - -Anyway, I opened [[sync_fast_import]]. -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_1_74923628f13c984c8223ea49092f47eb._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_1_74923628f13c984c8223ea49092f47eb._comment deleted file mode 100644 index 0a3daab3d4..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_1_74923628f13c984c8223ea49092f47eb._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-03-19T17:46:09Z" - content=""" -It would also be possible for listImportableContents to -return an url that can be used to publically download the content, -which git-annex could derive a URL key from (as well as recording the url). - -If the ContentIdentifier is something globally unique or using some kind -of proprietary hashing (like an S3 version ID), it could be used to -construct a key. (Note that it would be possible for a remote to include its -UUID in the ContentIdentifier if it's not otherwise globally unique.) -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_2_290fa11aa2733cb577c6dfea8b587d2e._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_2_290fa11aa2733cb577c6dfea8b587d2e._comment deleted file mode 100644 index a609006199..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_2_290fa11aa2733cb577c6dfea8b587d2e._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="annex.thin for importing from directory special remote" - date="2020-07-01T22:23:58Z" - content=""" -As a special case, when importing from a directory special remote, could there be an option to hardlink the files into the repo instead of copying them? -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_3_6c9c808ce7a50c71345be4969a6f7a90._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_3_6c9c808ce7a50c71345be4969a6f7a90._comment deleted file mode 100644 index 6776ab19a1..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_3_6c9c808ce7a50c71345be4969a6f7a90._comment +++ /dev/null @@ -1,41 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-07-02T18:18:57Z" - content=""" -Yeah, a directory special remote special case would be good. -It's kind of needed for [[remove_legacy_import_directory_interface]]. - -It could just as well hash the file in place in the directory, -and leave it there, not "downloading" it into the annex. Which avoids -me having to think about whether hard linking to files in a -special remote makes any kind of sense. (My gut feeling is it's not -the same as hard linking inside a git-annex repo.) - -This approach needs this interface to be added. - - importKey :: Maybe (ExportLocation -> ContentIdentifier -> ByteSize -> Annex Key) - -Then just use that, when it's available, rather than -retrieveExportWithContentIdentifier. Easy enough. - -And other remotes could use this interface too. -If some other remote has public urls, it could generate a URL key -and return that. And if a remote has server-side checksums, it can generate -a key from the checksum, as long as it's a checksum git-annex supports. -So this interface seems sufficiently general. - -This would be easy to add to the special remote protocol too, although -some new plumbing command might be needed to help generate a key -from information like the md5 and size. Eg, -`git annex genkey --type=MD5 --size=100 --value=3939393` and `git annex genkey ---type=URL value=http://example.com/foo` - ----- - -User interface changes: `git-annex import --from remote --fast` and -`git annex sync` without --content could import from a remote that -way, if it supports importKey. (Currently sync only imports with ---content so this is kind of a behavior change, but I think an ok one to -make.) -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_4_30a3ebad1c2a0129b7629842edc9095f._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_4_30a3ebad1c2a0129b7629842edc9095f._comment deleted file mode 100644 index a422d62b0a..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_4_30a3ebad1c2a0129b7629842edc9095f._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 4" - date="2020-07-02T20:22:25Z" - content=""" -Thanks -- this would solve (among other things) [[bugs/removeLink_failed_when_initializing_a_repo_in_a_VirtualBox_shared_folder]]: I could put the git-annex repo on the normal filesystem inside the VM, and only the directory special remote would then deal with the broken vboxsf filesystem. import-tree *with* copying isn't possible as the files are too big. -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_5_df0444ec9d9542d5ecadf6f84027930f._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_5_df0444ec9d9542d5ecadf6f84027930f._comment deleted file mode 100644 index 431a852517..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_5_df0444ec9d9542d5ecadf6f84027930f._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2020-07-03T01:46:46Z" - content=""" -Hmm, it would also be possible for a remote to generate a WORM key, -as long as there was a way for it to get a timestamp for the file being -imported. - -That might let it be implemented for several other special remotes. - -Although I'm wary about making git-annex ever use WORM without being -explicitly asked to. annex.eatworms? ;) -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_6_a04159b9c052e61483757ad69ba989db._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_6_a04159b9c052e61483757ad69ba989db._comment deleted file mode 100644 index cbfb7039b5..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_6_a04159b9c052e61483757ad69ba989db._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 6""" - date="2020-07-03T15:52:26Z" - content=""" -This has merge conflict potential, because the key generated by import ---fast is probably not be the same one generated by a regular import. So, if -two repositories are both importing from the same special remote, there will be -a need to resolve the resulting merge conflicts. - -Since git-annex sync is often run with and without --content, it's probably -the most likely problem point for this. Perhaps there should be another -config that controls whether sync does a fast import or not, and not -control it with --content? -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_7_87dda7df06cde981f142781849793c08._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_7_87dda7df06cde981f142781849793c08._comment deleted file mode 100644 index b340bd6954..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_7_87dda7df06cde981f142781849793c08._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 7""" - date="2020-07-03T16:27:10Z" - content=""" -Hmm, --fast is not very descriptive for this when it's used with a -directory special remote, because hashing is almost as slow as copying. - -Probably better to use --no-content and --content, same as sync. -(Though unfortunately with an opposite default though iirc there are plans -somewhere to transition sync to default to --content). -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_8_69d49ecfaf18d85a535fd7b52e443cc2._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_8_69d49ecfaf18d85a535fd7b52e443cc2._comment deleted file mode 100644 index 8d57141205..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_8_69d49ecfaf18d85a535fd7b52e443cc2._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 8""" - date="2020-07-03T17:39:19Z" - content=""" -Note that, since exporttree remotes are always untrusted, after importing ---no-content from one, fsck is going to complain about it being the only -location with the content. - -Which seems right.. That content could be overwritten at any time and the -only copy lost. But still worth keeping in mind. -"""]] diff --git a/doc/todo/importing_from_special_remote_without_downloading/comment_9_43054eb24fde3e229d8d75be0719371f._comment b/doc/todo/importing_from_special_remote_without_downloading/comment_9_43054eb24fde3e229d8d75be0719371f._comment deleted file mode 100644 index 1cc646fbfd..0000000000 --- a/doc/todo/importing_from_special_remote_without_downloading/comment_9_43054eb24fde3e229d8d75be0719371f._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 9""" - date="2020-07-03T18:29:05Z" - content=""" -implemented, directory remote only, but it could be added to adb easily, -and possibly to S3. Also added it to the proposed import extension to the -external special remote protocol. - -Still unsure what to do about git-annex sync without --content importing. -For now, sync doesn't do content-less imports still, but that could be -changed if the concerns in comment #6 are dealt with. -"""]] diff --git a/doc/todo/interruped_move_resume.mdwn b/doc/todo/interruped_move_resume.mdwn deleted file mode 100644 index 4c1226d701..0000000000 --- a/doc/todo/interruped_move_resume.mdwn +++ /dev/null @@ -1,83 +0,0 @@ -When a `git annex move` is interrupted at a point where the content has -been transferred, but not yet dropped from the remote, resuming the move -will often refuse to drop the content, because it would violate numcopies. - -Eg, if numcopies is 2, and there is only 1 extant copy, on a remote, -git-annex move --from remote will normally ignore numcopies (since it's not -getting any worse) and remove the content from the remote after -transferring it. But, on resume, git-annex sees there are 2 copies and -numcopies is 2, so it can't drop the copy from the remote. - -This happens to me often enough to be annoying. Note that being interrupted -during checksum verification makes it happen, so the window is relatively -wide. - -I think it can also happen with move --to, although I can't remember seeing -that. - -Perhaps some local state could avoid this problem? - ---[[Joey]] - -> One simple way would be to drop the content from the remote before moving -> it to annex/objects/. Then if the move were interrupted before the drop, -> it could resume the interrupted transfer, and numcopies would work the -> same as it did when the move started. -> -> > After an interrupted move, whereis would say the content is present, -> > but eg an annex link to it would be broken. That seems surprising, -> > and if the user doesn't think to resume the move, fsck would have to be -> > made to deal with it. I don't much like this approach, it seems to -> > change an invariant that usually existance of copy on disk is ground -> > truth, and location tracking tries to reflect it. With this, location -> > tracking would be correct, but only because the content is in an -> > unusual place on disk that it can be recovered from. -> -> Or: Move to annex/objects/ w/o updating local location log. -> Then do the drop, updating the remote's location log as now. -> Then update local location log. -> > -> > If interrupted, and then the move is resumed, it will see -> > there's a local copy, and drop again from the remote. Either that -> > finishes the interrupted drop, or the drop already happened and it's a -> > noop. Either way, the local location log then gets updated. -> > That should clean things up. -> > -> > But, if a sync is done with the remote first, and then the move -> > is resumed, it will no longer think the remote has a copy. This is -> > where the only copy can appear missing (in whereis). So a fsck -> > will be needed to recover. Or, move could be made to recover from -> > this too, noticing the local copy and updating the location log to -> > reflect it. -> > -> > Still, if the move is interrupted and never resumed, after a sync -> > with the remote, the only copy appears missing, which does seem -> > potentially confusing. - -> Local state could be a file listing keys that have had a move started -> but not finished. When doing the same move, it should be allowed to -> succeed even if numcopies would prevent it. More accurately, it -> should disregard the local copy when checking numcopies for a move -> --from. And for a move --to, it should disregard the remote copy. -> May need 2 separate lists for the two kinds of moves. -> -> > This is complex to implement, but it avoids the gotchas in the earlier -> > ideas, so I think is best. --[[Joey]] - -> > > Implementation will involve willDropMakeItWorse, -> > > which is passed a deststartedwithcopy that currently comes from -> > > inAnnex/checkPresent. Check the log, and if -> > > the interrupted move started with the move destination -> > > not having a copy, pass False. - -Are there any situations where this would be surprising? Eg, if git-annex -move were interrupted, and then a year later, run again, and proceeded -to apparently violate numcopies? - -Maybe, OTOH I've run into this problem probably weeks after the first move -got interrupted. Eg, if files are always moved from repo A to repo B, -leaving repo A empty, this problem can cause stuff to build up on repo A -unexpectedly. And in such a case, the timing of the resumed move does not -matter, the user expected files to always get eventually moved from A. - -[[fixed|done]] --[[Joey]] diff --git a/doc/todo/introspect_preferred_content_expressions.mdwn b/doc/todo/introspect_preferred_content_expressions.mdwn deleted file mode 100644 index 2d176eef23..0000000000 --- a/doc/todo/introspect_preferred_content_expressions.mdwn +++ /dev/null @@ -1,21 +0,0 @@ -Several todos need to examine preferred content expressions to see if -any of the terms in them match some criteria. - -That includes: - -* [[todo/sync_fast_import]] -* [[todo/faster_key_lookup_for_limits]] -* [[todo/skip_first_pass_in_git_annex_sync]] - -Internally, preferred content expressions are compiled -into a `Matcher (AssumeNotPresent -> MatchInfo -> Annex Bool)` - -The presence of the function there is a problem, because haskell does not -allow comparing functions for equality. So probably what is needed is -something that contains that function but also indicates which preferred -content term it's for. - -Or, perhaps, not the term, but the specific criteria needed by each such -todo. - -> [[done]] --[[Joey]] diff --git a/doc/todo/keep_annexed_files_for_a_while.mdwn b/doc/todo/keep_annexed_files_for_a_while.mdwn deleted file mode 100644 index f6b91cbf1a..0000000000 --- a/doc/todo/keep_annexed_files_for_a_while.mdwn +++ /dev/null @@ -1,14 +0,0 @@ -I don't want files that I dropped to immediately disappear from my local or all of my remotes repos on the next sync. Especially in situations where changes to the git-annex repo get automatically and immediately replicated to remote repos, I want a configurable "grace" period before files in .git/annex/objects get really deleted. - -This has similarities to the "trash" on a desktop. It might also be nice to - -* configure a maximum amount of space of the "trash" -* have a way to see the contents of the trash to easily recover deleted files - -Maybe it would make sense to just move dropped files to the desktops trash? "git annex trash" as an alternative to drop? - -> This seems likely to have been a misunderstanding of what drop does, -> since dropping from the local repo would not remove the content from a -> remote. -> -> closing as there's no clear todo here. [[done]] --[[Joey]] diff --git a/doc/todo/limit_forwardRetry.mdwn b/doc/todo/limit_forwardRetry.mdwn deleted file mode 100644 index 9162b08fd4..0000000000 --- a/doc/todo/limit_forwardRetry.mdwn +++ /dev/null @@ -1,29 +0,0 @@ -The forwardRetry RetryDecider keeps retrying a transfer as long as at least -one more byte got transferred than in the previous, failed try. - -Suppose that a transfer was restarting from the beginning each time, and it -just so happened that each try got a tiny little bit further before -failing. Then transferring an `N` byte object could result in `sum [1..N]` -bytes being sent. Worst case. (Real world it involves the size of chunks -sent in a failing operation, so probably `sum [1..N/1024]` or so.) - -So I think forwardRetry should cap after some amount of automatic retrying. -Ie, it could give up after 5 retries. --[[Joey]] - -Of course, the real use case for forwardRetry is remotes that use eg, rsync -and can really resume at the last byte. But, forwardRetry can't tell -if a remote is doing that (unless some timing heuristics were used). Around -5 retries seems fairly reasonable for that case too, it would be unlikely -for a rsync transfer to keep failing so many times while still making -forward progess. --[[Joey]] - -> Or could add data to remotes about this, but it would need to be added -> for external special remotes too, and this does not really seem worth the -> complication. -> -> I think, even if a remote does not support resuming like -> rsync, it makes sense to retry a few failed transfers if it's getting -> closer to success each time, because forward progress suggests whatever -> made it fail is becoming less of a problem. - -[[done]] --[[Joey]] diff --git a/doc/todo/limit_to_low_cost_remotes.mdwn b/doc/todo/limit_to_low_cost_remotes.mdwn deleted file mode 100644 index 9e38f67a82..0000000000 --- a/doc/todo/limit_to_low_cost_remotes.mdwn +++ /dev/null @@ -1,19 +0,0 @@ -Add --maximum-cost=N which prevents trying to access any remotes with a -larger cost. May as well add --minimum-cost too for completeness. - -My use case: Want to git annex get --auto and pull from any of 3 usb -drives, but not from the network. --[[Joey]] - -> Hmm, [[todo/to_and_from_multiple_remotes]] might be another way to do -> that. Put the 3 drives in a git remote group, or list the remotes on the -> fly. -> -> There could still be benefit in avoiding high cost remotes. But, the cost -> numbers are only intended to create a local ordering, so making them part of a -> user interface is kind of weird. While 50 might be a high cost in one -> repository, in another repository it could be a fairly low cost. The user -> would need to examine all the costs to pick the cost they want; using -> remote names seems better UI. --[[Joey]] - -> > that seems convincing reason not to implement this and instead -> > implement remote groups. [[wontfix|done]] --[[Joey]] diff --git a/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative.mdwn b/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative.mdwn deleted file mode 100644 index 2978049ab1..0000000000 --- a/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative.mdwn +++ /dev/null @@ -1,33 +0,0 @@ -ATM upon `get` of a file for which no remote in .git/config provides its content, git-annex spits out a message like - -[[!format sh """ -/tmp/najafi-2018-nwb > git annex get data/FN_dataSharing/nwb/mouse1_fni16_150817_001_ch2-PnevPanResults-170808-190057.nwb -(merging origin/git-annex into git-annex...) -(recording state in git...) -(scanning for unlocked files...) -get data/FN_dataSharing/nwb/mouse1_fni16_150817_001_ch2-PnevPanResults-170808-190057.nwb - Remote origin not usable by git-annex; setting annex-ignore -(not available) - Try making some of these repositories available: - 2cca1320-6f51-4acf-a778-efdc79f87ab3 -- smaug:/mnt/btrfs/datasets/datalad/crawl/labs/churchland/najafi-2018-nwb - e513795e-1311-431d-8106-917d9528cfbd -- datasets.datalad.org - - (Note that these git remotes have annex-ignore set: origin) -failed -(recording state in git...) -git-annex: get: 1 failed -"""]] - -although those remote descriptions/names give an idea for an informed user, they do not event differentiate between regular and special remotes. Special remotes could just be "enabled", some of them might even have `autoenable` set. May be it could separate them and provide a message like - -[[!format sh """ -... - Try making some of these repositories available: - 2cca1320-6f51-4acf-a778-efdc79f87ab3 -- smaug:/mnt/btrfs/datasets/datalad/crawl/labs/churchland/najafi-2018-nwb - or enable (using git annex enableremote ) one of: - e513795e-1311-431d-8106-917d9528cfbd -- datasets.datalad.org -"""]] - -[[!meta author=yoh]] - -> implemented as shown. [[done]] --[[Joey]] diff --git a/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_1_23d4ae943a4f4389ef21993907cf38ae._comment b/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_1_23d4ae943a4f4389ef21993907cf38ae._comment deleted file mode 100644 index e5aefb2307..0000000000 --- a/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_1_23d4ae943a4f4389ef21993907cf38ae._comment +++ /dev/null @@ -1,38 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-09-30T17:55:22Z" - content=""" -I'm not sure that the distinction between regular and special remotes is -likely to matter in general? - -If I intuit correctly, in your use case, you may have special remotes that -are extremely easy to enable. (Auto-enabling seems a red herring since it -didn't get autoenabled). While conversely some random repository -might be on a LAN/device the user doesn't have access to. - -But it seems just as likely that a user might have a special remote that -needs installing extra software to access, or needs a password or other -authentication method that's a pain, but it be easy enough to add a ssh -remote pointing at another repository on the LAN, or to mount a drive. - -Or in my personal setup, some repositories are on offline drives and a pain -to access, others are on network attached storage and easy, and special -remotes are a distant third choice. (I use repo descriptions to -differentiate.) - -I also feel that this message is already really too verbose, and adding -lots more instructions to it will overall hurt usability. Bear in mind -there can be many such messages displayed by a single command. - -Also, the proposed output suggesting to run git-annex enableremote doesn't -make sense if the special remote is actually already enabled, but was still -not able to be accessed for whatever reason. The existing message is -intentionally worded so it works in either case, disambiguated by -displaying the names of the remotes that are enabled. - -It might be that more metadata about repositories would help, like it -already separates out untrusted repositories into a separate list. -But it would have to be metadata that applies to all users of a repository, -or is somehow probed at runtime. -"""]] diff --git a/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_2_5a23cff0de7af9d1792969b29e9ac564._comment b/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_2_5a23cff0de7af9d1792969b29e9ac564._comment deleted file mode 100644 index b6a98e53e2..0000000000 --- a/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_2_5a23cff0de7af9d1792969b29e9ac564._comment +++ /dev/null @@ -1,34 +0,0 @@ -[[!comment format=mdwn - username="yarikoptic" - avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" - subject="comment 2" - date="2019-09-30T20:35:59Z" - content=""" -> I'm not sure that the distinction between regular and special remotes is likely to matter in general? - -Those (regular git repositories, and special remotes) are technically completely different beasts, and \"made available\" using different mechanisms (`git remote add` vs `git annex enableremote`). Listing them in one list makes it hard-to-impossible for a user to choose a correct command without background knowledge. Indeed some of them (regardless of the type) would be harder to \"make available\" than the others, but that is different type of information which annex unlikely to ever contain and thus to express in the message. `autoenabled` ones though are more likely to be the \"easy ones\". - -> (Auto-enabling seems a red herring since it didn't get autoenabled) - -`datalad install` autoenables by default since we call `git annex init` on a fresh clone (IIRC if we see `git-annex` branch on remote). With pure `git annex`, I believe it is only if you run `git annex init` explicitly after cloning, you would get it autoenabled. So `git clone https://github.com/dandi/najafi-2018-nwb && cd najafi-2018-nwb && git annex get data/FN_dataSharing/nwb/mouse1_fni16_150817_001_ch2-PnevPanResults-170808-190057.nwb` wouldn't work, while the one with `git annex init` before `git annex get` would. -So I wouldn't say it is `red herring` per se - I (user) can end up in a situation where a special remote was not enabled since I did not explicitly `git annex init` locally. - -> ... Bear in mind there can be many such messages displayed by a single command. - -yeah, that is what I (as a user) dislike as well. I even thought that in `datalad` (e.g. [#3078](https://github.com/datalad/datalad/issues/3078)) we could parse those and provide a single summary statement... I think that splitting here into two wouldn't be the straw to break the camel's back. Some more generic (re)solution is needed. - -> Also, the proposed output suggesting to run git-annex enableremote doesn't make sense if the special remote is actually already enabled, but was still not able to be accessed for whatever reason. - -Indeed. But `git annex` \"knows\" either any given special remote was or was not available/tried, correct? To a user (if we forget about the verbosity for a moment) most informative message then could be - -1. a list of remotes which tried but failed (thus might need to be \"made available) - may be even with some reason for each (e.g. \"connection time out\", \"file is missing\", ...) -2. a list of regular remotes (to be added via `git remote add`) -3. a list of special remotes (to be enabled via `git annex enableremote`) - -from `1.` I would see if I should do something about what I had already connected to, from 2. and 3. I would immediately see what and how to enable (if I see that I potentially has access to it) - -> It might be that more metadata about repositories would help, like it already separates out untrusted repositories into a separate list. - -Besides considering untrusted repos last (could be placed last in any corresponding list) I personally do not see such separation as useful. - -"""]] diff --git a/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_3_49591a74c990c12823671926680cbd4f._comment b/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_3_49591a74c990c12823671926680cbd4f._comment deleted file mode 100644 index d4fce2c2ad..0000000000 --- a/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_3_49591a74c990c12823671926680cbd4f._comment +++ /dev/null @@ -1,25 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-09-22T16:15:49Z" - content=""" -Yes, it knows which remotes are configured, and every configured remote -that it's going to list will have been tried and not been accessible -when there's such a message. So, the list can be split into repos -that have a remote and those without one. Eg: - - Try making some of these remotes accessible: - 2370e576-fcef-11ea-a46e-7fce4739e70f -- joey@localhost:/media/usb [usbdrive] - 346cad24-fcef-11ea-a275-d3951b734346 -- joey@server:repo [origin] - 9808c3da-fcf0-11ea-b47f-cfa6e90a9d4a -- amazon S3 - Maybe enable some of these special remotes (git annex enableremote): - e513795e-1311-431d-8106-917d9528cfbd -- datasets.datalad.org - Maybe add some of these git remotes (git remote add): - 2cca1320-6f51-4acf-a778-efdc79f87ab3 -- smaug:/mnt/btrfs/datasets/datalad/crawl/labs/churchland/najafi-2018-nwb - -So only 2 lines longer at most. - -(The "Maybe" wording is because "And/or" is so ugly, and yet -the user may need to only do one, or more than one, depending on what -they're doing.) -"""]] diff --git a/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_4_aefdd17cbfc249ac9b3c6880cabca6d1._comment b/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_4_aefdd17cbfc249ac9b3c6880cabca6d1._comment deleted file mode 100644 index b28de85723..0000000000 --- a/doc/todo/make___34____Try_making_some_of_these_repositories_available__34___more_informative/comment_4_aefdd17cbfc249ac9b3c6880cabca6d1._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-09-22T16:45:19Z" - content=""" -This risks changing the --json output. Eg currently it has: - - {"command":"get","wanted":[{"here":false,"uuid":"7f03b57d-5923-489a-be26-1ab254d0620d","description":"archive-13 [house]"}],"note":"from house...\nrsync failed -- run git annex again to resume file transfer\nUnable to access these remotes: house\nTry making some of these repositories available:\n\t7f03b57d-5923-489a-be26-1ab254d0620d -- archive-13 [house]\n","skipped":[] - -The "wanted" list comes from the display of the list of -uuids, but now there would be up to 3 lists displayed. - -I doubt anything uses that, but I don't want to change the json, -so I suppose it would need to keep the current behavior when json is -enabled, ugh. -"""]] diff --git a/doc/todo/make_http_special_remote_support_exporttree_remotes.mdwn b/doc/todo/make_http_special_remote_support_exporttree_remotes.mdwn deleted file mode 100644 index 8faa4858da..0000000000 --- a/doc/todo/make_http_special_remote_support_exporttree_remotes.mdwn +++ /dev/null @@ -1,6 +0,0 @@ -The http special remote doesn't currently support being used with a ---sameas remote that uses exporttree=yes. - -It seems like this should be fairly easy to implement. --[[Joey]] - -> [[done]] --[[Joey]] diff --git a/doc/todo/making_it_easier_to_smudge_dotfiles.mdwn b/doc/todo/making_it_easier_to_smudge_dotfiles.mdwn deleted file mode 100644 index 6a5e1f41fc..0000000000 --- a/doc/todo/making_it_easier_to_smudge_dotfiles.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -I want to add some dotfiles in the root of my repository to git-annex as unlocked annexed files. So I edited `.git/info/attributes` to remove the line `.* !filter`, such that it only contains the line `* filter=annex`. This seems to be working fine. - -I was thinking that it might make sense to have a `git annex config` option to tell git-annex not to add the `.* !filter` line to `.git/info/attributes` when initialising other clones of this repo. In the meantime, I've worked around it using a `post_checkout` hook in my `~/.mrconfig` which edits `.git/info/attributes`. - ---spwhitton - -> annex.dotfiles added, [[done]] --[[Joey]] diff --git a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_1_73ad5cd7f65c94a1db859d22eb6eece4._comment b/doc/todo/making_it_easier_to_smudge_dotfiles/comment_1_73ad5cd7f65c94a1db859d22eb6eece4._comment deleted file mode 100644 index 96b75c3d10..0000000000 --- a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_1_73ad5cd7f65c94a1db859d22eb6eece4._comment +++ /dev/null @@ -1,32 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-12-19T16:08:09Z" - content=""" -Hmm, it used to be that `git add .` would smudge all dotfiles without that -line, but now annex.largefiles has to be configured for it to smudge -anything. - -So, this could be dealt with in annex.largefiles. Both `anything` and -`include=*` currently match dotfiles. It's kind of weird really that `*` -matches dotfiles; it does not in the shell. If `*` did not match dotfiles -(and `anything` is just an alias for `include=*`), it would be fairly safe -to remove the `.* !filter` line by default. (If annex.largefiles has a -content-based setting, and a dotfile is large enough or the right mime type -or whatever, it's reasonable to default to smudging it.) - -Then, you could set annex.largfiles to match the dotfiles you want, -eg `include=* or include=.mydotfile`. You could put the config in -.gitattributes if you want to configure it globally. - -This change to annex.largefiles would also let `git-annex add` -stop skipping dotfiles by default; instead annex.largefiles would not match -dotfiles unless the user explicitly configured it to, and so the dotfiles -would be added as small files, directly to git. - -I like this because it unifies the behaviors of the two ways of adding, -and it reduces the complexity, rather than adding more. - -Removing the `.* !filter` line by default -would need to be done as part of the v8 upgrade, or a later upgrade. -"""]] diff --git a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_2_3912c1194c3f4f0e4c372e7603e0a3e5._comment b/doc/todo/making_it_easier_to_smudge_dotfiles/comment_2_3912c1194c3f4f0e4c372e7603e0a3e5._comment deleted file mode 100644 index 11cd3e4441..0000000000 --- a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_2_3912c1194c3f4f0e4c372e7603e0a3e5._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2019-12-19T17:17:31Z" - content=""" -`*` is not only used in annex.largefiles, but other pagespecs too. -Like preferred content: - - exclude=archive/* - -So changing `*` to not match dotfiles would have wide reaching effects, -and it's really not good for different versions of git-annex to parse -preferred content expressions differently. And it seems too confusing to -have `*` match differently in annex.largefiles than in other pagespecs. - -Having a single config that controls both kinds of adds still seems like a -good idea, but I don't know what that config should be. -annex.largedotfiles? -"""]] diff --git a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_3_10e03e3c90370e83f6f0999f3cd3a89f._comment b/doc/todo/making_it_easier_to_smudge_dotfiles/comment_3_10e03e3c90370e83f6f0999f3cd3a89f._comment deleted file mode 100644 index e368f0e7a9..0000000000 --- a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_3_10e03e3c90370e83f6f0999f3cd3a89f._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2019-12-19T20:37:18Z" - content=""" -How about annex.dotfiles=true enables annexing dotfiles, but only -if annex.largefiles is set to something. Leave it up -to the user to configure annex.largefiles according to their use case. -And if the user neglects to annex.largefiles, this avoids blowing their foot -off by default. - -annex.dotfiles could certainly go in the global `git-annex config`; -annex.largefiles would then make sense to be set in .gitattributes, -or also add support for storing in in `git-annex config` (which avoids -the syntatic hacks needed to shoehorn it into .gitattributes, and makes it -be repo-global the same as the annex.dotfiles config). -"""]] diff --git a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_4_82fe6d1e4a71cdb6c283a21e52f96f88._comment b/doc/todo/making_it_easier_to_smudge_dotfiles/comment_4_82fe6d1e4a71cdb6c283a21e52f96f88._comment deleted file mode 100644 index c9e97bf6dc..0000000000 --- a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_4_82fe6d1e4a71cdb6c283a21e52f96f88._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="spwhitton" - avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb" - subject="comment 4" - date="2019-12-21T23:03:46Z" - content=""" -Hmm, what would the default value of `annex.dotfiles` be? If the default is false (so that there is no behavioural change unless the user explicitly requests it), then why not have it take effect even if annex.largefiles has not been set? -"""]] diff --git a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_5_ef22ff9e2e507931baee09bf536fc5df._comment b/doc/todo/making_it_easier_to_smudge_dotfiles/comment_5_ef22ff9e2e507931baee09bf536fc5df._comment deleted file mode 100644 index 123bcd7a50..0000000000 --- a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_5_ef22ff9e2e507931baee09bf536fc5df._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2019-12-26T20:34:02Z" - content=""" -I've implemented annex.dotfiles in the `v8` branch. - -I did not tie it to annex.largefiles in the end, spwhitton is right. - -`git-annex add` behavior around dotfiles did change in a potentially -surprising way, since dotfiles are assumed to be non-large, they get added -to git. Users who have dotfiles that are large (or dotdirs containing large -files) will need to configure annex.largefiles and annex.dotfiles to avoid -those files being added to git. But, I don't think that will affect many -users, and it avoided a lot of complexity. At least such users can use this -and other semi-recent git-annex configs to avoid `git add` adding their -large dotfiles directly to git. -"""]] diff --git a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_6_62d03e7fccb9e8b824e145d5458e9796._comment b/doc/todo/making_it_easier_to_smudge_dotfiles/comment_6_62d03e7fccb9e8b824e145d5458e9796._comment deleted file mode 100644 index ecef0f67af..0000000000 --- a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_6_62d03e7fccb9e8b824e145d5458e9796._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="spwhitton" - avatar="http://cdn.libravatar.org/avatar/9c3f08f80e67733fd506c353239569eb" - subject="comment 6" - date="2020-02-29T16:34:16Z" - content=""" -Thank you for working on this. Just upgraded my repo to v8 and removed my hack to edit .git/info/attributes. -"""]] diff --git a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_7_66e313842f388bad020d212885eeffd0._comment b/doc/todo/making_it_easier_to_smudge_dotfiles/comment_7_66e313842f388bad020d212885eeffd0._comment deleted file mode 100644 index 893f79bf5f..0000000000 --- a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_7_66e313842f388bad020d212885eeffd0._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="adding dotfiles" - date="2020-02-29T21:35:06Z" - content=""" -Thanks -- this also resolves [[bugs/dotfiles_handled_differently]] (which wasn't actually a bug, just an unexpected special case). - -[[Manpage|git-annex]] says \"Setting annex.dotfiles to true makes dotfiles be added to the annex the same as any other file\" -- \"same as any other file\" means that if `annex.largefiles` is *not* set, or if `annex.gitaddtoannex` is set to `false`, `git add .mydotfile` still adds to git, correct? So, basically, if `annex.dotfiles=false` (default), dotfiles are handled specially (always added to git regardless of other settings), while if `annex.dotfiles=true`, dotfiles are handled the same as non-dotfiles? -"""]] diff --git a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_8_4de0189eba81ce6b23bfe45280985519._comment b/doc/todo/making_it_easier_to_smudge_dotfiles/comment_8_4de0189eba81ce6b23bfe45280985519._comment deleted file mode 100644 index 6557cbfefa..0000000000 --- a/doc/todo/making_it_easier_to_smudge_dotfiles/comment_8_4de0189eba81ce6b23bfe45280985519._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 8""" - date="2020-03-02T19:00:27Z" - content=""" -Yes, `git add` default behavior with dotfiles has not changed -and you have to set both annex.gitaddtoannex and annex.dotfiles -to change it. -"""]] diff --git a/doc/todo/memory_use_increase.mdwn b/doc/todo/memory_use_increase.mdwn deleted file mode 100644 index f161d15054..0000000000 --- a/doc/todo/memory_use_increase.mdwn +++ /dev/null @@ -1,34 +0,0 @@ -Max memory use running `git-annex find` in a repo with 100k annexed files -has increased significantly since 8.20200330, from 36 mb to 118 mb: - -old: - - 5.16user 3.36system 0:16.91elapsed 50%CPU (0avgtext+0avgdata 37188maxresident)k - 897424inputs+0outputs (263major+17954minor)pagefaults 0swaps - -current: - - 7.51user 1.76system 0:06.50elapsed 142%CPU (0avgtext+0avgdata 121812maxresident)k - 0inputs+0outputs (0major+34283minor)pagefaults 0swaps - -Note that it has at the same time gotten nearly 3x faster! - -I don't think this is a memory leak, but instead is probably a change in -buffering behavior, likely connected to optimisations. It may be that it's -an acceptable tradeoff, but it seems to have been somewhat unintentional, -so it needs to be investigated and understood. - -Version 8.20200617 (just before the cat-file --buffer changes): - - 4.78user 3.81system 0:17.94elapsed 47%CPU (0avgtext+0avgdata 37188maxresident)k - 1144592inputs+0outputs (249major+18142minor)pagefaults 0swaps - -Version 8.20200720 (cat-file --buffer) - - annex/git-annex find >/dev/null7.82user 3.01system 0:08.60elapsed 126%CPU (0avgtext+0avgdata 122016maxresident)k - 350752inputs+0outputs (1444major+17634minor)pagefaults 0swaps - -Smoking gun. And probably reasonable. But, why exactly does that optimisation -change the memory use in this way? - -[[done]] diff --git a/doc/todo/memory_use_increase/comment_10_362b878949e691e45fac0ff1aa240688._comment b/doc/todo/memory_use_increase/comment_10_362b878949e691e45fac0ff1aa240688._comment deleted file mode 100644 index f1525e182e..0000000000 --- a/doc/todo/memory_use_increase/comment_10_362b878949e691e45fac0ff1aa240688._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 10""" - date="2020-10-19T17:33:49Z" - content=""" -[[/profiling]] has a history of `+RTS -p` profiles in the same repo. -Comparing against the 10 month old one there, current git-annex find -runs in same time, and actually allocates slightly less memory, 583357880 -bytes down from 608475328. That's memory churn, not max memory usage, -so doesn't rule out a memory leak. But if there is one, it's memory that -was allocated before, so it would need to be a laziness bug I think. -And the profiles are not showing another such leak. - -My feeling is, what's left now is all due to a change to haskell runtime's -memory management, or a library. So not worth keeping this open for since I -can't do anything about it except for keep an eye on it. -"""]] diff --git a/doc/todo/memory_use_increase/comment_1_854006a91f114775797c7c332b2ad83f._comment b/doc/todo/memory_use_increase/comment_1_854006a91f114775797c7c332b2ad83f._comment deleted file mode 100644 index 484391527b..0000000000 --- a/doc/todo/memory_use_increase/comment_1_854006a91f114775797c7c332b2ad83f._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Lukey" - avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" - subject="comment 1" - date="2020-10-11T14:17:47Z" - content=""" -I guess commit de3d7d044d237999c0a16eb51c79625f904dd60d could be the culprit as the list of files could become large if \"git cat-file\" has a large delay between the request and the response. -"""]] diff --git a/doc/todo/memory_use_increase/comment_2_21a2d5a3090a5bbdad413f2221b887b0._comment b/doc/todo/memory_use_increase/comment_2_21a2d5a3090a5bbdad413f2221b887b0._comment deleted file mode 100644 index e497e5bb48..0000000000 --- a/doc/todo/memory_use_increase/comment_2_21a2d5a3090a5bbdad413f2221b887b0._comment +++ /dev/null @@ -1,28 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-10-12T15:42:31Z" - content=""" -That is good thinking. Since that commit, it has since been changed to use -a Chan. Commit bd2d304064915b3785dbd0725bf81643277530ec actually did think -about this a little: - - Chan will be faster than DList here. Bearing in mind, it is unbounded, - but in reality will be bounded by the size of the stdio buffer through - git cat-file. - -I added debug prints of all readChan and writeChan, and they seemed more or -less evenly interleaved throughout the run, although at the start -there were 274 writeChans before the first readChan and at the end -a similar run of readChans. That does not seem large enough to explain the -memory use though. - -Next I tried converting to a TBChan, bounded to 1000 items. Did not affect -memory use. Dropped the bound to 10 items. That actually hung, possibly -the pipeline needs to be fuller than that before stuff comes out the other -end? Also hung with 100 items. With 300 items, it ran, with the same memory -use. - -(Also noticed that the memory use is sometimes more like 225 mb, probably due -to differences in caching and timing.) -"""]] diff --git a/doc/todo/memory_use_increase/comment_3_af1c49a3919e23b15832d205534d7c58._comment b/doc/todo/memory_use_increase/comment_3_af1c49a3919e23b15832d205534d7c58._comment deleted file mode 100644 index 4b65da8068..0000000000 --- a/doc/todo/memory_use_increase/comment_3_af1c49a3919e23b15832d205534d7c58._comment +++ /dev/null @@ -1,40 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-10-12T17:14:05Z" - content=""" -Made a profiling build. Profiles show memory use is growing linearly, a -clear space leak. - -Heap profiling shows main use is PINNED (probably ByteString), -seekHelper/seekFilteredKeys is also large, and takeByteString/decimal -and extractSha next largest. (Those last 3 are clearly from -parseStagedDetails.) - -Allocation profiling shows `ARR_WORDS` (ByteString internal), then -ByteString. - -The profile only shows 35mb in use max, not the full 200+mb. -That and the PINNED and the overall shape seems very similar to the leak described -here http://neilmitchell.blogspot.com/2013/02/chasing-space-leak-in-shake.html -Profiling with "-xt -hy" suggested from that, did not show anything new -though. - -I think there could be two different problems here, whatever is NOT -showing up on the profiling is one, and the other looks to be -a problem in seekHelper. - -I tried a simplified seekHelper (not suitable for production use) -and with that version the memory use is 122mb like I originally -reported, not the 200+mb I see now. - - seekHelper c ww a (WorkTreeItems l) = do - os <- seekOptions ww - (r, cleanup) <- inRepo $ a os (map toRawFilePath l) - return (map (mk Nothing) r, cleanup) - -I don't much like the way the lsfiles code path is reading a lazy bytestring, -copying lines of it to a strict bytestring and parsing each with -attoparsec. Using strict bytestrings and incremental attoparsec would be -better, and might just resolve the problem that profiling is failing to spot. -"""]] diff --git a/doc/todo/memory_use_increase/comment_4_774d540ce6f5c3ffda924159e146721e._comment b/doc/todo/memory_use_increase/comment_4_774d540ce6f5c3ffda924159e146721e._comment deleted file mode 100644 index f877c89000..0000000000 --- a/doc/todo/memory_use_increase/comment_4_774d540ce6f5c3ffda924159e146721e._comment +++ /dev/null @@ -1,32 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-10-12T19:19:40Z" - content=""" -Thinking a little more about this, the lazy bytestring it reads is probably -in around 32kb chunks. The git ls-files --stage output segment for a file -is 50 bytes plus the filename, so probably under 200 bytes. - -The lazy bytestring is split into those segments, and then each segment -is copied to a strict bytestring with L.toStrict. - -How does a lazy bytestring get split on null? L.split uses L.take. -L.take uses S.take on the chunk. S.take simply updates the length of -the bytestring, but the result still keeps the rest of it allocated. -(And similar for drop I assume.) - -So, if L.toStrict is run on a lazy bytestring consisting of a single chunk -that's a strict bytestring, that's had its size reduced by L.split, -the rest is still allocated. And in L.toStrict, there's a special case for a -single chunk input, that bypasses the usual copying: - - goLen1 _ bs Empty = bs - -That keeps the original strict bytestring, not copying it. And so -the rest of it, after the NULL, remains allocated for as long as the result -is in use. - -Hmm, this doesn't explain the memory leak (throwing in a S.copy didn't fix -it either) or why profiling doesn't show the full memory use, but it does -start to explain the PINNED memory use. -"""]] diff --git a/doc/todo/memory_use_increase/comment_5_786f9bfe6de60d199cb4ff56fe55cdde._comment b/doc/todo/memory_use_increase/comment_5_786f9bfe6de60d199cb4ff56fe55cdde._comment deleted file mode 100644 index 138743bb8f..0000000000 --- a/doc/todo/memory_use_increase/comment_5_786f9bfe6de60d199cb4ff56fe55cdde._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2020-10-13T16:22:53Z" - content=""" -Profiling a non-optimised build, the PINNED went up to 90+mb. - -So, it seems likely that the profile is missing some of the pinned memory -due to it being used too briefly to show get recorded, and that it's the -main culprit. -"""]] diff --git a/doc/todo/memory_use_increase/comment_6_8329ffed8b4e84440ae5326d40ece517._comment b/doc/todo/memory_use_increase/comment_6_8329ffed8b4e84440ae5326d40ece517._comment deleted file mode 100644 index ab2c8ba61e..0000000000 --- a/doc/todo/memory_use_increase/comment_6_8329ffed8b4e84440ae5326d40ece517._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 6""" - date="2020-10-13T18:59:08Z" - content=""" -In the alternatepipenullsplit branch, I made what I think is the relevant -part of LsFiles not use a lazy bytestring, but instead read chunks -directly to strict bytestrings. It did not change the PINNED amount or reduce -overall memory use. - -So the pinning must be happening somewhere other than where I had assumed -it was happening. -"""]] diff --git a/doc/todo/memory_use_increase/comment_7_68f580f2df32ee9027f70b4407451937._comment b/doc/todo/memory_use_increase/comment_7_68f580f2df32ee9027f70b4407451937._comment deleted file mode 100644 index 9b5a0db4da..0000000000 --- a/doc/todo/memory_use_increase/comment_7_68f580f2df32ee9027f70b4407451937._comment +++ /dev/null @@ -1,16 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 7""" - date="2020-10-13T19:15:41Z" - content=""" -The seekHelper version in comment 3 significantly changes the -shape of the profile. PINNED is still there, but only a couple mb, -and it's a constant amount of memory, not an upward slope. - -So I think that identifies the main culprit for the mysterious PINNED. -runSegmentPaths is supposed to stream, but it seems it somehow is hanging -on to values longer than it should. - -Although the max memory use of that version is still around 90 mb -according to `time`. The profile is flat at 3500 kb. -"""]] diff --git a/doc/todo/memory_use_increase/comment_8_244acef6b20ac8fc281372f0898a49be._comment b/doc/todo/memory_use_increase/comment_8_244acef6b20ac8fc281372f0898a49be._comment deleted file mode 100644 index 693852e049..0000000000 --- a/doc/todo/memory_use_increase/comment_8_244acef6b20ac8fc281372f0898a49be._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 8""" - date="2020-10-13T19:52:50Z" - content=""" -[[!commit f624876dc21fed141eee368c86f4b209852ab91c]] is where seekHelper -went bad. Fixed that. - -git-annex is still allocating 120 mb in this situation, though the -profile now shows a max memory use of 4 mb. Still don't understand that, so -this remains open. -"""]] diff --git a/doc/todo/memory_use_increase/comment_9_a04a9de3b767d644c0f359af39ee857b._comment b/doc/todo/memory_use_increase/comment_9_a04a9de3b767d644c0f359af39ee857b._comment deleted file mode 100644 index f3c6ff0dca..0000000000 --- a/doc/todo/memory_use_increase/comment_9_a04a9de3b767d644c0f359af39ee857b._comment +++ /dev/null @@ -1,43 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 9""" - date="2020-10-13T20:31:30Z" - content=""" -100k files: 122364k -40k files: 115476k -1k files: 97920k -0 files: 94808k - -So, it's using a little bit more for more files, reasonable if something -is scaling up under load. - -But where did that use of 90mb for 0 files come from? - -My earlier numbers for 8.20200617 were for the debian package of it. I -built that version with a current toolchain: - -100k files: 127240k (vs 37188k with old toolchain) -0 files: 105748k - -So, much of the increase is due to a change in ghc or the toolchain or -libraries. Maybe its memory manager is preallocating more or a buffer has -started to default to significantly larger? - -For that matter, the old build should not have needed 37mb and I'm pretty -sure it used to be more like 2-5mb? - -A haskell hello world still only needs a few kb of memory. - -Made git-annex's `main = print "hello"` -- 61716k - -Made git-annex exit before it runs git ls-files -- 66892k - -Made it exit after starting ls-files and cat-file -- 67320k - -Made it exit just before running the first action on the first file, but -after seekFilteredKeys did its processing -- 93300k - -So this is partly toolchain or compiler caused, but I don't know if the -extra 30mb it grows by that point is because of the toolchain changing, -or another memory leak. -"""]] diff --git a/doc/todo/metadata_batch_command_should_allow_changes_by_key.mdwn b/doc/todo/metadata_batch_command_should_allow_changes_by_key.mdwn deleted file mode 100644 index 64ec724c0e..0000000000 --- a/doc/todo/metadata_batch_command_should_allow_changes_by_key.mdwn +++ /dev/null @@ -1,19 +0,0 @@ -while writing a script that links related files via meta-data i found that meta-data lookup by key works in batch mode, but meta-data storage by key doesn't - -my scripts output was lines like: - - {"key": "SHA256E-s13238976--7f9f459b99e36e9d50e6da349258525c9f63a33bd5f5bf5a3284cc0e4bda5fd8.ARW", "fields": {"linked.image": "SHA256Es3983492--0fef9eba38c21629e2ae06bd7f64dae104d8576e3c4d31b2caf7337a1ed8c3a9.JPG"}} - {"key": "SHA256E-s3983492--0fef9eba38c21629e2ae06bd7f64dae104d8576e3c4d31b2caf7337a1ed8c3a9.JPG", "fields": {"linked.raw": "SHA256E-s13238976--7f9f459b99e36e9d50e6da349258525c9f63a33bd5f5bf5a3284cc0e4bda5fd8.ARW"}} - -and the full pipe was - - git annex info */*.* --json --fast| python3 ./check_has_all_raws.py |git annex metadata --batch --json - -to my surprise all i got was the retrial of the existing meta-data instead of the addition of the linked fields - -IHO git annex should allow to store metadata in batch mode by key - -[[!meta title="metadata --batch parses json strictly, loosen?"]] - -> [[done]] I guess, as there's been no response to my question in over a -> year. --[[Joey]] diff --git a/doc/todo/metadata_batch_command_should_allow_changes_by_key/comment_1_753ae21e1c43130159681c4e4e0b5cf3._comment b/doc/todo/metadata_batch_command_should_allow_changes_by_key/comment_1_753ae21e1c43130159681c4e4e0b5cf3._comment deleted file mode 100644 index f58ea5f1bf..0000000000 --- a/doc/todo/metadata_batch_command_should_allow_changes_by_key/comment_1_753ae21e1c43130159681c4e4e0b5cf3._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="RonnyPfannschmidt" - avatar="http://cdn.libravatar.org/avatar/c5379a3fe2188b7571858c49f9db63c6" - subject="comment 1" - date="2018-07-29T21:41:56Z" - content=""" -i iterated a bit more on the script, -i experienced the same issue when i ran with just field and added a few more filds to the output and changed from a string to a list - -the working version outputs lines like - - {\"command\": \"set\", \"file\": \"DCIM/23980708/DSC04071.ARW\", \"fields\": {\"linked.image\": [\"SHA256E-s3983492--0fef9eba38c21629e2ae06bd7f64dae104d8576e3c4d31b2caf7337a1ed8c3a9.JPG\"]}} - {\"command\": \"set\", \"file\": \"DCIM/23980708/DSC04071.JPG\", \"fields\": {\"linked.raw\": [\"SHA256E-s13238976--7f9f459b99e36e9d50e6da349258525c9f63a33bd5f5bf5a3284cc0e4bda5fd8.ARW\"]}} - -"""]] diff --git a/doc/todo/metadata_batch_command_should_allow_changes_by_key/comment_2_fd80a5425bef1aface37c9836b387fdd._comment b/doc/todo/metadata_batch_command_should_allow_changes_by_key/comment_2_fd80a5425bef1aface37c9836b387fdd._comment deleted file mode 100644 index 51d11d1c42..0000000000 --- a/doc/todo/metadata_batch_command_should_allow_changes_by_key/comment_2_fd80a5425bef1aface37c9836b387fdd._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2018-10-01T16:09:13Z" - content=""" -It does. - - git annex metadata --batch --json - {"key": "SHA256E-s30--56c5f90f308696d997525622df4103a31d50ef70d22ceb457d5f87a8b72283cc", "fields":{"author":["bar"]}} - {"command":"metadata","note":"author=bar\nauthor-lastchanged=2018-10-01@16-08-50\nlastchanged=2018-10-01@16-08-50\n","success":true,"key":"SHA256E-s30--56c5f90f308696d997525622df4103a31d50ef70d22ceb457d5f87a8b72283cc","file":null,"fields":{"author":["bar"],"lastchanged":["2018-10-01@16-08-50"],"author-lastchanged":["2018-10-01@16-08-50"]}} - -Your input is malformed, the value of a field needs to be inside `[]` like `["bar"]` above. -The same format as it's output. -Since that part of your json did not parse it thinks you want to query the metadata, not set -it. - -Does your json library actually convert a list containing a -single value into that value not in a list and vice-versa? -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures.mdwn b/doc/todo/more_extensive_retries_to_mask_transient_failures.mdwn deleted file mode 100644 index c030f0da70..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures.mdwn +++ /dev/null @@ -1,5 +0,0 @@ -In a number of scenarios (e.g. [[bugs/still_seeing_errors_with_parallel_git-annex-add]], [[bugs/parallel_copy_fails]], [[git-annex-sync/#comment-aceb18109c0a536e04bcdd3aa04bda29]]), `git-annex` operations may fail or hang due to transient conditions. It would help a lot if `git-annex` could be configured to fail timed-out operations, and to retry failed operations after a delay. This would especially help when using `git-annex` in a script or a higher-level tool. I've tried wrapping some retry logic around `git-annex` calls, but it seems `git-annex` itself is in the best position to do that sensibly (e.g. only retrying idempotent operations, or capping retries per remote). This would be a catch-all fix for unusual conditions that are hard to test for. - -`git-annex` already has config options `annex.retry` and `annex.retry-delay`, but it seems that they don't cover all failure types. - -> Added annex.stalldetection, [[done]] --[[Joey]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_10_f39b7dad4fa1bf4a70bde2e74372fe00._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_10_f39b7dad4fa1bf4a70bde2e74372fe00._comment deleted file mode 100644 index f4756dd94d..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_10_f39b7dad4fa1bf4a70bde2e74372fe00._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 10""" - date="2020-06-03T15:51:41Z" - content=""" -Just using async is not enough, if thread A asyncs thread B which asyncs -thread C, and then A cancels B, that leaves C running. - -withAsync avoids that problem. So around 30 calls to `async` need to be -converted. Also, any library that might use `async` would be a problem. - -(Seems that `concurrently` and `race` also avoid the problem, being built on -withAsync.) - -(Some uses of `async` do avoid the problem because the async thread exits -for some other reason when the outer thread is terminated. Eg, -Database.Handle auto-terminates its worker thread when the handle gets -garbage collected.) -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_11_9c289d93f49ff59fddd214d51356b1f9._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_11_9c289d93f49ff59fddd214d51356b1f9._comment deleted file mode 100644 index a61eca8677..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_11_9c289d93f49ff59fddd214d51356b1f9._comment +++ /dev/null @@ -1,33 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 11""" - date="2020-06-03T16:10:44Z" - content=""" -Ah, withCreateProcess is the bracketing version that's needed, it uses -cleanupProcess. And readProcess already uses it. It was easy to make -safeSystem use it, so I've already done so. Other calls to waitForProcess -probably indicate places that need to use it. - -Reading its code, I found a couple of gotchas: - -* If another thread has a Handle for the process, and are "holding the - handle lock", and lead to deadlock when cleanupProcess tries to close - them. This would seem to involve something that uses withHandle and - blocks for some reason, maybe just reading/writing to the Handle? - I've tried a few things like passing the handle into another thread - started with async, that uses hGetLine, but could not produce a deadlock, - the process was always killed anyway. - - Seems like using withAsync would help make sure any thread the handle - is passed to does get killed. - -* It sends SIGTERM, which doesn't necessarily kill every process. So - withCreateProcess might exit cleanly but leave the process hanging - around. - - I doubt this is really a problem in anything git-annex runs. - And if some program did ignore SIGTERM, wouldn't ctrl-c of git-annex - also leave that running? Never seen that happen which confirms my feeling - it's likely not a problem in something git-annex runs, but you never - know. -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_12_bd90d3cd846186d9fc6b96199d2e951b._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_12_bd90d3cd846186d9fc6b96199d2e951b._comment deleted file mode 100644 index 50368aa33e..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_12_bd90d3cd846186d9fc6b96199d2e951b._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 12""" - date="2020-06-04T18:44:52Z" - content=""" -Also, anything that catches SomeException could catch an async exception -that was intended to interrupt the thread. Probably need to make a specific -exception for thread interrupts, and make everything that catches -SomeException mask out that exception. - -And, there may be data structures that get filled with a value and need to -be emptied again on an async exception. Things like the lock pool. Need -to somehow audit for those. - -And for that matter, should audit for any code that opens file handles w/o -bracketing, because I'm not really positive it's all bracketed. -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_13_6e5fb676ae08026abeb500d01ab86414._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_13_6e5fb676ae08026abeb500d01ab86414._comment deleted file mode 100644 index 57114323a1..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_13_6e5fb676ae08026abeb500d01ab86414._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 13""" - date="2020-06-04T19:39:23Z" - content=""" -I've converted everything to withCreateProcess, except for process pools -(P2P.IO, Assistant.TransferrerPool, Utility.CoProcess, -Remote.External (update: done), -and P2PSshConnectionPool), which need to be handled as discussed in -comment 8. And also Git.Command.pipeReadLazy may (or may not) need to be -converted. - -During this conversion, I did not watch out for interactive processes that -might block on a password, so any timeout would also affect them. Really, -I don't see a good way to avoid that. Any ssh may or may not need a -password. I guess timeouts will need to affect things stuck on passwords -too, which argues for no default timeout, but otherwise is probably ok -as long as timeouts can be configured on a per-remote basis. -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_14_ee9b52d830d682f0090acc64458fb278._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_14_ee9b52d830d682f0090acc64458fb278._comment deleted file mode 100644 index 898bcab7bb..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_14_ee9b52d830d682f0090acc64458fb278._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 14" - date="2020-06-05T14:51:07Z" - content=""" -Thanks @joeyh for working on this. In my experience ssh password prompts aren't a big issue as you pretty much always want to use ssh-agent. -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_15_f371b5cbb216967d803dfa04707e6ba2._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_15_f371b5cbb216967d803dfa04707e6ba2._comment deleted file mode 100644 index 19e47f9d0e..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_15_f371b5cbb216967d803dfa04707e6ba2._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 15""" - date="2020-06-05T19:00:07Z" - content=""" -Some progress: All threads that `async` or `forkOS` starts are now -confirmed to get shut down when an async exception reaches the code that -uses them. All uses of SomeException are confirmed to not catch async -exceptions. All file opening is confirmed to bracket and close (except for -the lock pool). - -Still to do: - -* process things noted in comment #13 -* data structures that get modified while an action is running and need - to be cleaned up on an async exception (eg the lock pool) -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_16_870fb821a48487d7d4a457eda42c6ebd._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_16_870fb821a48487d7d4a457eda42c6ebd._comment deleted file mode 100644 index 21f87a3d38..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_16_870fb821a48487d7d4a457eda42c6ebd._comment +++ /dev/null @@ -1,55 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 16""" - date="2020-06-09T17:53:08Z" - content=""" -A possible problem is code like this: - - bracket setup cleanup - where - setup = blockingOperationFoo >>= blockingOperationBar - -The problem is that async exceptions are not fully -masked by bracket. See Control.Exception's -discussion of "Interruptible operations". - -All the blocking operations in that example can still receive an async -exception. - -If blockingOperationFoo opens a handle say, and then blockingOperationBar -is hit by an async exception, then cleanup will never run! The handle -will be left open. - -Solution is `uninterruptibleMask_ setup`. Although of course this means -it will block until both operations finish, so they should not be long -duration operations. Alternatively, untangle the two operations, and nest -brackets. - ---- - -Also, if the cleanup action of bracket is an interruptable operation, it too -can receive an async exception. - -This doesn't result in withFile leaking file handles when it brackets hClose, -but I think perhaps only because the GC closes them. -But it looks like -[withCreateProcess has a bug](https://github.com/haskell/process/issues/183), -and maybe [http-client too](https://github.com/snoyberg/http-client/issues/436). - -And it's easy to construct cases where a compound cleanup action leaks: - - main = forever $ do - let p = proc "cat" [] - a <- async $ bracket - (createProcess p) - (\p -> threadDelay 10000 >> cleanupProcess p) - (\_ -> print "started process") - threadDelay 1000 - cancel a - ---- - -So, seems all uses of bracket and things like it (onException etc) -will need to be audited for blocking operations, or just make -uninterruptibleMask_ be used everywhere. Ugh. -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_17_825334555b637a07f93caf3f519d817d._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_17_825334555b637a07f93caf3f519d817d._comment deleted file mode 100644 index 96d714c784..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_17_825334555b637a07f93caf3f519d817d._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="separate processes for parallel transfers" - date="2020-06-12T15:47:28Z" - content=""" -\"separate processes for would of course work, it would add quite a lot of overhead. Especially in places like Annex.Queue\" -- would it be hard to measure how much that overhead is in practice? -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_18_1e1f99966d4d870057d2be4353c20028._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_18_1e1f99966d4d870057d2be4353c20028._comment deleted file mode 100644 index 7ed7871c72..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_18_1e1f99966d4d870057d2be4353c20028._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 18""" - date="2020-07-06T19:11:14Z" - content=""" -Changes to Utility.CoProcess for async exception safety -introduced some very strange behavior and had to be reverted in -[[!commit d66fc1a4646e354a8ae2514183b3d45a20ae8681]]. -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_19_980f4a04543166677ef41fcefd57da03._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_19_980f4a04543166677ef41fcefd57da03._comment deleted file mode 100644 index 11abd5156d..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_19_980f4a04543166677ef41fcefd57da03._comment +++ /dev/null @@ -1,35 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 19""" - date="2020-10-08T13:45:37Z" - content=""" -New plan for this: Extend `git-annex transferkeys` to send back progress -and success/failure info. Make it stop updating location tracking -and other state (the caller should instead) and only handle the actual -data movement and possibly low-level locking associated with it. - -Then running multiple concurrent processes of it should be reasonably -efficient. There will still be some added overhead from pipe communication. -And some remotes will need to do more work. Eg, ones that use a tcp connection -will need to open more connections (one per -J level), even if earlier -non-transfer use already cached one. So git-annex should only use -transferkeys when annex.retry is enabled, and do the transfers in-process -otherwise. - -Some remote backends should not be used via transferkeys. One, I think, is -the external special remote backend, at least when the async extension is -used. Instead, the external backend can expose something (ie a TChan) that -can be used to stop or restart a transfer that its running, and can just -kill the external process to do it. And if this is implemented for the async -extension it will work just as well for the other externals too, so may as -well do it for all of them. (When the async extension is used, that will -also stop other concurrent transfers, unless that gets extended with the -ability to cancel specific jobs.) - -Also, all parts of Messages that remotes use will need to be serialized -over the transferkeys protocol. Eg, Messages.prompt when the transferkeys -process is doing something like ssh that is prompting for a password. -So transferkeys will need its own OutputType to make Messages communicate -over the pipe. (This is a hard thing, so added a todo for it: -[[todo/serialize_Messages_for_transferkeys]] -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_1_938d2c43a9a1b8432b54bcecaa14c20a._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_1_938d2c43a9a1b8432b54bcecaa14c20a._comment deleted file mode 100644 index 2863a81d0f..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_1_938d2c43a9a1b8432b54bcecaa14c20a._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="automatic retries if index is locked" - date="2019-10-08T16:01:52Z" - content=""" -As a concrete example, if the `index.lock` file exists and has relatively recent mtime and a git process is running, it would help if git-annex could be configured to retry, up to a given number of times with increasing delays between retries, the operation that failed because the index is locked. - -Also, from the log -[[!format sh \"\"\" -add metadata_orig.json ok -(recording state in git...) -fatal: Unable to create '/ssd/crogrun_191008_043145__8684__/viral-ngs-benchmarks/.git/index.lock': File exists. -\"\"\"]] - -it looks like the index.lock conflict is due to writing the [[git-annex branch|internals/#The_git-annex_branch]]? I thought git-annex used a separate index for that? - -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_2_b0260afbe4c8c49caa20df4f5122a00a._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_2_b0260afbe4c8c49caa20df4f5122a00a._comment deleted file mode 100644 index 9df4bb277c..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_2_b0260afbe4c8c49caa20df4f5122a00a._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-01-30T17:45:39Z" - content=""" -I don't think that git-annex can generally abort an operation that is -outright hung. While it's certianly possible to kill a worker thread, if -that thread has other threads associated with it, they could keep on using -resources. And if an external command is hung, the command would keep -running. The only way to guarantee such an abort is to kill the whole -git-annex process and let the signal reap its children. That's what the -assistant does when the UI is used to stop a transfer, it kills the whole -`git-annex transferkeys` process. - -(A locked git index file does not prevent git-annex from making transfers -so AFAICS the comment above is not relevant.) -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_3_281e9061a7940171a1febde6c3dca95f._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_3_281e9061a7940171a1febde6c3dca95f._comment deleted file mode 100644 index 0539b94fac..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_3_281e9061a7940171a1febde6c3dca95f._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-01-30T17:52:31Z" - content=""" -Moving a similar todo I wrote to here: - -I'd like an option that makes transfers (get,copy,etc) of files fail if the -transfer speed falls below a given rate. - -My use case is that I'm pulling files off a hard drive and read -errors/retries (by the kernel) are slowing it down to a crawl for some -files. I'd like to make a first pass getting all the files it can transfer -at the usual speed and skipping the ones that are coming too slow. Then -I can see what files it failed on and either resume those or see if I have -a copy of them somewhere else. - -(Unfortunatly implementing that has the same problems..) -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_4_418a9cbc38142ec1b2d08fd617a3e4d4._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_4_418a9cbc38142ec1b2d08fd617a3e4d4._comment deleted file mode 100644 index 8141141765..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_4_418a9cbc38142ec1b2d08fd617a3e4d4._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="aborting stuck operations so they can be retried" - date="2020-02-05T16:39:36Z" - content=""" -\"The only way to guarantee such an abort is to kill the whole git-annex process and let the signal reap its children\" -- then maybe the initial `git-annex` command can be made a wrapper that starts a separate `git-annex` process to do the actual work, monitors its progress, and kills/reaps/restarts it if it gets stuck? Or `-Jn` could work by starting up several separate git-annex processes, [[each handling a subset of files|parallel_possibilities/#comment-304240ba804513291c1a996b8eb3fd1c]], and the original process could kill/reap/restart any sub-process that gets stuck. This of course presumes idempotent operations. -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_5_69b132f465851421acb7e5edf009995d._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_5_69b132f465851421acb7e5edf009995d._comment deleted file mode 100644 index e764bdd56b..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_5_69b132f465851421acb7e5edf009995d._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="retries due to locked index file" - date="2020-02-05T16:59:40Z" - content=""" -\"A locked git index file does not prevent git-annex from making transfers\" -- by \"mask transient failures\" I meant all types of failures, not just transfers. So e.g. if concurrent operations fail due to contention for the index file lock, retries (after increasing, randomized intervals) could mask the failure. This would help especially for writing scripts/tools on top of git-annex. Logically, some operations -- like `git-annex-add` -- should never fail, and being able to assume that makes scripting easier. -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_6_5a7771a6169caa90fd91bc6ea7b4fe3d._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_6_5a7771a6169caa90fd91bc6ea7b4fe3d._comment deleted file mode 100644 index 51b9a693b3..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_6_5a7771a6169caa90fd91bc6ea7b4fe3d._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="example of where retries could help" - date="2020-02-05T22:19:26Z" - content=""" -As one example, I just had a `git-annex-copy` command fail twice with `git-annex: thread blocked indefinitely in an STM transaction`, then have the same command succeed (or at least get much further -- still running) on the third try. I can write my own wrappers to mask such errors, but a built-in implementation seems generally useful and would know better which failures are likely transient. -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_7_53c2820a0a60ab1efe4560a58ecd4f0b._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_7_53c2820a0a60ab1efe4560a58ecd4f0b._comment deleted file mode 100644 index 5a587f0be6..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_7_53c2820a0a60ab1efe4560a58ecd4f0b._comment +++ /dev/null @@ -1,17 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""Re: retries due to locked index file""" - date="2020-02-20T15:32:11Z" - content=""" -If you have a bug where concurrent adds fail due to some locking problem, -please file a bug report. This is not the correct place to discuss that -problem. - -[So far all you've shown is that a git index file lock, -which could be a stale lock or a lock due to some git operation not under -the control of git-annex, causes git annex add to fail. Since git-annex -takes it own lock before any operation that touches the index file, -I'm quite confident that concurrent git-annex adds do not interfere with -one-another. See Annex.Queue which uses withExclusiveLock when flushing -the queue.] -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_8_b302f24d3234b1c46ed1566cfdf696fb._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_8_b302f24d3234b1c46ed1566cfdf696fb._comment deleted file mode 100644 index cab6838e95..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_8_b302f24d3234b1c46ed1566cfdf696fb._comment +++ /dev/null @@ -1,84 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 8""" - date="2020-02-20T15:43:59Z" - content=""" -While separate processes for would of course work, it would add quite a -lot of overhead. Especially in places like Annex.Queue where multiple -threads can currently cooperate in building up something that gets flushed -to disk together, and separate processes would need to do more work. - -Of course, it's also entirely possible to write your own program that runs -concurrent git-annex processes and kills them if they seem stuck or -whatever. - ----- - -Thinking some more about what would be necessary to make worker threads -cancellable, they would first of all need to use async whenever they fork -any threads of their own. That would be fairly easy to arrange for in the -git-annex code, although it's currently not the case. (Remote.Git at least -forks a thread w/o using async.) - -If a special remote uses a library that itself uses worker threads, that -library would also need to use async. But I am pretty sure that the -libraries in question (S3 and DAV) don't spawn off threads. Also, it would -be an easy sell in the haskell community that any such library that -spawns off its own threads use async or something similar that causes -cancelation of an API call to cancel the threads. - -That leaves processes, and it occurs to me that there all at least 3 -different types of processes a remote might run. - -1. Interactive processes. Eg, ssh prompting for a password, or - git-credential. Killing such a process in the middle of user input - or after it's output a prompt would not be good. Also, being blocked - by a prompt is not the same as having stalled a download. - - Locking already prevents more than one thread from running - such an interactive process; the actions are run inside `prompt`. - So, something would need to be done to prevent killing threads that - are in `prompt`. - -2. Processes in worker pools shared amoung threads of the remote. - The ExternalState pool is an example of this, the P2PSshConnectionPool - is another. - - Killing a thread needs to kill whatever external process - it's currently using, but on the other hand, it could have started an - external process that's idle, or that is now being used by some other - thread, and that process should not be killed. - - I think these all work the same: A process is removed from the pool - while it's being used, and then gets put back into the pool once - it's idle again. So, register the pid as belonging to a thread when - the thread removes it from the pool, and deregister it when the thread - returns it. If the thread gets killed, don't add it back to the pool, - but instead kill the process. - -3. All the rest. For these what's needed is some way to register - the pid of the process that a thread starts as belonging to the thread, - so that on killing the thread that pid can also be killed. - - Starting the process and registering it needs to be done - in an exception-safe way; if a cancelation exception is thrown - as the process is started and before registering its pid, - the process needs to be killed. - - This seems like it would need wrappers for everything in - Utility.Process, to gather and register the pids. - - It might be that a remote runs a child thread using withAsync, and - then that thread starts a process. So the process would be - registered as belonging to the child thread. Then, if the parent - thread gets killed, the signal would propigate to kill the child thread - due to async being used. Resulting in the process being left running, - because it was not registered as belonging to the parent process. - This is difficult to solve, because the child thread does not know what - thread is its parent. - -Wow, this looks like a lot of work, and it would be fragile -- -any mistake would not be noticed until git-annex tried to kill a worker -thread, and left a process behind -- and the consequence of a mistake -could potentially be a (slowish) fork bomb. -"""]] diff --git a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_9_4fe7beb47e88259a72a1cd42c85761a5._comment b/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_9_4fe7beb47e88259a72a1cd42c85761a5._comment deleted file mode 100644 index 8df1b98a7d..0000000000 --- a/doc/todo/more_extensive_retries_to_mask_transient_failures/comment_9_4fe7beb47e88259a72a1cd42c85761a5._comment +++ /dev/null @@ -1,22 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 9""" - date="2020-02-20T18:31:59Z" - content=""" -Hmm, the reason I'm not worried about leaking open FDs -when killing threads is because `bracket` is generally used -when opening files, with a cleanup action that closes them. -(when not doing lazy IO and letting the GC close the file once the value -goes out of scope). So when a thread is canceled, the cleanup actions -automatically run. - -So what's needed is a way to use `bracket` for the non-interactive, -non-pooled processes. - -Currently processes are run by things like `safeSystem` -and `readProcess`, neither of which use a bracketing operation. -But they could use a bracketOnError with an cleanup action that -is passed the pid and kill -9's it. - -That avoids the registration and the child thread problem. -"""]] diff --git a/doc/todo/operate_on_files_affected_by_a_commit_range.mdwn b/doc/todo/operate_on_files_affected_by_a_commit_range.mdwn deleted file mode 100644 index e6455e95db..0000000000 --- a/doc/todo/operate_on_files_affected_by_a_commit_range.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -Sometimes you want to operate on files touched by commits in a range, e.g. to `git-annex-copy` files added in the last 10 commits to an S3 special remote. Could the option be added, to commands that take a path to operate on, to give a commit range, with the meaning "operate on files changed by these commits"? - -> Since my comment gives a way to do it, and there was no followup, I think -> this is [[done]] --[[Joey]] diff --git a/doc/todo/operate_on_files_affected_by_a_commit_range/comment_1_b1cbd5a804a6068213be02fb725c6b0a._comment b/doc/todo/operate_on_files_affected_by_a_commit_range/comment_1_b1cbd5a804a6068213be02fb725c6b0a._comment deleted file mode 100644 index 0ca8d1b080..0000000000 --- a/doc/todo/operate_on_files_affected_by_a_commit_range/comment_1_b1cbd5a804a6068213be02fb725c6b0a._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2019-10-08T16:09:35Z" - content=""" -If you have a command that generates a list of files changed by a series of -commits, you can simply pass that list into --batch which is supported by -most commands. -"""]] diff --git a/doc/todo/optimise_by_converting_Ref_to_ByteString.mdwn b/doc/todo/optimise_by_converting_Ref_to_ByteString.mdwn deleted file mode 100644 index 8b9f79c2da..0000000000 --- a/doc/todo/optimise_by_converting_Ref_to_ByteString.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -Profiling of `git annex find --not --in web` suggests that converting Ref -to contain a ByteString, rather than a String, would eliminate a -fromRawFilePath that uses about 1% of runtime. - -[[!tag confirmed]] - -> Well, I did the conversion. Running that command does not seem to have -> sped up by any appreciable amount. But, I did notice several places where -> using ByteString is certainly more efficient. It may just be that it -> doesn't matter if git-annex is IO bound etc. Still, glad I did it. -> [[done]] --[[Joey]] diff --git a/doc/todo/optimise_by_using_RawFilePath_for_gitAnnexIndex.mdwn b/doc/todo/optimise_by_using_RawFilePath_for_gitAnnexIndex.mdwn deleted file mode 100644 index 527f97e865..0000000000 --- a/doc/todo/optimise_by_using_RawFilePath_for_gitAnnexIndex.mdwn +++ /dev/null @@ -1,28 +0,0 @@ -Profiling `git annex find --in web` found that a single decodeFilePath in -gitAnnexIndex uses 1.4% of time. - -The call path is getRef to withIndex to gitAnnexIndex. - -The FilePath is put into the environment. Switching to use RawFilePath -environment when running processes would be a bit involved, because -System.Process does not support it and would need to be modified/forked. -(The rawfilepath package did it, but unfortunately also changed its API in -other ways.) - -However... git-annex starts up a single, long-running git cat-file -process. The only reason it needs to get gitAnnexIndex after that is -running is to select the git process that is using the right index file. - -So, one way would be to make withIndexFile less generic, -eg a withAnnexIndexFile that does not need the filename to be calculated -each time. - -Or, keep withIndexFile generic but change Annex.cachedgitenv to contain -ByteStrings, and convert to FilePath only when that environment is used to -start a new process. (This seems like it would be a little slower than -the other alternative, since constructing a RawFilePath is also not -entirely without cost, although significantly faster.) - ---[[Joey]] - -> [[done]], and benchmarking shows at least 1.75% speedup --[[Joey]] diff --git a/doc/todo/optimise_journal_access.mdwn b/doc/todo/optimise_journal_access.mdwn deleted file mode 100644 index 848a5c8998..0000000000 --- a/doc/todo/optimise_journal_access.mdwn +++ /dev/null @@ -1,33 +0,0 @@ -Often a command will need to read a number of files from the git-annex -branch, and it uses getJournalFile for each to check for any journalled -change that has not reached the branch. But typically, the journal is empty -and in such a case, that's a lot of time spent trying to open journal files -that DNE. - -Profiling eg, `git annex find --in web` shows things called by getJournalFile -use around 5% of runtime. - -What if, once at startup, it checked if the journal was entirely empty. -If so, it can remember that, and avoid reading journal files. -Perhaps paired with staging the journal if it's not empty. - -When a process writes to the journal, it will need to update its state -to remember it's no longer empty. - -This could lead to behavior changes in some cases where one command is -writing changes and another command used to read them from the journal and -may no longer do so. But any such behavior change is of a behavior that -used to involve a race; the reader could just as well be ahead of the -writer and it would have already behaved as it would after the change. - -> Hmm, not so fast. If the user has two --batch processes, one that makes -> changes and the other that queries, they will expect the querying process -> to see the changes after they were made. There's no race, the user can -> control which process runs by feeding batch inputs to them. -> -> So, --batch and the assistant, as well as batch-like things that don't -> use --batch will need to disable this optimisation it seems. --[[Joey]] - -[[!tag confirmed]] - ->> [[done]] speedup was around 5% --[[Joey]] diff --git a/doc/todo/option_to_add_user-specified_string_to_key.mdwn b/doc/todo/option_to_add_user-specified_string_to_key.mdwn deleted file mode 100644 index 4ed4ce1071..0000000000 --- a/doc/todo/option_to_add_user-specified_string_to_key.mdwn +++ /dev/null @@ -1,20 +0,0 @@ -Add a config (and gitattributes) option annex.userkeystring , such that git-annex-add (and calckey) will include this string in the key. If the string is 'UUID' then a uuid (or shorter random string) will be included instead. - -From [[todo/support_longer_file_extensions]]: - -"The way git-annex picks extensions doesn't need to be stable across all versions of git-annex, because it's only done when initially adding a file, and then the key, with whatever extension, is added as-is and git-annex does not care about the extension thereafter." - -Then, for an MD5E key, the userkeystring can be included before the extension: MD5E-s0--d41d8cd98f00b204e9800998ecf8427e.USERKEYSTRING.txt . -Cleaner would be to add a field to the key, as in MD5E-s0-uUSERKEYSTRING--d41d8cd98f00b204e9800998ecf8427e.txt , if that wouldn't break compatibility. - - -This enables attaching metadata not to file contents, but to the file itself; or partitioning keys (and therefore key metadata) into namespaces. The downside is some loss of -deduplication. This loss may be acceptable. The loss can be mitigated for local repo and non-special remotes: after storing an object with e.g. MD5 d41d8cd98f00b204e9800998ecf8427e under .git/annex/objects, check if there is a symlink .git/annex/contenthash/d41d8cd98f00b204e9800998ecf8427e ; if not, make this a symlink to the object just stored; if yes, -erase the object just stored, and hardlink the symlink's target instead. - -> Closing since [[external_backends]] is implemented, and you could do this -> using it. Whether that's a good idea, I'm fairly doubtful about. Be sure -> to read "considerations for generating keys" in -> -> -> [[done]] --[[Joey]] diff --git a/doc/todo/option_to_add_user-specified_string_to_key/comment_1_9858fb711d788db96efa03c389a2ba52._comment b/doc/todo/option_to_add_user-specified_string_to_key/comment_1_9858fb711d788db96efa03c389a2ba52._comment deleted file mode 100644 index eef5e4966f..0000000000 --- a/doc/todo/option_to_add_user-specified_string_to_key/comment_1_9858fb711d788db96efa03c389a2ba52._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 1" - date="2018-10-02T17:38:10Z" - content=""" -Example of where the ability to add a string to key would help: -annex.genmetadata stores year, month, and day metadata, from the file's modification date; but this will be silently overwritten if later other files with the same key but different modtime are added. On the other hand, if I specified in .gitattributes that a random string uXXXXXX be included in the key, git-annex metadata would in effect become per-file, so one could attach metadata without worrying that it could get silently overwritten. - -An heuristic alternative is to add an option to metadata --get to use the field value from the time the file was committed, rather than the current value. But this doesn't cover the case -of two same-content files in the same commit. -"""]] diff --git a/doc/todo/option_to_add_user-specified_string_to_key/comment_2_915c1c31fcb23455aa11331cd34aa92d._comment b/doc/todo/option_to_add_user-specified_string_to_key/comment_2_915c1c31fcb23455aa11331cd34aa92d._comment deleted file mode 100644 index f0b61e3cf8..0000000000 --- a/doc/todo/option_to_add_user-specified_string_to_key/comment_2_915c1c31fcb23455aa11331cd34aa92d._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 2" - date="2018-10-19T17:54:49Z" - content=""" -Another use case for a user-defined string field in a key is, for URL or WORM keys, to include additional information unique to the object. E.g. for an s3:// URI (handled by an external special remote) could embed the ETag in the key. Or for a dx:// URI to include the DNAnexus file ID . - -From [[internals/key_format]], it seems like adding -uXXXXXX- (with XXXXXX a user-specified string) would not break compatibility? - - - -"""]] diff --git a/doc/todo/option_to_add_user-specified_string_to_key/comment_3_1f028eb4d73516918a83959570180f0d._comment b/doc/todo/option_to_add_user-specified_string_to_key/comment_3_1f028eb4d73516918a83959570180f0d._comment deleted file mode 100644 index c6ace8526f..0000000000 --- a/doc/todo/option_to_add_user-specified_string_to_key/comment_3_1f028eb4d73516918a83959570180f0d._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2019-01-11T20:52:31Z" - content=""" -A very big problem with this idea is that it would defeat git-annex's -protections against a SHA1 collision attack against git, which rely -on keys not containing user-supplied data. - -If this were implemented, annex.securehashesonly would need to prohibit use -of the resulting keys. -"""]] diff --git a/doc/todo/option_to_add_user-specified_string_to_key/comment_4_f7a57038a1ccce664d30b545444fc145._comment b/doc/todo/option_to_add_user-specified_string_to_key/comment_4_f7a57038a1ccce664d30b545444fc145._comment deleted file mode 100644 index 7e6c2eed19..0000000000 --- a/doc/todo/option_to_add_user-specified_string_to_key/comment_4_f7a57038a1ccce664d30b545444fc145._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 4" - date="2019-01-11T22:23:06Z" - content=""" -\"annex.securehashesonly would need to prohibit use of the resulting keys\" -- that's fine, I only use URL/WORM/MD5 keys already, and so does DataLad by default. Btw, if letting keys contain user-provided content is insecure, then all keys that contain file extensions should also be deemed insecure? -"""]] diff --git a/doc/todo/option_to_add_user-specified_string_to_key/comment_5_f2701eabebd95db02497ee6191897198._comment b/doc/todo/option_to_add_user-specified_string_to_key/comment_5_f2701eabebd95db02497ee6191897198._comment deleted file mode 100644 index cdff4fb16a..0000000000 --- a/doc/todo/option_to_add_user-specified_string_to_key/comment_5_f2701eabebd95db02497ee6191897198._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2020-01-30T18:58:13Z" - content=""" -Is there any reason to leave this todo open since [[external_backends]] -would presumably let it be implemented? -"""]] diff --git a/doc/todo/per-branch_git-annex_branch.mdwn b/doc/todo/per-branch_git-annex_branch.mdwn deleted file mode 100644 index 059144fa63..0000000000 --- a/doc/todo/per-branch_git-annex_branch.mdwn +++ /dev/null @@ -1,3 +0,0 @@ -Currently, if I do some work on an experimental branch, creating some annexed files, then abandon the branch, information about keys created on the experimental branch will remain in the git-annex branch. This breaks git's normal notion of lightweight branching, where you can work on an experimental branch and, if you later decide to abandon that work, it'll be as if the experimental branch never existed. Maybe, it would make sense to have, for each branch mybranch, a corresponding branch git-annex-b/mybranch , which would hold the state of the git-annex branch reflecting work on mybranch? Then, if you decide to merge mybranch into master, git-annex-b/mybranch would get union-merged into the git-annex branch (or into git-annex-b/master). But if you decide to abandon/delete mybranch, git-annex-b/mybranch can be abandoned/deleted with no trace left in the main git-annex branch. - -> [[wontfix|done]] --[[Joey]] diff --git a/doc/todo/per-branch_git-annex_branch/comment_1_a6218eeed79ba2e068b28535e781bf79._comment b/doc/todo/per-branch_git-annex_branch/comment_1_a6218eeed79ba2e068b28535e781bf79._comment deleted file mode 100644 index 5c5353b2e4..0000000000 --- a/doc/todo/per-branch_git-annex_branch/comment_1_a6218eeed79ba2e068b28535e781bf79._comment +++ /dev/null @@ -1,28 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-01-30T18:10:49Z" - content=""" -That won't work, and here's why: - -You're in master, and you git checkout -b tmp - -Now you're in tmp, and you git-annex move foo --from origin - -Now you git checkout master. You delete tmp and tmp/git-annex. - -Except, foo has been moved from origin to the local repo. So now the local -repo doesn't know it contains foo, at least until git-annex fsck notices -it's there. Worse, no other repo knows where foo went, only that it was -deleted from origin. - -Notice also that, even if you keep tmp around, tmp/git-annex must never get -pushed, unless tmp get merged back into master. So even without deleting -tmp, you get into this situation where other clones don't know where the -file went. - ---- - -git-annex v0 behaved just like this, and it quickly became apparent that it -was not a good idea due to this kind of scenario. -"""]] diff --git a/doc/todo/precache_logs_for_speed_with_cat-file_--buffer.mdwn b/doc/todo/precache_logs_for_speed_with_cat-file_--buffer.mdwn deleted file mode 100644 index 76c0ab0f50..0000000000 --- a/doc/todo/precache_logs_for_speed_with_cat-file_--buffer.mdwn +++ /dev/null @@ -1,54 +0,0 @@ -Like --all was sped up 2x-16x in -[[!commit d010ab04be5a8d74fe85a2fa27a853784d1f9009]], commands -that ls-files the worktree may be sped up by using cat-file --buffer -to get location logs (and maybe other logs) more efficiently, -and precache them. - -> The precachelog branch adds location log precaching for `git annex get` -> only. But it benchmarks 4x *slower*. (Even if it were faster, it would -> have needed more work, because limits are matched before location log -> precaching, so if any limit like --in is used that uses the location -> log, it will actually be read twice.) This is a surprising result, -> and I don't understand why it's slower, but backburnered this -> optimisation for now. -> -> > Hmm, but `git annex get` does not need location log access when -> > the object is already present. So that was the wrong thing to benchmark -> > this with. `copy --to` would be a better benchmark. To speed up get, -> > it would need to check inAnnex before caching the location log. -> > -> > Benchmarking `copy --fast --to`, precaching location logs -> > did yield a 30% speedup, in a 10k repo where all objects were present. -> > But, in a 10k repo where no objects were present, it was over 14x slower, -> > again because the inAnnex check really needs to come before the -> > location log precaching. -> > -> > So, this needs some more work, but is promising. - -> > > Second try at this, results: -> > > -> > > * `get` in a full repo is not any slower. And presumably in an -> > > empty repo, `get` is faster, but I didn't try it and the transfers -> > > will dominate that anyway -> > > * `sync --content` 2x speedup! -> > > * `fsck --fast` 1.5x speedup -> > > * `whereis` 1.5x speedup -> > > * `copy --to` 2x speedup when remote has all files -> > > -> > > [[done]] - -Another thing that the same cat-file --buffer approach could be used with -is to cat the annex links. Git.LsFiles.inRepoDetails provides the Sha -of file contents, which can be fed through cat-file --buffer to get keys. -A complication is that, non-symlinks could be large files that are not -annexed but in git; don't want to cat those when looking for annex links. -That would probably need pre-filtering through a cat-file --buffer that -only gets the size of the blob, not its content. - -> This was a win! Nearly 2x faster `git-annex get` seeking. - -Some calls to lookupKey remain, and the above could -be used to remove them and make it faster. The ones in Annex.View and -Command.Unused seem most likely to be able to be converted. - -See also [[faster_key_lookup_for_limits]] diff --git a/doc/todo/reinit_should_work_without_arguments.mdwn b/doc/todo/reinit_should_work_without_arguments.mdwn deleted file mode 100644 index 0b347d7a4b..0000000000 --- a/doc/todo/reinit_should_work_without_arguments.mdwn +++ /dev/null @@ -1,68 +0,0 @@ -Sometimes it may happen that you have multiple git-annex repositories -with the same UUID. This is usually because you did something special, -like copying a repository with `cp -a` or `dd` instead of cloning it, -or you just replaced a drive with a fresh one. In my case, the latter -happened: I ran out of space on my media drive and replaced it with a -larger drive. Since it had multiple git-annex repositories (and data -*not* managed by git-annex), I simply used `rsync` to copy the drive -over, which created duplicate git-annex repositories with the same -UUIDs. - -It may still be useful to reuse those repositories as distinct -entities, and it is therefore important to assign new UUIDs to those -cloned repositories so that content tracking works properly and you do -not lose data. - -In this case, you actually do *not* want to specify an existing UUID -when you run reinit: you want git-annex to generate a new one for -you. So, in that context, I've always wondered why -[[git-annex-reinit]] absolutely requires an argument. I understand -there may be *other* use cases where you may want to `reinit` a -repository to an existing UUID, but that seems like a much *less* -common use case, and one that may bring more trouble than is worth. - -So I believe there should be an easy way to assign a fresh new UUID to -a repository. `reinit` should allow doing that when no arguments are -given: it should show the old and new UUID and maybe a warning message -indicating that tracking information may be wrong now if the old UUID -is not in use anymore. - -As shown below, I also wonder if `reinit` should recommend the user -perform a `fsck` to make sure the location logs reflect the change... - -Workaround ----------- - -It is obviously possible to assign a new UUID with the current -command, by generating one by hand. - -Git-annex doesn't have a user-visible way of generating a new UUID, so -we'll have to improvise something. It uses the -[Data.UUID](https://hackage.haskell.org/package/uuid-1.3.13/docs/Data-UUID.html) -Haskell module, in V4 mode, which is the [standard, random -way](https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_.28random.29) -of generating UUIDs. I believe that the `uuidgen` command, when ran in -`--random` mode, will produce similarly unique UUIDs that are good -enough for our purpose. So I have used this to reassign new UUIDs to -cloned copies of repositories: - - git annex reinit $(uuidgen) - -The next step is to fix the location log so that the UUID change is -reflected in the tracking information: - - git annex fsck --fast - -You may also want to set a new description for the new clone: - - git annex describe here "bare 2TB Seagate barracuda HD" - -Then, optionally, you will want to propagate that change to other -repositories: - - git annex sync - -Thanks for any feedback or comments... -- [[anarcat]] - -> [[done]], as duplicate of [[todo/reinit_current_repo_to_new_uuid]] -> --[[Joey]] diff --git a/doc/todo/reinit_should_work_without_arguments/comment_1_46ba213e20eee4a1a970c545eef82285._comment b/doc/todo/reinit_should_work_without_arguments/comment_1_46ba213e20eee4a1a970c545eef82285._comment deleted file mode 100644 index 1a332074a0..0000000000 --- a/doc/todo/reinit_should_work_without_arguments/comment_1_46ba213e20eee4a1a970c545eef82285._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2017-01-30T17:24:27Z" - content=""" -I don't know that reinit is the best place to put this. -It's a different use case, and a different operation than the current -reinit; making a single command do two opposite things is often a bad idea. - -One way to accomplish what you want is to just delete the old -annex.uuid from git config, and then `git annex init` will -allocate a new uuid. -"""]] diff --git a/doc/todo/reinit_should_work_without_arguments/comment_2_5efe582a5aacc8ae1ec0c4ff6f8bed08._comment b/doc/todo/reinit_should_work_without_arguments/comment_2_5efe582a5aacc8ae1ec0c4ff6f8bed08._comment deleted file mode 100644 index bd081e6865..0000000000 --- a/doc/todo/reinit_should_work_without_arguments/comment_2_5efe582a5aacc8ae1ec0c4ff6f8bed08._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="https://anarc.at/openid/" - nickname="anarcat" - avatar="http://cdn.libravatar.org/avatar/b36dcf65657dd36128161355d8920a99503def9461c1bb212410980fe6f07125" - subject="usability..." - date="2017-01-30T18:06:22Z" - content=""" -This is all about usability. `reinit` is where I looked for this feature, and I didn't find it, so I figured it could be expanded to cover that use case. - -If `init` can do this, why not have a `overwrite` option or something that will actually remove the uuid key from the git config first? - -I don't think we should assume users know that they can edit `.git/config` safely. I certainly didn't think of doing that and sometimes shot myself in the foot playing in there... ;) -"""]] diff --git a/doc/todo/reinit_should_work_without_arguments/comment_3_f4e790b5298b8fb1578ce86cd7fd060e._comment b/doc/todo/reinit_should_work_without_arguments/comment_3_f4e790b5298b8fb1578ce86cd7fd060e._comment deleted file mode 100644 index e3d833e66f..0000000000 --- a/doc/todo/reinit_should_work_without_arguments/comment_3_f4e790b5298b8fb1578ce86cd7fd060e._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-01-30T19:19:53Z" - content=""" -The same thing is being also dicussed at [[todo/reinit_current_repo_to_new_uuid]] -so I'm closing this todo in favor of that one. -"""]] diff --git a/doc/todo/restore_--include-dotfiles_as_a_no-op_for_backwards_compatibility.mdwn b/doc/todo/restore_--include-dotfiles_as_a_no-op_for_backwards_compatibility.mdwn deleted file mode 100644 index f7975282ac..0000000000 --- a/doc/todo/restore_--include-dotfiles_as_a_no-op_for_backwards_compatibility.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -Thanks for [[addressing|news/version_8.20200226]] "[[bugs/dotfiles_handled_differently]]". One small thing: to preserve backwards compatibility, could `--include-dotfiles` be made a no-op, rather than an invalid flag? - -> I think that my comment shows why this should not happen, so [[done]]. -> --[[Joey]] diff --git a/doc/todo/restore_--include-dotfiles_as_a_no-op_for_backwards_compatibility/comment_1_30f00e53b02f31f68a6ecc5af027585b._comment b/doc/todo/restore_--include-dotfiles_as_a_no-op_for_backwards_compatibility/comment_1_30f00e53b02f31f68a6ecc5af027585b._comment deleted file mode 100644 index 31d35c5044..0000000000 --- a/doc/todo/restore_--include-dotfiles_as_a_no-op_for_backwards_compatibility/comment_1_30f00e53b02f31f68a6ecc5af027585b._comment +++ /dev/null @@ -1,35 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-02T18:45:27Z" - content=""" -With old git-annex, --include-dotfiles made dotfiles be added to the -annex. With current git-annex, the default is to add them to git. -Two such different behaviors do not seem like preserving backwards -compatability. - -Consider if a user actually has dotfiles that are large -files, and was in the habit of running eg "git-annex add . --include-dotfiles". -Currently, they will get an error message about an unknown option, and can then -read about the behavior change, and learn they want to set -annex.dotfiles=true. - -But, if --include-dotfiles is still supported as a no-op, that user would -instead add a large file and bloat their git repo with its content as it is -not annexed. - -Now, the same kind of thing can happen if the user instead was in the -habit of running "git-annex add .bigfile" with the old git-annex -to annex the file, as that will now add it to git instead. -That's a bit concerning and so I hope users will notice the news about -the behavior change of that. But adding back --include-dotfiles -only widens the amount of users who will see an unexpected behavior change. - -I did consider fully preserving back-compat when committing that change: - - The git add behavior changes could be avoided if it turns out to be - really annoying, but then it would need to behave the old way when - annex.dotfiles=false and the new way when annex.dotfiles=true. I'd - rather not have the config option result in such divergent behavior as - `git annex add .` skipping a dotfile (old) vs adding to annex (new). -"""]] diff --git a/doc/todo/serialize_Messages_for_transferkeys.mdwn b/doc/todo/serialize_Messages_for_transferkeys.mdwn deleted file mode 100644 index 18a1aafcaa..0000000000 --- a/doc/todo/serialize_Messages_for_transferkeys.mdwn +++ /dev/null @@ -1,47 +0,0 @@ -For [[todo/more_extensive_retries_to_mask_transient_failures]], -multiple transferkeys processes need to be run. So all console IO needs to -be serialized and sent back to the main git-annex process from them, -to avoid concurrent output. - -Using --json and --json-progress and --json-error-messages is pretty close -to what would be needed. That handles progress of transfers, and overall -success/failure. - -Using --json-error-messages makes most error messages be output in json. -However, that uses an error-messages array in the json object, -which error messages are added to as an action runs, and it's only sent at -the end. So a problem for eg, a relay of stderr output from a command -(eg ssh password prompt), or for anything that displays a warning that -should be displayed before the transfer is complete. - -Since the existing json is not a perfect fit, it might be better to use a -custom protocol, implemented as a new output type. Then anything in -Messages and child modules that looks at output types will support it. -(Although perhaps some of that stuff is not used by remotes.) - -A few notes on implementing that: - -* Messages.Progress.mkOutputHandler, which uses mkStderrEmitter, - outputs to stderr directly no matter the output type currently. - It would need to be changed to support the new output type. - (And probably should for concurrent output mode too actually!) - - > It's true, this is not concurrent output safe. However, that's already - > the case, and output to stderr doesn't affect the piping of serialized - > messages on stdout. So, punted on this. - -* So does warningIO, though it's only used in a couple of remotes - and rarely. It would be good to find a way to eliminate it. - - > Eliminated except for one call in a non-relevant code path. - -* Messages.prompt. Which is used by remotes, and would need to - communicate over the pipe to the parent git-annex bidirectionally. - Eg, send a message saying the parent needs to prepare for prompt, - wait for it to reply saying it has, and then send a message when the - prompting is done. (Note that the parent would need to detect if the child - process crashed to avoid being locked waiting for the prompt.) - - > Done. - -[[done]] diff --git a/doc/todo/skip_first_pass_in_git_annex_sync.mdwn b/doc/todo/skip_first_pass_in_git_annex_sync.mdwn deleted file mode 100644 index a70b0df14d..0000000000 --- a/doc/todo/skip_first_pass_in_git_annex_sync.mdwn +++ /dev/null @@ -1,23 +0,0 @@ -Hello Joey, -So I have another idea to speed up git annex sync --content --all. As far as I understand, the first pass with --all (walking the worktree) is just to make preferred content expressions which include paths (and lackingcopies=) happy. Now, if the preferred content expression of the local repo and of none of the remotes contain paths, that first pass can be skipped. - -I did some benchmarking by replacing the following in seekSyncContent: - - Just WantAllKeys -> Just <$> genBloomFilter (seekworktree mvar (WorkTreeItems [])) - -with: - - Just WantAllKeys -> pure Nothing - -and it led to a 2x speedup (with warm cache): - - before: - real 0m41.186s - - after: - real 0m22.182s - - -This repo has 25641 keys and all of them are in the worktree too. - -> [[done]]! --[[Joey]] diff --git a/doc/todo/skip_first_pass_in_git_annex_sync/comment_1_95752fa9589819f7bd5860aa8ee1a708._comment b/doc/todo/skip_first_pass_in_git_annex_sync/comment_1_95752fa9589819f7bd5860aa8ee1a708._comment deleted file mode 100644 index 113f82c9d9..0000000000 --- a/doc/todo/skip_first_pass_in_git_annex_sync/comment_1_95752fa9589819f7bd5860aa8ee1a708._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-09-24T16:42:36Z" - content=""" -Yes. That would need a way to introspect a preferred content expression -to see if the terms used in it match that criteria. - -There are some other todos that also are blocked by the ability to do that, -so I've created a new todo for it. -[[todo/introspect_preferred_content_expressions]] -"""]] diff --git a/doc/todo/skip_first_pass_in_git_annex_sync/comment_2_9a93d321f955c3cdac5b8cbcd452f42e._comment b/doc/todo/skip_first_pass_in_git_annex_sync/comment_2_9a93d321f955c3cdac5b8cbcd452f42e._comment deleted file mode 100644 index 6802de5e07..0000000000 --- a/doc/todo/skip_first_pass_in_git_annex_sync/comment_2_9a93d321f955c3cdac5b8cbcd452f42e._comment +++ /dev/null @@ -1,15 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-09-24T19:04:32Z" - content=""" -One side effect of this optimisation is that, while sync --all used to -tell the filenames it was getting or dropping, when operating on files -in the working tree, when the optimsation is enabled it will only -display the keys. So, its behavior in 2 different repos might seem -inconsistent to a user, who doesn't know about all these gory 2 pass details. - -I think, if that became a problem, the best fix would be to only display -the keys, and never the worktree filenames, even when running the first -pass. But I'll wait and see if that needs to be done, I suppose. -"""]] diff --git a/doc/todo/skip_first_pass_in_git_annex_sync/comment_3_57303f7b1399b408deabb10c32a8db8a._comment b/doc/todo/skip_first_pass_in_git_annex_sync/comment_3_57303f7b1399b408deabb10c32a8db8a._comment deleted file mode 100644 index 165f7973e1..0000000000 --- a/doc/todo/skip_first_pass_in_git_annex_sync/comment_3_57303f7b1399b408deabb10c32a8db8a._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Lukey" - avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" - subject="comment 3" - date="2020-09-25T16:33:40Z" - content=""" -Works great, Thanks! -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all.mdwn b/doc/todo/speed_up_git_annex_sync_--content_--all.mdwn deleted file mode 100644 index 8700f3959f..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -Hello Joeyh, -Overall the performance of git-annex is good for me. However, one case where git-annex could improve is with "git annex sync --content --all", as it takes 20 minutes just to traverse all keys without uploading/downloading anything in my repo. I've looked at the code (learnig some haskell along the way) and I think it's due to getting the location logs via git cat-file. I see two ways how performance could be improved: - -1. Use "git cat-file --batch-all-objects --unordered" and traverse the keys in whatever order that outputs the location logs. -2. Cache the location logs in the sqlite database - -Other than that, git-annex has really solved all my file syncing and archival needs and is just awesome! - -> [[done]] --[[Joey]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_10_93311b7f89c28e1b5d3be562a584e25f._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_10_93311b7f89c28e1b5d3be562a584e25f._comment deleted file mode 100644 index 4f95176b18..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_10_93311b7f89c28e1b5d3be562a584e25f._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Lukey" - avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" - subject="comment 10" - date="2020-07-06T21:20:58Z" - content=""" -I'm seeing larger improvements in my repo: ~40% speedup with -J2 and even ~200% speedup without jobs. Good work! -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_10_d1d462a22ad849c62e8620a12f3ba064._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_10_d1d462a22ad849c62e8620a12f3ba064._comment deleted file mode 100644 index f92977a34f..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_10_d1d462a22ad849c62e8620a12f3ba064._comment +++ /dev/null @@ -1,19 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 10""" - date="2020-07-07T17:13:46Z" - content=""" -Wow, I implemented your --buffer trick, and `get --all` -is over 2x faster. `sync --content --all` somewhat less, but still -another decent improvement there. (cold cache timings) - -And some warm cache times are *much* faster than my cold cache benchmarks. -`get --all` is 17x faster in a 10k file repo, which makes it only 3x slower -than `get` without --all. - -I think I will still leave this open because it's still worth considering -sqlite caching or finding a way to speed up the second sync --all pass... -but would be interested to know how your use case is improved now. - -Please feel free to find optimisations anytime, I really appreciate it. -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_12_6e8863e4d039a6eb97751a9a58149151._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_12_6e8863e4d039a6eb97751a9a58149151._comment deleted file mode 100644 index 71a44fb048..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_12_6e8863e4d039a6eb97751a9a58149151._comment +++ /dev/null @@ -1,36 +0,0 @@ -[[!comment format=mdwn - username="Lukey" - avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" - subject="comment 12" - date="2020-08-04T14:10:32Z" - content=""" -I finally got around to test this and it is a lot faster. Interestingly it now -J2 is slower than without jobs. -Benchmark with warm cache: - - git annex sync --content --all - - 8.20200330: - real 23m48.059s - user 20m23.320s - sys 3m20.854s - - 8.20200720.2-g465842f3f - real 3m41.064s - user 3m23.319s - sys 0m49.159s - - - git annex sync --content --all -J2 - - 8.20200330: - real 10m47.484s - user 16m3.024s - sys 1m49.373s - - 8.20200720.2-g465842f3f - real 8m55.808s - user 12m44.766s - sys 1m25.181s - -Thanks! -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_1_dc1850b3ec3e4775a52f4cd9e311c5a9._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_1_dc1850b3ec3e4775a52f4cd9e311c5a9._comment deleted file mode 100644 index bcb41f9bd0..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_1_dc1850b3ec3e4775a52f4cd9e311c5a9._comment +++ /dev/null @@ -1,24 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-06-30T16:27:26Z" - content=""" -I am surprised by 30m, unless you have a very large number of files -or keys. Benchmarking in a repo with 100k files and keys, git-annex -sync --content took 7m, and with --all 14m. (both cold cache) - -How many files and keys does git-annex info say are in your repository? - ---all is slower because it does two passes, first over all files in the -current branch, and a second pass over all keys. (Necessary to handle -preferred content filename matching correctly.) - -git cat-file --batch-all-objects --unordered would not work, because -there would be no way to know if a particular blob was a current version of -the location log for a key, or an old version. -(For that matter, it doesn't even say what file the blob belongs to, so -it would have no idea what it's a location log for.) - -[[design/caching_database]] talks about speedups by sqlite caching. But -that's a hard road. -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_2_5ab6db1e8b5f3131cfc61626bab7b8d9._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_2_5ab6db1e8b5f3131cfc61626bab7b8d9._comment deleted file mode 100644 index 44026911ee..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_2_5ab6db1e8b5f3131cfc61626bab7b8d9._comment +++ /dev/null @@ -1,32 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-06-30T18:53:52Z" - content=""" -I wonder if the second pass could avoid looking at the location log at all -for keys handled in the first pass. Currently it only checks the bloom -filter to skip dropping those keys, but not to skip transferring the keys. If it -also skipped transferring, it would not need to read the location log a second -time for the same key. - -That would speed it up by around 2x, in fairly typical cases where there are a -lot of files, but not a lot of old versions of each file. - -Problem with that is, it's a bloom filter, there can be false positives. -Currently a false positive means it does not drop a key that it should want -to, annoying maybe but unlikely to happen and not a big problem. But -consulting the bloom filter for transfers would risk it not making as many -copies of a key as it is configured to, which risks data loss, or at least not -having all desired data available after sync. - -But, if it could use something other than a bloom filter to keep track -of the keys processed in the first pass, that would be a good optimisation. -Sqlite database maybe, have to consider the overhead of querying it. Just -keeping the keys in memory w/o a bloom filter maybe, and only use the bloom -filter if there are too many keys. - -The bloom filter used currently uses around 16 mb of ram. A typical key is -80 bytes or so. So, up to around 200,000 keys in a set is probably the same -ballpark amount of ram. (32 mb would be needed to construct the -bloom filter from the set, probably.) -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_3_c4a3d92d6467dcd1c735f27f51fa0b35._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_3_c4a3d92d6467dcd1c735f27f51fa0b35._comment deleted file mode 100644 index 9d701031ed..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_3_c4a3d92d6467dcd1c735f27f51fa0b35._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="Lukey" - avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" - subject="comment 3" - date="2020-07-01T14:32:11Z" - content=""" -All in all there are 258465 files on two branches (master (8921) and manyfiles (249544)) and about as many keys. I mostly work on the master branch (The manyfiles branch is more or less an archive), so the two passes are not an problem (yet). - -Regarding \"git cat-file --batch-all-objects --unordered\", the blob can be mapped back to the key by the objectname and \"git ls-tree -r git-annex\". - -For reference: \"time git cat-file --batch --batch-all-objects --unordered --buffer > /dev/zero\" takes 40 seconds on my repo. -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_4_690c0dcbfc112f6abd94d02c248ce68b._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_4_690c0dcbfc112f6abd94d02c248ce68b._comment deleted file mode 100644 index 8ac734863f..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_4_690c0dcbfc112f6abd94d02c248ce68b._comment +++ /dev/null @@ -1,24 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2020-07-01T16:13:23Z" - content=""" -It's 80s in my big repo. But of course it would also have to be read back -in and parsed, so seems it would take 160s or so. (It's going to be a dozen -or so gb of data anywhere the speed of git-annex sync --all is a problem.) - -Cross-referencing it with `git ls-tree -r git-annex` to get filenames -would mean git-annex would take more memory the more keys are stored in it. -Which is something I have been careful to avoid. - -An sqlite database could surely be faster, especially if it's designed so -it can be queried for things like "all keys in repo A that are not in repo -B". But a sqlite database shouldn't only benefit --all, so it also needs to -be able to do queries like "all keys that have files in HEAD, that are in -repo A and not in repo B". With that, `git annex get` etc could also get -faster. - -Anyway, it seems like --all is not really the problem for you; I guess -you would see similar runtime if you ran git-annex sync --content with the -larger of your two branches checked out than you do with --all. -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_4_d35cfe01d9093c047b0f8ed0a6265258._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_4_d35cfe01d9093c047b0f8ed0a6265258._comment deleted file mode 100644 index 715220e3dc..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_4_d35cfe01d9093c047b0f8ed0a6265258._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 4" - date="2020-07-01T17:33:49Z" - content=""" -Possibly related: [[todo/keep_git-annex_branch_checked_out?]] -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_5_e4d3a2592d1df3b8953603fccffe8f41._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_5_e4d3a2592d1df3b8953603fccffe8f41._comment deleted file mode 100644 index 61fb3f78b1..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_5_e4d3a2592d1df3b8953603fccffe8f41._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 5""" - date="2020-07-01T18:36:17Z" - content=""" -Tried implementing avoiding the redundant location log lookups in the -second pass of --all. But, a Set holding 100000 keys uses 60 mb of memory -(mostly for the tree, not the elements). If it has to convert back to a -bloom filter, add 32 mb memory more, and this is looking too memory hungry. - -The Set could be converted at a smaller size than 100000, eg 50000 keys -needs 25mb. But git-annex sync --content --all with 50000 keys takes -7 minutes w/o this optimisation, so it would probably only save a few -minutes. So, abandoning this approach for now. - -([[!commit 7e2c4ed21622a4fefab58e709528a14866e0d133]] has a nice data -type I built that starts off a Set and converts to a Bloom filter.) -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_6_4e307b9488b207fa24fa6b80de1623ae._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_6_4e307b9488b207fa24fa6b80de1623ae._comment deleted file mode 100644 index 4eb7a900bc..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_6_4e307b9488b207fa24fa6b80de1623ae._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="Lukey" - avatar="http://cdn.libravatar.org/avatar/c7c08e2efd29c692cc017c4a4ca3406b" - subject="comment 6" - date="2020-07-01T20:37:13Z" - content=""" -The memory consumption is indeed problematic. - -Digging further, I found a even better solution: -\"time (git ls-tree -r git-annex | awk '/SHA256.*.log$/{print $3\" \"$4}' | git cat-file --batch='%(objectname) %(objecttype) %(objectsize) %(rest)' --buffer > /dev/null)\" takes just 7 seconds in my repo. -Note the '%(rest)' in the batch format, awk feeds the objectname and the key (separated by space) to cat-file, which outputs the key in place of '%(rest)'. So a git-annex thread reading from cat-file has all the information (location log and key) readily available. This can be extended to also feed metadata at the same time by making the location log directly followed by metadata. -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_8_f640b622376d087732db9f2880fe7f80._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_8_f640b622376d087732db9f2880fe7f80._comment deleted file mode 100644 index 8436262f6b..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_8_f640b622376d087732db9f2880fe7f80._comment +++ /dev/null @@ -1,31 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 8""" - date="2020-07-06T14:51:24Z" - content=""" -Nice trick with the %rest! - -This looks like it could work. However, here the same command without ---buffer is almost the same speed (0:2:16 vs 0:2:11, nearly lost in the -noise). If there's any real benefit here, it seems it would be in keeping -git cat-file more active, avoiding round trips in sending queries. - -I was thinking it would be hard to implement because all location log -queries would need to be changed to use this different source of the -information. But, I think it can be finessed by having a small cache of -contents of annex branch files, and prepopulate that cache with information -about each key as it's read in from git cat-file. - -Actually, there used to be a git-annex branch cache like that, caching just -the most recently read file, but it was removed in -[[!commit 3417c55189275d038bc445fe3ef71090d518e79e]]. - -I had forgotten that was removed, and it also seems possible that bringing -that cache back would improve perf generally, because I think there are -probably situations where eg the location log is looked at repeatedly. - -.. In fact, loggedKeys calls checkDead on each key, so that's one extra -location log lookup right there! Indeed, instrumenting getRef, -I see sync --content getting the same location log 3x per key w/o --all, -so probably 4x with --all! -"""]] diff --git a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_9_c723aacf622de83477a424ebcb3a444a._comment b/doc/todo/speed_up_git_annex_sync_--content_--all/comment_9_c723aacf622de83477a424ebcb3a444a._comment deleted file mode 100644 index a0cb340203..0000000000 --- a/doc/todo/speed_up_git_annex_sync_--content_--all/comment_9_c723aacf622de83477a424ebcb3a444a._comment +++ /dev/null @@ -1,18 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 9""" - date="2020-07-06T16:23:07Z" - content=""" -Cache brought back in [[!commit e72ec8b9b23346c4b741a071e9edae4460b233c9]]. - -Benchmarking git-annex sync --content (w/o --all) with 10k files, there -was very little change, it's only 0.4% faster. With --all, it's 20% faster. - -That's kind of weird, because w/o --all it was doing 3 redundant queries, -and --all only adds one more. The first result makes me think that a) git -cat-file is doing its own caching or takes good advantage of disk cache and -b) roundtrip time communicating with it does not seem significant so ---buffer would not be much of an improvement. But the second result muddies -the waters, and I guess it would still be worth trying the --buffer -optimisation. -"""]] diff --git a/doc/todo/sqlite_database_improvements.mdwn b/doc/todo/sqlite_database_improvements.mdwn deleted file mode 100644 index e0afc0aefd..0000000000 --- a/doc/todo/sqlite_database_improvements.mdwn +++ /dev/null @@ -1,170 +0,0 @@ -The `sqlite` branch changes the databases, updating annex.version to 8. -The branch is ready to merge, but it might be deferred until Q1 2020 -to avoid user upgrade fatigue. Discussion below is about the motivation for -these changes. - ----- - -Collection of non-ideal things about git-annex's use of sqlite databases. -Would be good to improve these sometime, but it would need a migration -process. - - -* Database.Keys.SQL.isInodeKnown has some really ugly SQL LIKE queries. - Probably an index would not speed them up. They're only needed when - git-annex detects inodes are not stable, eg on fat or probably windows. - A better database - schema should be able to eliminate the need for those LIKE queries. - Eg, store the size and allowable mtimes in a separate table that is - queried when necessary. - - Fixed. - -* Several selects were not able to use indexes, so would be slow. - - Fixed by adding indexes. - -* Database.Types has some suboptimal encodings for Key and InodeCache. - They are both slow due to being implemented using String - (which may be fixable w/o changing the DB schema), - and the VARCHARs they generate are longer than necessary - since they look like eg `SKey "whatever"` and `I "whatever"` - - Fixed. - -* SFilePath is stored efficiently, and has to be a String anyway, - (until ByteStringFilePath is used) - but since it's stored as a VARCHAR, which sqlite interprets using the - current locale, there can be encoding problems. This is at least worked - around with a hack that escapes FilePaths that contain unusual - characters. It would be much better to use a BLOB. - - Also, when LANG=C is sometimes used, the hack can result in duplicates with - different representations of the same filename, like this: - - INSERT INTO associated VALUES(4,'SHA256E-s30--7d51d2454391a40e952bea478e45d64cf0d606e1e8c0652bb815a22e0e23419a,'foo.ü'); - INSERT INTO associated VALUES(5,'SHA256E-s30--7d51d2454391a40e952bea478e45d64cf0d606e1e8c0652bb815a22e0e23419a','"foo.\56515\56508"'); - - See - for an example of how this can happen. - - And it seems likely that a query by filename would fail if the filename - was in the database but with a different encoding. - - Fixed by converting to blob. - -* SKey and IKey could fail to round-trip as well, when a Key contains something - (eg, a filename extension) that is not valid in the current locale, - for similar reasons to SFilePath. Using BLOB would be better. - - See [[!commit cf260d9a159050e2a7e70394fdd8db289c805ec3]] for details - about the encoding problem for SFilePath. I reproduced a similar problem - for IKey by making a file `foo.ü` and running `git add` on it in a unicode - locale. Then with LANG=C, `git annex drop --force foo.ü` thinks - it drops the content, but in fact the work tree file is left containing - the dropped content. The database then contained: - - INSERT INTO associated VALUES(8,'SHA256E-s30--59594eea8d6f64156b3ce6530cc3a661739abf2a0b72443de8683c34b0b19344.ü','foo.ü'); - INSERT INTO associated VALUES(9,'SHA256E-s30--59594eea8d6f64156b3ce6530cc3a661739abf2a0b72443de8683c34b0b19344.��','"foo.\56515\56508"'); - - Fixed by converting to blob. - -* migration - -> Investigated this in more detail, and I can't find a way to -> solve the encoding problem other than changing the encoding -> SKey, IKey, and SFilePath in a non-backwards-compatible way. -> -> Probably the encoding problem is actually not in sqlite, but -> in persistent's use of Text internally. I did some tests with sqlite3 -> command and it did not seem to vary query results based on the locale -> when using VARCHAR values. I was able to successfully insert an -> invalid unicode `ff` byte into it, and get the same byte back out. -> -> Unfortunately, it's not possible to make persistent not use Text -> for VARCHAR. While its PersistDbSpecific lets a non-Text value be stored -> as VARCHAR, any VARCHAR value coming out of the database gets converted -> to a PersistText. -> -> So that seems to leave using a BLOB to store a ByteString for -> SKey, IKey, and SFilePath. But old git-annex won't be able to -> read the updated databases, and won't know that it can't read them! -> -> This seems to call for a flag day, throwing out the old database -> contents and regenerating them from other data: -> -> * Fsck (SKey) -> can't rebuild? Just drop and let incremental fscks re-do work -> -> (done) - -> * ContentIdentifier (IKey) -> rebuild with updateFromLog, will need to diff from empty tree to -> current git-annex branch, may be expensive to do! - - (done; will be done automatically by the first command that needs to - use the db) -> -> * Export (IKey, SFilePath) -> difficult to rebuild, what if in the middle of an interrupted -> export? -> -> updateExportTreeFromLog only updates two tables (ExportTree and -> ExportTreeCurrent), not others (Exported and ExportedDirectory). -> -> Conceptually, this is the same as the repo being lost and another -> clone being used to update the export. The clone can only learn -> export state from the log. It's supposed to recover from such -> situations, the next time an export is run, so should be ok. -> -> An interrupted export won't resume where it left off, since the -> information about a partial export doesn't reach the git-annex branch. -> So it will re-send some files on resume. Documenting this should -> be good enough. -> -> Testing this an exporting to a directory with both exporttree=yes and -> importtree=yes, it refused to let an interrupted export proceed after -> upgrade, with "unsafe to overwrite file". An import resolved the -> problem. Guess the problem is that the content idenfifier did not -> get recorded in the git-annex branch and the db value was lost on upgrade. -> -> If the export is not interrupted before upgrade, later exports are able -> to overwrite the exported files as they should. -> -> Hmm, testing again, with a script, and interrupting the export, I am -> not able to reproduce the problem either. The cids of exported files -> get written to the journal, so make their way into the git-annex branch -> and so the cid db gets repopulated. Perhaps my earlier manual test -> was mistaken somehow. -> -> * Keys (IKey, SFilePath, SInodeCache) -> Use scanUnlockedFiles to repopulate the Associated table. -> -> But that does not repopulate the Content table. Doing so needs -> to iterate over the unlocked files, filter out any that are modified, -> and record the InodeCaches of the unmodified ones. Seems that it would -> have to use git's index to know which files are modified. -> -> There is a race; a file could be modified after getting the list of -> modified files. To completely avoid that race is tricky. To mostly -> eliminate it, just generate the InodeCache, then check -> if the file is still unmodified, then check if the InodeCache is still -> valid. That leaves some much less likely races where files are being -> repeatedly swapped and the InodeCache generations see one file while -> the git ls-files --modified see the other one. -> -> To fully avoid the race, use git ls-files --cached --debug, -> and parse the debug output into a InodeCache! This way the info -> from git's index is simply copied over into the git-annex database. -> One little problem: The --debug format is not specified and may change. -> However, it has never actually changed since it was introduced in 2010 -> (git v1.8.3.1), except for a fix for an unsigned int overflow bug that -> was fixed in April 2019. -> -> (done) -> -> Alternatively, can keep the old database code and use it to read the old -> databases during the migration. But then bad data that got in due to the -> encoding problems will persist. - -[[done]] --[[Joey]] diff --git a/doc/todo/stalldetection_breaks_move.mdwn b/doc/todo/stalldetection_breaks_move.mdwn deleted file mode 100644 index d3cd728cdd..0000000000 --- a/doc/todo/stalldetection_breaks_move.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -Setting annex.stalldetection can break sync when it does a move, -which complains it cannot find enough copies to drop. -(Seems that git-annex move does work ok.) - -The problem is that the transferrer process updates the location log, but -the parent process doesn't see the update in time. So, the location log -update needs to move to the parent process. --[[Joey]] - -> [[fixed|done]] --[[Joey]] diff --git a/doc/todo/stalldetection_progress_bar_lacks_ETA.mdwn b/doc/todo/stalldetection_progress_bar_lacks_ETA.mdwn deleted file mode 100644 index a40d7fe1cf..0000000000 --- a/doc/todo/stalldetection_progress_bar_lacks_ETA.mdwn +++ /dev/null @@ -1,7 +0,0 @@ -Unsure why, but when annex.stalldetection is set, the progress info it -communicates results in a progress display w/o ETA sometimes. -In particular, it seems to happen downloading from ssh, when the key does -not have a size. Normally, the size is learned during download and used in -the progress bar, but somehow this does not happen. --[[Joey]] - -> [[fixed|done]] --[[Joey]] diff --git a/doc/todo/stop_using_createDirectoryIfMissing_True.mdwn b/doc/todo/stop_using_createDirectoryIfMissing_True.mdwn deleted file mode 100644 index f82555a47d..0000000000 --- a/doc/todo/stop_using_createDirectoryIfMissing_True.mdwn +++ /dev/null @@ -1,32 +0,0 @@ -`createDirectoryIfMissing True` creates any missing parent directories. -So if the git repository directory goes away suddenly, git-annex can create -a new empty directory in its place and start putting files in there. - -* I've seen this happen when a removable drive that's a remote is unmounted - and git-annex is running and stores a file on the remote. Since - git-annex's cwd is not on the remote and it may not have any files open - when it's unmounted, the unmount proceeds. When the mount point is - /media/username/drive, the automounter may delete the mount point, and - git-annex then recreates it and may write files to it, which is the wrong - location. This can also cause problems with later mounting of the drive; - some automounters fail if the mount point pre-exists. - -* A move of a directory to another path can also lead to git-annex writing - to the old path and re-creating the directory. While it mostly uses - relative paths to cwd, and so is fine, there may be cases where it needs - to use an absolute path. - -* Odd behavior has been reported if the repository is deleted while - git-annex is running it it. Including a git-annex seeming to hang or - spin. - - The sooner git-annex gives up in - such a situation, the less likely it is do get into an unusual state. - -What's needed is an action that creates directories only up to a given -point, which can be either .git/annex or the top of the worktree depending -on what's being done. --[[Joey]] - -[[!tag confirmed]] - -> [[fixed|done]], all relevant calls have been converted. --[[Joey]] diff --git a/doc/todo/stop_using_createDirectoryIfMissing_True/comment_1_09ea803b8f725071add33d03b8e3afa3._comment b/doc/todo/stop_using_createDirectoryIfMissing_True/comment_1_09ea803b8f725071add33d03b8e3afa3._comment deleted file mode 100644 index 2887656928..0000000000 --- a/doc/todo/stop_using_createDirectoryIfMissing_True/comment_1_09ea803b8f725071add33d03b8e3afa3._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 1""" - date="2020-03-05T17:54:38Z" - content=""" -Implemented createDirectoryUnder, now just have to change every -createDirectoryIfMissing True to it.. There are only 75 of them so not -super bad? - -Also, of course, some library might use it, but I doubt any that do -create directories inside the git repo, more likely they would be creating -tmp dirs or stuff like that. -"""]] diff --git a/doc/todo/stop_using_createDirectoryIfMissing_True/comment_2_1633fecf3fd5096c1bfcf77dfe471189._comment b/doc/todo/stop_using_createDirectoryIfMissing_True/comment_2_1633fecf3fd5096c1bfcf77dfe471189._comment deleted file mode 100644 index c6dc45fe00..0000000000 --- a/doc/todo/stop_using_createDirectoryIfMissing_True/comment_2_1633fecf3fd5096c1bfcf77dfe471189._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 2""" - date="2020-03-05T19:18:31Z" - content=""" -Most of the easy ones have been converted now. - -There's one in Annex.ReplaceFile that's hard, and is probably the only -important one left unconverted. -"""]] diff --git a/doc/todo/stop_using_createDirectoryIfMissing_True/comment_3_806bd4f8dd6ca4fcc898d0c02eb4c5ee._comment b/doc/todo/stop_using_createDirectoryIfMissing_True/comment_3_806bd4f8dd6ca4fcc898d0c02eb4c5ee._comment deleted file mode 100644 index f488b29c5c..0000000000 --- a/doc/todo/stop_using_createDirectoryIfMissing_True/comment_3_806bd4f8dd6ca4fcc898d0c02eb4c5ee._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2020-03-06T17:52:21Z" - content=""" -Fixed.. - -Also filed this bug report proposing the new function be added to -directory: -"""]] diff --git a/doc/todo/support_concurrency_for_export.mdwn b/doc/todo/support_concurrency_for_export.mdwn deleted file mode 100644 index fd817dfd30..0000000000 --- a/doc/todo/support_concurrency_for_export.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -Making git-annex export support -J should be doable and could speed up -exports a lot with some remotes. --[[Joey]] - -[[done]] diff --git a/doc/todo/support_disabling_git-annex___47___guard.mdwn b/doc/todo/support_disabling_git-annex___47___guard.mdwn deleted file mode 100644 index 876f1300fe..0000000000 --- a/doc/todo/support_disabling_git-annex___47___guard.mdwn +++ /dev/null @@ -1,9 +0,0 @@ -It would be nice if I could configure a git repository to be guarded against git-annex. Or even a global git config option to default to not use git-annex but only in specific repositories meant to be used with git-annex. - -We've had a source code repository added with git-annex branches (repeatedly) and the user had problems with unwanted git-annex hooks installed in his checkout. I suspect that this was a user error, i.e. running git-annex in the wrong directory, but I would like to put guards into place by disabling git-annex by default and only allowing it for certain specific repos. - -> Seems that .noannex was sufficient. I am doubtful about the utility -> of a global git config option to prevent git-annex init from being used. -> Seems like, if you had that config set, and were using git-annex, -> you would have to override it, and once you're in the habit of overriding -> it, you may as well not have the config at all... So [[done]] --[[Joey]] diff --git a/doc/todo/support_disabling_git-annex___47___guard/comment_1_0040fe8eb9df9f0a7cb7c4e490f41e6d._comment b/doc/todo/support_disabling_git-annex___47___guard/comment_1_0040fe8eb9df9f0a7cb7c4e490f41e6d._comment deleted file mode 100644 index af775ed88a..0000000000 --- a/doc/todo/support_disabling_git-annex___47___guard/comment_1_0040fe8eb9df9f0a7cb7c4e490f41e6d._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="kyle" - avatar="http://cdn.libravatar.org/avatar/7d6e85cde1422ad60607c87fa87c63f3" - subject="comment 1" - date="2020-05-27T15:19:13Z" - content=""" -If you put a .noannex file in the top-level directory of a repo, `git -annex init` will refuse to initialize the repo. -"""]] diff --git a/doc/todo/support_disabling_git-annex___47___guard/comment_2_0c3df9b8894937f2f3b4f90bee9cc096._comment b/doc/todo/support_disabling_git-annex___47___guard/comment_2_0c3df9b8894937f2f3b4f90bee9cc096._comment deleted file mode 100644 index 115c8a354d..0000000000 --- a/doc/todo/support_disabling_git-annex___47___guard/comment_2_0c3df9b8894937f2f3b4f90bee9cc096._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="mike@2d6d71f56ce2a992244350475251df87c26fe351" - nickname="mike" - avatar="http://cdn.libravatar.org/avatar/183fa439752e2f0c6f39ede658d81050" - subject=".noannex" - date="2020-05-27T15:34:54Z" - content=""" -Thanks, I didn't know about .noannex! -"""]] diff --git a/doc/todo/symlinks_for_not-present_unlocked_files.mdwn b/doc/todo/symlinks_for_not-present_unlocked_files.mdwn deleted file mode 100644 index b47567ba8f..0000000000 --- a/doc/todo/symlinks_for_not-present_unlocked_files.mdwn +++ /dev/null @@ -1,43 +0,0 @@ -Unlocked files with content not present appear as regular files containing -an annex pointer. This is not ideal as the user has no way to know if the -content is present other than file size. Broken symlinks are a better -representation. - -# smudge approach - -It would be possible to make `git annex smudge --update` write symlinks -for the missing content files. But then git will think they're modified, -and committing will lock them. - -Is it possible at all for the index to be in the state where a file that's -been converted to a symlink is not treated as modified? The index contains -an object type value, which will need to be set to symlink. But if -the tree for the checked out branch has the object type of regular file, -git diff-index would consider there to be a staged change. So probably -it's not possible. Still, it would be worth playing with the file format to -see if perhaps it can be tricked. - -Or such staged (non-)changes could be filtered out by the pre-commit hook. -This is probably possible, but git status, diff, etc would show the -non-present files as modified still. - -# adjusted branch approach - -Use an adjusted branch, where files with missing content are locked. - -Often the users who are affected by this are already using git-annex adjust ---unlock, so using a different adjustment would be a minor change for them. - -Such a branch is very conceptually similar to a --hide-missing branch. -And so certianly seems doable. Like a --hide-missing branch, there's an -open question of how to update the branch after get/drop of content. - -Needing to manually re-run git-annex adjust, as is done with --hide-missing, -means the user is still going to get surprised when files they open -turn out to have missing content. So for this to really be useful, -the branch needs to automatically get updated. - ---[[Joey]] - -> I've implemented this approach, as `git annex adjust --unlock-present` -> [[done]] --[[Joey]] diff --git a/doc/todo/symlinks_for_not-present_unlocked_files/comment_1_c433f7d3908248563aa5999c5946a625._comment b/doc/todo/symlinks_for_not-present_unlocked_files/comment_1_c433f7d3908248563aa5999c5946a625._comment deleted file mode 100644 index 60f5e2fe4a..0000000000 --- a/doc/todo/symlinks_for_not-present_unlocked_files/comment_1_c433f7d3908248563aa5999c5946a625._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 1" - date="2019-09-16T21:12:21Z" - content=""" -\"But if the tree for the checked out branch has the object type of regular file, git diff-index would consider there to be a staged change.\" -- maybe, setting the assume-unchanged bit or the skip-worktree bit in the index would cause git to ignore the change in the worktree? -"""]] diff --git a/doc/todo/symlinks_for_not-present_unlocked_files/comment_2_d31a23773bb9ec47ad7bde0b8b528520._comment b/doc/todo/symlinks_for_not-present_unlocked_files/comment_2_d31a23773bb9ec47ad7bde0b8b528520._comment deleted file mode 100644 index 316fa796d3..0000000000 --- a/doc/todo/symlinks_for_not-present_unlocked_files/comment_2_d31a23773bb9ec47ad7bde0b8b528520._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 2" - date="2019-09-16T23:44:48Z" - content=""" -You could also [git-replace](https://git-scm.com/docs/git-replace) the pointer file with the symlink. -"""]] diff --git a/doc/todo/symlinks_for_not-present_unlocked_files/comment_3_d185f9af10d2ba2ee059b78e38b2eee0._comment b/doc/todo/symlinks_for_not-present_unlocked_files/comment_3_d185f9af10d2ba2ee059b78e38b2eee0._comment deleted file mode 100644 index 76b13eaf71..0000000000 --- a/doc/todo/symlinks_for_not-present_unlocked_files/comment_3_d185f9af10d2ba2ee059b78e38b2eee0._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 3""" - date="2019-09-17T15:50:39Z" - content=""" -Goood thought about assume-unchanged or skip-worktree, that seems likely to -work. - -Using git-replace scares me, seems very likely to have unintended -consequences. -"""]] diff --git a/doc/todo/symlinks_for_not-present_unlocked_files/comment_4_185110a8c01ccc094e5bbb11f31b3a09._comment b/doc/todo/symlinks_for_not-present_unlocked_files/comment_4_185110a8c01ccc094e5bbb11f31b3a09._comment deleted file mode 100644 index cc41b4ebf4..0000000000 --- a/doc/todo/symlinks_for_not-present_unlocked_files/comment_4_185110a8c01ccc094e5bbb11f31b3a09._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 4""" - date="2019-09-17T15:51:21Z" - content=""" -Note that direct mode used to behave this way, with broken symlinks to -missing objects. But only on filesystems supporting symlinks, and direct -mode was mostly for filesystems w/o that (as well as used by the assistant -to let users edit arbitrary file contents w/o bothering with unlocking). - -So, I don't consider v7 not having this yet to be a big regression from v5. -"""]] diff --git a/doc/todo/symlinks_for_not-present_unlocked_files/comment_5_1a11280fbed406677e30cc5e149c55a4._comment b/doc/todo/symlinks_for_not-present_unlocked_files/comment_5_1a11280fbed406677e30cc5e149c55a4._comment deleted file mode 100644 index 0f91982202..0000000000 --- a/doc/todo/symlinks_for_not-present_unlocked_files/comment_5_1a11280fbed406677e30cc5e149c55a4._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 5" - date="2019-09-18T21:30:46Z" - content=""" -One more option is to add the paths for missing unlocked files to .git/info/exclude . -"""]] diff --git a/doc/todo/symlinks_for_not-present_unlocked_files/comment_6_c76cc5b2b61177cdd79b0ca4024acca1._comment b/doc/todo/symlinks_for_not-present_unlocked_files/comment_6_c76cc5b2b61177cdd79b0ca4024acca1._comment deleted file mode 100644 index d231463a27..0000000000 --- a/doc/todo/symlinks_for_not-present_unlocked_files/comment_6_c76cc5b2b61177cdd79b0ca4024acca1._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="Ilya_Shlyakhter" - avatar="http://cdn.libravatar.org/avatar/1647044369aa7747829c38b9dcc84df0" - subject="comment 6" - date="2019-10-08T20:23:06Z" - content=""" -One more approach might be to configure `core.fsmonitor` to a custom one that reports missing unlocked files as unchanged even though they've been changed to symlinks. -"""]] diff --git a/doc/todo/symlinks_for_not-present_unlocked_files/comment_7_37e6192b8f4410058f7b3f34b0787ec6._comment b/doc/todo/symlinks_for_not-present_unlocked_files/comment_7_37e6192b8f4410058f7b3f34b0787ec6._comment deleted file mode 100644 index 2f9af4fd0f..0000000000 --- a/doc/todo/symlinks_for_not-present_unlocked_files/comment_7_37e6192b8f4410058f7b3f34b0787ec6._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="joey" - subject="""comment 7""" - date="2020-09-01T19:32:13Z" - content=""" -annex.thin could be combined with this. Imagine a `git annex adjust` mode -that represents present files as annex pointer files to git, and that makes -them hardlinks to the object files, but with the write bit removed, so -they're still effectively locked. And non-present files are represented as -dangling symlinks. -"""]] diff --git a/doc/todo/sync_--content_with_borg_does_not_get_content.mdwn b/doc/todo/sync_--content_with_borg_does_not_get_content.mdwn deleted file mode 100644 index 9636925b6e..0000000000 --- a/doc/todo/sync_--content_with_borg_does_not_get_content.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -Subject says it all really, sync does not try to get content -from remotes that are thirdPartyPopulated yet. - -> [[done]] --[[Joey]] diff --git a/doc/todo/sync_fast_import.mdwn b/doc/todo/sync_fast_import.mdwn deleted file mode 100644 index 5ac51a03dd..0000000000 --- a/doc/todo/sync_fast_import.mdwn +++ /dev/null @@ -1,52 +0,0 @@ -git-annex import --no-content from a directory special remote is -implemented, but git-annex sync, when run without --content, does not -operate on import/export special remotes. This is inconsistent, and it -would be useful if it did. - - -describes a problem with doing that, involving merge conflicts. That -should not actually happen with the directory special remote, because -it generates the same key with importKey as git-annex import would. -But other special remotes later using this interface might generate a key -using a different hash than usual. - -The suggestion there is that there could be a separate config that controls -whether sync does a fast import or an import with content. Then when sync -is run without --content, it can do a fast import. And when run with ---content, it can do a fast import, followed by getting the content. - -Or maybe that should just be what it always does, when a remote supports -importKey? (If so, git-annex import should do the same.) Yeah, this seems -better than a config. Look at it like this: The special remote makes pseudo -"commits" when changes are made to it. And maybe it choses to use a -different kind of key than the local repository would use. Same could -happen when pulling from someone else's repo, if they've configured -git-annex to use a different backend. - -> There could be future transition problems here. If a remote does not -> support importkey, and imports are done from it, and then in a new -> version, it does support importkey, there would be the same risk of -> conflicts. -> -> Could be solved by the remote's code indicating if -> importKey is safe to use by default. If a remote started off implementing -> only imports w/o importKey, and then added importKey, and importKey -> generates different keys than the keys generated by hashing downloaded -> content, then the remote could say, don't use importKey by default. -> (Or more likely, only the directory remote will be able to support -> importKey by default..) -> -> Problem: When annex.largefiles matches file content, -> cannot use importKey. So then should sync --content not use importKey -> then, risking generating a different tree? Or should it fail, even -> though importing with content is possible? -> -> > Well, different annex.largefiles settings in different clones -> > can already risk generating a different tree on import. So, -> > the former option seems preferable. - ---- - -See also, [[todo/import_--no-content_largefiles_conflict]] - -> [[done]] --[[Joey]] diff --git a/doc/todo/sync_my_local_git-annex_from_a_dump_remote.mdwn b/doc/todo/sync_my_local_git-annex_from_a_dump_remote.mdwn deleted file mode 100644 index 2cd967c848..0000000000 --- a/doc/todo/sync_my_local_git-annex_from_a_dump_remote.mdwn +++ /dev/null @@ -1,13 +0,0 @@ -As discussed on debconf, I have the following use case: - -* I have a dump remote, a folder on my webserver where files are uploaded through the web app. I don't have git on the webserver, just a plain folder. -* I have git-annex repo on a development server. The development server polls the webserver (ssh/ftp) once in an hour and synchronizes the state of the local git-annex repo with the state found on the webserver and commits that. -* This is not meant to be backup facility. I just want to be able to have a state on my development machine that is very likely to the state on the webserver. - -> Closing because it's really not clear what the request is, I don't -> remember the in person conversation. -> -> Possibly [[import_tree_from_rsync_special_remote]] would cover the use -> case. -> -> [[done]] --[[Joey]] diff --git a/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_1_81d63854f89f00855cda5ace0fc8262a._comment b/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_1_81d63854f89f00855cda5ace0fc8262a._comment deleted file mode 100644 index d9abb3a3cb..0000000000 --- a/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_1_81d63854f89f00855cda5ace0fc8262a._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="http://joeyh.name/" - ip="2001:4978:f:21a::2" - subject="comment 1" - date="2013-08-13T21:44:17Z" - content=""" -We had a conversation about this IRL. At the time, I thought I understood what you wanted. Reading the above, I am not so sure. - -What I thought you wanted was something like `git annex mirror --from remote`, which would, for each object known to git-annex that the location log said was present on the remote, make sure that the local repo had the object too, and for each object that the location log said was not present on the remote, drop it from the local repo (if numcopies etc allowed). - -`git annex mirror --to remote` could also be used as the complement of the above. - -If that's not the sort of thing you meant, let me know. -"""]] diff --git a/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_2_66822b72b1450e79e8edd0c6c21d5aa6._comment b/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_2_66822b72b1450e79e8edd0c6c21d5aa6._comment deleted file mode 100644 index 3d459371f5..0000000000 --- a/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_2_66822b72b1450e79e8edd0c6c21d5aa6._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="http://thkoch2001.myopenid.com/" - nickname="thkoch" - subject="pseudocode" - date="2013-08-14T04:58:22Z" - content=""" -lets say my local annex is in direct mode, then the following might already do what I want: - -cd $LOCAL_ANNEX -rsync --recursive --delete $REMOTE . -git annex add && git commit - -It would however be nice if I could do the same with an annex in indirect mode or even a bare annex. -"""]] diff --git a/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_3_b9f73375e2c732e798495f8ee6970c7c._comment b/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_3_b9f73375e2c732e798495f8ee6970c7c._comment deleted file mode 100644 index df4be033bb..0000000000 --- a/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_3_b9f73375e2c732e798495f8ee6970c7c._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="http://joeyh.name/" - ip="4.154.0.63" - subject="comment 3" - date="2013-08-24T16:35:33Z" - content=""" -Seems to me that this could easily be dealt with by installing git-annex on the webserver, making the directory there a git repository, and using either a cron job or `git annex watch` to commit files as they were changed there. - -Then you can make a direct mode, indirect mode, or even a bare clone on your local machine and use git-annex to get the files. - -Maybe you have good reasons for not wanting to go that route. And rsync on a direct mode repository should work, provided to tell it to not delete `.git`. :P I don't see any way to make rsync work in an indirect mode repository. As for trying to make git-annex handle this import over rsync itself in a way that would work in an indirect mode repository, let alone a bare repository -- I don't see a good way to do it and it seems quite special case and likely to get quite complicated to implement. - -In the meantime, I did implement `git annex mirror`, which I think is a much more interesting and generally useful tool to have. And could even be used in my recommended solution above. -"""]] diff --git a/doc/todo/test_testremote.mdwn b/doc/todo/test_testremote.mdwn deleted file mode 100644 index 6c1ca7a03e..0000000000 --- a/doc/todo/test_testremote.mdwn +++ /dev/null @@ -1,14 +0,0 @@ -git-annex test could run git-annex testremote with a single directory -remote. This would both find bugs in testremote, and probably generally -bugs involving special remote operations, of course especially with the -directory remote. - -To avoid confusing output, running testremote itself is not ideal, -instead, it would be great to have Command.TestRemote export a TestTree, -that could then be added to the test suite's TestTree. - -(See [[bugs/testremote_failures_starting_with_aeca7c220]]) - ---[[Joey]] - -> [[done]] --[[Joey]] diff --git a/doc/todo/transferkeys_optimisation.mdwn b/doc/todo/transferkeys_optimisation.mdwn deleted file mode 100644 index 4e595de11e..0000000000 --- a/doc/todo/transferkeys_optimisation.mdwn +++ /dev/null @@ -1,14 +0,0 @@ -Some of the things git-annex transferkeys does are suboptimal, especially -when -J has many of them running. - -In particular, it writes location logs when downloading (but not -uploading), and so it flushes the journal etc. - -> [[done]] --[[Joey]] - -It may also do some queries of data from git that could be avoided with -some refactoring of what code runs in it, which could avoid it needing to -start up git helper processes like catkey. --[[Joey]] - -> Still starts some git catkey processes, but I guess it's for good reason. -> --[[Joey]] diff --git a/doc/todo/use_same_vector_clock_for_content_identifier_updates_in_import.mdwn b/doc/todo/use_same_vector_clock_for_content_identifier_updates_in_import.mdwn deleted file mode 100644 index 0200e40a0d..0000000000 --- a/doc/todo/use_same_vector_clock_for_content_identifier_updates_in_import.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -When importing a tree from a special remote, often content identifiers need -to be written for all files that it finds. Currently, the timestamp varies -a little bit while writing these. If the same timestamp were used for all -the logs, they would delta-compress better. - -This probably extends well past content identifiers, but they would be a -probably relatively easy place to do it since they're all written in the -same place. - -As a bonus, since borg uses the same content identifiers for all keys -(""), implementing this would make the content of the logs all identical, -and so avoid any overhead entirely. Well.. perhaps that's my actual -motivation. ;-) --[[Joey]] - -> [[done]], though reuseVectorClockWhile could be used in other places -> perhaps. --[[Joey]] diff --git a/doc/todo/utilising_the_mklink_command_on_windows_to_utilise_symlinks_and_therefore_indirect_mode_on_windows.mdwn b/doc/todo/utilising_the_mklink_command_on_windows_to_utilise_symlinks_and_therefore_indirect_mode_on_windows.mdwn deleted file mode 100644 index b8e0151321..0000000000 --- a/doc/todo/utilising_the_mklink_command_on_windows_to_utilise_symlinks_and_therefore_indirect_mode_on_windows.mdwn +++ /dev/null @@ -1,12 +0,0 @@ -It seems like it is possible to achieve this now on later versions of Windows (7 and above). - -The mklink tool creates a symlink that works on windows. - -There would be some work required so that a linux symlink and a windows symlink are considered the same, a user has recommend 'git update-index --assume-unchanged' for this. - -There is a good write up of someone using this approach on vanilla git here: http://stackoverflow.com/questions/5917249/git-symlinks-in-windows - -> I think that with adjusted unlocked branches being used by default on -> windows, instead of the old direct mode with its limitations, I don't -> want to delve into the arcanea of windows's 57 different kinds of -> symlink equivilants. So, [[done]] --[[Joey]] diff --git a/doc/todo/utilising_the_mklink_command_on_windows_to_utilise_symlinks_and_therefore_indirect_mode_on_windows/comment_1_f974b0fc908277fcc35ee6c8073b65c8._comment b/doc/todo/utilising_the_mklink_command_on_windows_to_utilise_symlinks_and_therefore_indirect_mode_on_windows/comment_1_f974b0fc908277fcc35ee6c8073b65c8._comment deleted file mode 100644 index 7e44497e8f..0000000000 --- a/doc/todo/utilising_the_mklink_command_on_windows_to_utilise_symlinks_and_therefore_indirect_mode_on_windows/comment_1_f974b0fc908277fcc35ee6c8073b65c8._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="Karl" - subject="Symbolic links are not supported by Windows applications" - date="2015-02-06T22:27:27Z" - content=""" -Hi! - -I did research the practical usage of (real) NTFS-links (junktions + links) when I was programming http://tagstore.org since it would have been a clean solution for that purpose as well. - -However, I have to say that NTFS-links only work in theory. First of all, you have to be Administrator to create links by mklink. And secondly, most end-user applications can't cope with links. - -The latter one is the real fun-killing issue: many applications (even from Microsoft) are renaming a freshly opened file to a temporary file name. When the user is appending data, the temporary file gets updated. Only when the user is (manually) saving, a *new* file with the original file name is created. This results in *replacing* the original file with the new copy. Unfortunately, links are not handled properly. This way, many applications end up replacing the original linked file with an ordinary file when saving. -"""]] diff --git a/doc/todo/utilising_the_mklink_command_on_windows_to_utilise_symlinks_and_therefore_indirect_mode_on_windows/comment_2_575d96ffa2a8f2cb1ba3d0e5c4ba4e6f._comment b/doc/todo/utilising_the_mklink_command_on_windows_to_utilise_symlinks_and_therefore_indirect_mode_on_windows/comment_2_575d96ffa2a8f2cb1ba3d0e5c4ba4e6f._comment deleted file mode 100644 index ad37ac8b05..0000000000 --- a/doc/todo/utilising_the_mklink_command_on_windows_to_utilise_symlinks_and_therefore_indirect_mode_on_windows/comment_2_575d96ffa2a8f2cb1ba3d0e5c4ba4e6f._comment +++ /dev/null @@ -1,11 +0,0 @@ -[[!comment format=mdwn - username="jason.dixon.email@aa0e536a2ec2877d6f666108dbbc6e39bbe87ac0" - nickname="jason.dixon.email" - avatar="http://cdn.libravatar.org/avatar/fbe9050fc83bbd536d307d87ea14d4bc" - subject="Does this not work counter to locking / unlocking files?" - date="2017-03-03T13:40:06Z" - content=""" -Perhaps I understand this wrong. But if you're in a position to be viewing symlinks, then the file is in effect, locked right? So in the situation that a windows user is trying to edit a file that is currently symlinked, they shouldn't be allowed to save at all. They should have unlocked the file first, thus replacing the link with the actual file. A simple fix would be to have the symlink read-only just as the file tucked away in .git/annex/objects is. - -As I said, I'm probably looking at this in entirely the wrong way. -"""]] diff --git a/doc/todo/wishlist__58___more_info_in_the_standard_commit_message_of___96__sync__96__.mdwn b/doc/todo/wishlist__58___more_info_in_the_standard_commit_message_of___96__sync__96__.mdwn deleted file mode 100644 index 836cc53b99..0000000000 --- a/doc/todo/wishlist__58___more_info_in_the_standard_commit_message_of___96__sync__96__.mdwn +++ /dev/null @@ -1,10 +0,0 @@ -Could you include the REPO and UUID information in the "automatic sync" commit message? - -This would make troubleshooting easier. (not there was much trouble with git-annex!) - -> This was [[done]] at some point, the message includes the hostname -> and repo location. -> -> Also git-annex sync --message can provide a custom message. There is also -> even an annex.commitmessage that can be used to provide custom messages -> to git-annex branch commits. --[[Joey]] diff --git a/doc/todo/wishlist__58___more_info_in_the_standard_commit_message_of___96__sync__96__/comment_1_b9c241cf94a35aa6a45f4d44334694b0._comment b/doc/todo/wishlist__58___more_info_in_the_standard_commit_message_of___96__sync__96__/comment_1_b9c241cf94a35aa6a45f4d44334694b0._comment deleted file mode 100644 index bede77efcd..0000000000 --- a/doc/todo/wishlist__58___more_info_in_the_standard_commit_message_of___96__sync__96__/comment_1_b9c241cf94a35aa6a45f4d44334694b0._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="http://olivier.mehani.name/" - nickname="olivier-mehani" - subject="comment 1" - date="2014-01-15T02:26:31Z" - content=""" -See also [[wishlist: more info in commit messages in general]] for a similar but more generic wish. -"""]] diff --git a/doc/todo/wishlist__58___option_to_print_more_info_with___39__unused__39__.mdwn b/doc/todo/wishlist__58___option_to_print_more_info_with___39__unused__39__.mdwn deleted file mode 100644 index 1abadcd3d6..0000000000 --- a/doc/todo/wishlist__58___option_to_print_more_info_with___39__unused__39__.mdwn +++ /dev/null @@ -1,40 +0,0 @@ -It would be nice if the 'unused' command could optionally display info about the actual files behind its cryptic keys. - -I created a (very rough) bash script that simply splices in some info from git log -S'KEY' --numstat into the unused list, like so: - - arand@mas:~/annex(master)$ bash ~/utv/scripts/annex-vunused - unused . (checking for unused data...) (checking master...) (checking synced/master...) (checking origin/HEAD...) (checking seagate/master...) - Some annexed data is no longer used by any files: - NUMBER KEY - 1 SHA256E-s1073741824--49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14.img - 8f479a4 Sat Feb 23 16:14:12 2013 +0100 remove bigfile - 0 1 dummy_bigfile.img - 2988d18 Sat Feb 23 16:13:48 2013 +0100 dummy file - 1 0 dummy_bigfile.img - (To see where data was previously used, try: git log --stat -S'KEY')To remove unwanted data: git-annex dropunused NUMBER - ok -The script: - - #!/bin/bash - - pipe="$(mktemp -u)" - mkfifo "$pipe" - - git annex unused >"$pipe" || exit 1 & - - while read -r line - do - key="$(echo "$line" | sed 's/^[^-]*-\([^-]*\)-.*/\1/')" - echo -n "$line" - test -n "$key" && \ - echo && \ - git log --format="%h %cd %s" --numstat -S"$key" | \ - sed '/^$/d;/git-annex automatic sync/,/^ /d;s/^/\t\t/' - - done < "$pipe" - rm "$pipe" - -It would be nice if something like this was available as an option, since it's good way to get a quick overview of what the content is, and if it's safe to drop it. - -> There is an earlier todo, [[show_me_where_unused_file_was__44___i_can_wait]] -> about the same thing, so closing this as dup [[done]] --[[Joey]]