git-annex/doc/bugs/withOtherTmp_file_escapes.mdwn

This was reported with concurrent processes, I think probably during a `git
annex add` but not sure:

	git-annex: .git/annex/othertmp/inge59014-3: getFileStatus: does not exist (No such file or directory)
	failed
 
	git-annex: .git/annex/othertmp/ingest-assemble_den59014-8: removeLink: does not exist (No such file or directory)
	failed

From the "ingest" in the name, these files are from
`Annex.Ingest.lockDown'`, which uses `withOtherTmp`, hard links the file
into the temp directory, and then returns a LockedDown that includes the
temp filepath. So, it's escaped the locking that `withOtherTmp` does, and
another process can clean up the temp files at the wrong point in time.
This will need some significant code reworking to fix.

> [[fixed|done]] --[[Joey]]

This is a fairly new problem because the code to have other processes
cleanup stale othertmp files was only added a couple months back.

I audited other uses of withOtherTmp, and the only other problem I found is
similar, in Annex.Branch.stageJournal. Fixed that one.
--[[Joey]]
bug report 2019-05-07 15:40:45 +00:00			This was reported with concurrent processes, I think probably during a `git
			annex add` but not sure:

			`git-annex: .git/annex/othertmp/inge59014-3: getFileStatus: does not exist (No such file or directory)`
			`failed`

			`git-annex: .git/annex/othertmp/ingest-assemble_den59014-8: removeLink: does not exist (No such file or directory)`
			`failed`

			`From the "ingest" in the name, these files are from`
			`Annex.Ingest.lockDown'`, which uses `withOtherTmp`, hard links the file
			`into the temp directory, and then returns a LockedDown that includes the`
			temp filepath. So, it's escaped the locking that `withOtherTmp` does, and
			`another process can clean up the temp files at the wrong point in time.`
			`This will need some significant code reworking to fix.`

avoid ingest lockdown file escaping the withOtherTmp call Fixes bug that caused git-annex to fail to add a file when another git-annex process cleaned up the temp directory it was using. Solution is just to push withOtherTmp out to a higher level, so that the whole ingest process can be completed inside it. But in the assistant, that was not practical to do, since withOtherTmp runs in the Annex monad and the assistant does not. Worked around by introducing a separate temp directory that only the assistant uses for lockdown. Since only one assistant can run at a time, it's easy to clean up that directory of old cruft at startup. 2019-05-07 17:04:39 +00:00			`> [[fixed\|done]] --[[Joey]]`

bug report 2019-05-07 15:40:45 +00:00			`This is a fairly new problem because the code to have other processes`
			`cleanup stale othertmp files was only added a couple months back.`

			`I audited other uses of withOtherTmp, and the only other problem I found is`
avoid stageJournal escaping withOtherTmp This is only done for correctness sake; I don't see any way that it would have caused a problem here. The jlog file escaped withOtherTmp so another process could swoop in and delete it, but the file is only used as a buffer for a list of filenames, and its handle gets rewound and they're read back out, which will still work even if it's already been deleted. The only reason I didn't just pre-delete the file and keep the handle open is I'm not sure that works on all OS's (eg Windows). If there was a problem that this fixed it might involve an OS that doesn't support deleting an open file or something like that. 2019-05-07 15:57:12 +00:00			`similar, in Annex.Branch.stageJournal. Fixed that one.`
bug report 2019-05-07 15:40:45 +00:00			`--[[Joey]]`