fdToHandle does not set the usual system locale encoding,
so when the Handle is used for any String IO, it needs to be done
manually for correctness.
I don't know if this fixes any bugs. It might eg, fix a bug with
multicast receive of a file.
Sponsored-by: Leon Schuermann
Works around this bug https://github.com/haskell/file-io/issues/45
The fix is in Utility.FileIO.CloseOnExec because all use of file-io is
already wrapped through that module. Although perhaps that ought to be
refactored at this point.
I'd hope that file-io will eventually fix this bug, and also provide
CloseOnExec variants of its functions. That would allow depending on the
fixed version, and removing this ugly code.
Note that, functions like readFile that don't care about the encoding
due to reading/writing a ByteString were kept optimally fast by not
setting the encoding. This avoids an IORef read and write per open.
Sponsored-by: Graham Spencer
Exported by Common, so they will be available everywhere.
These are the same as readFile, writeFile, appendFile.
But have two benefits:
* They take OsPath, so using them avoids converting back and forth
unncessarily.
* They use the close-on-exec flag so can't leak FDs to child processes.
Unlike the standard Haskell versions which unfortunately have that
wart currently. (I do hope the standard versions get fixed
eventually.)
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
Utility.FileIO.CloseOnExec is largely copied from
System.File.OsPath.Internal with the simple modification of
setting the flag.
Unfortunately, openTempFile does not set the flag when
opening the file, but afterwards, leaving it vulnerable to a race.
A lot of code, including posix and windows specific code,
would need to be copied from file-io in order to fix that.
Still, I consider this implementation a placeholder, it doesn't truely
fix all instances of the problem.
I hope that this will be addressed in file-io itself, see
https://github.com/haskell/file-io/issues/44
Utility.FileIO.CloseOnExec could form the basis of a
file-io-closeonexec library, depending on how things go with that
issue.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
This is groundwork for setting the close-on-exec flag when opening
files, which will be done in Utility.FileIO or a similar module using the same
function names. The bytestring library does not set that flag, even though it
could, and IMHO should.
Note that there are many calls to the Prelude's readFile/writeFile/appendFile
still in git-annex, and this does not address those.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
Made all uses of openFd and dup set the close-on-exec flag, with a few
exceptions when starting a git-annex daemon.
Made openFdWithMode be used everywhere, rather than openFd.
Adding a new parameter to it ensures I checked everything.
And will help to make sure this gets considered in the future when
opening fds.
In lockPidFile, the only thing that keeps the pid file locked, once
daemonize re-runs the command in a new session, is that the fd is
inherited.
In Utility.LogFile.redir, the new fd it dups to does not have the
close-on-exec flag set, because this is used to set up the stdout and
stderr fds, which need to be inherited by child processes.
Same in Assistant.startDaemon where the browser gets started with the
original stdout and stderr.
This does nothing about uses of openFile and similar!
Sponsored-By: mycroft
unix-2.8.0 adds cloexec to OpenFileFlags, rather than needing to call
setFdOption after opening.
This also might avoid a race, if another thread started a process at
just the wrong time, before the flag got set, it could inherit the FD.
Unfortunately, the arm64-ancient build needs support for older versions
of unix than that, so had to ifdef. That build is still needed to
support using git-annex in termux on phones like mine, I have confirmed
today.
Sponsored-by: Luke T. Shumaker
add: Fix crash adding filenames that are exactly 21 bytes long and begin
with a utf-8 character.
Also longer filenames that start with "....." would cause the same crash.
I also audited for other calls to truncateFilePath that could truncate it
to "". Most use pathmax so are not a problem. Backend.Utilities.genKeyName
could possibly truncate it like that, but appends the md5 so would not be a
problem either.
Sponsored-by: Kevin Mueller
RemoteDaemon.Transport.Tor was refactored into this, and most of the
code is reused between them.
getSocketFile does not yet deal with repositories on crippled
filesystems that don't support sockets. Annex.Ssh detects that and
allows the user to set an environment variable, and something similar
could be done here.
And it does not deal with a situation where there is no path to the
socket file that is not too long. In that situation it would crash out
I suppose. Probably though, remotedaemon is ran from the top of the
repo, and in that case the path is just ".git/annex/p2p/<md5>" so nice
and short.
This seems to mostly work. But I don't yet have a working git-annex-p2p-
command to test it with.
And with my not quite working git-annex-p2p-foo test script, running
remotedaemon results in an ever-growing number of zombie processes
that it's not waiting on.
Avoid a problem with temp file names ending in whitespace on filesystems
like VFAT that don't support such filenames.
See a6eb7d7339 previously for the same but
with "."
At some point relatedTemplate is more bother than it's worth and it would
be simpler to just use "temp" as the basename of all temp files. We seem to
be approaching that point, since my interest in absurd ancient filesystem
limitations is limited.
Sponsored-by: unqueued on Patreon
Added annex.fastcopy and remote.name.annex-fastcopy config setting. When
set, this allows the copy_file_range syscall to be used, which can eg allow
for server-side copies on NFS. (For fastest copying, also disable
annex.verify or remote.name.annex-verify.)
This is a simple implementation, that does not handle resuming as well as
it possibly could.
It can be used with both local git remotes (including on NFS), and
directory special remotes. Other types of remotes could in theory also
support it, so I've left the config documented as a general thing.
This dates back to commit 625303226d,
where a cross-device moveFile on Windows was made to fall back to copying
to the destination, but forgot to delete the source file.
Should fix the following test suite failure on Windows:
import: FAIL (2.52s)
.\Test\Framework.hs:383:
C:\Users\RUNNER~1\AppData\Local\Temp\importtest.0\import1\f exists unexpectedly
Use -p '/import/' to rerun this test only.
Which was seen here, running the test suite in the github action environment.
https://github.com/psychoinformatics-de/git-annex-wheel/issues/5
This allows rejecting output filenames that are outside the repository,
and also handles converting eg "-foo" to "./-foo" to prevent a command
that it's passed to interpreting the output filename as a dashed option.
However, filepath-bytestring is still in Setup-Depends.
That's because Utility.OsPath uses it when not built with OsPath.
It would be maybe possible to make Utility.OsPath fall back to using
filepath, and eliminate that dependency too, but it would mean either
wrapping all of System.FilePath's functions, or using `type OsPath = FilePath`
Annex.Import uses ifdefs to avoid converting back to FilePath when not
on windows. On windows it's a bit slower due to that conversion.
Utility.Path.Windows.convertToWindowsNativeNamespace got a bit
slower too, but not really worth optimising I think.
Note that importing Utility.FileSystemEncoding at the same time as
System.Posix.ByteString will result in conflicting definitions for
RawFilePath. filepath-bytestring avoids that by importing RawFilePath
from System.Posix.ByteString, but that's not possible in
Utility.FileSystemEncoding, since Setup-Depends does not include unix.
This turned out not to affect any code in git-annex though.
Sponsored-by: Leon Schuermann
These are too widly used for other things to make sense to export OsPath
versions of them. And OsString also provides them and gets imported
qualified when needed.
If the filepath starts with something that is not valid utf-8, it would
have returned "". And if the filepath was all non-valid utf-8, it would
also return "".
