git-annex/doc/todo/windows_support.mdwn

The git-annex Windows port is beta, but rapidly becoming polished and
usable!


## status

* Local pairing seems to fail, after acking on Linux box, it stalls.
  (Also, of course, the Windows box is unlikely to have a ssh server,
  so only pairing with a !Windows box will work.)

* gcrypt is not ported to windows (and as a shell script, may need
  to be rewritten)

* Deleting a git repository from inside the webapp fails "RemoveDirectory
  permision denied ... file is being used by another process"

* Tor remotes are not supported yet. Should not be very hard to get it working.

## potential encoding problems

[[bugs/Unicode_file_names_ignored_on_Windows]] is fixed, but some potential
problems remain, since the FileSystemEncoding that git-annex relies on
seems unreliable/broken on Windows.

* When git-annex displays a filename that it's acting on, there
  can be mojibake on Windows. For example, "háčky.txt" displays
  the accented characters as instead the pairs of bytes making
  up the utf-8. Tried doing various things to the stdout handle
  to avoid this, but only ended up with encoding crashes, or worse
  mojibake than this.

  > This may be use to windows actually using utf-16, but git-annex uses
  > utf-8 for filename encoding when on windows.

* If interoperating with a git-annex repository from a unix system, it's
  possible for a key to contain some invalid utf-8, which means its filename
  cannot even be represented on Windows, so who knows what will happen in that
  case -- probably it will fail in some way when adding the object file
  to the Windows repo. 

* If data from the git repo does not have a unicode encoding, it will be
  mangled in various places on Windows, which can lead to undefined behavior.

## minor problems

* webapp lets user choose to encrypt repo, and generate gpg key,
  before checking that gcrypt is not installed
* Ssh connection caching does not work on Windows, so `git annex get`
  has to connect twice to the remote system over ssh per file, which
  is much slower than on systems supporting connection caching.
* glacier-cli is not easily available (probably)

## stuff needing testing

* test that adding a repo on a removable drive works; that git is synced to
  it and files can be transferred to it and back
* Does stopping in progress transfers work in the webapp?

## do we need this port anymore?

See <http://blog.dustinkirkland.com/2016/03/ubuntu-on-windows.html>

If windows has transparent support for running linux executables, and those
executables can access files in "." which are on the windows system, then
you could just use this to run linux git-annex on windows. No port needed.

That would be great!

Seems like this would need Windows 10.

> The latest builds of Windows 10 (build 15063) can run git-annex in the
> Windows Subsystem for Linux. After following the instructions at
> <https://msdn.microsoft.com/en-us/commandline/wsl/about>, run:
> `sudo apt-get install git-annex`
> 
> git-annex in WSL passes its full test suite, and it avoids all
> the problems discussed in sections above.
> 
> git-annex can access Windows files in eg `/mnt/c`, so a git-annex
> repository can be stored there. However, if the git-annex repository uses
> indirect mode, the symlinks used by git-annex won't be usable by Windows
> programs. Use either direct mode, or v6 mode to avoid the symlink
> problem.
> 
> Also, see this important caveat:
> <https://blogs.msdn.microsoft.com/commandline/2016/11/17/do-not-change-linux-files-using-windows-apps-and-tools/>
> 
> March 2017: WSL is currently rather annoying to enable. *If* it became easy enough
> to enable, note that "bash -c git-annex" works from a windows command
> prompt, and would probably work in a .bat file as well, so git-annex from
> the WSL could be transparently used on the windows side.
> 
> The webapp does not currently work. It doesn't know how to open a web
> browser from the linux side. There are also what look like some emulation
> problems around the daemonization code. `git annex assistant
> --foreground` does run, but while it notices when new files are added, it
> does not notice when existing files get modified. Probably an inotify
> emulation bug. --[[Joey]]
> 
> > Update May 2018: With the latest MS Edge VirtualBox image, 
> > enabling WSL was much easier than before, just a matter of checking a
> > single box and a reboot. No need to enable developer mode first.
> > (Don't know if this will apply to commercial Windows 10 or only this
> > VM image.)
> > 
> > Then bash was available in the menu, but it said no distro installed,
> > and gave an url to a "Store" where Ubuntu, Debian, etc 
> > could be installed.
> > 
> > git-annex test had a failure this time, something to do with disk
> > IO error and sqlite.
> > 
> > The assistant can run as a daemon now. It still doesn't notice some changes. 
> > Eg with "rm foo; echo new > foo", it got an inotify event for the removal,
> > but missed the new creation.
> >
> > Running `git annex assistant` when the assistant is running
> > complains of a stale pid file, with the wrong pid number, but it
> > doesn't start a second one.
> > 
> > webapp still can't open a web browser, but there's a way to do
> > it from within WSL that it should be possible for it to use:
> > `powershell.exe Start http://google.com`
> > ("powershell.exe" is in PATH)
> > 
> > --[[Joey]]
> > 
> > > The sqlite problem may be fixed in git-annex 7. I fixed some similar,
> > > less frequent crashes on Linux. Need to test on windows. --[[Joey]]

> > > > But here's a bug about sqlite in WSL:
> > > > [[bugs/WSL_adjusted_braches__58___smudge_fails_with_sqlite_thread_crashed_-_locking_protocol]] --[[Joey]]

## Update Oct 2021: Steps for using git-annex on NTFS with WSL1 (an experimental setup for those adventurous enough)

The following steps are tested on Windows 10 21h1 with Ubuntu 18.04/20.04 and are specifically designed to work around these two bugs:

* [[bugs/WSL_adjusted_braches__58___smudge_fails_with_sqlite_thread_crashed_-_locking_protocol]]
* [[bugs/WSL1__58___git-annex-add_fails_in_DrvFs_filesystem]]

Do the following:

1. Enable Developer mode in Windows settings so that symlinks can be created without elevated privileges.

2. Mount the NTFS drive with metadata option. [`/etc/wsl.conf`](https://docs.microsoft.com/en-us/windows/wsl/wsl-config) can be used or a line such as `C: /mnt/c drvfs metadata` can be added in `/etc/fstab`.

3. Create an empty directory where your repo will be. Then enable case sensitivity `setfattr -n system.wsl_case_sensitive -v 1 <path>`. This attribute will be inherited by new subdirectories. If setfattr(1) errs out with permission denied, you can also effect the same change in CMD.EXE / Windows Powershell as admin with `fsutil file setCaseSensitiveInfo <path> enable`.[^1] You can check that the setting is enabled with `getfattr -n system.wsl_case_sensitive <path>` under WSL1.

    If you do not have files which may be differ only case, you do not need to set this on the entire repository, only for `.git/annex`. If in addition you have repository [[tuning]] set to use only lowercase hash directories, you do not need to set this at all.

4. Create the repo however you like (see steps below for cloning a repo with ssh). Immediately after `git annex init`, do `git config annex.crippledfilesystem true`. If you set `crippledfilesystem` before init, then git annex will try to enter an adjusted branch and trigger the first bug. If you do not set `crippledfilesystem` after init, you will trigger the second bug when doing `git annex add`.

[^1]: This works because Administrators usually have Full Control over most files. What Windows actually looks for is "Write attributes", "Create files", "Create folders" and "Delete subfolders and files" permissions on the directory required for changing case-sensitivity. As a regular user (or without UAC) you might not have those permissions by default for instance on external drives, so adjust accordingly. For more info about about the `system.wsl_case_sensitive` attribute see this blog post: [[https://devblogs.microsoft.com/commandline/improved-per-directory-case-sensitivity-support-in-wsl/]]

### Cloning a repo with ssh

When cloning a repo with ssh, `git annex init` will fail to enable ssh remotes if `crippledfilesystem` is not set, but you also cannot set it before init. Follow these steps to avoid unrelated history in the `git-annex` branch.

    git clone <sshpath> annex
    cd annex
    git branch git-annex origin/git-annex
    git remote remove origin
    git annex init
    git config annex.crippledfilesystem true
    git remote add origin <sshpath>
    git annex sync

### Using symlinks and locked files

* You can now use symlinks and locked files but please remember that locked files can still be overwritten. So make sure to unlock them before you edit them.
* After you `git annex get` files, the symlinks for those files will still be broken. Recreate the symlinks to fix them. You can make a script or delete them and `git checkout`.
* It can be difficult to use symlinks on Windows because programs will see the link target rather than the link, which makes it impossible to do things like navigating between files in the same directory or using opened file history. You can unlock the files or access them through another filesystem layer such as SMB.