# Importing Shared Texture

This document describes the design of the `sharedTexture` API that imports external shared textures into Electron as a `VideoFrame`. Written based on Electron 37 and Chromium 137. Note that Chromium's `SharedImage` interfaces may iterate quickly, especially regarding the lifecycle management of `SharedImage`-related resources, including `GpuMemoryBuffer` and `VideoFrame`.

## Design

The main goal of the current implementation is to import external shared texture descriptions while reusing as much as possible of the existing Chromium infrastructure. I chose `SharedImage` as the underlying holder of the external texture because it provides `SharedImageInterface::CreateSharedImage`, which accepts a `GpuMemoryBufferHandle` containing native shared handle types: NT HANDLE for Windows, `IOSurfaceRef` for macOS, and `NativePixmapHandle` with file descriptors for each plane for Linux.

However, I encountered multiple challenges during implementation. The current implementation was designed by overcoming these obstacles:

1. Shared handle is local to process

   For Windows, a shared D3D11 texture can be created by `GetSharedHandle` (deprecated) or `CreateSharedHandle`. The deprecated method generates a non-NT HANDLE that can be globally accessed, while the newer one generates an NT HANDLE that is local to the current process. To share it with other processes, you need to call `DuplicateHandle` to create this handle in the remote process.

   For macOS, `IOSurface` can also be global by setting `kIOSurfaceIsGlobal`, which is also a deprecated option. If you want to share an `IOSurface` with other processes, you need to create a `mach_port` from it and pass the `mach_port` through a previously created IPC (which also uses `mach_port` as transport), making it significantly more complex than Windows.

   Given these obstacles, you must ensure that when calling `sharedTexture.importSharedTexture`, the handle is already available to the current process. You might be able to use global `IOSurface`, but a non-NT HANDLE is not an option as described in problem 2 below. In fact, Chromium's IPC internally handles all the concerns about this (duplicate handles for remote processes, passing `mach_port` through `mach_port`), which is why the OSR `paint` event can use the handle directly - because IPC has transparently handled these issues.

2. Shared handle ownership management

   Chromium takes ownership of the `GpuMemoryBuffer` and its native representation. Once the resource is destroyed, the handle will be closed.

   For Windows, calling `CloseHandle` on a non-NT HANDLE is an invalid operation and will cause Chromium to crash. Therefore, you cannot use a non-NT HANDLE when importing. In the future, we may provide a helper for this. To work with this design, when calling `importSharedTexture`, an NT HANDLE will be duplicated.

   For macOS, `IOSurface` is a reference-counted resource. When calling `importSharedTexture`, instead of taking ownership, we can simply let Chromium retain this resource and increment the reference count.

3. Transfer the shared texture between processes

   I initially considered using WebGPU Dawn Native API to import the external texture as a `WGPUSharedTextureMemory`, but encountered more problems. For example, it was unable to export, difficult to manage the lifetime of a frame, and working with WebGPU was non-ideal.

   `SharedImage` has advantages when it comes to sharing across processes because it holds a reference to a `Mailbox`, which points to the corresponding `SharedImageBacking` in the GPU process. Therefore, I use `SharedImageInterface->ImportSharedImage` and `ClientSharedImage->Export` to serialize sufficient information to retrieve the `SharedImage` reference in another process, and it can also reuse the mojo serializer to serialize as a string.

4. Interprocess resource management

   The final obstacle is managing the lifetime of a frame. Currently, I use OSR to get an exported shared texture generated by Chromium itself. As the documentation states, the texture needs to be manually released. By importing this texture into a `SharedTextureImported` in Electron, we must ensure the imported one is released before the source texture is released.

   What's more challenging is that the `paint` event occurs in the main process, while we have to render in renderer processes, so we must use `startTransferSharedTexture` to pass the underlying `SharedImage` to another process.

   Most GPU calls are asynchronous, sending to the command buffer of the GPU process through the `GpuChannel` of each client process. When we have two `SharedImage` instances referencing the same `Mailbox` in two different processes, we don't know when the GPU has finished using the resources. Typically, this is guaranteed by `SyncToken`. For example, we can simply schedule the destruction of a `SharedImage` with an empty `SyncToken` in the main process (where the texture was first imported), but when we use the same `SharedImage` in the renderer process and use it in WebGPU (or WebGL) pipelines, the destruction token will be generated by WebGPU to prevent destruction before the GPU uses it. The destruction won't occur until WebGPU rendering finishes.

   Ideally, if we are in Chromium, we can use mojo to create a `PendingRemote` callback and update the main process destruction `SyncToken` to prevent the main process from releasing the frame before the GPU starts working on it. In the future, we may implement this in Electron code to wrap the mojo functionality. Eventually, I found a way to use `gpu::ContextSupport` and register a callback when a specific `SyncToken` is released (signaled). When you call `release()` on the imported shared texture object, if you've used `VideoFrame` and imported it into a WebGPU pipeline, it will wait for WebGPU to finish rendering, then run a callback to notify you to release dependent resources, such as the original imported object in the main process, the source texture, etc.

## Example

I use built-in OSR `offscreen: { useSharedTexture: true }` to obtain a exported shared texture, but you can use whatever you want with same requirements, you can even import at renderer process, in case you choose to load your native code in renderer process and correctly put the handle under that process. However, you need to make sure the shared handle is visible to that process.

To read the example, visit the test spec of this feature, start from [here](https://github.com/electron/electron/blob/main/spec/api-shared-texture-spec.ts). There's a detailed step by step comment in it, you'll also need to read [preload](https://github.com/electron/electron/blob/main/spec/fixtures/api/shared-texture/preload.js) and [renderer](https://github.com/electron/electron/blob/main/spec/fixtures/api/shared-texture/renderer.js) code to navigate throught all steps.
