feat: GPU shared texture offscreen rendering (#42953)

* feat: GPU shared texture offscreen rendering

* docs: clarify texture infos that passed by the paint event.

* feat: make gpu osr spec test optional

* fix: osr image compare

* fix: remove duplicate test

* fix: update patch file

* fix: code review

* feat: expose more metadata

* feat: use better switch design

* feat: add warning when user forget to release the texture.

* fix: typo

* chore: update patch

* fix: update patch

* fix: update patch description

* fix: update docs

* fix: apply suggestions from code review

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* fix: apply suggested fixes

---------

Co-authored-by: Charles Kerr <charles@charleskerr.com>

docs/api/structures/offscreen-shared-texture.md

@@ -0,0 +1,24 @@
+# OffscreenSharedTexture Object
+
+* `textureInfo` Object - The shared texture info.
+  * `widgetType` string - The widget type of the texture. Can be `popup` or `frame`.
+  * `pixelFormat` string - The pixel format of the texture. Can be `rgba` or `bgra`.
+  * `codedSize` [Size](size.md) - The full dimensions of the video frame.
+  * `visibleRect` [Rectangle](rectangle.md) - A subsection of [0, 0, codedSize.width(), codedSize.height()]. In OSR case, it is expected to have the full section area.
+  * `contentRect` [Rectangle](rectangle.md) - The region of the video frame that capturer would like to populate. In OSR case, it is the same with `dirtyRect` that needs to be painted.
+  * `timestamp` number - The time in microseconds since the capture start.
+  * `metadata` Object - Extra metadata. See comments in src\media\base\video_frame_metadata.h for accurate details.
+    * `captureUpdateRect` [Rectangle](rectangle.md) (optional) - Updated area of frame, can be considered as the `dirty` area.
+    * `regionCaptureRect` [Rectangle](rectangle.md) (optional) - May reflect the frame's contents origin if region capture is used internally.
+    * `sourceSize` [Rectangle](rectangle.md) (optional) - Full size of the source frame.
+    * `frameCount` number (optional) - The increasing count of captured frame. May contain gaps if frames are dropped between two consecutively received frames.
+  * `sharedTextureHandle` Buffer _Windows_ _macOS_ - The handle to the shared texture.
+  * `planes` Object[] _Linux_ - Each plane's info of the shared texture.
+    * `stride` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
+    * `offset` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
+    * `size` number - Size in bytes of the plane. This is necessary to map the buffers.
+    * `fd` number - File descriptor for the underlying memory object (usually dmabuf).
+  * `modifier` string _Linux_ - The modifier is retrieved from GBM library and passed to EGL driver.
+* `release` Function - Release the resources. The `texture` cannot be directly passed to another process, users need to maintain texture lifecycles in
+  main process, but it is safe to pass the `textureInfo` to another process. Only a limited number of textures can exist at the same time, so it's important
+  that you call `texture.release()` as soon as you're done with the texture.

docs/api/structures/web-preferences.md

@@ -79,10 +79,14 @@
   [browserWindow](../browser-window.md) has disabled `backgroundThrottling` then
   frames will be drawn and swapped for the whole window and other
   [webContents](../web-contents.md) displayed by it. Defaults to `true`.
-* `offscreen` boolean (optional) - Whether to enable offscreen rendering for the browser
+* `offscreen` Object | boolean (optional) - Whether to enable offscreen rendering for the browser
   window. Defaults to `false`. See the
   [offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
   more details.
+  * `useSharedTexture` boolean (optional) _Experimental_ - Whether to use GPU shared texture for accelerated
+     paint event. Defaults to `false`. See the
+    [offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
+    more details.
 * `contextIsolation` boolean (optional) - Whether to run Electron APIs and
   the specified `preload` script in a separate JavaScript context. Defaults
   to `true`. The context that the `preload` script runs in will only have

docs/api/web-contents.md

@@ -869,12 +869,12 @@ app.whenReady().then(() => {
 
 Returns:
 
-* `event` Event
+* `details` Event\<\>
+  * `texture` [OffscreenSharedTexture](structures/offscreen-shared-texture.md) (optional) _Experimental_ - The GPU shared texture of the frame, when `webPreferences.offscreen.useSharedTexture` is `true`.
 * `dirtyRect` [Rectangle](structures/rectangle.md)
 * `image` [NativeImage](native-image.md) - The image data of the whole frame.
 
-Emitted when a new frame is generated. Only the dirty area is passed in the
-buffer.
+Emitted when a new frame is generated. Only the dirty area is passed in the buffer.
 
 ```js
 const { BrowserWindow } = require('electron')
@@ -886,6 +886,33 @@ win.webContents.on('paint', (event, dirty, image) => {
 win.loadURL('https://github.com')
 ```
 
+When using shared texture (set `webPreferences.offscreen.useSharedTexture` to `true`) feature, you can pass the texture handle to external rendering pipeline without the overhead of
+copying data between CPU and GPU memory, with Chromium's hardware acceleration support. This feature is helpful for high-performance rendering scenarios.
+
+Only a limited number of textures can exist at the same time, so it's important that you call `texture.release()` as soon as you're done with the texture.
+By managing the texture lifecycle by yourself, you can safely pass the `texture.textureInfo` to other processes through IPC.
+
+```js
+const { BrowserWindow } = require('electron')
+
+const win = new BrowserWindow({ webPreferences: { offscreen: { useSharedTexture: true } } })
+win.webContents.on('paint', async (e, dirty, image) => {
+  if (e.texture) {
+    // By managing lifecycle yourself, you can handle the event in async handler or pass the `e.texture.textureInfo`
+    // to other processes (not `e.texture`, the `e.texture.release` function is not passable through IPC).
+    await new Promise(resolve => setTimeout(resolve, 50))
+
+    // You can send the native texture handle to native code for importing into your rendering pipeline.
+    // For example: https://github.com/electron/electron/tree/main/spec/fixtures/native-addon/osr-gpu
+    // importTextureHandle(dirty, e.texture.textureInfo)
+
+    // You must call `e.texture.release()` as soon as possible, before the underlying frame pool is drained.
+    e.texture.release()
+  }
+})
+win.loadURL('https://github.com')
+```
+
 #### Event: 'devtools-reload-page'
 
 Emitted when the devtools window instructs the webContents to reload

docs/tutorial/offscreen-rendering.md

@@ -3,7 +3,8 @@
 ## Overview
 
 Offscreen rendering lets you obtain the content of a `BrowserWindow` in a
-bitmap, so it can be rendered anywhere, for example, on texture in a 3D scene.
+bitmap or a shared GPU texture, so it can be rendered anywhere, for example,
+on texture in a 3D scene.
 The offscreen rendering in Electron uses a similar approach to that of the
 [Chromium Embedded Framework](https://bitbucket.org/chromiumembedded/cef)
 project.
@@ -17,22 +18,39 @@ the dirty area is passed to the `paint` event to be more efficient.
 losses with no benefits.
 * When nothing is happening on a webpage, no frames are generated.
 * An offscreen window is always created as a
-[Frameless Window](../tutorial/window-customization.md)..
+[Frameless Window](../tutorial/window-customization.md).
 
 ### Rendering Modes
 
 #### GPU accelerated
 
-GPU accelerated rendering means that the GPU is used for composition. Because of
-that, the frame has to be copied from the GPU which requires more resources,
-thus this mode is slower than the Software output device. The benefit of this
-mode is that WebGL and 3D CSS animations are supported.
+GPU accelerated rendering means that the GPU is used for composition. The benefit
+of this mode is that WebGL and 3D CSS animations are supported. There are two
+different approaches depending on the `webPreferences.offscreen.useSharedTexture`
+setting.
+
+1. Use GPU shared texture
+
+    Used when `webPreferences.offscreen.useSharedTexture` is set to `true`.
+
+    This is an advanced feature requiring a native node module to work with your own code.
+    The frames are directly copied in GPU textures, thus this mode is very fast because
+    there's no CPU-GPU memory copies overhead, and you can directly import the shared
+    texture to your own rendering program.
+
+2. Use CPU shared memory bitmap
+
+    Used when `webPreferences.offscreen.useSharedTexture` is set to `false` (default behavior).
+
+    The texture is accessible using the `NativeImage` API at the cost of performance.
+    The frame has to be copied from the GPU to the CPU bitmap which requires more system
+    resources, thus this mode is slower than the Software output device mode. But it supports
+    GPU related functionalities.
 
 #### Software output device
 
 This mode uses a software output device for rendering in the CPU, so the frame
-generation is much faster. As a result, this mode is preferred over the GPU
-accelerated one.
+generation is faster than shared memory bitmap GPU accelerated mode.
 
 To enable this mode, GPU acceleration has to be disabled by calling the
 [`app.disableHardwareAcceleration()`][disablehardwareacceleration] API.