feat: GPU shared texture offscreen rendering (#42953)

* feat: GPU shared texture offscreen rendering

* docs: clarify texture infos that passed by the paint event.

* feat: make gpu osr spec test optional

* fix: osr image compare

* fix: remove duplicate test

* fix: update patch file

* fix: code review

* feat: expose more metadata

* feat: use better switch design

* feat: add warning when user forget to release the texture.

* fix: typo

* chore: update patch

* fix: update patch

* fix: update patch description

* fix: update docs

* fix: apply suggestions from code review

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* fix: apply suggested fixes

---------

Co-authored-by: Charles Kerr <charles@charleskerr.com>
This commit is contained in:
reito 2024-08-23 08:23:13 +08:00 committed by GitHub
parent b481966f02
commit 1aeca6fd0e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
34 changed files with 1009 additions and 102 deletions

View file

@ -0,0 +1,24 @@
# OffscreenSharedTexture Object
* `textureInfo` Object - The shared texture info.
* `widgetType` string - The widget type of the texture. Can be `popup` or `frame`.
* `pixelFormat` string - The pixel format of the texture. Can be `rgba` or `bgra`.
* `codedSize` [Size](size.md) - The full dimensions of the video frame.
* `visibleRect` [Rectangle](rectangle.md) - A subsection of [0, 0, codedSize.width(), codedSize.height()]. In OSR case, it is expected to have the full section area.
* `contentRect` [Rectangle](rectangle.md) - The region of the video frame that capturer would like to populate. In OSR case, it is the same with `dirtyRect` that needs to be painted.
* `timestamp` number - The time in microseconds since the capture start.
* `metadata` Object - Extra metadata. See comments in src\media\base\video_frame_metadata.h for accurate details.
* `captureUpdateRect` [Rectangle](rectangle.md) (optional) - Updated area of frame, can be considered as the `dirty` area.
* `regionCaptureRect` [Rectangle](rectangle.md) (optional) - May reflect the frame's contents origin if region capture is used internally.
* `sourceSize` [Rectangle](rectangle.md) (optional) - Full size of the source frame.
* `frameCount` number (optional) - The increasing count of captured frame. May contain gaps if frames are dropped between two consecutively received frames.
* `sharedTextureHandle` Buffer _Windows_ _macOS_ - The handle to the shared texture.
* `planes` Object[] _Linux_ - Each plane's info of the shared texture.
* `stride` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
* `offset` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
* `size` number - Size in bytes of the plane. This is necessary to map the buffers.
* `fd` number - File descriptor for the underlying memory object (usually dmabuf).
* `modifier` string _Linux_ - The modifier is retrieved from GBM library and passed to EGL driver.
* `release` Function - Release the resources. The `texture` cannot be directly passed to another process, users need to maintain texture lifecycles in
main process, but it is safe to pass the `textureInfo` to another process. Only a limited number of textures can exist at the same time, so it's important
that you call `texture.release()` as soon as you're done with the texture.

View file

@ -79,10 +79,14 @@
[browserWindow](../browser-window.md) has disabled `backgroundThrottling` then
frames will be drawn and swapped for the whole window and other
[webContents](../web-contents.md) displayed by it. Defaults to `true`.
* `offscreen` boolean (optional) - Whether to enable offscreen rendering for the browser
* `offscreen` Object | boolean (optional) - Whether to enable offscreen rendering for the browser
window. Defaults to `false`. See the
[offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
more details.
* `useSharedTexture` boolean (optional) _Experimental_ - Whether to use GPU shared texture for accelerated
paint event. Defaults to `false`. See the
[offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
more details.
* `contextIsolation` boolean (optional) - Whether to run Electron APIs and
the specified `preload` script in a separate JavaScript context. Defaults
to `true`. The context that the `preload` script runs in will only have

View file

@ -869,12 +869,12 @@ app.whenReady().then(() => {
Returns:
* `event` Event
* `details` Event\<\>
* `texture` [OffscreenSharedTexture](structures/offscreen-shared-texture.md) (optional) _Experimental_ - The GPU shared texture of the frame, when `webPreferences.offscreen.useSharedTexture` is `true`.
* `dirtyRect` [Rectangle](structures/rectangle.md)
* `image` [NativeImage](native-image.md) - The image data of the whole frame.
Emitted when a new frame is generated. Only the dirty area is passed in the
buffer.
Emitted when a new frame is generated. Only the dirty area is passed in the buffer.
```js
const { BrowserWindow } = require('electron')
@ -886,6 +886,33 @@ win.webContents.on('paint', (event, dirty, image) => {
win.loadURL('https://github.com')
```
When using shared texture (set `webPreferences.offscreen.useSharedTexture` to `true`) feature, you can pass the texture handle to external rendering pipeline without the overhead of
copying data between CPU and GPU memory, with Chromium's hardware acceleration support. This feature is helpful for high-performance rendering scenarios.
Only a limited number of textures can exist at the same time, so it's important that you call `texture.release()` as soon as you're done with the texture.
By managing the texture lifecycle by yourself, you can safely pass the `texture.textureInfo` to other processes through IPC.
```js
const { BrowserWindow } = require('electron')
const win = new BrowserWindow({ webPreferences: { offscreen: { useSharedTexture: true } } })
win.webContents.on('paint', async (e, dirty, image) => {
if (e.texture) {
// By managing lifecycle yourself, you can handle the event in async handler or pass the `e.texture.textureInfo`
// to other processes (not `e.texture`, the `e.texture.release` function is not passable through IPC).
await new Promise(resolve => setTimeout(resolve, 50))
// You can send the native texture handle to native code for importing into your rendering pipeline.
// For example: https://github.com/electron/electron/tree/main/spec/fixtures/native-addon/osr-gpu
// importTextureHandle(dirty, e.texture.textureInfo)
// You must call `e.texture.release()` as soon as possible, before the underlying frame pool is drained.
e.texture.release()
}
})
win.loadURL('https://github.com')
```
#### Event: 'devtools-reload-page'
Emitted when the devtools window instructs the webContents to reload

View file

@ -3,7 +3,8 @@
## Overview
Offscreen rendering lets you obtain the content of a `BrowserWindow` in a
bitmap, so it can be rendered anywhere, for example, on texture in a 3D scene.
bitmap or a shared GPU texture, so it can be rendered anywhere, for example,
on texture in a 3D scene.
The offscreen rendering in Electron uses a similar approach to that of the
[Chromium Embedded Framework](https://bitbucket.org/chromiumembedded/cef)
project.
@ -17,22 +18,39 @@ the dirty area is passed to the `paint` event to be more efficient.
losses with no benefits.
* When nothing is happening on a webpage, no frames are generated.
* An offscreen window is always created as a
[Frameless Window](../tutorial/window-customization.md)..
[Frameless Window](../tutorial/window-customization.md).
### Rendering Modes
#### GPU accelerated
GPU accelerated rendering means that the GPU is used for composition. Because of
that, the frame has to be copied from the GPU which requires more resources,
thus this mode is slower than the Software output device. The benefit of this
mode is that WebGL and 3D CSS animations are supported.
GPU accelerated rendering means that the GPU is used for composition. The benefit
of this mode is that WebGL and 3D CSS animations are supported. There are two
different approaches depending on the `webPreferences.offscreen.useSharedTexture`
setting.
1. Use GPU shared texture
Used when `webPreferences.offscreen.useSharedTexture` is set to `true`.
This is an advanced feature requiring a native node module to work with your own code.
The frames are directly copied in GPU textures, thus this mode is very fast because
there's no CPU-GPU memory copies overhead, and you can directly import the shared
texture to your own rendering program.
2. Use CPU shared memory bitmap
Used when `webPreferences.offscreen.useSharedTexture` is set to `false` (default behavior).
The texture is accessible using the `NativeImage` API at the cost of performance.
The frame has to be copied from the GPU to the CPU bitmap which requires more system
resources, thus this mode is slower than the Software output device mode. But it supports
GPU related functionalities.
#### Software output device
This mode uses a software output device for rendering in the CPU, so the frame
generation is much faster. As a result, this mode is preferred over the GPU
accelerated one.
generation is faster than shared memory bitmap GPU accelerated mode.
To enable this mode, GPU acceleration has to be disabled by calling the
[`app.disableHardwareAcceleration()`][disablehardwareacceleration] API.