feat: GPU shared texture offscreen rendering (#42953)
* feat: GPU shared texture offscreen rendering * docs: clarify texture infos that passed by the paint event. * feat: make gpu osr spec test optional * fix: osr image compare * fix: remove duplicate test * fix: update patch file * fix: code review * feat: expose more metadata * feat: use better switch design * feat: add warning when user forget to release the texture. * fix: typo * chore: update patch * fix: update patch * fix: update patch description * fix: update docs * fix: apply suggestions from code review Co-authored-by: Charles Kerr <charles@charleskerr.com> * fix: apply suggested fixes --------- Co-authored-by: Charles Kerr <charles@charleskerr.com>
This commit is contained in:
parent
b481966f02
commit
1aeca6fd0e
34 changed files with 1009 additions and 102 deletions
24
docs/api/structures/offscreen-shared-texture.md
Normal file
24
docs/api/structures/offscreen-shared-texture.md
Normal file
|
@ -0,0 +1,24 @@
|
|||
# OffscreenSharedTexture Object
|
||||
|
||||
* `textureInfo` Object - The shared texture info.
|
||||
* `widgetType` string - The widget type of the texture. Can be `popup` or `frame`.
|
||||
* `pixelFormat` string - The pixel format of the texture. Can be `rgba` or `bgra`.
|
||||
* `codedSize` [Size](size.md) - The full dimensions of the video frame.
|
||||
* `visibleRect` [Rectangle](rectangle.md) - A subsection of [0, 0, codedSize.width(), codedSize.height()]. In OSR case, it is expected to have the full section area.
|
||||
* `contentRect` [Rectangle](rectangle.md) - The region of the video frame that capturer would like to populate. In OSR case, it is the same with `dirtyRect` that needs to be painted.
|
||||
* `timestamp` number - The time in microseconds since the capture start.
|
||||
* `metadata` Object - Extra metadata. See comments in src\media\base\video_frame_metadata.h for accurate details.
|
||||
* `captureUpdateRect` [Rectangle](rectangle.md) (optional) - Updated area of frame, can be considered as the `dirty` area.
|
||||
* `regionCaptureRect` [Rectangle](rectangle.md) (optional) - May reflect the frame's contents origin if region capture is used internally.
|
||||
* `sourceSize` [Rectangle](rectangle.md) (optional) - Full size of the source frame.
|
||||
* `frameCount` number (optional) - The increasing count of captured frame. May contain gaps if frames are dropped between two consecutively received frames.
|
||||
* `sharedTextureHandle` Buffer _Windows_ _macOS_ - The handle to the shared texture.
|
||||
* `planes` Object[] _Linux_ - Each plane's info of the shared texture.
|
||||
* `stride` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
|
||||
* `offset` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
|
||||
* `size` number - Size in bytes of the plane. This is necessary to map the buffers.
|
||||
* `fd` number - File descriptor for the underlying memory object (usually dmabuf).
|
||||
* `modifier` string _Linux_ - The modifier is retrieved from GBM library and passed to EGL driver.
|
||||
* `release` Function - Release the resources. The `texture` cannot be directly passed to another process, users need to maintain texture lifecycles in
|
||||
main process, but it is safe to pass the `textureInfo` to another process. Only a limited number of textures can exist at the same time, so it's important
|
||||
that you call `texture.release()` as soon as you're done with the texture.
|
|
@ -79,10 +79,14 @@
|
|||
[browserWindow](../browser-window.md) has disabled `backgroundThrottling` then
|
||||
frames will be drawn and swapped for the whole window and other
|
||||
[webContents](../web-contents.md) displayed by it. Defaults to `true`.
|
||||
* `offscreen` boolean (optional) - Whether to enable offscreen rendering for the browser
|
||||
* `offscreen` Object | boolean (optional) - Whether to enable offscreen rendering for the browser
|
||||
window. Defaults to `false`. See the
|
||||
[offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
|
||||
more details.
|
||||
* `useSharedTexture` boolean (optional) _Experimental_ - Whether to use GPU shared texture for accelerated
|
||||
paint event. Defaults to `false`. See the
|
||||
[offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
|
||||
more details.
|
||||
* `contextIsolation` boolean (optional) - Whether to run Electron APIs and
|
||||
the specified `preload` script in a separate JavaScript context. Defaults
|
||||
to `true`. The context that the `preload` script runs in will only have
|
||||
|
|
|
@ -869,12 +869,12 @@ app.whenReady().then(() => {
|
|||
|
||||
Returns:
|
||||
|
||||
* `event` Event
|
||||
* `details` Event\<\>
|
||||
* `texture` [OffscreenSharedTexture](structures/offscreen-shared-texture.md) (optional) _Experimental_ - The GPU shared texture of the frame, when `webPreferences.offscreen.useSharedTexture` is `true`.
|
||||
* `dirtyRect` [Rectangle](structures/rectangle.md)
|
||||
* `image` [NativeImage](native-image.md) - The image data of the whole frame.
|
||||
|
||||
Emitted when a new frame is generated. Only the dirty area is passed in the
|
||||
buffer.
|
||||
Emitted when a new frame is generated. Only the dirty area is passed in the buffer.
|
||||
|
||||
```js
|
||||
const { BrowserWindow } = require('electron')
|
||||
|
@ -886,6 +886,33 @@ win.webContents.on('paint', (event, dirty, image) => {
|
|||
win.loadURL('https://github.com')
|
||||
```
|
||||
|
||||
When using shared texture (set `webPreferences.offscreen.useSharedTexture` to `true`) feature, you can pass the texture handle to external rendering pipeline without the overhead of
|
||||
copying data between CPU and GPU memory, with Chromium's hardware acceleration support. This feature is helpful for high-performance rendering scenarios.
|
||||
|
||||
Only a limited number of textures can exist at the same time, so it's important that you call `texture.release()` as soon as you're done with the texture.
|
||||
By managing the texture lifecycle by yourself, you can safely pass the `texture.textureInfo` to other processes through IPC.
|
||||
|
||||
```js
|
||||
const { BrowserWindow } = require('electron')
|
||||
|
||||
const win = new BrowserWindow({ webPreferences: { offscreen: { useSharedTexture: true } } })
|
||||
win.webContents.on('paint', async (e, dirty, image) => {
|
||||
if (e.texture) {
|
||||
// By managing lifecycle yourself, you can handle the event in async handler or pass the `e.texture.textureInfo`
|
||||
// to other processes (not `e.texture`, the `e.texture.release` function is not passable through IPC).
|
||||
await new Promise(resolve => setTimeout(resolve, 50))
|
||||
|
||||
// You can send the native texture handle to native code for importing into your rendering pipeline.
|
||||
// For example: https://github.com/electron/electron/tree/main/spec/fixtures/native-addon/osr-gpu
|
||||
// importTextureHandle(dirty, e.texture.textureInfo)
|
||||
|
||||
// You must call `e.texture.release()` as soon as possible, before the underlying frame pool is drained.
|
||||
e.texture.release()
|
||||
}
|
||||
})
|
||||
win.loadURL('https://github.com')
|
||||
```
|
||||
|
||||
#### Event: 'devtools-reload-page'
|
||||
|
||||
Emitted when the devtools window instructs the webContents to reload
|
||||
|
|
|
@ -3,7 +3,8 @@
|
|||
## Overview
|
||||
|
||||
Offscreen rendering lets you obtain the content of a `BrowserWindow` in a
|
||||
bitmap, so it can be rendered anywhere, for example, on texture in a 3D scene.
|
||||
bitmap or a shared GPU texture, so it can be rendered anywhere, for example,
|
||||
on texture in a 3D scene.
|
||||
The offscreen rendering in Electron uses a similar approach to that of the
|
||||
[Chromium Embedded Framework](https://bitbucket.org/chromiumembedded/cef)
|
||||
project.
|
||||
|
@ -17,22 +18,39 @@ the dirty area is passed to the `paint` event to be more efficient.
|
|||
losses with no benefits.
|
||||
* When nothing is happening on a webpage, no frames are generated.
|
||||
* An offscreen window is always created as a
|
||||
[Frameless Window](../tutorial/window-customization.md)..
|
||||
[Frameless Window](../tutorial/window-customization.md).
|
||||
|
||||
### Rendering Modes
|
||||
|
||||
#### GPU accelerated
|
||||
|
||||
GPU accelerated rendering means that the GPU is used for composition. Because of
|
||||
that, the frame has to be copied from the GPU which requires more resources,
|
||||
thus this mode is slower than the Software output device. The benefit of this
|
||||
mode is that WebGL and 3D CSS animations are supported.
|
||||
GPU accelerated rendering means that the GPU is used for composition. The benefit
|
||||
of this mode is that WebGL and 3D CSS animations are supported. There are two
|
||||
different approaches depending on the `webPreferences.offscreen.useSharedTexture`
|
||||
setting.
|
||||
|
||||
1. Use GPU shared texture
|
||||
|
||||
Used when `webPreferences.offscreen.useSharedTexture` is set to `true`.
|
||||
|
||||
This is an advanced feature requiring a native node module to work with your own code.
|
||||
The frames are directly copied in GPU textures, thus this mode is very fast because
|
||||
there's no CPU-GPU memory copies overhead, and you can directly import the shared
|
||||
texture to your own rendering program.
|
||||
|
||||
2. Use CPU shared memory bitmap
|
||||
|
||||
Used when `webPreferences.offscreen.useSharedTexture` is set to `false` (default behavior).
|
||||
|
||||
The texture is accessible using the `NativeImage` API at the cost of performance.
|
||||
The frame has to be copied from the GPU to the CPU bitmap which requires more system
|
||||
resources, thus this mode is slower than the Software output device mode. But it supports
|
||||
GPU related functionalities.
|
||||
|
||||
#### Software output device
|
||||
|
||||
This mode uses a software output device for rendering in the CPU, so the frame
|
||||
generation is much faster. As a result, this mode is preferred over the GPU
|
||||
accelerated one.
|
||||
generation is faster than shared memory bitmap GPU accelerated mode.
|
||||
|
||||
To enable this mode, GPU acceleration has to be disabled by calling the
|
||||
[`app.disableHardwareAcceleration()`][disablehardwareacceleration] API.
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue