electron/shell/browser/osr
trop[bot] 85a8bfaa31
chore: bump chromium to 139.0.7256.0 (38-x-y) (#47615)
* chore: bump chromium in DEPS to 139.0.7242.0

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>

* chore: update render_widget_host_view_mac.patch

no code changes; just updating patch context

Do a cleanup pass on the history swiper code | 6604367

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: update mas_avoid_private_macos_api_usage.patch.patch

no code changes; just updating patch context

[tracing] Delete base/trace_event/base_tracing.h | 6624012

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: update chore_provide_iswebcontentscreationoverridden_with_full_params.patch

no manual changes; just updating patch context

[ActorFramework] Refactor Actor Task Management | 6618684

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: update fix_move_autopipsettingshelper_behind_branding_buildflag.patch

[pip] Tuck picture-in-picture windows when a file dialog is open | 6449682

Reland "[document pip] Restrict the size that a website can request" | 6372104

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: update feat_corner_smoothing_css_rule_and_blink_painting.patch

Xref: corner-shape: constraint radii based on opposite corner overlap | 6592572

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: update revert_code_health_clean_up_stale_macwebcontentsocclusion.patch

no manual changes; just updating patch context

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: update fix_rename_sqlite_win32_exports_to_avoid_conflicts_with_node_js.patch

no code changes; just updating patch context

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: e patches all

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* Plumb Verify2QwacBinding and hook it up in QwacWebContentsObserver

6624719

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* Remove host delegate OnMainFrameCreatedForBackgroundPage

6631123

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* Extensions: Rename GetResourceURL to ResolveExtensionURL

6625053

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* Consolidate NativeFrameViewMac

6614239

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* ICWYU

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* Remove dead code WidgetAXTreeIDMap

6619701

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* Reland "extensions: Add `WillPrepareForEvaluation` to setup MojoJS"

6630056

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* NavigationThrottleRunner2: Remove MaybeAddThrottle

6628079

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* Tuck picture-in-picture windows when a file dialog is open

6449682

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* build: fix snapshot_blob.bin build error

xref: https://issues.chromium.org/issues/416540976

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* chore: e patches all

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* build: freeup disk space on macos

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* chore: bump chromium in DEPS to 139.0.7244.0

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>

* chore: update printing.patch

no manual changes; just updating patch context

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: remove upstreamed ignore_parse_errors_for_resolveshortcutproperties.patch

Prevent Windows crash on unexpected shortcut type | 6633298

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: e patches all

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* Revert "Reland "extensions: Add `WillPrepareForEvaluation` to setup MojoJS""

This reverts commit 77c4f967a637f7e8970114f91311f9fddede0f7c.

Revert CL for the high confidence crash culprit for http://crash/28f897bb9743dfe0 | 6641819

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* Fix spec's expected base64-encoded PNG strings to match upstream changes.

[rust png] Enable by default. | 6085801

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: bump chromium in DEPS to 139.0.7246.0

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>

* chore: e patches all

Co-authored-by: Charles Kerr <charles@charleskerr.com>

* chore: bump chromium in DEPS to 139.0.7248.0

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>

* chore: update patches

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* siso: Enable Siso by default for non-Google builds

6638830

Disabling for now until we are ready to build siso on all platforms.

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* Revert "revert Don't use static variable for UseExternalPopupMenus"

This reverts commit e91e3894e6c34cc0ffe69ed45417c0ebec882fb1.

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* Update mac_sdk_min to match minimum required SDK version

6493969
(cherry picked from commit 3e7cbe912d8fe1062d68ed06968aaee22013985f)

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* Use default window styling on Mac

6648665

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* Reland "Force the unintentional renderer process creation check by default"

6626905

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* fixup: Reland "Force the unintentional renderer process creation check by default

6626905

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* chore: bump chromium in DEPS to 139.0.7249.0

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>

* fixup: Reland "Force the unintentional renderer process creation check by default

6626905

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* chore: update patches

Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>

* chore: bump chromium in DEPS to 139.0.7250.0

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>

* chore: bump chromium in DEPS to 139.0.7252.0

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>

* chore: bump chromium in DEPS to 139.0.7254.0

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>

* 6638187: browser level TOCTOU check for coordinate target

6638187

Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>

* chore: fixup patch indices

Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>

* chore: add missing base/notimplemented includes

Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>

* 6652910: [Frame Cleanup] Push down/hide implementation-specific API

6652910

Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>

* chore: bump chromium in DEPS to 139.0.7256.0

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>

* chore: fix lint

Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>

* fixup! 6652910: [Frame Cleanup] Push down/hide implementation-specific API

Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>

* fix: move HandleScope location

Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>

* chore: bump chromium in DEPS to 139.0.7258.0

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>

* fixup! [NonClientFrameView] Consolidate NativeFrameViewMac

Co-authored-by: deepak1556 <hop2deep@gmail.com>

* Revert "chore: bump chromium in DEPS to 139.0.7258.0"

This reverts commit 264b2e934f4b2705c47d9761010052b95d9dd5de.

Co-authored-by: deepak1556 <hop2deep@gmail.com>

* chore: update patches

---------

Co-authored-by: trop[bot] <37223003+trop[bot]@users.noreply.github.com>
Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>
Co-authored-by: Charles Kerr <charles@charleskerr.com>
Co-authored-by: John Kleinschmidt <jkleinsc@electronjs.org>
Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>
Co-authored-by: deepak1556 <hop2deep@gmail.com>
Co-authored-by: patchup[bot] <73610968+patchup[bot]@users.noreply.github.com>
2025-06-30 12:29:54 -04:00
..
osr_host_display_client.cc chore: bump chromium to 134.0.6968.0 (main) (#45172) 2025-01-23 23:07:43 -05:00
osr_host_display_client.h feat: GPU shared texture offscreen rendering (#42953) 2024-08-22 19:23:13 -05:00
osr_host_display_client_mac.mm feat: GPU shared texture offscreen rendering (#42953) 2024-08-22 19:23:13 -05:00
osr_paint_event.cc feat: GPU shared texture offscreen rendering (#42953) 2024-08-22 19:23:13 -05:00
osr_paint_event.h feat: GPU shared texture offscreen rendering (#42953) 2024-08-22 19:23:13 -05:00
osr_render_widget_host_view.cc chore: bump chromium to 139.0.7256.0 (38-x-y) (#47615) 2025-06-30 12:29:54 -04:00
osr_render_widget_host_view.h refactor: remove CreateViewForWidget patch (#46981) 2025-05-14 21:43:23 +09:00
osr_video_consumer.cc chore: bump chromium to 138.0.7154.0 (main) (#46872) 2025-05-01 23:53:40 -05:00
osr_video_consumer.h feat: GPU shared texture offscreen rendering (#42953) 2024-08-22 19:23:13 -05:00
osr_view_proxy.cc refactor: inline simple getters (#41125) 2024-01-29 20:43:28 -06:00
osr_view_proxy.h chore: remove more unused #include calls (#43000) 2024-07-25 11:25:45 +02:00
osr_web_contents_view.cc refactor: remove CreateViewForWidget patch (#46981) 2025-05-14 21:43:23 +09:00
osr_web_contents_view.h chore: bump chromium to 132.0.6807.0 (main) (#44360) 2024-11-04 09:10:00 -05:00
osr_web_contents_view_mac.mm chore: bump chromium to 136.0.7095.0 (main) (#46118) 2025-04-03 19:02:49 -05:00
README.md docs: add docs about shared texture mode osr (#45670) 2025-02-27 11:10:58 -05:00

Offscreen Rendering

Shared Texture Mode

This section provides a brief summary about how an offscreen frame is generated and how to handle it in native code. This only applies to the GPU-accelerated mode with shared texture, when webPreferences.offscreen.useSharedTexture is set to true.

Life of an Offscreen Frame

This is written at the time of Chromium 134 / Electron 35. The code may change in the future. The following description may not completely reflect the procedure, but it generally describes the process.

Initialization

  1. Electron JS creates a BrowserWindow with webPreferences.offscreen set to true.
  2. Electron C++ creates OffScreenRenderWidgetHostView, a subclass of RenderWidgetHostViewBase.
  3. It instantiates an OffScreenVideoConsumer, passing itself as a reference as view_.
  4. The OffScreenVideoConsumer calls view_->CreateVideoCapturer(), which makes Chromium code use HostFrameSinkManager to communicate with FrameSinkManagerImpl (in the Renderer Process) to create a ClientFrameSinkVideoCapturer and a FrameSinkVideoCapturerImpl (in the Renderer Process). It stores ClientFrameSinkVideoCapturer in video_capturer_.
  5. The OffScreenVideoConsumer registers the capture callback to OffScreenRenderWidgetHostView::OnPaint.
  6. It sets the target FPS, size constraints for capture, and calls video_capturer_->Start with the parameter viz::mojom::BufferFormatPreference::kPreferGpuMemoryBuffer to enable shared texture mode and start capturing.
  7. The FrameSinkVideoCapturerImpl accepts kPreferGpuMemoryBuffer and creates a GpuMemoryBufferVideoFramePool to copy the captured frame. The capacity is kFramePoolCapacity, currently 10, meaning it can capture at most 10 frames if the consumer doesn't consume them in time. It is stored in frame_pool_.
  8. The GpuMemoryBufferVideoFramePool creates a RenderableGpuMemoryBufferVideoFramePool using GmbVideoFramePoolContext as the context provider, responsible for creating GpuMemoryBuffer (or MappableSharedImage, as the Chromium team is removing the concept of GpuMemoryBuffer and may replace it with MappableSI in the future).
  9. The GmbVideoFramePoolContext initializes itself in both the Renderer Process and GPU Process.

Capturing

  1. The FrameSinkVideoCapturerImpl starts a capture when it receives an OnFrameDamaged event or is explicitly requested to refresh the frame. All event sources are evaluated by VideoCaptureOracle to see if the capture frequency meets the limit.
  2. If a frame is determined to be captured, FrameSinkVideoCapturerImpl calls frame_pool_->ReserveVideoFrame() to make the pool allocate a frame. The GmbVideoFramePoolContext then communicates with the GPU Process to create an actual platform-dependent texture (e.g., ID3D11Texture2D or IOSurface) that supports being shared across processes.
  3. The GPU Process wraps it into a GpuMemoryBuffer and sends it back to the Renderer Process, and the pool stores it for further usage.
  4. The FrameSinkVideoCapturerImpl then uses this allocated (or reused) frame to create a CopyOutputRequest and calls resolved_target_->RequestCopyOfOutput to copy the frame to the target texture. The resolved_target_ is a CapturableFrameSink that was previously resolved when calling CreateVideoCapturer using OffScreenRenderWidgetHostView.
  5. The GPU Process receives the request and renders the frame to the target texture using the requested format (e.g., RGBA). It then sends a completed event to the Renderer Process FrameSinkVideoCapturerImpl.
  6. The FrameSinkVideoCapturerImpl receives the completed event, provides feedback to the VideoCaptureOracle, and then calls frame_pool_->CloneHandleForDelivery with the captured frame to get a serializable handle to the frame (HANDLE or IOSurfaceRef). On Windows, it calls DuplicateHandle to create a new handle.
  7. It then creates a VideoFrameInfo with the frame info and the handle and calls consumer_->OnFrameCaptured to deliver the frame to the consumer.

Consuming

  1. OffScreenVideoConsumer::OnFrameCaptured is called when the frame is captured. It creates an Electron C++ struct OffscreenSharedTextureValue to extract the required info and handle from the callback. It then creates an OffscreenReleaserHolder to take ownership of the handle and the mojom remote releaser to prevent releasing.
  2. It calls the callback_ with the OffscreenSharedTextureValue, which goes to OffScreenRenderWidgetHostView::OnPaint. When shared texture mode is enabled, it directly redirects the callback to an OnPaintCallback target set during the initialization of OffScreenRenderWidgetHostView, currently set by OffScreenWebContentsView, whose callback_ is also set during initialization. Finally, it goes to WebContents::OnPaint.
  3. The WebContents::OnPaint uses gin_converter (osr_converter.cc) to convert the OffscreenSharedTextureValue to a v8::Object. It converts most of the value to a corresponding v8::Value, and the handle is converted to a Buffer. It also creates a release function to destroy the releaser and free the frame we previously took ownership of. The frame can now be reused for further capturing. Finally, it creates a release monitor to detect if the release function is called before the garbage collector destroys the JS object; if not, it prints a warning.
  4. The data is then emitted to the paint event of webContents. You can now grab the data and pass it to your native code for further processing. You can pass the textureInfo to other processes using Electron IPC, but you can only release it in the main process. Do not keep it for long, or it will drain the buffer pool.

Native Handling

You now have the texture info for the frame. Here's how you should handle it in native code. Suppose you write a node native addon to handle the shared texture.

Retrieve the handle from the textureInfo.sharedTextureHandle. You can also read the buffer in JS and use other methods.

auto textureInfo = args[0];
auto sharedTextureHandle =
    NAPI_GET_PROPERTY_VALUE(textureInfo, "sharedTextureHandle");

size_t handleBufferSize;
uint8_t* handleBufferData;
napi_get_buffer_info(env, sharedTextureHandle,
                      reinterpret_cast<void**>(&handleBufferData),
                      &handleBufferSize);

Import the handle to your rendering program.

// Windows
HANDLE handle = *reinterpret_cast<HANDLE*>(handleBufferData);
Microsoft::WRL::ComPtr<ID3D11Texture2D> shared_texture = nullptr;
HRESULT hr = device1->OpenSharedResource1(handle, IID_PPV_ARGS(&shared_texture)); 

// Extract the texture description
D3D11_TEXTURE2D_DESC desc;
shared_texture->GetDesc(&desc);

// Cache the staging texture if it does not exist or size has changed
if (!cached_staging_texture || cached_width != desc.Width ||
    cached_height != desc.Height) {
  if (cached_staging_texture) {
    cached_staging_texture->Release();
  }

  desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
  desc.Usage = D3D11_USAGE_STAGING;
  desc.BindFlags = 0;
  desc.MiscFlags = 0;

  std::cout << "Create staging Texture2D width=" << desc.Width
            << " height=" << desc.Height << std::endl;
  hr = device->CreateTexture2D(&desc, nullptr, &cached_staging_texture);

  cached_width = desc.Width;
  cached_height = desc.Height;
}

// Copy to a intermediate texture
context->CopyResource(cached_staging_texture.Get(), shared_texture.Get());
// macOS
IOSurfaceRef handle = *reinterpret_cast<IOSurfaceRef*>(handleBufferData);

// Assume you have created a GL context.

GLuint io_surface_tex;
glGenTextures(1, &io_surface_tex);
glEnable(GL_TEXTURE_RECTANGLE_ARB);
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, io_surface_tex);

CGLContextObj cgl_context = CGLGetCurrentContext();

GLsizei width = (GLsizei)IOSurfaceGetWidth(io_surface);
GLsizei height = (GLsizei)IOSurfaceGetHeight(io_surface);

CGLTexImageIOSurface2D(cgl_context, GL_TEXTURE_RECTANGLE_ARB, GL_RGBA8, width,
                        height, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV,
                        io_surface, 0);

glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);

// Copy to a intermediate texture from io_surface_tex
// ...

As introduced above, the shared texture is not a single fixed texture that Chromium draws on every frame (CEF before Chromium 103 works that like, please note this significate difference). It is a pool of textures, so every frame Chromium may pass a different texture to you. As soon as you call release, the texture will be reused by Chromium and may cause picture corruption if you keep reading from it. It is also wrong to only open the handle once and directly read from that opened texture on later events. Be very careful if you want to cache the opened texture(s), on Windows, the duplicated handle's value will not be a reliable mapping to the actual underlying texture.

The suggested way is always open the handle on every event, and copy the shared texture to a intermediate texture that you own and release it as soon as possible. You can use the copied texture for further rendering whenever you want. Opening a shared texture should only takes couple microseconds, and you can also use the damaged rect to only copy a portion of the texture to speed up.

You can also refer to these examples: