6936d74fb5
The sub-device format documentation documented scaling configuration through formats. Instead the compose selection rectangle is elsewhere documented to be used for the purpose. Remove scaling related part of the documentation. Signed-off-by: Sakari Ailus <sakari.ailus@iki.fi> Acked-by: Hans Verkuil <hans.verkuil@cisco.com> Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Mauro Carvalho Chehab <mchehab@osg.samsung.com>
484 lines
22 KiB
XML
484 lines
22 KiB
XML
<title>Sub-device Interface</title>
|
|
|
|
<note>
|
|
<title>Experimental</title>
|
|
<para>This is an <link linkend="experimental">experimental</link>
|
|
interface and may change in the future.</para>
|
|
</note>
|
|
|
|
<para>The complex nature of V4L2 devices, where hardware is often made of
|
|
several integrated circuits that need to interact with each other in a
|
|
controlled way, leads to complex V4L2 drivers. The drivers usually reflect
|
|
the hardware model in software, and model the different hardware components
|
|
as software blocks called sub-devices.</para>
|
|
|
|
<para>V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver
|
|
implements the media device API, they will automatically inherit from media
|
|
entities. Applications will be able to enumerate the sub-devices and discover
|
|
the hardware topology using the media entities, pads and links enumeration
|
|
API.</para>
|
|
|
|
<para>In addition to make sub-devices discoverable, drivers can also choose
|
|
to make them directly configurable by applications. When both the sub-device
|
|
driver and the V4L2 device driver support this, sub-devices will feature a
|
|
character device node on which ioctls can be called to
|
|
<itemizedlist>
|
|
<listitem><para>query, read and write sub-devices controls</para></listitem>
|
|
<listitem><para>subscribe and unsubscribe to events and retrieve them</para></listitem>
|
|
<listitem><para>negotiate image formats on individual pads</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
|
|
<para>Sub-device character device nodes, conventionally named
|
|
<filename>/dev/v4l-subdev*</filename>, use major number 81.</para>
|
|
|
|
<section>
|
|
<title>Controls</title>
|
|
<para>Most V4L2 controls are implemented by sub-device hardware. Drivers
|
|
usually merge all controls and expose them through video device nodes.
|
|
Applications can control all sub-devices through a single interface.</para>
|
|
|
|
<para>Complex devices sometimes implement the same control in different
|
|
pieces of hardware. This situation is common in embedded platforms, where
|
|
both sensors and image processing hardware implement identical functions,
|
|
such as contrast adjustment, white balance or faulty pixels correction. As
|
|
the V4L2 controls API doesn't support several identical controls in a single
|
|
device, all but one of the identical controls are hidden.</para>
|
|
|
|
<para>Applications can access those hidden controls through the sub-device
|
|
node with the V4L2 control API described in <xref linkend="control" />. The
|
|
ioctls behave identically as when issued on V4L2 device nodes, with the
|
|
exception that they deal only with controls implemented in the sub-device.
|
|
</para>
|
|
|
|
<para>Depending on the driver, those controls might also be exposed through
|
|
one (or several) V4L2 device nodes.</para>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Events</title>
|
|
<para>V4L2 sub-devices can notify applications of events as described in
|
|
<xref linkend="event" />. The API behaves identically as when used on V4L2
|
|
device nodes, with the exception that it only deals with events generated by
|
|
the sub-device. Depending on the driver, those events might also be reported
|
|
on one (or several) V4L2 device nodes.</para>
|
|
</section>
|
|
|
|
<section id="pad-level-formats">
|
|
<title>Pad-level Formats</title>
|
|
|
|
<warning><para>Pad-level formats are only applicable to very complex device that
|
|
need to expose low-level format configuration to user space. Generic V4L2
|
|
applications do <emphasis>not</emphasis> need to use the API described in
|
|
this section.</para></warning>
|
|
|
|
<note><para>For the purpose of this section, the term
|
|
<wordasword>format</wordasword> means the combination of media bus data
|
|
format, frame width and frame height.</para></note>
|
|
|
|
<para>Image formats are typically negotiated on video capture and
|
|
output devices using the format and <link
|
|
linkend="vidioc-subdev-g-selection">selection</link> ioctls. The
|
|
driver is responsible for configuring every block in the video
|
|
pipeline according to the requested format at the pipeline input
|
|
and/or output.</para>
|
|
|
|
<para>For complex devices, such as often found in embedded systems,
|
|
identical image sizes at the output of a pipeline can be achieved using
|
|
different hardware configurations. One such example is shown on
|
|
<xref linkend="pipeline-scaling" />, where
|
|
image scaling can be performed on both the video sensor and the host image
|
|
processing hardware.</para>
|
|
|
|
<figure id="pipeline-scaling">
|
|
<title>Image Format Negotiation on Pipelines</title>
|
|
<mediaobject>
|
|
<imageobject>
|
|
<imagedata fileref="pipeline.pdf" format="PS" />
|
|
</imageobject>
|
|
<imageobject>
|
|
<imagedata fileref="pipeline.png" format="PNG" />
|
|
</imageobject>
|
|
<textobject>
|
|
<phrase>High quality and high speed pipeline configuration</phrase>
|
|
</textobject>
|
|
</mediaobject>
|
|
</figure>
|
|
|
|
<para>The sensor scaler is usually of less quality than the host scaler, but
|
|
scaling on the sensor is required to achieve higher frame rates. Depending
|
|
on the use case (quality vs. speed), the pipeline must be configured
|
|
differently. Applications need to configure the formats at every point in
|
|
the pipeline explicitly.</para>
|
|
|
|
<para>Drivers that implement the <link linkend="media-controller-intro">media
|
|
API</link> can expose pad-level image format configuration to applications.
|
|
When they do, applications can use the &VIDIOC-SUBDEV-G-FMT; and
|
|
&VIDIOC-SUBDEV-S-FMT; ioctls. to negotiate formats on a per-pad basis.</para>
|
|
|
|
<para>Applications are responsible for configuring coherent parameters on
|
|
the whole pipeline and making sure that connected pads have compatible
|
|
formats. The pipeline is checked for formats mismatch at &VIDIOC-STREAMON;
|
|
time, and an &EPIPE; is then returned if the configuration is
|
|
invalid.</para>
|
|
|
|
<para>Pad-level image format configuration support can be tested by calling
|
|
the &VIDIOC-SUBDEV-G-FMT; ioctl on pad 0. If the driver returns an &EINVAL;
|
|
pad-level format configuration is not supported by the sub-device.</para>
|
|
|
|
<section>
|
|
<title>Format Negotiation</title>
|
|
|
|
<para>Acceptable formats on pads can (and usually do) depend on a number
|
|
of external parameters, such as formats on other pads, active links, or
|
|
even controls. Finding a combination of formats on all pads in a video
|
|
pipeline, acceptable to both application and driver, can't rely on formats
|
|
enumeration only. A format negotiation mechanism is required.</para>
|
|
|
|
<para>Central to the format negotiation mechanism are the get/set format
|
|
operations. When called with the <structfield>which</structfield> argument
|
|
set to <constant>V4L2_SUBDEV_FORMAT_TRY</constant>, the
|
|
&VIDIOC-SUBDEV-G-FMT; and &VIDIOC-SUBDEV-S-FMT; ioctls operate on a set of
|
|
formats parameters that are not connected to the hardware configuration.
|
|
Modifying those 'try' formats leaves the device state untouched (this
|
|
applies to both the software state stored in the driver and the hardware
|
|
state stored in the device itself).</para>
|
|
|
|
<para>While not kept as part of the device state, try formats are stored
|
|
in the sub-device file handles. A &VIDIOC-SUBDEV-G-FMT; call will return
|
|
the last try format set <emphasis>on the same sub-device file
|
|
handle</emphasis>. Several applications querying the same sub-device at
|
|
the same time will thus not interact with each other.</para>
|
|
|
|
<para>To find out whether a particular format is supported by the device,
|
|
applications use the &VIDIOC-SUBDEV-S-FMT; ioctl. Drivers verify and, if
|
|
needed, change the requested <structfield>format</structfield> based on
|
|
device requirements and return the possibly modified value. Applications
|
|
can then choose to try a different format or accept the returned value and
|
|
continue.</para>
|
|
|
|
<para>Formats returned by the driver during a negotiation iteration are
|
|
guaranteed to be supported by the device. In particular, drivers guarantee
|
|
that a returned format will not be further changed if passed to an
|
|
&VIDIOC-SUBDEV-S-FMT; call as-is (as long as external parameters, such as
|
|
formats on other pads or links' configuration are not changed).</para>
|
|
|
|
<para>Drivers automatically propagate formats inside sub-devices. When a
|
|
try or active format is set on a pad, corresponding formats on other pads
|
|
of the same sub-device can be modified by the driver. Drivers are free to
|
|
modify formats as required by the device. However, they should comply with
|
|
the following rules when possible:
|
|
<itemizedlist>
|
|
<listitem><para>Formats should be propagated from sink pads to source pads.
|
|
Modifying a format on a source pad should not modify the format on any
|
|
sink pad.</para></listitem>
|
|
<listitem><para>Sub-devices that scale frames using variable scaling factors
|
|
should reset the scale factors to default values when sink pads formats
|
|
are modified. If the 1:1 scaling ratio is supported, this means that
|
|
source pads formats should be reset to the sink pads formats.</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
|
|
<para>Formats are not propagated across links, as that would involve
|
|
propagating them from one sub-device file handle to another. Applications
|
|
must then take care to configure both ends of every link explicitly with
|
|
compatible formats. Identical formats on the two ends of a link are
|
|
guaranteed to be compatible. Drivers are free to accept different formats
|
|
matching device requirements as being compatible.</para>
|
|
|
|
<para><xref linkend="sample-pipeline-config" />
|
|
shows a sample configuration sequence for the pipeline described in
|
|
<xref linkend="pipeline-scaling" /> (table
|
|
columns list entity names and pad numbers).</para>
|
|
|
|
<table pgwide="0" frame="none" id="sample-pipeline-config">
|
|
<title>Sample Pipeline Configuration</title>
|
|
<tgroup cols="3">
|
|
<colspec colname="what"/>
|
|
<colspec colname="sensor-0 format" />
|
|
<colspec colname="frontend-0 format" />
|
|
<colspec colname="frontend-1 format" />
|
|
<colspec colname="scaler-0 format" />
|
|
<colspec colname="scaler-0 compose" />
|
|
<colspec colname="scaler-1 format" />
|
|
<thead>
|
|
<row>
|
|
<entry></entry>
|
|
<entry>Sensor/0 format</entry>
|
|
<entry>Frontend/0 format</entry>
|
|
<entry>Frontend/1 format</entry>
|
|
<entry>Scaler/0 format</entry>
|
|
<entry>Scaler/0 compose selection rectangle</entry>
|
|
<entry>Scaler/1 format</entry>
|
|
</row>
|
|
</thead>
|
|
<tbody valign="top">
|
|
<row>
|
|
<entry>Initial state</entry>
|
|
<entry>2048x1536/SGRBG8_1X8</entry>
|
|
<entry>(default)</entry>
|
|
<entry>(default)</entry>
|
|
<entry>(default)</entry>
|
|
<entry>(default)</entry>
|
|
<entry>(default)</entry>
|
|
</row>
|
|
<row>
|
|
<entry>Configure frontend sink format</entry>
|
|
<entry>2048x1536/SGRBG8_1X8</entry>
|
|
<entry><emphasis>2048x1536/SGRBG8_1X8</emphasis></entry>
|
|
<entry><emphasis>2046x1534/SGRBG8_1X8</emphasis></entry>
|
|
<entry>(default)</entry>
|
|
<entry>(default)</entry>
|
|
<entry>(default)</entry>
|
|
</row>
|
|
<row>
|
|
<entry>Configure scaler sink format</entry>
|
|
<entry>2048x1536/SGRBG8_1X8</entry>
|
|
<entry>2048x1536/SGRBG8_1X8</entry>
|
|
<entry>2046x1534/SGRBG8_1X8</entry>
|
|
<entry><emphasis>2046x1534/SGRBG8_1X8</emphasis></entry>
|
|
<entry><emphasis>0,0/2046x1534</emphasis></entry>
|
|
<entry><emphasis>2046x1534/SGRBG8_1X8</emphasis></entry>
|
|
</row>
|
|
<row>
|
|
<entry>Configure scaler sink compose selection</entry>
|
|
<entry>2048x1536/SGRBG8_1X8</entry>
|
|
<entry>2048x1536/SGRBG8_1X8</entry>
|
|
<entry>2046x1534/SGRBG8_1X8</entry>
|
|
<entry>2046x1534/SGRBG8_1X8</entry>
|
|
<entry><emphasis>0,0/1280x960</emphasis></entry>
|
|
<entry><emphasis>1280x960/SGRBG8_1X8</emphasis></entry>
|
|
</row>
|
|
</tbody>
|
|
</tgroup>
|
|
</table>
|
|
|
|
<para>
|
|
<orderedlist>
|
|
<listitem><para>Initial state. The sensor source pad format is
|
|
set to its native 3MP size and V4L2_MBUS_FMT_SGRBG8_1X8
|
|
media bus code. Formats on the host frontend and scaler sink
|
|
and source pads have the default values, as well as the
|
|
compose rectangle on the scaler's sink pad.</para></listitem>
|
|
|
|
<listitem><para>The application configures the frontend sink
|
|
pad format's size to 2048x1536 and its media bus code to
|
|
V4L2_MBUS_FMT_SGRBG_1X8. The driver propagates the format to
|
|
the frontend source pad.</para></listitem>
|
|
|
|
<listitem><para>The application configures the scaler sink pad
|
|
format's size to 2046x1534 and the media bus code to
|
|
V4L2_MBUS_FMT_SGRBG_1X8 to match the frontend source size and
|
|
media bus code. The media bus code on the sink pad is set to
|
|
V4L2_MBUS_FMT_SGRBG_1X8. The driver propagates the size to the
|
|
compose selection rectangle on the scaler's sink pad, and the
|
|
format to the scaler source pad.</para></listitem>
|
|
|
|
<listitem><para>The application configures the size of the compose
|
|
selection rectangle of the scaler's sink pad 1280x960. The driver
|
|
propagates the size to the scaler's source pad
|
|
format.</para></listitem>
|
|
|
|
</orderedlist>
|
|
</para>
|
|
|
|
<para>When satisfied with the try results, applications can set the active
|
|
formats by setting the <structfield>which</structfield> argument to
|
|
<constant>V4L2_SUBDEV_FORMAT_ACTIVE</constant>. Active formats are changed
|
|
exactly as try formats by drivers. To avoid modifying the hardware state
|
|
during format negotiation, applications should negotiate try formats first
|
|
and then modify the active settings using the try formats returned during
|
|
the last negotiation iteration. This guarantees that the active format
|
|
will be applied as-is by the driver without being modified.
|
|
</para>
|
|
</section>
|
|
|
|
<section id="v4l2-subdev-selections">
|
|
<title>Selections: cropping, scaling and composition</title>
|
|
|
|
<para>Many sub-devices support cropping frames on their input or output
|
|
pads (or possible even on both). Cropping is used to select the area of
|
|
interest in an image, typically on an image sensor or a video decoder. It can
|
|
also be used as part of digital zoom implementations to select the area of
|
|
the image that will be scaled up.</para>
|
|
|
|
<para>Crop settings are defined by a crop rectangle and represented in a
|
|
&v4l2-rect; by the coordinates of the top left corner and the rectangle
|
|
size. Both the coordinates and sizes are expressed in pixels.</para>
|
|
|
|
<para>As for pad formats, drivers store try and active
|
|
rectangles for the selection targets <xref
|
|
linkend="v4l2-selections-common" />.</para>
|
|
|
|
<para>On sink pads, cropping is applied relative to the
|
|
current pad format. The pad format represents the image size as
|
|
received by the sub-device from the previous block in the
|
|
pipeline, and the crop rectangle represents the sub-image that
|
|
will be transmitted further inside the sub-device for
|
|
processing.</para>
|
|
|
|
<para>The scaling operation changes the size of the image by
|
|
scaling it to new dimensions. The scaling ratio isn't specified
|
|
explicitly, but is implied from the original and scaled image
|
|
sizes. Both sizes are represented by &v4l2-rect;.</para>
|
|
|
|
<para>Scaling support is optional. When supported by a subdev,
|
|
the crop rectangle on the subdev's sink pad is scaled to the
|
|
size configured using the &VIDIOC-SUBDEV-S-SELECTION; IOCTL
|
|
using <constant>V4L2_SEL_TGT_COMPOSE</constant>
|
|
selection target on the same pad. If the subdev supports scaling
|
|
but not composing, the top and left values are not used and must
|
|
always be set to zero.</para>
|
|
|
|
<para>On source pads, cropping is similar to sink pads, with the
|
|
exception that the source size from which the cropping is
|
|
performed, is the COMPOSE rectangle on the sink pad. In both
|
|
sink and source pads, the crop rectangle must be entirely
|
|
contained inside the source image size for the crop
|
|
operation.</para>
|
|
|
|
<para>The drivers should always use the closest possible
|
|
rectangle the user requests on all selection targets, unless
|
|
specifically told otherwise.
|
|
<constant>V4L2_SEL_FLAG_GE</constant> and
|
|
<constant>V4L2_SEL_FLAG_LE</constant> flags may be
|
|
used to round the image size either up or down. <xref
|
|
linkend="v4l2-selection-flags" /></para>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Types of selection targets</title>
|
|
|
|
<section>
|
|
<title>Actual targets</title>
|
|
|
|
<para>Actual targets (without a postfix) reflect the actual
|
|
hardware configuration at any point of time. There is a BOUNDS
|
|
target corresponding to every actual target.</para>
|
|
</section>
|
|
|
|
<section>
|
|
<title>BOUNDS targets</title>
|
|
|
|
<para>BOUNDS targets is the smallest rectangle that contains all
|
|
valid actual rectangles. It may not be possible to set the actual
|
|
rectangle as large as the BOUNDS rectangle, however. This may be
|
|
because e.g. a sensor's pixel array is not rectangular but
|
|
cross-shaped or round. The maximum size may also be smaller than the
|
|
BOUNDS rectangle.</para>
|
|
</section>
|
|
|
|
</section>
|
|
|
|
<section>
|
|
<title>Order of configuration and format propagation</title>
|
|
|
|
<para>Inside subdevs, the order of image processing steps will
|
|
always be from the sink pad towards the source pad. This is also
|
|
reflected in the order in which the configuration must be
|
|
performed by the user: the changes made will be propagated to
|
|
any subsequent stages. If this behaviour is not desired, the
|
|
user must set
|
|
<constant>V4L2_SEL_FLAG_KEEP_CONFIG</constant> flag. This
|
|
flag causes no propagation of the changes are allowed in any
|
|
circumstances. This may also cause the accessed rectangle to be
|
|
adjusted by the driver, depending on the properties of the
|
|
underlying hardware.</para>
|
|
|
|
<para>The coordinates to a step always refer to the actual size
|
|
of the previous step. The exception to this rule is the source
|
|
compose rectangle, which refers to the sink compose bounds
|
|
rectangle --- if it is supported by the hardware.</para>
|
|
|
|
<orderedlist>
|
|
<listitem><para>Sink pad format. The user configures the sink pad
|
|
format. This format defines the parameters of the image the
|
|
entity receives through the pad for further processing.</para></listitem>
|
|
|
|
<listitem><para>Sink pad actual crop selection. The sink pad crop
|
|
defines the crop performed to the sink pad format.</para></listitem>
|
|
|
|
<listitem><para>Sink pad actual compose selection. The size of the
|
|
sink pad compose rectangle defines the scaling ratio compared
|
|
to the size of the sink pad crop rectangle. The location of
|
|
the compose rectangle specifies the location of the actual
|
|
sink compose rectangle in the sink compose bounds
|
|
rectangle.</para></listitem>
|
|
|
|
<listitem><para>Source pad actual crop selection. Crop on the source
|
|
pad defines crop performed to the image in the sink compose
|
|
bounds rectangle.</para></listitem>
|
|
|
|
<listitem><para>Source pad format. The source pad format defines the
|
|
output pixel format of the subdev, as well as the other
|
|
parameters with the exception of the image width and height.
|
|
Width and height are defined by the size of the source pad
|
|
actual crop selection.</para></listitem>
|
|
</orderedlist>
|
|
|
|
<para>Accessing any of the above rectangles not supported by the
|
|
subdev will return <constant>EINVAL</constant>. Any rectangle
|
|
referring to a previous unsupported rectangle coordinates will
|
|
instead refer to the previous supported rectangle. For example,
|
|
if sink crop is not supported, the compose selection will refer
|
|
to the sink pad format dimensions instead.</para>
|
|
|
|
<figure id="subdev-image-processing-crop">
|
|
<title>Image processing in subdevs: simple crop example</title>
|
|
<mediaobject>
|
|
<imageobject>
|
|
<imagedata fileref="subdev-image-processing-crop.svg"
|
|
format="SVG" scale="200" />
|
|
</imageobject>
|
|
</mediaobject>
|
|
</figure>
|
|
|
|
<para>In the above example, the subdev supports cropping on its
|
|
sink pad. To configure it, the user sets the media bus format on
|
|
the subdev's sink pad. Now the actual crop rectangle can be set
|
|
on the sink pad --- the location and size of this rectangle
|
|
reflect the location and size of a rectangle to be cropped from
|
|
the sink format. The size of the sink crop rectangle will also
|
|
be the size of the format of the subdev's source pad.</para>
|
|
|
|
<figure id="subdev-image-processing-scaling-multi-source">
|
|
<title>Image processing in subdevs: scaling with multiple sources</title>
|
|
<mediaobject>
|
|
<imageobject>
|
|
<imagedata fileref="subdev-image-processing-scaling-multi-source.svg"
|
|
format="SVG" scale="200" />
|
|
</imageobject>
|
|
</mediaobject>
|
|
</figure>
|
|
|
|
<para>In this example, the subdev is capable of first cropping,
|
|
then scaling and finally cropping for two source pads
|
|
individually from the resulting scaled image. The location of
|
|
the scaled image in the cropped image is ignored in sink compose
|
|
target. Both of the locations of the source crop rectangles
|
|
refer to the sink scaling rectangle, independently cropping an
|
|
area at location specified by the source crop rectangle from
|
|
it.</para>
|
|
|
|
<figure id="subdev-image-processing-full">
|
|
<title>Image processing in subdevs: scaling and composition
|
|
with multiple sinks and sources</title>
|
|
<mediaobject>
|
|
<imageobject>
|
|
<imagedata fileref="subdev-image-processing-full.svg"
|
|
format="SVG" scale="200" />
|
|
</imageobject>
|
|
</mediaobject>
|
|
</figure>
|
|
|
|
<para>The subdev driver supports two sink pads and two source
|
|
pads. The images from both of the sink pads are individually
|
|
cropped, then scaled and further composed on the composition
|
|
bounds rectangle. From that, two independent streams are cropped
|
|
and sent out of the subdev from the source pads.</para>
|
|
|
|
</section>
|
|
|
|
</section>
|
|
|
|
&sub-subdev-formats;
|