L4T Multimedia API Reference

27.1 Release

 All Data Structures Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
nvee_backend.md
Go to the documentation of this file.
1 Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
2 
3 @page nvvid_backend_group 4 Ways to Video Decode with TensorRT Processing Sample
4 @{
5 
6  - [Overview](#overview)
7  - [Key Structure and Classes](#key)
8  - [Command Line Options](#options)
9 
10 - - - - - - - - - - - - - - -
11 <a name="overview">
12 ## Overview ##
13 
14 This use case application showcases a typical appliance performing intelligent video analytics.
15 Application areas include public safety, smart cities, and autonomous machines. This example demonstrates
16 four (4) concurrent video streams going through a decoding process using the on chip decoders, video scaling using on
17 chip scalar, and GPU compute. For simplicity of demonstration, only one of the channels will use
18 NVIDIA<sup>&reg;</sup> TensorRT<sup>&trade;</sup> to perform object identification
19 and generate bounding box around the identified object. This sample also uses video converter functions
20 to do various format convrsions and uses EGLImage to demonstrate buffer sharing and image display capability.
21 
22 In this sample, object detection is limited to identifying cars in video streams of 960 x 540 resolution,
23 running up to 14 FPS. The network is based on GoogleNet. The inference is performed on a frame-by-frame basis
24 and no object tracking is involved. Note that this network is intended to be purely an example to showcase
25 TensorRT usage to build the compute pipeline quickly. The trained GoogleNet is provided as source and was trained
26 using NVIDIA DIGITS with roughly 3000 frames taken from 5-10 feet elevation. Varying levels of detection accuracy are expected based on the video samples fed in. Given that this sample is locked to perform at half-HD resolutions under 10 FPS, video feeds with higher FPS for inference will show stuttering during playback.
27 
28 This sample does not require a camera.
29 
30 The image below shows a sample block diagram.
31 
32 ![ ](nvee_backend_one.jpg)
33 
34 The images below shows data flow details for the channel using TensorRT.
35 
36 ![ ](nvee_backend_two.jpg)
37 
38 `NvEGLImageFromFd` is NVIDIA defined API to return an `EGLImage` pointer form the file
39 descriptor buffer that is allocated by way of Tegra mechanism. `EGLImage` buffer is
40 then used by TensorRT to render the bounding box to the image.
41 
42 #### Prerequisites ####
43 Before running the sample, you must have the following:
44 
45 * CUDA Toolkit
46 * TensorRT (previously known as GPU Inference Engine (GIE))
47 * OpenCV4Tegra
48 * README that provides details on the environment requirements to build and run the sample
49 
50 - - - - - - - - - - - - - - -
51 <a name="key">
52 ## Key Structure and Classes ##
53 
54 The `context_t structure` (backend/v4l2_backend_test.h) manages all resources in sample applications.
55 
56 |Element|Description|
57 |-------|-----------|
58 |[NvVideoDecoder](classNvVideoDecoder.html)|Contains all video decoding-related elements and functions.|
59 |[NvVideoConverter](classNvVideoConverter.html)|Contains elements and functions for video format conversion.|
60 |NvEglRenderer|Contains all EGL display rendering-related functions.|
61 |EGLImageKHR|The EGLImage used for CUDA processing. This type is from the EGL open source graphical library.|
62 
63 ### %NvVideoDecoder ###
64 
65 The [NvVideoDecoder](classNvVideoDecoder.html) class creates a new V4L2 Video Decoder.
66 The following table describes the key %NvVideoDecoder members that this sample uses.
67 
68 |Member|Description|
69 |-------------|---|
70 |NvV4l2Element::output_plane |Holds the V4L2 output plane.|
71 |NvV4l2Element::capture_plane |Holds the V4L2 capture plane.|
72 |NvVideoDecoder::createVideoDecoder |Static function to create video decode object.|
73 |NvV4l2Element::subscribeEvent |Subscribes event.|
74 |NvVideoDecoder::setExtControls |Sets external control to V4L2 device.|
75 |NvVideoDecoder::setOutputPlaneFormat |Sets output plane format.|
76 |NvVideoDecoder::setCapturePlaneFormat |Sets capture plane format.|
77 |NvV4l2Element::getControl |Gets the value of a control setting.|
78 |NvV4l2Element::dqEvent |Dequeues the devent reported by the V4L2 device.|
79 |NvV4l2Element::isInError |Checks if under error state.|
80 
81 ### %NvVideoConverter ###
82 
83 The NvVideoConverter class packages all video
84 converting related elements and functions. It performs color space conversion,
85 scaling and conversion between hardware buffer memory and software buffer
86 memory. The following table describes the key %NvVideoConverter members that
87 this sample uses.
88 
89 |Member|Description|
90 |-------------|---|
91 |NvV4l2Element::output_plane |Holds the output plane.|
92 |NvV4l2Element::capture_plane |Holds the capture plane.|
93 |NvVideoConverter::waitForIdle |Waits until all the buffers queued on the output plane are converted and dequeued from the capture plane. This is a blocking method.|
94 |NvVideoConverter::setCapturePlaneFormat |Sets the format on the converter capture plane.|
95 |NvVideoConverter::setOutputPlaneFormat |Sets the format on the converter output plane.|
96 
97 `NvVideoDecoder` and `NvVideoConverter` contain two key elements:
98 `output_plane` and `capture_plane`. These objects are instantiated from the
99 NvV4l2ElementPlane class type.
100 
101 ### %NvV4l2ElementPlane ###
102 
103 [NvV4l2ElementPlane](group__ee__nvv4lelementplane__group.html) creates an [NVv4l2Element](classNvV4l2Element.html) plane.
104 The following table describes the key %NvV4l2ElementPlane members used in this
105 sample. `v4l2_buf` is a local variable inside the NvV4l2ElementPlane::dqThreadCallback
106 function and, thus, the scope exists only inside the callback function. If other
107 functions of the sample must access this buffer, a prior copy of the buffer
108 inside callback function is required.
109 
110 |Member |Description|
111 |-------------------|---|
112 |NvV4l2ElementPlane::setupPlane |Sets up the plane of V4l2 element.|
113 |NvV4l2ElementPlane::deinitPlane |Destroys the plane of V4l2 element.|
114 |NvV4l2ElementPlane::setStreamStatus |Starts/Stops the stream.|
115 |NvV4l2ElementPlane::setDQThreadCallback |Sets the callback function of the `dqueue` buffer thread.|
116 |NvV4l2ElementPlane::startDQThread |Starts the thread of the `dqueue` buffer.|
117 |NvV4l2ElementPlane::stopDQThread |Stops the thread of the `dqueue` buffer.|
118 |NvV4l2ElementPlane::qBuffer |Queues a V4l2 buffer from the plane.|
119 |NvV4l2ElementPlane::dqBuffer |Dequeues a V4l2 buffer from the plane.|
120 |NvV4l2ElementPlane::getNumBuffers |Gets the number of the V4l2 buffer.|
121 |NvV4l2ElementPlane::getNumQueuedBuffers |Gets the number of the V4l2 buffer in the queue.|
122 |NvV4l2ElementPlane::getNthBuffer |Gets the \c %NvBuffer queue object at index N.|
123 
124 
125 ### %GIE_Context ###
126 
127 GIE_Context provides a
128 series of interfaces to load Caffe model and perform inference. The following
129 table describes the key %GIE_Context members used in this sample.
130 
131 |%GIE_Context|Description|
132 |-----------|-----------|
133 |GIE_Context::destroyGieContext |Destroys the GIE_context.|
134 |GIE_Context::getNumGieInstances |Gets the number of GIE_context instances.|
135 |GIE_Context::doInference |Interface for inference after TensorRT model is loaded.|
136 
137 ### Functions to Create/Destroy EGLImage ##
138 
139 The sample uses 2 global functions to create and destroy EGLImage from `dmabuf`
140 file descriptor. These functions are defined in nvbuf_utils.h.
141 
142 |Global Function|Description|
143 |---------------|-----------|
144 |NvEGLImageFromFd() |Creates EGLImage from dmabuf fd.|
145 |NvDestroyEGLImage() |Destroys the EGLImage.|
146 
147 - - - - - - - - - - - - - - -
148 <a name="options">
149 ## Command Line Options ##
150 
151 To run the sample, execute:
152 
153  backend <channel-num> <in-file1> <in-file2>... <in-format> [options]
154 
155 The following video formats are supported for use with command line options:
156 
157  * H.264
158  * H.265
159 
160 ### Options ####
161 
162 Use the `-h` option to view the currently supported options.
163 
164 For X11 technical details, see:
165 
166 http://www.x.org/docs/X11/xlib.pdf
static NvVideoDecoder * createVideoDecoder(const char *name, int flags=0)
Creates a new V4L2 Video Decoder object named name.
void destroyGieContext(bool bUseCPUBuf=false)
Defines a helper class for V4L2 Video Decoder.
int dqEvent(struct v4l2_event &event, uint32_t max_wait_ms)
Dequeues an event from the element.
Defines a helper class for operations performed on a V4L2 Element plane.
void doInference(queue< vector< cv::Rect > > *rectList_queue, float *input=NULL)
int getNumGieInstances() const
int setCapturePlaneFormat(uint32_t pixfmt, uint32_t width, uint32_t height, enum v4l2_nv_buffer_layout type)
Sets the format on the converter output plane.
NvV4l2ElementPlane capture_plane
Sets the capture plane.
Class representing a buffer.
Definition: NvBuffer.h:85
NvV4l2ElementPlane output_plane
Sets the output plane.
int setOutputPlaneFormat(uint32_t pixfmt, uint32_t sizeimage)
Sets the format on the decoder output plane.
virtual int isInError()
EGLImageKHR NvEGLImageFromFd(EGLDisplay display, int dmabuf_fd)
This method must be used for getting EGLImage from dmabuf-fd.
int getControl(uint32_t id, int32_t &value)
Gets the value of a control.
Defines a helper class for V4L2 Video Converter.
int setCapturePlaneFormat(uint32_t pixfmt, uint32_t width, uint32_t height)
Sets the format on the decoder output plane.
int setExtControls(struct v4l2_ext_controls &ctl)
Sets the value of several controls.
int waitForIdle(uint32_t max_wait_ms)
Waits until all buffers queued on the output plane are converted and dequeued from the capture plane...
NvEglRenderer is a helper class for rendering using EGL and OpenGL ES 2.0.
Definition: NvEglRenderer.h:74
int subscribeEvent(uint32_t type, uint32_t id, uint32_t flags)
Subscribes to an V4L2 event.
int setOutputPlaneFormat(uint32_t pixfmt, uint32_t width, uint32_t height, enum v4l2_nv_buffer_layout type)
Sets the format on the converter output plane.
int NvDestroyEGLImage(EGLDisplay display, EGLImageKHR eglImage)
This method must be used for destroying EGLImage object.