1 Copyright (c) 2016-2017, NVIDIA CORPORATION. All rights reserved.
2 <!-- Sample is in the backend dir. -->
4 @page nvvid_backend_group Backend
7 - [Overview](#overview)
8 - [Building and Running](#build_and_run)
10 - [Command Line Options](#options)
11 - [Key Structure and Classes](#key)
13 - - - - - - - - - - - - - - -
17 This application implements a typical appliance performing intelligent video analytics.
18 Application areas include
public safety, smart cities, and autonomous machines. This example demonstrates
19 four (4) concurrent video streams going through a decoding process
using the on-chip decoders, video scaling
using on
20 chip scalar, and GPU compute. For simplicity of demonstration, only one of the channels uses
21 NVIDIA<sup>®</sup> TensorRT<sup>™</sup> to perform
object identification
22 and generate bounding box around the identified
object. This sample also uses video converter functions
23 for various format conversions. It also uses EGLImage to demonstrate buffer sharing and image display.
25 In
this sample,
object detection is limited to identifying cars in video streams
26 of 960 x 540 resolution, running up to 14 FPS. The network is based on
27 GoogleNet. The inference is performed on a frame-by-frame basis and no
object
28 tracking is involved. Note that
this network is intended to be an example
29 that shows how to use TensorRT to quickly build the compute pipeline. The sample
30 includes trained GoogleNet, which was trained with NVIDIA Deep Learning GPU
31 Training System (DIGITS). The training was done with roughly
32 3000 frames taken from 5-10 feet elevation. Varying levels of detection accuracy
33 are expected based on the video samples fed in. Given that
this sample is locked
34 to perform at half-HD resolutions under 10 FPS, video feeds with higher FPS
for
35 inference will show stuttering during playback.
37 This sample does not require a camera.
39 <a name=
"build_and_run">
40 ## Building and Running ##
42 #### Prerequisites ####
43 * You have followed Steps 1-3 in @ref mmapi_build.
47 * Optionally, you have installed NVIDIA<sup>®</sup>
48 TensorRT<sup>™</sup> (previously known as GPU Inference Engine (GIE))
51 1. If you want to run the sample without TensorRT, set the following in the Makefile:
54 By
default, TensorRT is enabled.
64 $ ./backend 1 ../../data/Video/sample_outdoor_car_1080p_10fps.h264 H264 \
65 --trt-deployfile ../../data/Model/GoogleNet_one_class/GoogleNet_modified_oneClass_halfHD.prototxt \
66 --trt-modelfile ../../data/Model/GoogleNet_one_class/GoogleNet_modified_oneClass_halfHD.caffemodel \
67 --trt-forcefp32 0 --trt-proc-interval 1 -fps 10
69 @note The TensorRT batch size can be configured from the third line of the following file:
71 ../../data/Model/GoogleNet_one_class/GoogleNet_modified_oneClass_halfHD.prototxt
72 Where the valid values are 1(
default), or 2, or 4.
74 . <!-- leave
this period alone. It forces Doxygen to stop indenting. -->
79 ### To view command-line options
87 - - - - - - - - - - - - - - -
90 The following image shows the movement of data through the sample when TensorRT
93 
95 The following image shows data flow details
for the channel
using TensorRT.
97 
99 `
NvEGLImageFromFd` is an NVIDIA API that returns an `EGLImage` pointer from the file
100 descriptor buffer that is allocated via the Tegra mechanism. TensorRT then uses
101 the `EGLImage` buffer to render the bounding box to the image.
104 For X11 technical details, see:
108 - - - - - - - - - - - - - - -
110 ## Key Structure and Classes ##
112 The `context_t structure` (backend/v4l2_backend_test.h) manages all resources in sample applications.
114 |Element|Description|
115 |-------|-----------|
116 |[
NvVideoDecoder](classNvVideoDecoder.html)|Contains all video decoding-related elements and functions.|
117 |[
NvVideoConverter](classNvVideoConverter.html)|Contains elements and functions
for video format conversion.|
118 |
NvEglRenderer|Contains all EGL display rendering-related functions.|
119 |EGLImageKHR|The EGLImage used
for CUDA processing. This type is from the EGL open source graphical library.|
123 The [
NvVideoDecoder](classNvVideoDecoder.html)
class creates a new V4L2 Video Decoder.
124 The following table describes the key %
NvVideoDecoder members that
this sample uses.
139 ### %NvVideoConverter ###
142 converting related elements and functions. It performs color space conversion,
143 scaling and conversion between hardware buffer memory and software buffer
144 memory. The following table describes the key %
NvVideoConverter members that
151 |
NvVideoConverter::waitForIdle |Waits until all the buffers queued on the output plane are converted and dequeued from the capture plane. This is a blocking method.|
156 `output_plane` and `capture_plane`. These objects are instantiated from the
161 [
NvV4l2ElementPlane](group__l4t_mm__nvv4lelementplane__group.html) creates an [NVv4l2Element](classNvV4l2Element.html) plane.
163 sample. `v4l2_buf` is a local variable inside the
NvV4l2ElementPlane::dqThreadCallback
164 function and, thus, the scope exists only inside the callback function. If other
165 functions of the sample must access this buffer, a prior copy of the buffer
166 inside callback function is required.
168 |Member |Description|
169 |-------------------|---|
173 |
NvV4l2ElementPlane::setDQThreadCallback |Sets the callback function of the `dqueue` buffer thread.|
179 |
NvV4l2ElementPlane::getNumQueuedBuffers |Gets the number of the V4l2 buffer in the queue.|
186 series of interfaces to load Caffe model and perform inference. The following
187 table describes the key %
TRT_Context members used in
this sample.
190 |-----------|-----------|
195 ### Functions to Create/Destroy EGLImage ##
197 The sample uses 2 global functions to create and destroy EGLImage from `dmabuf`
198 file descriptor. These functions are defined in nvbuf_utils.h.
200 |Global Function|Description|
201 |---------------|-----------|
static NvVideoDecoder * createVideoDecoder(const char *name, int flags=0)
Creates a new V4L2 Video Decoder object named name.
void destroyTrtContext(bool bUseCPUBuf=false)
void doInference(queue< vector< cv::Rect > > *rectList_queue, float *input=NULL)
Defines a helper class for V4L2 Video Decoder.
int dqEvent(struct v4l2_event &event, uint32_t max_wait_ms)
Dequeues an event from the element.
Defines a helper class for operations performed on a V4L2 Element plane.
int setCapturePlaneFormat(uint32_t pixfmt, uint32_t width, uint32_t height, enum v4l2_nv_buffer_layout type)
Sets the format on the converter output plane.
NvV4l2ElementPlane capture_plane
Sets the capture plane.
Class representing a buffer.
NvV4l2ElementPlane output_plane
Sets the output plane.
int setOutputPlaneFormat(uint32_t pixfmt, uint32_t sizeimage)
Sets the format on the decoder output plane.
EGLImageKHR NvEGLImageFromFd(EGLDisplay display, int dmabuf_fd)
This method must be used for getting EGLImage from dmabuf-fd.
int getControl(uint32_t id, int32_t &value)
Gets the value of a control.
Defines a helper class for V4L2 Video Converter.
int setCapturePlaneFormat(uint32_t pixfmt, uint32_t width, uint32_t height)
Sets the format on the decoder output plane.
int setExtControls(struct v4l2_ext_controls &ctl)
Sets the value of several controls.
uint32_t getNumTrtInstances() const
int waitForIdle(uint32_t max_wait_ms)
Waits until all buffers queued on the output plane are converted and dequeued from the capture plane...
NvEglRenderer is a helper class for rendering using EGL and OpenGL ES 2.0.
int subscribeEvent(uint32_t type, uint32_t id, uint32_t flags)
Subscribes to an V4L2 event.
int setOutputPlaneFormat(uint32_t pixfmt, uint32_t width, uint32_t height, enum v4l2_nv_buffer_layout type)
Sets the format on the converter output plane.
int NvDestroyEGLImage(EGLDisplay display, EGLImageKHR eglImage)
This method must be used for destroying EGLImage object.