Video Processing with Graph Mode

This tutorial demonstrates video processing with graph mode.

The tutorial uses the following pipeline as an example:

(source image) -> [Median 3x3] -> [Equalize Hist] -> [Canny Edge Detector] -> (edges)

VisionWorks has two modes for vision primitive execution : immediate mode and graph mode. In graph mode, primitives are instantiated as graph nodes. The graph then is built and verified, and can be executed multiple times without re-verification. The graph mode is more suitable for video processing since it gives the best performance.

First, the video is opened using the NVXIO library and an vx_image object is created to hold frames from the video stream.

std::unique_ptr<nvxio::FrameSource> frameSource(
    nvxio::createDefaultFrameSource(context, sourceUri));

if (!frameSource->open())
{
    std::cerr << "Error: Can't open source URI " << sourceUri << std::endl;
    exit(1);
}

nvxio::FrameSource::Parameters frameConfig = frameSource->getConfiguration();
vx_image frame = vxCreateImage(context, frameConfig.frameWidth, frameConfig.frameHeight, frameConfig.format);

Second, the graph is created. Note that virtual images are used for intermediate results. Virtual images cannot be accessed by application and can be optimized by graph manager. Also, for virtual images, metadata can be skipped if the meta-data can be deduced from the node's input.

vx_graph graph = vxCreateGraph(context);

vx_image virt_grayscale = vxCreateVirtualImage(graph, 0, 0, VX_DF_IMAGE_U8);
vx_image virt_filtered = vxCreateVirtualImage(graph, 0, 0, VX_DF_IMAGE_VIRT);
vx_image virt_equalized = vxCreateVirtualImage(graph, 0, 0, VX_DF_IMAGE_VIRT);

vxColorConvertNode(graph, frame, virt_grayscale);
vxMedian3x3Node(graph, virt_grayscale, virt_filtered);
vxEqualizeHistNode(graph, virt_filtered, virt_equalized);
vxCannyEdgeDetectorNode(graph, virt_equalized, hyst, 3, VX_NORM_L1, edges);

After the graph is created, it must be verified. During the verification stage, the graph checks connections between nodes, checks input and output parameters, performs additional optimizations, allocates memory for data objects, and more. This is done ahead of time so that the processing stage can avoid this overhead.

if (vxVerifyGraph(graph) != VX_SUCCESS)
{
    printf("Graph verification failed, see [NVX LOG] for details\n");
    fflush(stdout);
    exit(1);
}

To display results NVXIO library is used. For that purpose the nvxio::Render object is created, which will be represented as a window by default.

std::unique_ptr<nvxio::Render> render(
    nvxio::createDefaultRender(context, "Basic tutorial",
                               frameConfig.frameWidth, frameConfig.frameHeight));

The next step is a loop that grabs frame from the video source. nvxio::FrameSource::fetch method is used for this purpose. The method has a timeout to wait for the next frame; thus, you must check that the fetch call was successful.

for (;;)
{
    nvxio::FrameSource::FrameStatus frameStatus = frameSource->fetch(frame);

    if (frameStatus == nvxio::FrameSource::TIMEOUT)
    {
        continue;
    }

    if (frameStatus == nvxio::FrameSource::CLOSED)
    {
        printf("Finished processing\n");
        fflush(stdout);
        break;
    }

    printf("Processing frame\n");
    fflush(stdout);

After the next frame is grabbed, it can be processed by the graph.

    vxProcessGraph(graph);

Finally, results are displayed using the Render object. Note that the nvxio::Render::flush method must be called at the end to ensure that all rendering commands are finished and to process OS events.

    render->putImage(edges);

    if (!render->flush())
    {
        printf("Finished processing\n");
        fflush(stdout);
        break;
    }
}