Delay Object Usage

This tutorial demonstrates delay object usage.

A delay is a special kind of VisionWorks framework objects that contains a temporally-delayed fixed-size list of other data objects. All objects in delay have the same metadata (dimensions, format, etc.). Delay object might be used in scenarios like feature tracking or video stabilization, which requires processing a sequence of frames from input video stream.

This tutorial implements simple feature tracking pipeline based on the Optical Flow PyrLK vision primitive. It tracks features from Frame # (i - 2) to Frame # i. For this purpose, three delay objects are used: the first for input frames, the second for gaussian pyramids, and the third for features arrays.

The first step is to create the delay objects. The vxCreateDelay method accepts the number of slots in the delay object and exemplar object. The exemplar is used only to pass metadata to the delay's constructor. Note that vxCreateImage and similar methods do not allocate memory for objects (VisionWorks uses delayed allocation scheme), so the exemplar creation is a fast operation.

int num_slots = 3;

vx_image frame_exemplar = vxCreateImage(context, frameWidth, frameHeight, format);
vx_delay frame_delay = vxCreateDelay(context, (vx_reference)frame_exemplar, num_slots);
vxReleaseImage(&frame_exemplar);

vx_pyramid pyramid_exemplar = vxCreatePyramid(context, num_levels, VX_SCALE_PYRAMID_HALF,
                                              frameWidth, frameHeight, VX_DF_IMAGE_U8);
vx_delay pyramid_delay = vxCreateDelay(context, (vx_reference)pyramid_exemplar, num_slots);
vxReleasePyramid(&pyramid_exemplar);

vx_array array_exemplar = vxCreateArray(context, NVX_TYPE_POINT2F, 1000);
vx_delay array_delay = vxCreateDelay(context, (vx_reference)array_exemplar, num_slots);
vxReleaseArray(&array_exemplar);

The next step is to initialize the delay objects with the first two frames from video sequence. To get the data object from delay's slot by index, the vxGetReferenceFromDelay method is used. The delay index is in the range \( [-num\_slots + 1, 0] \), where 0 corresponds to the current object.

nvxio::FrameSource::FrameStatus frameStatus;
vx_image frame_grayscale = vxCreateImage(context, frameWidth, frameHeight, VX_DF_IMAGE_U8);

for (int i = -num_slots + 1; i < 0; ++i)
{
    vx_image frame_i = (vx_image)vxGetReferenceFromDelay(frame_delay, i);
    vx_pyramid pyramid_i = (vx_pyramid)vxGetReferenceFromDelay(pyramid_delay, i);
    vx_array array_i = (vx_array)vxGetReferenceFromDelay(array_delay, i);

    do
    {
        frameStatus = frameSource->fetch(frame_i);

        if (frameStatus == nvxio::FrameSource::CLOSED)
        {
            printf("Not enough frames in video stream\n");
            fflush(stdout);
            exit(1);
        }
    } while (frameStatus != nvxio::FrameSource::OK);

    vxuColorConvert(context, frame_i, frame_grayscale);
    vxuGaussianPyramid(context, frame_grayscale, pyramid_i);
    vxuHarrisCorners(context, frame_grayscale, strength_threshold, min_distance,
                     sensitivity, gradient_size, block_size, array_i, NULL);
}

vxReleaseImage(&frame_grayscale);

After the initialization, the delay has the following structure:

The third step is graph creation for the following pipeline:

Delay's slots can be used as nodes parameters. In this case, nodes are bound to a specific delay slot and not to its content. When delay is aged, nodes use the same slot with updated data.

vx_image current_frame = (vx_image)vxGetReferenceFromDelay(frame_delay, 0);

vx_pyramid previous_pyramid = (vx_pyramid)vxGetReferenceFromDelay(pyramid_delay, -num_slots + 1);
vx_pyramid current_pyramid = (vx_pyramid)vxGetReferenceFromDelay(pyramid_delay, 0);

vx_array previous_array = (vx_array)vxGetReferenceFromDelay(array_delay, -num_slots + 1);
vx_array current_array = (vx_array)vxGetReferenceFromDelay(array_delay, 0);

vx_graph graph = vxCreateGraph(context);

vx_image virt_grayscale = vxCreateVirtualImage(graph, 0, 0, VX_DF_IMAGE_U8);
vxColorConvertNode(graph, current_frame, virt_grayscale);
vxGaussianPyramidNode(graph, virt_grayscale, current_pyramid);
vxOpticalFlowPyrLKNode(graph, previous_pyramid, current_pyramid,
                       previous_array, previous_array, current_array,
                       termination, epsilon, num_iterations, use_initial_estimate,
                       window_dimension);

if (vxVerifyGraph(graph) != VX_SUCCESS)
{
    printf("Graph verification failed, see [NVX LOG] for details\n");
    fflush(stdout);
    exit(1);
}

Now you can write a loop that fetches frames from a video stream and processes them.

for (;;)
{
    frameStatus = frameSource->fetch(current_frame);
    if (frameStatus == nvxio::FrameSource::TIMEOUT)
    {
        continue;
    }
    if (frameStatus == nvxio::FrameSource::CLOSED)
    {
        printf("Finished processing\n");
        fflush(stdout);
        break;
    }

    vxProcessGraph(graph);

After the first fetch, the delay has the following structure:

The graph uses Frame #0 and Frame #2 as input for the Optical Flow primitive.

The last step is to age delay. This means that the reference from Slot 0 goes to Slot -1, reference from Slot -1 goes to Slot -2, etc.

    vxAgeDelay(frame_delay);
    vxAgeDelay(pyramid_delay);
    vxAgeDelay(array_delay);
}

After the aging, the delay has the following structure:

The next fetch fills Slot 0 with Frame #3 and the graph uses Frame #1 and Frame #3 as input for Optical Flow.