# GstAnalyticsSegmentationMtd

This type of metadata holds information on which pixels belongs to a region of the image representing a type of object.

It supports two types of segmentation, semantic or instance:

- Semantic: All objects of the same type have the same id
- Instance: Each instance of an object has a different id

The results of the segmentation are stored in a GstBuffer that has a GstVideoMeta associated with it. This buffer is stored in the GstAnalyticsSegmentationMtd using gst_analytics_relation_meta_add_segmentation_mtd(). The GstBuffer containing the segmentation mask is image-like but the color values are arbitrary values, referred by region-id in this API, without meaning beyond specifying that two pixels in the original image with the same values in their corresponding mask value belong to the same region.

To further describe a region, the GstAnalyticsSegmentationMtd can be associated with other GstAnalyticsMtd. Since region ids are generated by the segmentation process itself and are not always sequential, we use a map of indexes to region ids starting with 0 without discontinuity which facilitate N-to-N mapping with other GstAnalyticsMtd. For example it can be associated with GstAnalyticsClsMtd to describe the class of object matching the pixels of a segmented region.

Example: Associate Instance Segmentation with Classification

In the following example the segmentation process will fill segmask with values of 0 for background, 12 for the first region which correspond to a to a strawberry, 7 for the second region that also correspond to a strawberry in the image and 31 for the third region that correspond to a leaf in the image. region_ids is fill during segmentation post-processing

region_ids: |region-index | region-id |

0 | 0
1 | 12
2 | 7
3 | 31

region_count = 4

      GstAnalyticsSegmentationMtd segmtd;
      GstAnalyticsClassificationMtd clsmtd;
      GstBuffer *segmask, *img;
      guint *region_ids;
      gsize region_count, class_count;
      gfloat *class_confidence;
      GQuark *classes;

      ... (segmentation filling segmask based on img)

      gst_analytics_relation_meta_add_segmentation_mtd (rmeta, segmask,
        GST_SEGMENTATION_TYPE_INSTANCE, region_count, region_ids, &segmtd);
      class_count = region_count;

      ... (class-index must match and correspond to region-index)
      classes [0]  = g_quark_from_string ("background");
      classes [1]  = g_quark_from_string ("strawberry");
      classes [2] = g_quark_from_string ("strawberry");
      classes [3] = g_quark_from_string ("leaf");

      ... (set confidence level for each class associated with a region
      ... where -1.0 mean undefined.)
      class_confidence [0] = -1.0;
      class_confidence [1] = 0.6;
      class_confidence [2] = 0.9;
      class_confidence [3] = 0.8;

      gst_analytics_relation_meta_add_cls_mtd (rmeta, class_count,
        class_confidence, classes, &clsmtd);

      gst_analytics_relation_meta_set_relation (rmeta,
       GST_ANALYTICS_REL_TYPE_RELATE_TO, segmtd.id, clsmtd.id);

Example: Associate Semantic Segmentation with Classification Assuming the same context as for Instance Segmentation above but instead a semantic segmentation is performed, therefore region-id-12 and region-id-7 are now represented by the same region-id-12

region_ids: (here |region-index | region-id |

0 | 0
1 | 12
2 | 31

Code remain the same except that we set all confidence level to undefined (-1.0).

      ... (class-index must match and correspond to region-index)
      classes [0]  = g_quark_from_string ("background");
      classes [1]  = g_quark_from_string ("strawberry");
      classes [2] = g_quark_from_string ("leaf");

      ... (set confidence level for each class associated with a region
      ... where -1.0 mean undefined.)
      class_confidence [0] = -1.0;
      class_confidence [1] = -1.0;
      class_confidence [2] = -1.0;

      gst_analytics_relation_meta_add_cls_mtd (rmeta, class_count,
        class_confidence, classes, &clsmtd);

      gst_analytics_relation_meta_set_relation (rmeta,
       GST_ANALYTICS_REL_TYPE_RELATE_TO, segmtd.id, clsmtd.id);

Example: Retrieving class associated with a segmentation region-id-12 This the typical case for an overlay as we visit the segmentation mask we we find region-id values

    gsize idx;
    gst_analytics_segmentation_mtd_get_region_index (&segmtd, &idx, 12);
    gst_analytics_relation_meta_get_direct_related (rmeta, segmtd.id,
      GST_ANALYTICS_REL_TYPE_RELATE_TO, gst_analytics_cls_mtd_get_type (),
      NULL, &clsmtd);

     GQuark region_class = gst_analytics_cls_mtd_get_quark (&segmtd, idx)
     ...

## GstAnalyticsSegmentationMtd

Handle containing data required to use gst_analytics_segmentation_mtd APIs. This type is generally expected to be allocated on the stack.

### Members

`id` (guint) –

Instance identifier

`meta` (GstAnalyticsRelationMeta *) –

Instance of GstAnalyticsRelationMeta where the analytics-metadata identified by id is stored.

Since : 1.26

## GstAnalytics.SegmentationMtd

Handle containing data required to use gst_analytics_segmentation_mtd APIs. This type is generally expected to be allocated on the stack.

### Members

`id` (Number) –

Instance identifier

`meta` (GstAnalytics.RelationMeta) –

Instance of GstAnalytics.RelationMeta where the analytics-metadata identified by id is stored.

Since : 1.26

## GstAnalytics.SegmentationMtd

Handle containing data required to use gst_analytics_segmentation_mtd APIs. This type is generally expected to be allocated on the stack.

### Members

`id` (int) –

Instance identifier

`meta` (GstAnalytics.RelationMeta) –

Instance of GstAnalytics.RelationMeta where the analytics-metadata identified by id is stored.

Since : 1.26

### Methods

#### gst_analytics_segmentation_mtd_get_mask

    GstBuffer *
    gst_analytics_segmentation_mtd_get_mask (const GstAnalyticsSegmentationMtd * handle,
                                             gint * masks_loc_x,
                                             gint * masks_loc_y,
                                             guint * masks_loc_w,
                                             guint * masks_loc_h)

Get segmentation mask data.

Parameters:

`handle` –

Instance

`masks_loc_x` ( [out][nullable]) –

Left coordinate of the rectangle corresponding to the mask in the image.

`masks_loc_y` ( [out][nullable]) –

Top coordinate of the rectangle corresponding to the mask in the image.

`masks_loc_w` ( [out][nullable]) –

Width of the rectangle corresponding to the mask in the image.

`masks_loc_h` ( [out][nullable]) –

Height of the rectangle corresponding to the mask in the image.

Returns –

Segmentation mask data stored in a GstBuffer

Since : 1.26

#### GstAnalytics.SegmentationMtd.prototype.get_mask

    function GstAnalytics.SegmentationMtd.prototype.get_mask(): {
        // javascript wrapper for 'gst_analytics_segmentation_mtd_get_mask'
    }

Get segmentation mask data.

Parameters:

`handle` (GstAnalytics.SegmentationMtd) –

Instance

Returns a tuple made of:

(Gst.Buffer ) –

Segmentation mask data stored in a Gst.Buffer

`masks_loc_x` (Number ) –

Segmentation mask data stored in a Gst.Buffer

`masks_loc_y` (Number ) –

Segmentation mask data stored in a Gst.Buffer

`masks_loc_w` (Number ) –

Segmentation mask data stored in a Gst.Buffer

`masks_loc_h` (Number ) –

Segmentation mask data stored in a Gst.Buffer

Since : 1.26

#### GstAnalytics.SegmentationMtd.get_mask

    def GstAnalytics.SegmentationMtd.get_mask (self):
        #python wrapper for 'gst_analytics_segmentation_mtd_get_mask'

Get segmentation mask data.

Parameters:

`handle` (GstAnalytics.SegmentationMtd) –

Instance

Returns a tuple made of:

(Gst.Buffer ) –

Segmentation mask data stored in a Gst.Buffer

`masks_loc_x` (int ) –

Segmentation mask data stored in a Gst.Buffer

`masks_loc_y` (int ) –

Segmentation mask data stored in a Gst.Buffer

`masks_loc_w` (int ) –

Segmentation mask data stored in a Gst.Buffer

`masks_loc_h` (int ) –

Segmentation mask data stored in a Gst.Buffer

Since : 1.26

#### gst_analytics_segmentation_mtd_get_region_count

    gsize
    gst_analytics_segmentation_mtd_get_region_count (const GstAnalyticsSegmentationMtd * handle)

Get the regions count.

Parameters:

`handle` –

Instance

Returns –

Number of regions segmented

Since : 1.26

#### GstAnalytics.SegmentationMtd.prototype.get_region_count

    function GstAnalytics.SegmentationMtd.prototype.get_region_count(): {
        // javascript wrapper for 'gst_analytics_segmentation_mtd_get_region_count'
    }

Get the regions count.

Parameters:

`handle` (GstAnalytics.SegmentationMtd) –

Instance

Returns (Number) –

Number of regions segmented

Since : 1.26

#### GstAnalytics.SegmentationMtd.get_region_count

    def GstAnalytics.SegmentationMtd.get_region_count (self):
        #python wrapper for 'gst_analytics_segmentation_mtd_get_region_count'

Get the regions count.

Parameters:

`handle` (GstAnalytics.SegmentationMtd) –

Instance

Returns (int) –

Number of regions segmented

Since : 1.26

#### gst_analytics_segmentation_mtd_get_region_id

    guint
    gst_analytics_segmentation_mtd_get_region_id (const GstAnalyticsSegmentationMtd * handle,
                                                  gsize index)

Get id of the region corresponding to index, which should be smaller than the return value of gst_analytics_segmentation_mtd_get_region_count

Parameters:

`handle` –

Instance

`index` –

Region index

Returns –

The region ID

Since : 1.26

#### GstAnalytics.SegmentationMtd.prototype.get_region_id

    function GstAnalytics.SegmentationMtd.prototype.get_region_id(index: Number): {
        // javascript wrapper for 'gst_analytics_segmentation_mtd_get_region_id'
    }

Get id of the region corresponding to index, which should be smaller than the return value of GstAnalytics.SegmentationMtd.prototype.get_region_count

Parameters:

`handle` (GstAnalytics.SegmentationMtd) –

Instance

`index` (Number) –

Region index

Returns (Number) –

The region ID

Since : 1.26

#### GstAnalytics.SegmentationMtd.get_region_id

    def GstAnalytics.SegmentationMtd.get_region_id (self, index):
        #python wrapper for 'gst_analytics_segmentation_mtd_get_region_id'

Get id of the region corresponding to index, which should be smaller than the return value of GstAnalytics.SegmentationMtd.get_region_count

Parameters:

`handle` (GstAnalytics.SegmentationMtd) –

Instance

`index` (int) –

Region index

Returns (int) –

The region ID

Since : 1.26

#### gst_analytics_segmentation_mtd_get_region_index

    gboolean
    gst_analytics_segmentation_mtd_get_region_index (const GstAnalyticsSegmentationMtd * handle,
                                                     gsize * index,
                                                     guint id)

Get region index of the region identified by id.

Parameters:

`handle` –

Instance

`index` ( [out][not nullable]) –

Region index

`id` –

Region id

Returns –

TRUE if a region with id exist, otherwise FALSE

Since : 1.26

#### GstAnalytics.SegmentationMtd.prototype.get_region_index

    function GstAnalytics.SegmentationMtd.prototype.get_region_index(id: Number): {
        // javascript wrapper for 'gst_analytics_segmentation_mtd_get_region_index'
    }

Get region index of the region identified by id.

Parameters:

`handle` (GstAnalytics.SegmentationMtd) –

Instance

`id` (Number) –

Region id

Returns a tuple made of:

(Number ) –

TRUE if a region with id exist, otherwise FALSE

`index` (Number ) –

TRUE if a region with id exist, otherwise FALSE

Since : 1.26

#### GstAnalytics.SegmentationMtd.get_region_index

    def GstAnalytics.SegmentationMtd.get_region_index (self, id):
        #python wrapper for 'gst_analytics_segmentation_mtd_get_region_index'

Get region index of the region identified by id.

Parameters:

`handle` (GstAnalytics.SegmentationMtd) –

Instance

`id` (int) –

Region id

Returns a tuple made of:

(bool ) –

TRUE if a region with id exist, otherwise FALSE

`index` (int ) –

TRUE if a region with id exist, otherwise FALSE

Since : 1.26

### Functions

#### gst_analytics_segmentation_mtd_get_mtd_type

    GstAnalyticsMtdType
    gst_analytics_segmentation_mtd_get_mtd_type ()

Get an instance of GstAnalyticsMtdType that represent segmentation metadata type.

Returns –

A GstAnalyticsMtdType type

Since : 1.26

#### GstAnalytics.SegmentationMtd.prototype.get_mtd_type

    function GstAnalytics.SegmentationMtd.prototype.get_mtd_type(): {
        // javascript wrapper for 'gst_analytics_segmentation_mtd_get_mtd_type'
    }

Get an instance of Number that represent segmentation metadata type.

Returns (Number) –

A Number type

Since : 1.26

#### GstAnalytics.SegmentationMtd.get_mtd_type

    def GstAnalytics.SegmentationMtd.get_mtd_type ():
        #python wrapper for 'gst_analytics_segmentation_mtd_get_mtd_type'

Get an instance of int that represent segmentation metadata type.

Returns (int) –

A int type

Since : 1.26

## Enumerations

### GstSegmentationType

Enum value describing supported segmentation type

##### Members

`GST_SEGMENTATION_TYPE_SEMANTIC` (0) –

Segmentation where the belonging of each pixel to a class of objects is identified.

`GST_SEGMENTATION_TYPE_INSTANCE` (1) –

Segmentation where the belonging of each pixel to instance of an object is identified.

Since : 1.26

### GstAnalytics.SegmentationType

Enum value describing supported segmentation type

##### Members

`GstAnalytics.SegmentationType.SEMANTIC` (0) –

Segmentation where the belonging of each pixel to a class of objects is identified.

`GstAnalytics.SegmentationType.INSTANCE` (1) –

Segmentation where the belonging of each pixel to instance of an object is identified.

Since : 1.26

### GstAnalytics.SegmentationType

Enum value describing supported segmentation type

##### Members

`GstAnalytics.SegmentationType.SEMANTIC` (0) –

Segmentation where the belonging of each pixel to a class of objects is identified.

`GstAnalytics.SegmentationType.INSTANCE` (1) –

Segmentation where the belonging of each pixel to instance of an object is identified.

Since : 1.26

The results of the search are