标签:desktop lag clu 定义 总结 switch messages ram class
在上一篇文章中,我们介绍了如何将多个element连接起来构造一个pipline,进行数据传输。那么GStreamer是通过何种方式保证element之间能正常的进行数据传输?今天就将介绍GStreamer是如何利用Pad来控制数据的传输。
我们知道,pad是element之间的数据的接口,一个src pad只能与一个sink pad相连。每个element可以通过pad过滤数据,接收自己支持的数据类型。Pad通过Pad Capabilities(简称为Pad Caps)来描述支持的数据类型。例如:
“video/x-raw,format=RGB,width=300,height=200,framerate=30/1”
“audio/x-raw,format=S16LE,rate=44100,channels=2”
“audio/x-vorbis” "video/x-vp8"
一个Pad可以支持多种类型的Caps(比如一个video sink可以同时支持RGB或YUV格式的数据),同时可以指定Caps支持的数据范围(比如一个audio sink可以支持1~48k的采样率)。但是,在一个Pipeline中,Pad之间所传输的数据类型必须是唯一的。GStreamer在进行element连接时,会通过协商(negotiation)的方式选择一个双方都支持的类型。
因此,为了能使两个Element能够正确的连接,双方的Pad Caps之间必须有交集,从而在协商阶段选择相同的数据类型,这就是Pad Caps的主要作用。在实际使用中,我们可以通过gst-inspect工具查看Element所支持的Pad Caps,从而才能知道在连接出错时如何处理。
我们曾使用gst_element_factory_make()接口创建Element,这个接口内部也会先创建一个Element 工厂,再通过工厂方法创建一个Element。由于大部分Element都需要创建类似的Pad,于是GStreame定义了Pad Template,Pad Template被包含中Element工厂中,在创建Element时,用于快速创建Pad。
Pad Template包含了一个Pad所能支持的所有Caps。通过Pad Template,我们可以快速的判断两个pad是否能够连接(比如两个elements都只提供了sink template,这样的element之间是无法连接的,这样就没必要进一步判断Pad Caps)。
由于Pad Template属于Element工厂,所以我们可以直接使用gst-inspect查看其属性,但Element实际的Pad会根据Element所处的不同状态来进行实例化,具体的Pad Caps会在协商后才会被确定。
我们看一个 “gst-inspect-1.0 alsasink”的例子(不同平台会有差异):
Pad Templates: SINK template: ‘sink‘ Availability: Always Capabilities: audio/x-raw format: S16LE layout: interleaved rate: [ 1, 48000 ] channels: [ 1, 2 ] audio/x-ac3 framed: true
alsasink只提供了一个sink template,可以创建sink pad,并且是一直存在的。支持两种类型的音频数据:16位的PCM(audio/x-raw),采样率1~48k,1-2通道和AC3(audio/x-ac3)的帧数据。
再看一个 “gst-inspect-1.0 videotestsrc”的例子:
Pad Templates: SRC template: ‘src‘ Availability: Always Capabilities: video/x-raw format: { I420, YV12, YUY2, UYVY, AYUV, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, v210, v216, NV12, NV21, NV16, NV24, GRAY8, GRAY16_BE, GRAY16_LE, v308, RGB16, BGR16, RGB15, BGR15, UYVP, A420, RGB8P, YUV9, YVU9, IYU1, ARGB64, AYUV64, r210, I420_10LE, I420_10BE, I422_10LE, I422_10BE, Y444_10LE, Y444_10BE, GBR, GBR_10LE, GBR_10BE } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] video/x-bayer format: { bggr, rggb, grbg, gbrg } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ]
videotestsrc只提供了一个src template用于创建src pad,pad支持多种格式,可以通过参数指定输出的数据类型或Caps Filter指定。
上面的例子中显示的Pad Template都是一直存在的(Availability: Always),创建的Pad也是一直有效的。但有些Element会根据输入数据以及后续的Element动态增加或删除Pad,因此GStreamer提供了3种Pad有效性的状态:Always,Sometimes,On request。
在element被初始化后就存在的pad,被称为always pad或static pad。
根据输入数据的不同而产生的pad,被称为sometimes pad,常见于各种文件格式解析器。例如用于解析mp4文件的qtdemux:"gst-inspect-1.0 qtdemux"
Pad Templates: SINK template: ‘sink‘ Availability: Always Capabilities: video/quicktime video/mj2 audio/x-m4a application/x-3gp SRC template: ‘video_%u‘ Availability: Sometimes Capabilities: ANY SRC template: ‘audio_%u‘ Availability: Sometimes Capabilities: ANY SRC template: ‘subtitle_%u‘ Availability: Sometimes Capabilities: ANY
只有我们从mp4文件中读取数据时,我们才能知道这个文件中包含多少音频,视频,字幕,所以这些src pad都是sometimes pad。
按需创建的pad被称为request pad,常见于合并或生成多路数据。例如,用于1到N转换的tee:"gst-inspect-1.0 tee"
Pad Templates: ... SRC template: ‘src_%u‘ Availability: On request Has request_new_pad() function: gst_tee_request_new_pad Capabilities: ANY
当我们需要将同一路视频流同时进行显示和存储,这时候我们就需要用到tee,在创建tee element的时候,我们不知道pipeline需要多少个src pad,需要后续element来请求一个src pad。
GStreamer提供了gst-inspect工具来查看element所提供的Pad Templates,但无法查看element在不同状态时其Pad所支持的数据类型,通过下面的代码,我们可以看到Pad Caps在不同状态下的变化。
#include <gst/gst.h> /* Functions below print the Capabilities in a human-friendly format */ static gboolean print_field (GQuark field, const GValue * value, gpointer pfx) { gchar *str = gst_value_serialize (value); g_print ("%s %15s: %s\n", (gchar *) pfx, g_quark_to_string (field), str); g_free (str); return TRUE; } static void print_caps (const GstCaps * caps, const gchar * pfx) { guint i; g_return_if_fail (caps != NULL); if (gst_caps_is_any (caps)) { g_print ("%sANY\n", pfx); return; } if (gst_caps_is_empty (caps)) { g_print ("%sEMPTY\n", pfx); return; } for (i = 0; i < gst_caps_get_size (caps); i++) { GstStructure *structure = gst_caps_get_structure (caps, i); g_print ("%s%s\n", pfx, gst_structure_get_name (structure)); gst_structure_foreach (structure, print_field, (gpointer) pfx); } } /* Prints information about a Pad Template, including its Capabilities */ static void print_pad_templates_information (GstElementFactory * factory) { const GList *pads; GstStaticPadTemplate *padtemplate; g_print ("Pad Templates for %s:\n", gst_element_factory_get_longname (factory)); if (!gst_element_factory_get_num_pad_templates (factory)) { g_print (" none\n"); return; } pads = gst_element_factory_get_static_pad_templates (factory); while (pads) { padtemplate = pads->data; pads = g_list_next (pads); if (padtemplate->direction == GST_PAD_SRC) g_print (" SRC template: ‘%s‘\n", padtemplate->name_template); else if (padtemplate->direction == GST_PAD_SINK) g_print (" SINK template: ‘%s‘\n", padtemplate->name_template); else g_print (" UNKNOWN!!! template: ‘%s‘\n", padtemplate->name_template); if (padtemplate->presence == GST_PAD_ALWAYS) g_print (" Availability: Always\n"); else if (padtemplate->presence == GST_PAD_SOMETIMES) g_print (" Availability: Sometimes\n"); else if (padtemplate->presence == GST_PAD_REQUEST) g_print (" Availability: On request\n"); else g_print (" Availability: UNKNOWN!!!\n"); if (padtemplate->static_caps.string) { GstCaps *caps; g_print (" Capabilities:\n"); caps = gst_static_caps_get (&padtemplate->static_caps); print_caps (caps, " "); gst_caps_unref (caps); } g_print ("\n"); } } /* Shows the CURRENT capabilities of the requested pad in the given element */ static void print_pad_capabilities (GstElement *element, gchar *pad_name) { GstPad *pad = NULL; GstCaps *caps = NULL; /* Retrieve pad */ pad = gst_element_get_static_pad (element, pad_name); if (!pad) { g_printerr ("Could not retrieve pad ‘%s‘\n", pad_name); return; } /* Retrieve negotiated caps (or acceptable caps if negotiation is not finished yet) */ caps = gst_pad_get_current_caps (pad); if (!caps) caps = gst_pad_query_caps (pad, NULL); /* Print and free */ g_print ("Caps for the %s pad:\n", pad_name); print_caps (caps, " "); gst_caps_unref (caps); gst_object_unref (pad); } int main(int argc, char *argv[]) { GstElement *pipeline, *source, *sink; GstElementFactory *source_factory, *sink_factory; GstBus *bus; GstMessage *msg; GstStateChangeReturn ret; gboolean terminate = FALSE; /* Initialize GStreamer */ gst_init (&argc, &argv); /* Create the element factories */ source_factory = gst_element_factory_find ("audiotestsrc"); sink_factory = gst_element_factory_find ("autoaudiosink"); if (!source_factory || !sink_factory) { g_printerr ("Not all element factories could be created.\n"); return -1; } /* Print information about the pad templates of these factories */ print_pad_templates_information (source_factory); print_pad_templates_information (sink_factory); /* Ask the factories to instantiate actual elements */ source = gst_element_factory_create (source_factory, "source"); sink = gst_element_factory_create (sink_factory, "sink"); /* Create the empty pipeline */ pipeline = gst_pipeline_new ("test-pipeline"); if (!pipeline || !source || !sink) { g_printerr ("Not all elements could be created.\n"); return -1; } /* Build the pipeline */ gst_bin_add_many (GST_BIN (pipeline), source, sink, NULL); if (gst_element_link (source, sink) != TRUE) { g_printerr ("Elements could not be linked.\n"); gst_object_unref (pipeline); return -1; } /* Print initial negotiated caps (in NULL state) */ g_print ("In NULL state:\n"); print_pad_capabilities (sink, "sink"); /* Start playing */ ret = gst_element_set_state (pipeline, GST_STATE_PLAYING); if (ret == GST_STATE_CHANGE_FAILURE) { g_printerr ("Unable to set the pipeline to the playing state (check the bus for error messages).\n"); } /* Wait until error, EOS or State Change */ bus = gst_element_get_bus (pipeline); do { msg = gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE, GST_MESSAGE_ERROR | GST_MESSAGE_EOS | GST_MESSAGE_STATE_CHANGED); /* Parse message */ if (msg != NULL) { GError *err; gchar *debug_info; switch (GST_MESSAGE_TYPE (msg)) { case GST_MESSAGE_ERROR: gst_message_parse_error (msg, &err, &debug_info); g_printerr ("Error received from element %s: %s\n", GST_OBJECT_NAME (msg->src), err->message); g_printerr ("Debugging information: %s\n", debug_info ? debug_info : "none"); g_clear_error (&err); g_free (debug_info); terminate = TRUE; break; case GST_MESSAGE_EOS: g_print ("End-Of-Stream reached.\n"); terminate = TRUE; break; case GST_MESSAGE_STATE_CHANGED: /* We are only interested in state-changed messages from the pipeline */ if (GST_MESSAGE_SRC (msg) == GST_OBJECT (pipeline)) { GstState old_state, new_state, pending_state; gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state); g_print ("\nPipeline state changed from %s to %s:\n", gst_element_state_get_name (old_state), gst_element_state_get_name (new_state)); /* Print the current capabilities of the sink element */ print_pad_capabilities (sink, "sink"); } break; default: /* We should not reach here because we only asked for ERRORs, EOS and STATE_CHANGED */ g_printerr ("Unexpected message received.\n"); break; } gst_message_unref (msg); } } while (!terminate); /* Free resources */ gst_object_unref (bus); gst_element_set_state (pipeline, GST_STATE_NULL); gst_object_unref (pipeline); gst_object_unref (source_factory); gst_object_unref (sink_factory); return 0; }
将源码保存为basic-tutorial-3.c,执行下列命令可得到编译结果:
$ gcc basic-tutorial-3.c -o basic-tutorial-3 `pkg-config --cflags --libs gstreamer-1.0`
print_field, print_caps and print_pad_templates_information实现类似功能,打印GStreamer的数据结构,可以查看相应GStreamer GstCaps 接口了解更多信息。
/* Shows the CURRENT capabilities of the requested pad in the given element */ static void print_pad_capabilities (GstElement *element, gchar *pad_name) { GstPad *pad = NULL; GstCaps *caps = NULL; /* Retrieve pad */ pad = gst_element_get_static_pad (element, pad_name); if (!pad) { g_printerr ("Could not retrieve pad ‘%s‘\n", pad_name); return; } /* Retrieve negotiated caps (or acceptable caps if negotiation is not finished yet) */ caps = gst_pad_get_current_caps (pad); if (!caps) caps = gst_pad_query_caps (pad, NULL); /* Print and free */ g_print ("Caps for the %s pad:\n", pad_name); print_caps (caps, " "); gst_caps_unref (caps); gst_object_unref (pad); }
因为我们使用的source和sink都具有static(always)pad,所以这里使用gst_element_get_static_pad()获取Pad, 其他情况可以使用gst_element_foreach_pad()或gst_element_iterate_pads()获取动态创建的Pad。
接着使用gst_pad_get_current_caps()获取pad当前的caps,根据不同的element状态会有不同的结果,甚至可能不存在caps。如果没有,我们通过gst_pad_query_caps()获取当前可以支持的caps,当element处于NULL状态时,这个caps为Pad Template所支持的caps,其值可随状态变化而变化。
/* Create the element factories */ source_factory = gst_element_factory_find ("audiotestsrc"); sink_factory = gst_element_factory_find ("autoaudiosink"); if (!source_factory || !sink_factory) { g_printerr ("Not all element factories could be created.\n"); return -1; } /* Print information about the pad templates of these factories */ print_pad_templates_information (source_factory); print_pad_templates_information (sink_factory); /* Ask the factories to instantiate actual elements */ source = gst_element_factory_create (source_factory, "source"); sink = gst_element_factory_create (sink_factory, "sink");
在使用gst_element_factory_make()接口创建element时,应用不需要关心element工厂。在这里,由于Pad Template数据Element工程,因此我们首先根据工厂名创建了相应工厂实例(GstElementFactory ),再由其获取Pad Template以及创建element。
此处使用gst_element_factory_find()查找"audiotestsrc"工厂,再通过gst_element_factory_create()创建source element。以前使用的gst_element_factory_make()是gst_element_factory_find() + gst_element_factory_create()的简化版。
Pipeline的创建过程与其他示例相同,此例新增了状态变化的处理。
case GST_MESSAGE_STATE_CHANGED: /* We are only interested in state-changed messages from the pipeline */ if (GST_MESSAGE_SRC (msg) == GST_OBJECT (pipeline)) { GstState old_state, new_state, pending_state; gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state); g_print ("\nPipeline state changed from %s to %s:\n", gst_element_state_get_name (old_state), gst_element_state_get_name (new_state)); /* Print the current capabilities of the sink element */ print_pad_capabilities (sink, "sink"); } break;
因为我们在gst_bus_timed_pop_filtered()中加入了GST_MESSAGE_STATE_CHANGED,所以我们会收到状态变化的消息。在状态变化时,输出sink element的pad caps中当前状态的信息。
Pad Templates for Audio test source: SRC template: ‘src‘ Availability: Always Capabilities: audio/x-raw format: { S16LE, S32LE, F32LE, F64LE } layout: interleaved rate: [ 1, 2147483647 ] channels: [ 1, 2 ] Pad Templates for Auto audio sink: SINK template: ‘sink‘ Availability: Always Capabilities: ANY
首先是“audiotestsrc”和“autoaudiosink”的pad templates信息,这个与gst-inspect的输出相同。
In NULL state: Caps for the sink pad: ANY
NULL状态为Element的初始化状态,此时,“autoaudiosink”的sink pad caps与Pad Template相同,支持所有的格式。
Pipeline state changed from NULL to READY: Caps for the sink pad: audio/x-raw format: { S16LE, S16BE, F32LE, F32BE, S32LE, S32BE, S24LE, S24BE, S24_32LE, S24_32BE, U8 } layout: interleaved rate: [ 1, 2147483647 ] channels: [ 1, 32 ] audio/x-alaw rate: [ 1, 2147483647 ] channels: [ 1, 32 ] audio/x-mulaw rate: [ 1, 2147483647 ] channels: [ 1, 32 ]
状态从NULL转到READY时,GStreamer会获取音频输出设备所支持的所有类型,这里可以看到sink pad caps列出了输出设备所能支持的类型。
Pipeline state changed from READY to PAUSED: Caps for the sink pad: audio/x-raw format: S16LE layout: interleaved rate: 44100 channels: 1 Pipeline state changed from PAUSED to PLAYING: Caps for the sink pad: audio/x-raw format: S16LE layout: interleaved rate: 44100 channels: 1
状态从READY转到PAUSED时,GStreamer会协商一个所有element都支持的类型。当进入PLAYING状态时,sink会采用协商后的类型进行数据传输。
在本教程中,我们掌握了:
https://gstreamer.freedesktop.org/documentation/tutorials/basic/media-formats-and-pad-capabilities.html?gi-language=c
https://gstreamer.freedesktop.org/documentation/tutorials/basic/multithreading-and-pad-availability.html
https://gstreamer.freedesktop.org/documentation/gstreamer/gstcaps.html?gi-language=c
标签:desktop lag clu 定义 总结 switch messages ram class
原文地址:https://www.cnblogs.com/xleng/p/11113405.html