Linux.com Interviews the GStreamer Developers

Wim: I'm Wim Taymans, 28 years old. I live in Belgium, near the university in Leuven where I got a degree in computer science. I learned BASIC at the age of 9 on my first computer, a ZX80. I became active in the C64 demo scene later on. Installed my first Linux system on my Amiga1200.

I currently work in one of the biggest banks in Belgium, where I help to design and build the web banking system, using JAVA, JSP, HTML, etc...

Erik: I've been programming since 3rd grade, doing Logo on an Apple ][e. Taught myself Pascal in the 5th grade, and C in my freshman year in High School. Worked for Sequent Computers doing huge-systems testing, then went to the Oregon Graduate Institute to research Quality of Service issues. This is where things really began, as we worked on a home-grown streaming-media pipeline to do a lot of our research. This pipeline had many limitations, based on the fact that there were research points to make, not product requirements to fulfill. That's when I started thinking about how to build GStreamer.

Since then I did the required stint at a failed startup, and now work for RidgeRun, Inc., where I get paid to work on GStreamer <g>. I'll be bringing the GStreamer project to maturity, as well as applying it to specific embedded-systems problems such as cell phones, MP3 players, and whatever other hardware people start using DSPLinux for.

Linux.com: How did the GStreamer project start?

Erik: Well, I was spending a lot of time thinking about how to fix the OGI pipeline, and ended up deciding to just create my own. From there it started maturing, and more and more features were added. Then people like Wim started working on it, and it grew into the monster it is today .

Wim: I wanted to do a NLE (non-linear video editing application), so I joined the Trinity project. It became clear that we needed a solid multimedia library in order to be able to handle all the things we wanted to do. I was very impressed with the first GStreamer version (0.0.9) and I almost immediately started to add various video plug-ins for mpeg1/mpeg2/avi etc.

Linux.com: Can you explain us the "magic" behind GStreamer? How different is it from other multimedia frameworks available for Linux?

Wim: First of all, GStreamer is a real framework. This means that it can be used for a generic media player as well as serve as the core of large multimedia render farms. The GStreamer core is built in such a way that it is media agnostic, it doesn't know or care what media data it is handling. The interpretation of the media types is entirely handled by the plug-ins.

This allows us to handle all possible media data you can imagine and allows us to create a new plug-in that integrates smoothly with the framework whenever a new multimedia types becomes available.

Gstreamer's job is to provide a clean interface to the plug-ins and to make sure they can interact with each other. We have very a powerful infrastructure to manage the scheduling of the various plug-ins and making sure that are able to talk to each other. GStreamer provides an API to do whatever multimedia task you want. It is in a sense a toolkit for building multimedia applications; very similar to what a widget toolkit is for building GUI applications. We are very focused on building a solid foundation instead of building 'yet another media player'.

We also provide an editor to visually create a multimedia pipeline that can be saved to XML and used inside an application. We also use XML to store the information about the plug-ins. This makes it possible to have thousands of plug-ins without the overhead of having to load them all into memory.

Erik: Well, in the strict sense, there really aren't any other multimedia frameworks for Linux. All of the multimedia programs I've seen so far have, at most, a modularity sufficient to avoid pain while bringing in disparate media types. When things can't be done easily, things get messy.

GStreamer is designed to be generic, so it handles any media type, and just about any way of dealing with the media. We can do an MP3 player, or an NLE, or a DJ mixer, etc. The infrastructure is designed to support arbitrary graphs of elements, so you're limited only by your imagination.

The other half is that fact that since GStreamer is nothing but a framework (it doesn't try to be an application at all), we have a large set of tools to help application developers. This ranges from the auto-plugging system in the core, to a command line pipeline tester, to a complete graphical editor that can save XML that your application can load in one function call. Only DirectShow comes close, and I don't think we'll be seeing that on Linux any time soon, do you?

Linux.com: What would you say to the people coding on XMMS, XMPS, and such projects?

Erik: Hmmm, I'd say that they really should be using GStreamer! I should note that I haven't looked at either of those in depth, but I would assume that they have a structure designed for the sole purpose of playing the types of media they support. If they want to play other types of media, they would probably have to make significant changes.

The other half of it is that each of the players usually has their own codec collection, with local modifications and such. This is problematic for a number of reasons. First, there is significant effort expended to shim a random codec into any framework, and this effort is being expended for every single project out there. Second, any enhancements made to a codec used by one project are not generally propagated around to other projects to benefit the community as a whole.

An application based on GStreamer is very easy to code and expand. One of my first real test applications was a 602-line graphical media player with effects and such. Making it more robust is not very hard. Supporting new media types requires zero effort, as long as you have the decoder plug-in. Video is a matter of creating space in your GUI and letting GStreamer automatically create the whole decoder chain.

Wim: Well, they have a different set of goals, I think. They mainly want to create a media player and not necessarily a complete media framework. For media playback all you need is a simple src with a decoder and a video/audio output plug-in, possibly with an option to insert an effect between them. While GStreamer will of course be able to handle that very easily, we have not yet put all the necessary core infrastructure in place to handle the capabilities of those media players. Once the core is ready for it, I see no reason for them not to use GStreamer.

Linux.com: And that, even though creating plug-ins for these tools is much easier than with the GStreamer framework?

Wim: I don't think that's true. I think our plug-in API is quite simple and intuitive to use. In fact, you can also make the plug-in a lot smaller in GStreamer because lots of issues like scheduling, type negotiation, state change, seeking, disk/net sources are all handled by the framework and other plug-ins. You also have to take into account that the plug-in can be used in a much broader range of applications. While the easy plug-ins, like those found in most media players, are very easy to create with GStreamer, the harder ones, like video mixers with choma keying and plug-ins to handle 16 channel audio boards, are still possible. We are currently working on a decent plug-in writers manual so that people can see how the plug-ins have to be written.

Erik: Again, I haven't looked at how one goes about creating a plug-in for these players, but I would probably agree that it's easier to create a plug-in for them. But how much work is it to create the same plug-in for each of these different players? Quite a bit more work, I'd think. Instead, with GStreamer you create the plug-in once, to a standard interface, and it works with all applications that are set up to handle that class of elements (i.e. effects filters as a group). New format decoders are even easier for an application to take advantage of, since they are self-describing. As long as an application uses auto-plugging, it'll pick up new formats instantly.

As far as writing the plug-in itself, it is a little bit of a pain right now, but we've got quite a few ideas brewing under the title 'Filter Factory' that will make the construction of most plug-ins a very trivial task.

Linux.com: What applications would like to see based on GStreamer?

Erik: Well, a few good media players, of course. Audio and video processing and editing tools, stuff like that. My main interest personally is in live audio processing, where the computer handles a couple dozen audio channels, applies effects, mixes into main and monitor busses, etc. The main feature here would be scripting ability, in addition to the main GStreamer provided features of infinite 'hardware' within the bounds of your CPU(s).

Wim: A media player is on top of the list, of course, because that's what everyone will be using. I'm personally interested in the move heavy stuff like video/audio editing. I'd also like to create a distributed media application based on GStreamer to handle a big media-rendering farm. I'd also love to see a bonobo-ized media player component using GStreamer, that can be embedded in projects like Eazel's Nautilus and Evolution.

Linux.com: OpenDivX -- any plans on including a new input plug-in?

Wim: Well, to be correct, it will be a decoder plug-in. License issues aside, sure why not? Our priority for now is to get the core in a solid and feature full state before we do more plug-ins.

Erik: First of all, there's a terminology difference here. There is really no such thing as 'an input plug-in' in the current sense. There are sources, which read from disk, network, or whatever else, and there are filters. One of the biggest problems with xmms when I looked at it last (well before 1.0.0) was that the MP3 decoder input plug-in could read either from disk or from ShoutCast, nothing else. If you wanted to read from any other source, you'd have to hack the decoder plug-in, as well as the GUI.

That said, we do have an AVI decoder that loads Windoze DLLs to decode stuff. We can use the DiVX binary decoder, and even the encoder, though it's rather fragile. OpenDiVX itself has significant licensing issues that make it unlikely that a plug-in will be a part of the main GStreamer package. However, there's nothing stopping anyone from creating a separate package that wraps it. Since plug-in installation under GStreamer is a matter of dropping the .so in the right directory and running `gstreamer-register` to update the plug-in cache, there's nothing to it.

Linux.com: vi or emacs?

Erik: Um, neither ;-) /me shuts up.

Wim: Not sure what GStreamer and vi have in common, yet. We plan to create a lot of language bindings to access the GStreamer infrastructure so you might even see some emacs lisp soon. I do plan to rewrite cp(1) with GStreamer someday :-)

Linux.com: New release of GStreamer, what's new?

Wim: The 0.1.1 release is supposed to be a bug fix release but some interesting new features have slipped in because we only recently started to use CVS branches. No major changes for the end user since we didn't change the player or the plug-ins.

Erik: Well, 0.1.1 is mostly a bug fix release. There's a list in the mail archives, if you want the complete list. The short list is: lots of build and bug fixes, support for Alpha and ARM, some changes to the way ghost pads are handled, some state change updates, better handling of the EOS case, and various other minor things.

For the next release, we hope to have incorporated some major branches pending in CVS. These include a rewrite of the scheduling system to handle dynamic pipeline reconfiguration, the next generation of auto-plugging code, and capabilities negotiation. Other features may include an event system for handling end-of-stream (EOS) and other stuff.

Linux.com: Thank you guys for taking the time to talk with us.

Erik: Thanks; it was fun!

Wim: You're welcome :)

Bastien Nocera is a programmer interested in multimedia and GUIs. That's why he has a fish tank running 24/7... hum, an iMac, even. And all your computers belong to me, as well.

PS: I've been told Erik uses pico =)

Check out GStreamer at its homepage, GStreamer.net.