Video and Audio Streaming with Flash and Open Source Tools

Flash has always been developed and used for multimedia purposes, but until version 6 the possibilities for audio streaming was limited and also there was no video support. With Version 6 and 7 Macromedia introduced video support and a new file format to support various ways of streaming. This article covers only a streaming variant called “progressive download” which does not need server support. True streaming support is available with Macromedia's non-free Flash Communication Server (FCS).

FLV Streams

Streaming is built upon a new file format called FLV, which separates the streamable content and the flash movie. The result is a very compact flash movie acting as a multimedia player and a storage for streamable content from which the flash movie loads a stream on demand.

A single FLV stream contains at most one audio stream and at most one video stream. Flash supports uncompressed sound and various compressed formats like MP3 and ADPCM as well as the proprietary Nellymoser audio codec. With flash version 6 Macromedia also introduced video support for flash. In version 6 only the Sorenson H.263 video codec was supported which is a slightly modified version of the open H.263 standard. The latest flash version 7 introduced a second video format “Screen Video”, which is a simple, loss less video format, especially developed for screen capturing.

Converting and Creating Content

One method for creating FLV streams is converting existing audio and video content with FFmpeg [ffmpeg.sourceforge.net]. FFmpeg is a mature and very excellent software project for converting audio and video from and to various formats. Converting a video can be simply done with

ffmpeg -i infile.[avi|mpeg] stream.flV

FFmpeg uses the Sorenson H.263 video format for encoding video data. There is no support for the Screen Video format at this time. While the Screen Video format is mainly useful for screen capturing applications, Sorensons H.263 is multipurpose video codec with good compression rates, suitable especially for encoding motion pictures.

Another project dealing with FLV streams is called libflv [libflv.sourceforge.net]. While FFmpeg is a general audio and video converting suite, libflv is focused on working with FLV streams. The project is still in a very early stage, but is able encoding videos in the Screen Video format and allows simple FLV stream manipulations like (de-)multiplexing of audio and video streams. A simple GTK-based screen capturing application can be found in the example directory.

Building a Simple Multimedia Player

After having created some streamable content, a flash multimedia player is needed. One huge advantage of flash based players over other plug-in based multimedia-players is that there are no constraints about its look and how it is integrated in your sites design.MING is an open source library which is able to create flash files with almost all recent flash features, including Action Script, sound and video support. The library also has language bindings for a bunch of script and programming languages. The examples presented in this article are written in PHP4. Porting the examples to other supported languages like C/C++, Java, Python or Perl should be trivial.

To run the following example a current CVS snapshot of MING is needed. It is available either via Sourceforges anonymous CVS service or pre-packaged at klaus.geekserver.net/ming/.

First we create a new move instance and set dimension and background color:

ming_useswfversion(7);$movie=new SWFMovie(7);$movie->setDimension($width, $height);$movie->Background($r,$g,$b);

The new flash move can now be filled with flash objects called characters. For the multimedia player example we create a video canvas object and add it to the movie. The add() method takes a character and inserts it to the current frame and returns a handle to the object. This can be used to move, rotate, resize or remove an object. If the object is going to be used with ActionScript, a name can be assigned to it.

$stream = new SWFVideoStream();$stream->setDimension($width, $height);$item = $movie->add($stream);$item->moveTo($x, $y);$item->setname(“video”);

The SWFVideoStream() constructor can also take a FLV file as argument. In this case the video stream will be embedded to the flash file. However this approach has some drawbacks. First of all the resulting flash movie will get as least as big as the stream. But also the stream's frame rate must not exceed the flash movies frame rate and each flash file is limited to 16000 frames, which means that the embedded stream can contain at most 16000 frames.

A multimedia player application should be able to load and play streams dynamically. Therefore the SWFVideoStream() constructor is called with no arguments. Thus only an empty video canvas will be created, which will be controlled by the following ActionScript code:

connection = new NetConnection();connection.connect(null);stream = new NetStream(connection);video.attachVideo(stream);stream.setBufferTime(10);stream.play(‘http://localhost/mystream.flv');

The ActionScript first creates a pseudo connection by passing null to the connect() method of the NetConnection object. In contrast, a real connection to a Macromedia streaming server can be made by passing a valid url to the method. Having a NetConnection instance a newNetStream object can be created and attached to the empty video canvas. This object handles streaming and provides methods for controlling the stream. The above example loads a FLV stream from the local web server with a downloadbuffer of 10 seconds. The ActionScript code can be compiled and added to the movie with:

$action = new SWFAction($action_string);$movie->add($action);

Until now the flash movie just loads and plays a certain FLV stream. To control its behavior, a simple user interface consisting of some buttons and a seek-slider is missing. Flash has its own compressed loss less bitmap format called DBL. MING provides a small utility png2dbl to convert PNG images to DBL. Such images are used for the player's control buttons:

$button = new SWFButton();$flags = (SWFBUTTON_UP | SWFBUTTON_HIT | SWFBUTTON_OVER | SWFBUTTON_DOWN);$button->addShape(ImageShape(“images/pause.dbl”), $flags);

$action = new SWFAction(“stream.pause();”);$button->addAction($action, SWFBUTTON_MOUSEDOWN);

$button_ref = $movie->add($button);$button_ref->moveTo($x, $y);

The above example creates a pause button for the multimedia player. An interactive button is created in two steps. First its look has to be defined by adding shapes for certain mouse events. In flash a shape is the basic representation for graphic objects. For each mouse event a different shape object can be assigned to the button. In the above example the button looks always the same.In the second step the buttons action can be defined by assigning ActionScript to a special event.

One drawback using progressive download streaming without server support is that there is no possibility to get the stream's total length. Therefore the seek-sliders functionality is limited to seeking within the already loaded parts of the stream.

The dragable part of the seek-slider is realized as a movie-clip object. A movie-clip is running as an independent movie in the flash movie. It has an independent time line, can handles scripts and handles external events itself.

$mc = new SWFSprite();$shape = new SWFShape();$shape->setLine(4,25,0,0,128);$shape->movePenTo(0, 5);$shape->drawLineTo(0, 10);$mc->add($shape);$mc->nextFrame();

$slider = $movie->add($mc);$slider->moveTo($xMin, $y);

A movie clip (SWFSprite) has similar methods like a movie object. The add() method inserts a flash object to the current frame, nextFrame() finishes the current frame and creates a new one. The movie clip is also a normal flash object which can be added to a movie and placed on the stage. The functionality of the seek-slider is defined by three small scripts. The first two actions make the movie-clip dragable:

$a = new SWFAction(“startDrag(this, $xMin, $y, $xMax, $y, 1); drag = true;”);$slider->addAction($a, SWFACTION_MOUSEDOWN);

$a = new SWFAction(“stopDrag(); drag=flase;”);$slider->addAction($a, SWFACTION_MOUSEUP);

The third more lengthy script sets the stream position depending on sliders x-position if the slider is actually moved by the user or sets the sliders x-position depending on the streams current time:

// width in pxwidth = xMax – xMin;

paused = false;if(drag) {// pause stream while seeking_global.stream.pause(true);paused = true;

x = _root._xmouse – xMin;seekTo = (_global.streamLen / width) * x;_global.stream.seek(seekTo);}else {pos = (_global.stream.time * (width / _global.streamLen)) + xMin;this._x = pos;this._y = y;}

// restart paused streamif(paused){_global.stream.pause(false);}

This script is assigned to the $slider-handle with the SWFACTION_ENTERFRAME event.After having added all elements to the flash movie the first frame has to be closed with the nextFrame() call. Since we don not need another frame the movie can also be finished:$movie->nextFrame();$movie->save(“FLVPlayer.swf”);

The resulting multimedia player

 Example Video: Copyright by Thilo Weigel, University of Freiburg

Conclusion

With flash it is easy to create a lightweight, fully customized, embedded video and audio player. There are powerful open source tools available for creating content and also creating flash movies. This article introduced the basic concepts of flash streaming and working with MING. The here presented mediaplayer provides only the most basic features and was only intended as a simple example. It can be extended in many ways, which is left to the reader.

References

[1.] FFmpeg Project [ffmpeg.sourceforge.net][2.] libflv [libflv.sourceforge.net][3.] MING [ming.sourceforge.net][4.] MING CVS snapshots [klaus.geekserver.net/ming][5.] FLV player source tarball [Player Source]