Icecast.org

Description

Please note that this page is currently very much out of date.
Its contents should not be used for new implementations of clients or directories.

Please contact the icecast-dev mailing list instead if you have questions.

This document will describe the proposed protocol and components to be used for building a directory server (YP Server) for icecast2 and possibly other servers as well. This document will be a work in progress.

Overview

The need to centrally list streaming broadcasts has been done a bunch of times before with moderate degrees of success. This approach tries to take the good parts of each and hopefully create a better incarnation. The key aspect of this system will be simplicity. Simplicity, in the protocol and maintenance of the system, will take precedence in the design process.

Examples of existing directories can be found at :

* Shoutcast

* Live365

* CasterClub

* Radio Toolbox

Basics

To support a central directory listing of broadcasts, the minimum capabilities are required :

* Ability to create a new entry in the listing

* Ability to update an existing entry in the listing

* Ability to remove an entry from the listing

* Ability to group related broadcasts

* Ability to cluster relayed broadcasts into a single listing

* Support for icecast/icecast2 mountpoints

* Listing should be done by the streaming server (i.e. no manual effort required)

* Listings should not require registration or other prior sign-up

Additionally, the protocols which define the conversations above should NOT be proprietary, and should be well defined.

Protocol

HTTP protocol will be used for all communication between the listing client (icecast2 in this case) and the listing server (listing script(s))

A few notes about the protocol before you get into it....Listener counts will be provided by the listing client, however, the current plan is for this information to not be displayed in the actual YP. While this was at first a very difficult thing to decide upon (listener counts are always desirable), the tenets of "Simplicity" overrode the desire for this information. The reasoning is that if listener information is stored at the listing server, then people will want to rate broadcasts based off this information. Due to the "open" nature of the protocol, this information can be very easily faked (and has been in the past). So, in order to make hosting a directory server a nice easy task, and not a full-time policing job, metrics such as listener count have been excluded. Any ideas regarding this are welcomed.

The listing client will make HTTP requests on the listing server. The following types of requests will be supported :

Add Server

This type of request will add a new server entry to the directory.

The URL call will have the following mandatory parameters :

action	Action (add in this case)
sn	Server Name
type	Server Type (content type)
genre	Server Genre
b	Server Bitrate
listenurl	Listen URL (url which listeners use to listen)

The URL call will have the following *optional* parameters :

cpswd	Cluster Password (broadcasts with the same Server Name and cluster password will be displayed together in the directory server).
user	YP userid - not necessilarily all YP implementation will support users and passwords
pass	YP password - not necessilarily all YP implementation will support users and passwords
desc	Server Description
url	Stream URL (not the listen url, usually a link to the broadcasters website)
stype	Server Sub type. Used normally for multi-codec streams (ogg/theora, vp6/aac). Codecs should be separated by a '/' delimiter.

The listing scripts will respond with the following HTTP headers

YPResponse:	(0-failure or 1-success)
YPMessage:	Any error message
SID:	System Identifier which represents the unique identifier for the new listing entry. All futher communications must be made using this SID - the SID can be any alpha numeric string
TouchFreq:	The frequency (in seconds) in which the listing client needs to touch the server in order to prevent a stale record

The following is an example of the communication between the listing client and listing server

URL: http://dir.xiph.org/cgi-bin/yp-cgi?action=add&sn=Oddsock+Server&
     listenurl=http://localhost:8000/oddsock.ogg&genre=Rock&b=64&
     type=Ogg+Vorbis

Listing Client Request
-----------------
GET /yp-cgi?action=add&sn=Oddsock+Server&listenurl=http://localhost:8000/oddsock.ogg&
genre=Rock&b=64&type=application/ogg


Listing Server Response
-----------------
HTTP/1.0 200 OK
YPResponse: 1
YPMessage: Successfully Added
SID: 1041755216.363775
TouchFreq: 60

OR

HTTP/1.0 200 OK
YPResponse: 0
YPMessage: Problem with blah blah
YPSID: -1

Touch Server

This type of request will update a server entry with new information. This request will also cause the listing server to acknowledge this server as one that is still valid. Periodic cleanups of inactive servers will be performed on the listing server.

The URL call will have the following mandatory parameters :

action	Action (touch in this case)
sid	System Identifier received from the Add Server call

The URL call will have the following *optional* parameters :

st	Song Title
listeners	Number of listeners ***
max_listeners	Maximum capacity of station ***
alt	Average Listening time ***
ht	Stream Hits (tuneins) ***
cm	5 minute tunein ***
stype	Server Sub type. Used normally for multi-codec streams (ogg/theora, vp6/aac). Codecs should be separated by a '/' delimiter. - Note that since it is possible to change codecs mid stream in some container formats, so this field is updatable on a touch.

*** = due to the open-source nature of icecast, these metrics may not be reliable (i.e. they may be fake), it is up to the YP administrator to decide whether to include these stats.

The listing scripts will respond with the following HTTP headers

YPResponse:	(0-failure or 1-success)
YPMessage:	Any error message

The following is an example of the communication between the listing client and listing server

URL: http://dir.xiph.org/cgi-bin/yp-cgi?action=touch&sid=1041755216.363775&
st=Metallica-MasterOfPuppets

Listing Client Request
-----------------
GET /yp-cgi?action=touch&sid=1041755216.363775&st=Metallica-MasterOfPuppets


Listing Server Response
-----------------
HTTP/1.0 200 OK
YPResponse: 1
YPMessage: Updated Server Info

OR

HTTP/1.0 200 OK
YPResponse: 0
YPMessage: We had a problem...

Remove Server

This type of request will remove a server entry from the Directory. This request should be done when either the broadcast has stopped, or the icecast2 server has stopped.

The URL call will have the following mandatory parameters (all parameters are mandatory) :

action	Action (remove in this case)
sid	System Identifier received from the Add Server call

The listing scripts will respond with the following HTTP headers

YPResponse:	(0-failure or 1-success)
YPMessage:	Any error message

The following is an example of the communication between the listing client and listing server

URL: http://dir.xiph.org/cgi-bin/yp-cgi?action=remove&sid=1041755216.363775

Listing Client Request
-----------------
GET /yp-cgi?action=remove&sid=1041755216.363775


Listing Server Response
-----------------
HTTP/1.0 200 OK
YPResponse: 1
YPMessage: Deleted Server Info

OR

HTTP/1.0 200 OK
YPResponse: 0
YPMessage: We had a problem...

Server Implementation

This section describes my thoughts on the implementation of the listing server part of the system. This is the server script(s) which accept in the Add/Touch/Delete calls from icecast2 and process them.

Here are some requirements that I put together regarding this component :

* It needs access to a DB (probably mySQL due to it's common use and the fact that it's free :)

* It needs to be fast (the scripts shouldn't be scripts, but probably a C program)

* It needs to handle a potential large amount of traffic (this means not only does it have to be fast, but it has to scale up pretty good to high loads)

* It needs to be robust (yeah, well, thought I'd throw that in anyway).

Based off these requirements, I suggest that the server listing scripts be implemented as FastCGI C programs that access a mySQL to store the listing information. This database will also need to be accessed by other scripts (PHP/ASP/etc.) which will front-end the entire directory (directory browser). FastCGI is recommended due to the fact that it can be used to pool mySQL DB connections on the server and also hopefully provide the speed and scalability need to meet the requirements.

mySQL Database Schema


--
-- Table structure for table 'server_details'
--

CREATE TABLE if not exists server_details (
  id mediumint(9) NOT NULL auto_increment,
  parent_id mediumint(9) default NULL,
  server_name varchar(100) default NULL,
  listing_ip varchar(25) default NULL,
  description varchar(255) default NULL,
  genre varchar(100) default NULL,
  sid varchar(200) default NULL,
  cluster_password varchar(50) default NULL,
  url varchar(255) default NULL,
  current_song varchar(255) default NULL,
  listen_url varchar(200) default NULL,
  playlist_id mediumint(9) default NULL,
  server_type varchar(25) default NULL,
  server_subtype varchar(255) default NULL,
  bitrate varchar(25) default NULL,
  listeners int(11) default NULL,
  channels varchar(25) default NULL,
  samplerate varchar(25) default NULL,
  PRIMARY KEY  (id)
) TYPE=MyISAM;

create table if not exists playlists (
  id mediumint(9) NOT NULL,
  listen_url varchar(200) default NULL
) TYPE=MyISAM;

create table if not exists clusters (
  id mediumint(9) NOT NULL auto_increment,
  server_name varchar(255) default NULL,
  cluster_password varchar(50) default NULL,
  PRIMARY KEY  (id)
) TYPE=MyISAM;

--
-- Table structure for table 'servers'
--

CREATE TABLE if not exists servers (
  id mediumint(9) NOT NULL auto_increment,
  server_name varchar(100) default NULL,
  listing_ip varchar(25) default NULL,
  listeners int(11) default NULL,
  rank int(11) default NULL,
  PRIMARY KEY  (id)
) TYPE=MyISAM;

--
-- Table structure for table 'servers_touch'
--

CREATE TABLE if not exists servers_touch (
  id varchar(200) NOT NULL default '',
  server_name varchar(100) default NULL,
  listing_ip varchar(25) default NULL,
  last_touch datetime default NULL,
  PRIMARY KEY  (id)
) TYPE=MyISAM;

Closing

The purpose of this document was to start a basic specification for the directory listing that is badly needed in icecast2. It is by no means expected to be a complete document. All comments are welcome. :)

Icecast Directory Listing Specification

Description

Please note that this page is currently very much out of date. Its contents should not be used for new implementations of clients or directories.

Overview

Basics

Protocol

Add Server

Touch Server

Remove Server

Server Implementation

mySQL Database Schema

Closing

Please note that this page is currently very much out of date.
Its contents should not be used for new implementations of clients or directories.