En tête technical corner
Mediastreamer2 & oRTP
Technical corner
Mediastreamer2
Mediastreamer2 is a powerful lightweight streaming engine for voice/video telephony applications.
This open source library is responsible for receiving and sending all multimedia streams in Linphone, including voice/video capture, encoding and decoding, and rendering.
It includes oRTP
oRTP
oRTP is a library that implements Real-Time Transport Protocol (RFC 3550).
Mediastreamer2 and oRTP in Linphone architecture
Mediastreamer2 features
- Audio codecs: opus, speex, g711, g729, gsm, iLBC, AMR, AMR-WB, g722, SILK, iSAC, BV16, Codec2
- Video codecs: VP8, H.264 and H.265 with resolutions up to 1080P, MPEG4
- Hardware accelerated H.264 codec for Mac OSX / iOS (VideoToolbox) and Android (MediaCodec)
- Send and receive RTP/RTCP streams
- Audio conferencing API
- Features an innovative jitter buffer algorithm, which can quickly adapt to network conditions with a lot of jitter and improve control of the audio latency
- Adaptive bit rate control algorithm: congestion control and estimation of available bandwidth, in order to optimise the quality of audio and video
- ICE (RFC 5245), STUN and TURN (RFC 5766) for optimised NAT traversal, allowing peer-to-peer audio and video connections whenever possible
- SRTP, zRTP and SRTP-DTLS voice and video encryption
- RTP/AVPF support: RTCP control messages for video error recovery: PLI, SLI, RPSI, FIR
- TMMBR support (Temporary Maximum Media Stream Bit Rate Request)
- Acoustic echo cancellation using echo cancellers from libspeexdsp, webrtc AECm or AEC
- Capture and playback from various platform-dependent sound architectures (ALSA, PulseAudio, AudioUnit, AudioQueue, WaveApi, WASAPI, Android AudioTrack/AudioRecord, Android OpenSLES)
- Supports any webcam, based on the OS's camera API: QuickTime, DirectShow, Video4Linux, Android camera
- Play and record to or from raw, wav, or mkv (matroska) files
- Optimised rendering of YUV pictures, using OpenGL, DrawDib, X11/Xv
- Dual tone generation
- Custom tone detector
- Audio parametric equaliser
- Volume control, automatic gain control
- Can use plugins to add new codecs, for new sound input/output backend etc.
oRTP features
- Written in C, works under Linux (and probably any Unix) and Windows
- Implements the RFC 3550 (RTP) with an easy-to-use API with high- and low-level access
- Features an adaptive jitter algorithm that enables a receiver to adapt to the clock rate of the sender and network jitter
- Includes support for multiple profiles, (AV profile (RFC 3551) is the default)
- Supports part of RFC 4733 for telephone events over RTP
- Includes an API to parse and compose RTCP packets, including AVPF RTCP packets
- Features a congestion detector based on an analysis of the packet's arrival time
- Comprises a download capacity estimator that is usable for video streams
- Supports draft-ietf-mmusic-sdp-bundle-negotiation-54 : Negotiating Media Multiplexing Using the Session Description Protocol (SDP)
- Supports RFC 7941 : RTP Header Extension for the RTP Control Protocol (RTCP) Source Description Items
- Supports RFC 8285 : A General Mechanism for RTP Header Extensions
Portability
- GNU/Linux: x86, x86-64, ARM v5 to v7, arm64 ; Debian 8/9, Centos 7
- Windows Desktop: x86 (works also on x86_64), Windows 7 and later
- macOS: x86_64 ; 10.11 and later.
- GNU/Linux embedded: Linphone-daemon or Liblinphone are good candidates to provide the software stack for a hardware phone or hardware communication system.
- Apple iOS 9 to 12 (ARM v7, ARM 64)
- Google Android 4.1 to 8.1 (ARM v7-v8, x86)
- Windows 10 UWP : mobile and desktop (ARM v7)
Design and principles
Each processing entity is contained within an MSFilter object. MSFilter(s) have inputs and/or outputs that can be used to connect from and to other MSFilters.
A trivial example can illustrate this:
- MSRtpRecv is an MSFilter that can receive RTP packets from the network, unpacketise them and post them on its only output.
- MSSpeexDec is an MSFilter that takes everything on its input assuming these are Speex encoded packets, decodes them and puts the result on its output.
- MSFileRec is an MSFilter that takes everything on its input and writes it to a wav file (assuming the input is 16-bit linear pcm).
MSFilters can be connected together to form a filter chain. If we connect the three examples above, we obtain a processing chain that receives RTP packets, decodes them and write the uncompressed results into a wav file.
MSRtpRecv --> MSSpeexDec --> MSFileRec
The execution of the media processing is scheduled by a MSTicker object, a thread that wakes up every 10 ms to process data in all the MSFilter chains it manages. Several MSTickers can be used simultaneously. For example, one for audio filters, one for video filters, or one on each processor of the machine on which it is running.
Media protocols
- RTP: A Transport Protocol for Real-Time Applications, RFC 3550
- 5. RTP Data Transfer Protocol
- 6.4 Sender and Receiver Reports
- 6.5 SDES: Source Description RTCP Packet
- 6.6 BYE: Goodbye RTCP Packet
- 6.7 APP: Application-Defined RTCP Packet
- RTP Profile for Audio and Video Conference with Minimal Control, RFC 3551
- RTP Payload for Text Conversation, RFC 4103
- Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF), RFC 4585
- Temporary Maximum Media Stream Bit Rate Request and Notification, section 3.5.4. of RFC 5104
- Symmetric RTP / RTP Control Protocol (RTCP), RFC 4961
- Session Traversal Utilities for NAT (STUN), RFC 5389 (Basic procedures)
- Secure Real Time Transport Protocol (SRTP, RFC 3711)
- ZRTP, RFC 6189
- ICE, RFC 5245 & RFC 6336
- TURN, RFC 5766
- RTP Payload for Text conversation, RFC 4103
- rtcp-mux, RFC 5761
- SRTP-DTSL, RFC 5763
AUDIO PROTOCOLS
- RTP Payload Format for the Opus Speech and Audio Codec, RFC 7587
- RTP Payload Format for the Speex Codec, RFC 5574
- Real-Time Transport Protocol (RTP) Payload Format and File Storage Format for the Adaptive Multi-Rate (AMR) and Adaptive Multi-Rate Wideband (AMR-WB) Audio Codecs
- only RTP octet-align=1 mode, without interleaving, crc, single channel
- RTP Payload for DTMF Digits, Telephony Tones, and Telephony Signals, RFC 4733
- RTP Payload Format for Named Telephone Events
- RTP Payload Format and File Storage Format for SILK Speech and Audio Codec
- RTP Payload Format for BroadVoice Speech Codecs, RFC 4298
- A Real-time Transport Protocol (RTP) Header Extension for Client-to-Mixer Audio Level Indication, RFC 6464
- A Real-time Transport Protocol (RTP) Header Extension for Mixer-to-Client Audio Level Indication, RFC 6465
VIDEO PROTOCOLS
- XML Schema for Media Control, RFC 5168
- Sending and processing of picture fast update command in SIP INFO
- RTP Payload Format for MPEG-4 Audio/Visual Streams, RFC 3016
- 3. RTP Packetization of MPEG-4 Visual bitstream
- RTP Payload Format for ITU-T Rec. H.263 Video, RFC 4629
- RTP Payload Format for H.264 Video, RFC 3984
- RTP Payload Format for VP8 Video
License
Mediastreamer2 and oRTP are dual licensed. It can be licensed and distributed:
- Under a GNU GPLv3 license, for free (open source)
- Under a proprietary license, for use in closed source applications. Contact Belledonne Communications for costs and other service information.
Source code
Source code can be downloaded from our GitLab.
You can also use Git to retrieve the latest source code (recommended for developers):
|
Project |
Git repository |
|
Mediastreamer2 |
git clone https://gitlab.linphone.org/BC/public/mediastreamer2.git |
|
oRTP |
Please refer to public header files for Mediastreamer2 documentation.
The oRTP API documentation is generated here (using Doxygen).
Footer mentions
© Copyright 2023 - Linphone - Belledonne Communications SARL