Adding the liblinphone dependency to your iOS project

Liblinphone for iOS is available using Cocoapods, the de-facto standard in the Apple developer world for "dependency management for Swift and Objective-C". Liblinphone can also be compiled from the sources.

Using Cocoapods

For a project named "Myproject"

open Myproject/, use commands:

pod init

Modify the generated Podfile according to your project:

# Uncomment the next line to define a global platform for your project

##For macosx
# platform :osx, '10.9'
#source "https://gitlab.linphone.org/BC/public/podspec-macos.git"

#For iOS
platform :ios, '9.0'
source "https://gitlab.linphone.org/BC/public/podspec.git"

target 'Myproject' do
 use_frameworks!

  # Pods for Myproject
 pod 'linphone-sdk' , '5.0'
end

 To install or update the liblinphone version, do the following command 

pod repo update
pod install

And you're done.

Some alpha (development versions) of liblinphone may also be available. To get them, replace the line:

pod 'linphone-sdk' , '5.0'

by

pod 'linphone-sdk', '> 5.1.0-alpha' 

followed by:

pod install

Compiling linphone-sdk

Linphone-sdk is the name of the git project that contains liblinphone and all its dependencies.

Compilation instructions of the SDK are available at: linphone-sdk/README.md

Once done, cocoapods needs to be invoked to update the Xcode workspace to use your locally built linphone-sdk, as follows:

PODFILE_PATH=<path to linphone-sdk-ios> pod install
where <path to linphone-sdk-ios>  is your build directory of the linphone-sdk project, containing the linphone-sdk.podspec file and a linphone-sdk output directory comprising built frameworks and resources.

Using liblinphone

Liblinphone has a C API, suitable to be used with Objective-C, and a modern Swift API. To use in swift, import the liblinphone swift module in your source file using

import linphonesw

API documentation

You can find the liblinphone C API documentation here.

For Swift, online documentation is available here . In addition, this iOS sample app shows how to use liblinphone in a Swift project.

Guidelines for integrating with push notifications

Introduction

An iOS application has in general a very limited capability to run in background, for example to keep a connection to a SIP server in order to receive calls and IM at any time. When the application goes in background, the network connections are killed and the application no longer executes.

The solution to circumvent this limitation is to rely on Apple's push notification system. The push notification first launch or resumes the app, that can then connect to the SIP service and retrieve the pending INVITE or MESSAGE request. In iOS there are three kinds of push notifications:

  • The Remote Notification, for general purpose notifications
  • The PushKit notifications, dedicated to VoIP calls. Unlike Remote Push Notifications, they do not display anything by themselves.
  • The Background Update Notifications, used for service like notifications. They are not real-time and severely rate limited (no more than 3 per hour). They are currently not used by Linphone, liblinphone or Flexisip.

New limitations and restrictions were added by Apple with Xcode 11 and iOS 13:

  • applications can no longer use PushKit kind of push notifications for anything else than presenting a VoIP call with CallKit. This is disruptive because previously, communication apps tend to also use PushKit to get notified of incoming IM messages. Since they can wake up the app in background, it was convenient to perform background processing before displaying the message to the user.
  • applications receiving a PushKit notification are required to immediately invoke Callkit to display the call to the end-user. This is disruptive also, because at the time of receiving the push notification, the app has not yet the signaling information about the call (ie, the SIP INVITE). The end user may even accept the call while the INVITE is not yet arrived, in which case the app has to postpone the call acceptance to the actual receiving of the INVITE, and also update the CallKit view with the missing information.

The next chapters explain the detailed steps to create a Liblinphone based application that comply with the above requirements. It can be used of course as a guide to upgrade an existing liblinphone based application to iOS 13 / Xcode 11.

Please note that an iOS application compiled with Xcode 10 still executes normally on iOS 13. The disruption happens only while compiling with Xcode 11.

Technical solutions to advertise incoming calls

The table below summarizes the technical solutions offered to advertise incoming calls.

Use casePush notification typeSolution
App showing calls with Callkit, with access to internet.Pushkit

Integrate CallKit into your application according to the guidelines provided in the section "Callkit integration" of this document.

This solution is the one implemented in the Linphone application.

App showing calls with Callkit, in an isolated network.No push notification at all.

Use UIApplication's keepAliveTimeout to trigger periodic calls to Core.refreshRegisters() and background sockets. Use Callkit in your app, but without Pushkit. Liblinphone by default attempts to activate background sockets.

Using this solution requires a special authorization from Apple. The app must have its main usage done without connection to the internet.

Basic app not showing calls with Callkit.

This is the case where the Callkit UI is not adapted.

For example: an app that needs to establish video at first, home automation app showing video before the call is accepted.

Remote Push Notifications

Use Remote Push Notifications in a basic scheme.

  • They will play a sound and vibrate once.
  • They cannot be cancelled (for example if the caller cancels the call).
  • The caller identity must be added by the SIP server in the remote push notification in order to be shown to the user.

When pressed by the user, the notification will launch or resume the app. The app can then connect to the SIP service and display the incoming call in its UI.

Advanced app not showing calls with Callkit.

This is the case where the Callkit UI is not adapted.

For example: an app that needs to establish video at first, home automation app showing video before the call is accepted.

Remote Push Notifications used together with a Notification Content Extension

Use Remote Push Notifications, but handle them inside an app extension of type "Notification Content". The app extension is a companion bundled with main application. It executes in a different process with restricted permissions, for example it cannot access geolocation information.

  • The app extension can vibrate repeatedly
  • The app extension can wait for the INVITE to be received, present caller information, and can be cancelled if the caller cancelled the call.
  • The app extension can show video received as early-media in the notification area.

When the notification is pressed by the user, the main app is resumed or launched. The app extension stops, and the call needs to be received a second time by the main app. The main app can know if the user has accepted the call in the notification, and then immediately transition the call to a running state.

Flexisip SIP proxy server version >= 2.0 has support for all the push notification solutions described above.

Push integration in SDK > 5.0

Several features have been added in the version 5.0 of the linphone SDK to make the integration of push notifications (voip & remote) easier for the application developper. If you wish to enable them, you can have a look here : Enabling 5.0 Push Management in SDK

CallKit integration

Starting from linphone-sdk >= 4.3, Callkit must be integrated in the following way:

  • Add the CallKit framework into your application's dependencies in Xcode.
  • Implement the CXProviderDelegate protocol within your ApplicationDelegate
  • Add the configuration "use_callkit=1" in the section "app" or call linphone_core_enable_callkit() before the linphone core starts. 
  • Your CallKit delegate MUST inform the LinphoneCore when AVAudioSession is activated, as follows:
func provider(_ provider: CXProvider, didActivate audioSession: AVAudioSession) {
        lc.activateAudioSession(actived: true)
 }

func provider(_ provider: CXProvider, didDeactivate audioSession: AVAudioSession) {
        lc.activateAudioSession(actived: false)
 }
  •  Since linphone-sdk >= 4.5, your CallKit delegate MUST configure the AVAudioSession when a new call is incoming/outgoing, as follows:
func provider(_ provider: CXProvider, perform action: CXAnswerCallAction) {
    CallManager.instance().lc?.configureAudioSession()
   // accept call here
    action.fulfill()
}

func provider(_ provider: CXProvider, perform action: CXStartCallAction) {
   CallManager.instance().lc?.configureAudioSession()
  // start an outgoing call
   action.fulfill()
}

Display an incoming call

let update = CXCallUpdate()
update.remoteHandle = CXHandle(type:.generic, value: "Call incoming")
let uuid = UUID()
provider.reportNewIncomingCall(with: uuid, update: update) {error in}

Answer an incoming call

Since iOS 13, Apple requests CallKit to be invoked to display the incoming call immediately when a PushKit notification is received. So sometimes you can answer the CallKit before a LinphoneCall is received. In the callback CXAnswerCallAction,  if a LinphoneCall has not yet been received,  you need to configure your AVAudioSession and accept the call when you receive it. Otherwise, accept the call directly.

func provider(_ provider: CXProvider, perform action: CXAnswerCallAction) {
   if (call == nil || call.state != Call.State.IncomingReceived) {
           // configure audio session here. Use 48000 Hz as sampling rate.
    } else {
       // accept call here
        call.acceptWithParams(params: callParams)
    }
    action.fulfill()
}

Terminate a call

func provider(_ provider: CXProvider, perform action: CXEndCallAction) {
   // terminate call here
    call.terminate()
    action.fulfill()
}

Start an outgoing call

When starting an outgoing call from the application view, it must following these steps:

 Before starting an outgoing call, make sure others calls are paused by callkit.

 Then add these codes to your class when starting an outgoing call

let handle = CXHandle(type: .generic, value: displayName)
let startCallAction = CXStartCallAction(call: uuid, handle: handle)
let transaction = CXTransaction(action: startCallAction)
let callController = CXCallController()
callController.request(_ transaction: transaction, completion: @escaping (Error?) -> Void)

 Then this callback will be called.

func provider(_ provider: CXProvider, perform action: CXStartCallAction) {
  // start an outgoing call
   lc.inviteAddressWithParams(addr: addr, params: params)
   action.fulfill()
}

To ensure the audio session works properly for this outgoing call.

// When outgoing call is created
reportOutgoingCall(with UUID: UUID, startedConnectingAt dateStartedConnecting: Date?)
// When outgoing call is answered
reportOutgoingCall(with UUID: UUID, connectedAt dateConnected: Date?)

Hold/resume a call

Before resuming a call, make sure others calls are paused by callkit.

func provider(_ provider: CXProvider, perform action: CXSetHeldCallAction) {
  // If this is a conference, leave/enter the conference.
  // Otherwide, pause/resume the call.
   call.pause()
   action.fulfill()
}

Mute/un-mute a call

func provider(_ provider: CXProvider, perform action: CXSetMutedCallAction) {
   // Mute/un-mute a call
    lc.micEnabled = false
    action.fulfill()
}

Play DTMF

func provider(_ provider: CXProvider, perform action: CXPlayDTMFCallAction) {
   // Send DTMF
    call.sendDtmf(dtmf: digit)
    action.fulfill()
}

Group calls

func provider(_ provider: CXProvider, perform action: CXSetGroupCallAction) {
   // add all to conference
    lc.addAllToConference()
    action.fulfill()
}

Transfer a call

Callkit does not support call transfer when Linphone does. If you wan to realise the call transfer with Callkit, you must follow these tips:

  • For callkit, an unique uuid represents a call.
  • For Linphone, the referred call will use a different callid than the original call.

Here is the integration of callkit in  linphone-iphone/ProviderDelegate.swift.

Troubleshooting

Distorted voice when using bluetooth devices ( using SDK 4.4.x)

Error log: "Too many samples to drop, dropping entire frame."

Solution:  Add callback of AVAudioSessionRouteChangeNotification in your application, and call linphone_core_audio_route_changed(lc) in this callback. Here is the integration of Linphone linphone-iphone/LinphoneManager.m.

Technical solutions to advertise instant messages

The table below summarizes the possible options. Remember that using PushKit is no longer possible since iOS 13 / Xcode 11.

Use casePush notification typeSolution
Basic app, not using end to end encryption.Remote Push Notification

The sender identity and the text content must be placed into the push notification sent by the server. The push notification is displayed to the user. When pressed, the app is resumed or launched, and can display the conversation related to the received notification.

Please note that the app has no way to perform any kind of pre-processing on the notification, such as perform deciphering of the message or fetching additional content (photos for example).

Advanced app, with security requirements.Remote Push Notification used together with a Notification Service Extension.

Use Remote Push Notifications handled by an app extension of type "Notification Service Extension".

The app extension is bundled with the main app. It executes in a different process with restricted permissions, for example it cannot access geolocation information.

The Notification Service Extension will receive the remote push notification silently, in background. It has a constrained period of time to do some processing: connect to the SIP service, get the instant message, decipher it, and at the end update the notification to fill it with sender identity and clear text message. The notification is then displayed normally by iOS.

When pressed, the main application is launched and can display the conversation that relates to the received message. The app extension and the main app have a shared filesystem and messaging framework, with synchronisation mechanisms.

This solution is presented with details in the below section "Advertising instant messages with a Notification Service Extension". It is the one implemented in the Linphone application.

Advertising instant messages with a Notification Service Extension

This section describes a step by step procedure to implement state of art receiving of instant messages in a liblinphone based application, thanks to Notification Service Extension.

Create the UNNotificationServiceExtension App Extension

A Remote push notification cannot resume an application to perform tasks in background. However it can do so with an app extension. An app extension is a process that is linked to the app but it doesn't share the same container (not the same process, not the same file system). The first step is then to create a UNNotificationServiceExtension.

Its goal is to catch the Remote push notification and modify the user notification before it is displayed. For this we need to connect to the server, get the new message then we can display the user notification.

In Xcode, in the “capabilities” menu, you must add the App Group capability for both the app and the app extension with the same App Group identifier: the AppGroupId. This identifier is used to reach the file system that is shared between the main App and the UNNotificationServiceExtension.

Create a Shared Core in app

Since version 4.4, liblinphone has a new concept of Shared Linphone Core to be able to manage a LinphoneCore in both the app and the app extension. The goal of the Shared Cores is to be able to create several Cores at the same time with the same configuration while making sure that only one Shared Core can write to the configuration files and the databases at the same time.

There are two types of Shared Cores:
    - The Main Shared Core is the Core owned by the application. It can stop the Executor Shared Core when it needs to start. It is the one that has the priority.
    - The Executor Shared Core is the Core of the app extension. It can't start if another Shared Core is already running and it can be stopped by the Main Shared Core at any time.

The mutual exclusion between Shared Cores is handled by the liblinphone.

The liblinphone methods for creating the Shared Core exists in C and Swift. A newly written application will prefer to use Swift, while an already existing application written in Objective-C can use C.

The Main Shared Core is to be created as follows:

  1. Use linphone_config_new_for_shared_core(app_group_id, config_filename, factory_path) taking your AppGroupId and a config filename (i.e. linphonerc). It computes the path in the shared memory for your config file. The factory config path works as for a normal LinphoneCore. It returns a LinphoneConfig object suitable to instantiate a Shared Core.
  2. Use linphone_factory_create_shared_core_with_config(factory, config, system_context, app_group_id, main_core) to create the Shared Core. Set main_core=TRUE as we are creating a Main Shared Core for the app.

Note: linphone_factory_create_shared_core (factory, config_filename, factory_config_path, system_context, app_group_id, main_core) combines the two steps in a single function.

Stop the Main Shared Core when going to background

The app extension will need to start an Executor Shared Core to get the new messages. To allow that, you need to stop the Main Shared Core each time the app goes in background. Reciprocally you need to start it again when the app goes to foreground.

  • In (void)applicationDidEnterBackground: (UIApplication *)application : you need to call linphone_core_stop()
  • In (void)applicationWillEnterForeground: (UIApplication *)application : you need to call linphone_core_start()

Migrate all the configuration files to the shared file system (optional)

This section is applicable to apps using liblinphone before Xcode 11 arrived.

You have created a Main Shared Core instead of the usual Core. Now all the configuration files need to be moved to the shared file system in order to still have the SIP account information, the call logs, and the conversation history, etc.

For this we use these three functions:

- const char *linphone_factory_get_config_dir(LinphoneFactory *factory, void *context) that leads to /Library/Application Support/linphone/

- const char *linphone_factory_get_data_dir(LinphoneFactory *factory, void *context) that leads to /Library/Preferences/linphone/

- const char *linphone_factory_get_download_dir(LinphoneFactory *factory, void *context) that leads to /Library/Caches/

These functions return paths to the different iOS application configuration directories, depending on the context argument. If context = NULL, the returned path is in the application file system. If context = AppGroupId, the path leads to the file system shared between the app and the app extension.

Set up the app extension to get the message

The function didReceive() of the NotificationService app extension is called when a Remote push notification is received. It has ~30 seconds to get the message before the user notification is finally displayed by the system.

Please note that the push notification body must contain two string attributes (in its dictionary):

  • call_id : the call_id of the SIP MESSAGE carrying the IM, when the push notification is related to a message.
  • chat_room_addr : the SIP address of a chatroom in which the user is being INVITEd, when the push notification is related to an INVITE to join a user to chat room.

These are strong requirements. Flexisip proxy server of course takes care of adding these attributes since version 2.0.

An Executor Shared Linphone Core is required to retrieve the message. It is created as in step 2 but by setting main_core=FALSE. The main shared core running in the main application can stop the Executor Shared Core when the user puts the application in foreground.

These two methods are useful to perform the tasks of the app extension:

- LinphonePushNotificationMessage *linphone_core_get_new_message_from_callid(lc, call_id) : to get the message from the call_id attribute embedded in the push notification payload. 

- LinphoneChatRoom *linphone_core_get_new_chat_room_from_conf_addr(lc , chat_room_addr) : to get the new chat room from the chat_room_addr attribute embedded in the push notification. The app extension can use this information to indicate to the user that has been invited into a new chat room.

Don't call linphone_core_start() yourself. The above two functions will start the Shared Core if needed.

Once ever of the two functions has returned, linphone_core_stop() must be called, in order to allow the core to perform additional actions that usually follow the receiving of a message, such as sending a delivery notification (IMDN), and finally release the shared core lock, to let other push notification to be processed by other notification service extension instances.

iOS launches a new instance of the app extension for every push notification received. Proper synchronisation between these multiple instances and the main shared core is performed by liblinphone internally.

These two methods used by the app extension work when the main application is in background or in foreground. This means that the main application does not need to manage incoming messages. The app extension will take in charge the notification of ALL the incoming messages to the user.

If the application existed before XCode 11, it was probably user local notification to show IMs: this code needs to be removed now that the app extension is handling incoming IMs.

Adding actions in user notifications: UNNotificationContentExtension

Linphone also uses the UNNotificationContentExtension app extension in addition of the UNNotificationServiceExtension, in order to implement the reply and mark as seen buttons.

The user notifications displayed by the NotificationService extension need a NotificationContent extension to add a NotificationCategory as well as some actions attached to this category. The categories and actions declared in the main app won't work when the app is in background.

If the app is in foreground, the NotificationContent extension won't be able to start a Shared Core so the app will need to handle the actions.

To summarize, the notification categories and attached actions must be declared in both the NotificationContent extension and the main app, despite the Notification Service Extension is centralizing the incoming push notifications and the display of the IM.

Update Contact URI parameters

This section is only relevant if you did not enable Core.pushEnabled in your app and wish to implement push notification management yourself. If that is not the case, have a look over here to see how to use the linphone sdk push management.

In order to get push notifications, the iOS app must send its push token(s) to the SIP server.

These push paramers are sent in the Contact URI parameter in the REGISTER.

We recommend the syntax and principles of RFC8599 - Push Notification with the Session Initiation Protocol, on top of which we had unfortunately to plug extension to cover the specificities of iOS since version 13, because the app will actually need two different push notification tokens (one for Callkit, one for Remote Notifications).

The extensions we made are summarized by this ABNF grammar:

  • "pn-prid=“ token [ “:” service ]  *( “&” token “:” service )
  • "pn-param=" TeamId ”.” BundleId ”.” service *( “&” service )
  • "pn-provider=apns"
  • service = "voip" /  "remote"
  • token = <voip push token or remote push token>

Example with voip and remote push token:

  • pn-prid=00fc13adff78512:voip&c11292f7b74733d:remote
  • pn-param=DEF123GHIJ.org.linphone.phone.voip&remote
  • pn-provider=apns

Example only with remote push token:

  • pn-prid=c11292f7b74733d:remote
  • pn-param=DEF123GHIJ.org.linphone.phone.remote
  • pn-provider=apns

The "voip" service stands for PushKit kind of notifications, while the "remote" service is for the basic "Remote Notification" system.

These parameters must be added at application to the ProxyConfig object using linphone_proxy_config_set_contact_uri_parameters().

Bugs

It appears that in iOS < 12, if the UNNotificationServiceExtension takes more than 1 second to start, it is killed by the system. Once it has successfully started once, the process is kept in cache and works every time. It is removed from the cache after ~30 minutes without any push notification.

When the app extension fails to start, the push notification can't be caught to retrieve the message. We used the "loc-key" argument of the push notification JSON payload to put a key. This key is an entry in our translation files. So, when the app extension fails to start, we will display a localized message similar to "You have received a new message".

This limitation is not documented by Apple and we haven't been able to reproduce it in iOS versions >= 12.

Beyond the scene

This page describes the internal mechanisms used by liblinphone for Shared Core synchronisation.

Handling liblinphone log

In order to see liblinphone logs in your IOS app (for example in your Xcode console) follow these steps :

Put in your code at the launching of the app :

linphone_core_set_log_handler(your_log_handler);

This will make the liblinphone logs to call your_log_handler and so be processed as a log from your app.

You can set the liblinphone log level by using the functions documented here.

IOS log handler for liblinphone

Once you have set your log handler, you need to process liblinphone log in order to incorporate them into your app logs.

Here is a short example of how to manage liblinphone log into an IOS app :

void linphone_iphone_log_handler(const char *domain, OrtpLogLevel lev, const char *fmt, va_list args) {
NSString *format = [[NSString alloc] initWithUTF8String:fmt];
NSString *formatedString = [[NSString alloc] initWithFormat:format arguments:args];
NSString *lvl;

if (!domain)
  domain = "lib";
// since \r are interpreted like \n, avoid double new lines when logging network packets (belle-sip)
// output format is like: I/ios/some logs. We truncate domain to **exactly** DOMAIN_SIZE characters to have
// fixed-length aligned logs
switch (lev) {
 case ORTP_FATAL:
   lvl = @"Fatal";
  break;
 case ORTP_ERROR:
   lvl = @"Error";
  break;
 case ORTP_WARNING:
   lvl = @"Warning";
  break;
 case ORTP_MESSAGE:
   lvl = @"Message";
  break;
 case ORTP_DEBUG:
   lvl = @"Debug";
  break;
 case ORTP_TRACE:
   lvl = @"Trace";
  break;
 case ORTP_LOGLEV_END:
  return;
 }
if ([formatedString containsString:@"\n"]) {
 NSArray *myWords = [[formatedString stringByReplacingOccurrencesOfString:@"\r\n" withString:@"\n"]
  componentsSeparatedByString:@"\n"];
 for (int i = 0; i < myWords.count; i++) {
  NSString *tab = i > 0 ? @"\t" : @"";
  if (((NSString *)myWords[i]).length > 0) {
    NSLog(@"[%@] %@%@", lvl, tab, (NSString *)myWords[i]);
   }
  }
 } else {
  NSLog(@"[%@] %@", lvl, [formatedString stringByReplacingOccurrencesOfString:@"\r\n" withString:@"\n"]);
 }
}

 

Tags: IOS