Skip to content

Asymmetric Multiprocessing: RPMsg device and driver on Linux and Android

Asymmetric Multiprocessing: RPMsg device and driver on Linux and Android.

Introduction

A crucial component for Inter-core communication in an Asymmetric Architecture is the RPMsg device. The RPMsg char driver exposes RPMsg endpoints to user-space processes. Multiple user-space applications can use one RPMsg channel uniquely by requesting different interactions with the remote service. The RPMSg char driver supports the creation of multiple endpoints for each RPMsg device, enabling the use of the same channel for different purposes. Each created endpoint device shows up as a single character device in /dev and access rights can be set independently for each individual endpoint; this holds for a Linux or Android kernel patched with the support for RPMsg.

The RPMsg bus sits on top of the VirtIO bus. Each virtio name service announcement message creates a new RPMsg device, which is supposed to bind to a RPMsg driver. RPMsg channels are created dynamically:

  • the remote device announces the existence of a remote RPMsg service by sending a name service announcement message containing the name of the service (i.e. name of the channel), source and destination addresses
  • the message is handled by the RPMsg bus, which dynamically creates and registers an RPMsg channel which represents the remote service
  • as soon as a relevant RPMsg driver is registered, it is immediately probed by the bus and the two sides can start exchanging messages

Virtio RPMsg was not originally implemented to use the RPMsg char driver. Each RPMsg char driver can either be bound to a simple RPMsg driver or to the RPMsg char driver. A patch was proposed to support dynamic endpoints with a name service per each RPMsg device. The changes from the patch (not applied in mainline yet) are the following:

  • the RPMsg device is created when the VirtIO RPMsg driver is probed
  • when a remote service is announced, it is kept in the virtio proc remote services list
  • when a relevant RPMsg driver is registered, it is immediately probed by the RPMsg bus

The following image shows the current Linux implementation of the RPMsg framework and the RPMsg char driver functionalities. Note: in this example only one endpoint is exposed.

RPMsg implementation

The driver was first introduced in the Linux 4.11 version, and sources can be found in the $(KERNEL_SRC)/drivers/rpmsg folder of mainline kernel (v4.6 at the time of writing). The implementation is based on prior art by Texas Instruments, Google, and PetaLogix and was derived from a FreeRTOS performance statistics driver written by Michal Simek.

The control interface

The RPMsg char driver provides a control interface (in the form of a character device under /dev/rpmsg_ctrlX), allowing user-space to export an endpoint interface for each exposed endpoint. The control interface provides a dedicated ioctl to create an endpoint:

/**
  * IOW command: needs to write some data to kernel space
  * Arguments
  *     - magic number to differentiate the set of ioctl calls from the others
  *     - command number: number assigned to the ioctl operation to distinguish commands from one another
  *     - type of data that will be written
  */
  #define RPMSG_CREATE_EPT_IOCTL  _IOW(0xb5, 0x1, struct rpmsg_endpoint_info)

The ioctl is fed the following information:

/**
 * struct rpmsg_endpoint_info - endpoint info representation
 * @name: name of service
 * @src: local address
 * @dst: destination address
 */
struct rpmsg_endpoint_info {
    char name[32];
    uint32_t src;
    uint32_t dst;
};

The three fields are used by the RPMsg backend to create the actual endpoint and by the RPMsg char driver to create a RPMsg endpoint device (under /dev/rpmsgX), along with the same three parameters in sysfs for udev usage. Multiple endpoints can be created using the control interface. Each created endpoint device is assigned a sequential number, to be distinguishable among each other.

The endpoint interface

For each endpoint, the RPMsg char driver uses a socket buffer queue to hold the data received on the channel destined to that endpoint. When data from the remote device arrives on the RPMsg channel, the RPMsg char driver allocates a socket buffer to hold the received data and enqueues it in the destination endpoint socket buffer queue.

Userspace applications can interact with the remote service by reading/writing data to/from the endpoint, through the endpoint interface in /dev/rpmsgX (X indicates the endpoint number):

  • read: when a read operation is requested on an endpoint device, the backend dequeues a socket buffer from the socket buffer queue associated to that endpoint (if not empty) and copies data to the user's buffer.
  • write: when a write operation is requested on an endpoint device, the backend copies the user's buffer to a kernel buffer and sends an RPMsg message with the buffer content to the remote device.

The endpoint interface provides an ioctl for destroying the endpoint:

/**
  * IOW command: does not need to write data to kernel space
  * Arguments
  *     - magic number to differentiate the set of ioctl calls from the others
  *     - command number: number assigned to the ioctl operation to distinguish commands from one another
  */
#define RPMSG_DESTROY_EPT_IOCTL _IO(0xb5, 0x2)

The rpmsg endpoint will be created and destroyed following the opening and closing of the endpoint device, allowing rpmsg backends to open and close the physical channel, where supported.

Usage from user-space

The following code snippet shows how a user-space application can create and interact with an endpoint device using the control and endpoint interfaces provided by the RPMsg char driver:

/** user-space application source code
  * example channel: rpmsg-openamp-demo-channel [src=0x2 ----> dst=0x1]
  */

msg_data_t data_buf;

struct rpmsg_endpoint_info ept_info = {"rpmsg-openamp-demo-channel", 0x2, 0x1};
int fd = open("/dev/rpmsg_ctrl0", O_RDWR);

/* create endpoint interface */
ioctl(fd, RPMSG_CREATE_EPT_IOCTL, &ept_info);  // /dev/rpmsg0 is created 

/* create endpoint */
int fd_ept = open("/dev/rpmsg0", O_RDWR); // backend creates endpoint

/* receive data from remote device */
read(fd_ept, &data_buf, sizeof(data_buf));

/* send data to remote device */ 
write(fd_ept, &data_buf, sizeof(data_buf));

/* destroy endpoint */
ioctl(fd_ept, RPMSG_DESTROY_EPT_IOCTL);

close(fd_ept);
close(fd);

References: