This is the first part in a series of articles covering my learnings and experiments with the Interprocess Communication mechanisms in XNU.

I figured giving some structure to what I was learning would allow me to grasp the concepts better, so please keep in mind this is written from a learning person perspective. Providing a comprehensive guide on the topic is a non-goal of the series.

In this part, I’ll provide some background on XNU, the importance of Interprocess Communication, and introduce the core mechanism - Mach messages.

First up, I’d like to link some resources that I’ve been using to learn more about Mach IPC:

*OS Internals by Jonathan Levin¹
Mach Overview from Apple²
Mach documentation from the GNU Hurd project³
Mach 3 Kernel Principles/Interfaces books

Prelude

XNU is the computer operating system kernel, and the name is an acronym for X is Not Unix. Why is it relevant at all? Well, XNU is the heart of all Apple consumer operating systems, from macOS, through iOS all the way to watchOS. XNU is a hybrid kernel combining the best of monolithic and microkernel designs. It is based on the Mach kernel and components from FreeBSD, leading to the kernel having two personalities. For example, similar to the concept of a process in UNIX there’s a task in Mach, and both are present in XNU.

Why IPC?

XNU is based on Mach, the microkernel. The principle of a microkernel design is providing only the bare minimum of the required functionality in the kernel space, such as threads, memory management and IPC. Additional features are moved into separate modules that can communicate using IPC and don’t require running in a privileged kernel mode.
The main advantage of the microkernel design is modularity and security. When subsystems don’t need to run in a privileged environment, their vulnerabilities are less of a risk for the entire system. On the other hand, the major challenge the design has faced historically was performance. Kernel subsystems that’d otherwise run in the same privileged space now had to cross the execution mode boundary to communicate with other parts of the kernel or user space modules. This is also why IPC in Mach is so important. For this design to be feasible, the communication has to be performant. While IPC in Mach is reportedly fast, the cost of crossing the memory/execution mode can be significant for the code powering the entire system.
Given the challenges of a microkernel design, XNU uses a hybrid design. Much of the functionality lives in the kernel space, but also many subsystems are backed by user space daemons. Apple is also actively deprecating third party kernel extensions, making them significantly harder to use and recommending developers to use user space alternatives⁴.

Mach messages

Mach messages was the only IPC mechanism used in the Mach kernel. Others used nowadays, such as sockets, pipes were added in XNU as a part of the BSD layer. Some of those are built on top Mach messages, so this is why I chose it to start with.

Before messages, there were ports

Mach messages being a message queue type of an IPC mechanism, require endpoints to send and retrieve messages through. In Mach, those are called ports. They are a kernel level abstraction and not directly exposed to the user space. Instead, ports can be accessed through port rights, such as a SEND or a RECEIVE right.
Each task has its namespace of port rights, and rights are named using 32-bit integers. Therefore port APIs often take an explicit task argument to disambiguate the name, unless otherwise known from the usage context.

Ports are not only used to establish a communication channel for services or between processes, but they also serve as an endpoint for resources. In user space, all tasks, threads or even semaphore handles and more are really send rights to ports that represent the underlying resources. This is a truly powerful abstraction because the way to access and modify the state of the task of a running program is just the same as for other tasks. All that is needed is to obtain a send right to their port.

Here for example are some typedefs from mach/mach_types.h showcasing the abstraction:

1
2
3
4
5
6
7


typedef mach_port_t             task_t;
typedef mach_port_t             thread_t;
typedef mach_port_t             semaphore_t;
typedef mach_port_t             lock_set_t;
typedef mach_port_t             ledger_t;
typedef mach_port_t             alarm_t;
typedef mach_port_t             clock_serv_t;

Communication principles

Ports are a unidirectional communication channel. There can only exist one holder of a receive right to a port, but there can be multiple send right owners. Rights can be transferred over messages. There can be only one receive right so it can be only moved, but owners of a send right can freely move/copy it and grant it to other tasks.

Bidirectional

Only unidirectional communication would be of limited usefulness, but given a unidirectional channel, a bidirectional communication can be established.

Given a priori that Alice, an owner of a receive right, and Bob, an owner of a send right to the same port, this is how bidirectional communication can be established between them:

Bob allocated a Mach port with a receive right.
He creates a send right to the port.
He transfers the send right in a message sent over to Alice.

Now, both Alice and Bob own receive and send rights to ports they can communicate over. Syncing up the communication over different ports could be cumbersome, so each Mach message can optionally include a reply port — which can be used to send a message response over.

Establishing connection

Earlier, we assumed Alice and Bob both had rights to a common port, but how does that happen in the first place? Well, unlike with, for example, sockets, direct communication can’t be established between two tasks directly. Instead, there’s a centralised broker service, a bootstrap server, used to do precisely that. The role of the bootstrap server in XNU is nowadays fulfilled by launchd⁵, but before its introduction, the init process and bootstrap server were separate. Communication with the bootstrap also happens using Mach messages, and this is where another property of ports comes in — each spawned process has a send right to the broker. The bootstrap server is responsible for service registration and lookup, and allows to establish a connection between two parties. One way this could happen is as follows:

Alice registers itself with the bootstrap server using some known name, e.g., com.example.alice.
Registration in practice means that Alice grants the broker a send right to a port it holds a receive right to.
Next, when Bob queries the bootstrap server for the com.example.alice name, it can copy its send right, grant it to Bob, and then the connection is established.

There’s one important security culprit to this mechanism.
Self-registration means that a malicious actor spawned before Alice could claim to provide the service. launchd introduced an alternative check-in mechanism. It can spawn the service on demand and create a receive right on its behalf and then transfer it.

Messages

Whereas Mach ports serve as a communication channel, messages are used to exchange data over those channels, so let’s have a look now at how that works.

First up is the message format. Each message consists of a header and body. Header is defined as:

1
2
3
4
5
6
7
8


typedef struct {
  mach_msg_bits_t       msgh_bits;
  mach_msg_size_t       msgh_size;
  mach_port_t           msgh_remote_port;
  mach_port_t           msgh_local_port;
  mach_port_name_t      msgh_voucher_port;
  mach_msg_id_t         msgh_id;
} mach_msg_header_t;

1
2
3
4
5
6


msgh_bits - options and message metadata, such as disposition of port rights in the message
msgh_size - total message size, including header
msgh_remote_port - remote Mach port, used as the destination when sending a message, or a reply port when receiving
msgh_local_port - local Mach port, the port the message was received on, or a reply port when sending a message
msgh_voucher_port - port identifying a Mach Voucher, that's an optional field
msgh_id - user defined message identifier

While each message consists of a header and body, this is not the complete story. When receiving a message, they additionally contain a trailer. Trailers contain information about the message and are appended by the kernel. All that’s necessary to know about trailers, for now, is that they exist, and that there’s a default/null type to begin with that contains only its type and size:

1
2
3
4
5
6
7
8


// mach/message.h

#define MACH_RCV_TRAILER_NULL   0

typedef struct {
  mach_msg_trailer_type_t       msgh_trailer_type;
  mach_msg_trailer_size_t       msgh_trailer_size;
} mach_msg_trailer_t;

The last piece of the puzzle is how messages can be exchanged. Main userland API for communication over Mach ports is mach_msg. Notably this single function is used to send as well as receive messages, the exact behaviour is specified using its arguments. The signature is:

1
2
3
4
5
6
7
8


mach_msg_return_t mach_msg(
  mach_msg_header_t *msg,
  mach_msg_option_t option,
  mach_msg_size_t send_size,
  mach_msg_size_t rcv_size,
  mach_port_name_t rcv_name,
  mach_msg_timeout_t timeout,
  mach_port_name_t notify);

1
2
3
4
5
6
7


msg - pointer to the message. Note that type specifies a message header, but this is because every message starts with a header, but the message size is variable
option - message options, such as basic `MACH_SEND_MSG` and `MACH_RCV_MSG` to send and receive a message correspondingly, but there're more options
send_size - size of the message `msg` buffer to send a message
rcv_size - size of the message `msg` buffer to receive a message
rcv_name - port to receive a message over 
timeout - mach_msg is blocking, and timeout can be specified as a maximum waiting time before call gives up when using MACH_SEND_TIMEOUT and MACH_RCV_TIMEOUT options
notify - a notification port, similar to vouchers it's optional

Example

It’s now time to put all the theory into practice. The example demonstrates step by step how to create a simple server and a client program communicating using Mach messages.

First, the definition of the message for the client program:

1
2
3
4
5


typedef struct {
  mach_msg_header_t header;
  char bodyStr[32];
  int bodyInt;
} Message;

Message structure consists of the required Mach message header and two body fields.

On the server side message has the same base contents, but it also includes the required default trailer data:

1
2
3
4
5
6
7


typedef struct {
  mach_msg_header_t header;
  char bodyStr[32];
  int bodyInt;

  mach_msg_trailer_t trailer;
} Message;

Server

The message structures are now defined, and we can move to write the service server program. This entails the following steps:

Create a Mach port receive right
Add a send right to the port for the bootstrap server to use
Retrieving a Mach port to communicate with the broker, as it uses Mach messages for the service registration
Registering our service
Message receiving routine
Main server loop to receive messages

#1 Create a receive right

mach_port_allocate creates a port right in the task namespace, whereas creating a receive right is equivalent to the creation of a Mach port, as there can be only one owner of the right:

1
2
3
4
5
6
7


mach_port_t task = mach_task_self();

mach_port_name_t recvPort;
if (mach_port_allocate(task, MACH_PORT_RIGHT_RECEIVE, &recvPort) !=
  KERN_SUCCESS) {
  return EXIT_FAILURE;
}

#2 Add a send right

Service registration requires a send right to the port, as that will allow the bootstrap server to copy the right when clients request it. Notably, send and receive rights use the same name in the task namespace. Therefore instead of using mach_port_allocate to allocate a new right as in the previous step, there’s a mach_port_insert_right API to add the send right:

1
2
3
4


if (mach_port_insert_right(
      task, recvPort, recvPort, MACH_MSG_TYPE_MAKE_SEND) != KERN_SUCCESS) {
  return EXIT_FAILURE;
}

#3 Retrieve the bootstrap port

Port of the bootstrap server will be needed later to register the service, it can be retrieved using task_get_special_port API:

1
2
3
4
5


mach_port_t bootstrapPort;
if (task_get_special_port(task, TASK_BOOTSTRAP_PORT, &bootstrapPort) !=
  KERN_SUCCESS) {
  return EXIT_FAILURE;
}

#4 Service registration

1
2
3
4
5


if (bootstrap_register(
      bootstrapPort, "xyz.dmcyk.alice.as-a-service", recvPort) !=
  KERN_SUCCESS) {
  return EXIT_FAILURE;
}

#5 Message retrieval routing

Using the earlier allocated port and mach_msg function messages can be retrieved from the queue:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


mach_msg_return_t receive_msg(mach_port_name_t recvPort) {
  // Message buffer.
  Message message = {0};

  mach_msg_return_t ret = mach_msg(
      /* msg */ (mach_msg_header_t *)&message,
      /* option */ MACH_RCV_MSG,
      /* send size */ 0,
      /* recv size */ sizeof(message),
      /* recv_name */ recvPort,
      /* timeout */ MACH_MSG_TIMEOUT_NONE,
      /* notify port */ MACH_PORT_NULL);
  if (ret != MACH_MSG_SUCCESS) {
    return ret;
  }

  printf("got message!\n");
  printf("  id      : %d\n", message.header.msgh_id);
  printf("  bodyS   : %s\n", message.bodyStr);
  printf("  bodyI   : %d\n", message.bodyInt);

  return MACH_MSG_SUCCESS;
}

#6 Main program loop

Server will indefinitely retrieve messages, until stopped:

1
2
3
4
5
6


while (true) {
  mach_msg_return_t ret = receive_msg(recvPort);
  if (ret != MACH_MSG_SUCCESS) {
    printf("Failed to receive a message: %d\n", ret);
  }
}

Client

Now onto the client side:

Like in server, we need to retrieve the bootstrap server port to query it for the service port
Query bootstrap for the service port
Setup message header
Fill message body
Send the message

#1 Retrieve the bootstrap port

1
2
3
4
5
6
7


mach_port_name_t task = mach_task_self();

mach_port_t bootstrapPort;
if (task_get_special_port(task, TASK_BOOTSTRAP_PORT, &bootstrapPort) !=
  KERN_SUCCESS) {
  return EXIT_FAILURE;
}

#2 Query bootstrap for the service port

1
2
3
4
5


mach_port_t port;
if (bootstrap_look_up(bootstrapPort, "xyz.dmcyk.alice.as-a-service", &port) !=
  KERN_SUCCESS) {
  return EXIT_FAILURE;
}

#3 Setup message header

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


Message message = {0};
message.header.msgh_remote_port = port;

// copy send right on the remote port
message.header.msgh_bits = MACH_MSGH_BITS_SET(
    /* remote */ MACH_MSG_TYPE_COPY_SEND,
    /* local */ 0,
    /* voucher */ 0,
    /* other */ 0);
message.header.msgh_id = 4;
message.header.msgh_size = sizeof(message);

#4 Fill the message body

1
2


strcpy(message.bodyStr, "Hello Mach!");
message.bodyInt = 0xff;

And then send the message:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


mach_msg_return_t ret = mach_msg(
    /* msg */ (mach_msg_header_t *)&message,
    /* option */ MACH_SEND_MSG,
    /* send size */ sizeof(message),
    /* recv size */ 0,
    /* recv_name */ MACH_PORT_NULL,
    /* timeout */ MACH_MSG_TIMEOUT_NONE,
    /* notify port */ MACH_PORT_NULL);

if (ret != MACH_MSG_SUCCESS) {
  printf("Failed mach_msg: %d\n", ret);
  return EXIT_FAILURE;
}

The same message structure can be then reused to send an extra message:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


strcpy(message.bodyStr, "Hello Mach! #2");

ret = mach_msg(
    /* msg */ (mach_msg_header_t *)&message,
    /* option */ MACH_SEND_MSG,
    /* send size */ sizeof(message),
    /* recv size */ 0,
    /* recv_name */ MACH_PORT_NULL,
    /* timeout */ MACH_MSG_TIMEOUT_NONE,
    /* notify port */ MACH_PORT_NULL);
if (ret != MACH_MSG_SUCCESS) {
  printf("#2. Failed mach_msg: %d\n", ret);
  return EXIT_FAILURE;
}

Result

Now both, service server and client programs are ready and can be tested in action.

First, start the server program:

1

$ ./one_mach_messages/alice_server

Then run the client in another shell session:

1

$ ./one_mach_messages/bob_client

And in the server session, the messages will arrive:

1
2
3
4
5
6
7
8


got message!
  id      : 4
  bodyS   : Hello Mach!
  bodyI   : 255
got message!
  id      : 4
  bodyS   : Hello Mach! #2
  bodyI   : 255

So that’s a success! We were able to exchange messages between the client and server programs.

lsmp

macOS comes in with a tool called lsmp, which can inspect ports actively used by different processes.
Running lsmp as a root allows it to look up right names in task local namespaces and cross-reference them to show a neat overview. To see this in action, add a sleep call to the client program to keep it alive.

We can then find the PID of the server program and use lsmp to find the active ports:

1
2
3
4
5
6
7
8
9


$ pgrep alice
3361
$ sudo lsmp -p 3361
...
  name      ipc-object    rights     flags   boost  reqs  recv  send sonce oref  qlimit  msgcount  context            identifier  type
0x00000b03  0xb21a4fbb  recv,send   --------     0  ---      1     1         Y        5         0  0x0000000000000000
                  +     send        --------        D--            1         <-                                       0x00167353  (1) launchd
                  +     send        --------        ---            1         <-                                       0x00001303  (3577) bob_client
...

Here the server process has a recv - receive and send rights to a port named 0xb03. Alike launchd, serving as the bootstrap server, and bob_client have send rights to the service port. lsmp is able to match corresponding port names between tasks to show this neat overview.

This concludes our unidirectional communication example using Mach messages over Mach ports.

Summary

To recap what we went over in this first part:

A brief history of XNU
What makes IPC so important in Mach/XNU
Core principles of Mach IPC
Overview of Mach ports and messaging APIs
Example server and client program

Full implementation of the example can be found at GitHub.

Next up I’d like to look at how the reply port can be used to establish a bidirectional communication, and what other payload types are supported, so that will likely be the topic of the second part. Stay tuned!

Update 07-10

Added information about port rights task namespace and the usage of a right name in the context of the server program. Thanks to @janseredynski for bringing it up!

Thanks for reading! If you’ve got any questions, comments, or general feedback, you can find all my social links at the bottom of the page.

XNU IPC - Mach messages

Contents