This is the second article in the series covering my learnings and experiments with the Interprocess Communication mechanisms in XNU. In this part I’ll walk you through some new Mach messages APIs and cover:

  • Bidirectional communication.
  • Complex Mach messages with type descriptors.
  • Smaller Mach API extensions along the way.

Previous article was an introduction to Mach IPC. It covers Mach ports, messaging APIs, and an example client/server implementation. Unless you’re already familiar with those concepts I’d recommend to start from there.

Beyond unidirectional communication

Previous example has shown how to create a Mach IPC server and how clients can communicate with it. We’ll start where we’ve left of, gradually expanding the programs, with the goal of establishing a bidirectional communication. Along the way you’ll discover more functionalitites of Mach message APIs.

The server already has a Mach port and a receive right to it. It also granted a send right to that port to the bootstrap server to advertise its service. The client can receive a send right to the port through the bootstrap. This means we’ve established a client -> server communication channel.

Now we also want the server to be able to respond to those messages. Mach ports are a unidirectional communication channel, a single port won’t suffice to achieve this. The client has to create a new port and own a receive right to it, to receive messages. And just like server did for the client, the client has to somehow provide the server with a send right to that port. At this point it’s no longer necessary to involve the bootstrap server. Mach messages can be used to transfer port rights, so we can leverage the existing connection.

Below’s an overview of the entire process that will allow our programs to talk to each other:

Establishing bidirectional communication

You already know how to register serivces and do the lookup. You also know how to create ports and insert new rights. What’s left to figure out is how to send port rights over Mach messages. There are two ways to achieve this that we’ll explore:

  • Through the msgh_local_port header field, which serves as a reply port when sending a message.
  • Manually using port descriptors.

With the next steps layed out we can now move onto the examples. Go ahead and grab the previous example, if you’d like to follow along. We’ll build our new client and server programs on top of it, starting with the msgh_local_port based implementation.

Reply port

Before we dive into the code, first a quick recap of the message header structure.

1
2
3
4
5
6
7
8

typedef struct {
  mach_msg_bits_t       msgh_bits;
  mach_msg_size_t       msgh_size;
  mach_port_t           msgh_remote_port;
  mach_port_t           msgh_local_port;
  mach_port_name_t      msgh_voucher_port;
  mach_msg_id_t         msgh_id;
} mach_msg_header_t;

Header has the msgh_remote_port field to set the destination port when sending a message. Similarily, there’s the msgh_local_port, which is the source port when sending a message and what the receiver can use to send a response. Each of the port fields have their dedicated bits in the msgh_bits field which define the port disposition.

Client

Let’s start from where we queried bootstrap for the service port. So this part:

1
2
3
4
5

mach_port_t port;
if (bootstrap_look_up(bootstrapPort, "xyz.dmcyk.alice.as-a-service", &port) !=
  KERN_SUCCESS) {
  return EXIT_FAILURE;
}

First, you need to create a port, and a send right to it, to use as the local port.

#1 Create a new receive right

1
2
3
4
5

mach_port_t replyPort;
if (mach_port_allocate(task, MACH_PORT_RIGHT_RECEIVE, &replyPort) !=
    KERN_SUCCESS) {
  return EXIT_FAILURE;
}

#2 Add a send right

1
2
3
4
5

if (mach_port_insert_right(
        task, replyPort, replyPort, MACH_MSG_TYPE_MAKE_SEND) !=
    KERN_SUCCESS) {
  return EXIT_FAILURE;
}

#3 Prepare the message header

The message now includes both, the remote and local ports:

1
2
3

Message message = {0};
message.header.msgh_remote_port = port;
message.header.msgh_local_port = replyPort;

So you also need to specify the type of processing for both local and remote ports through the msgh_bits field. For the local port, we will use a new type of a message right.
Thus far, we’ve talked about send and receive rights, but there’s one more - a send once right. As the name indicates, it is a send right that can be used only once. Unlike the send right, the send once right can only be moved, and once used, it’s invalidated. This means the client allows the server to respond only one time to its message.

1
2
3
4
5

message.header.msgh_bits = MACH_MSGH_BITS_SET(
    /* remote */ MACH_MSG_TYPE_COPY_SEND,
    /* local */ MACH_MSG_TYPE_MAKE_SEND_ONCE,
    /* voucher */ 0,
    /* other */ 0);

#4 Send the message

1
2
3
4
5
6
7
8

mach_msg_return_t ret = mach_msg(
    /* msg */ (mach_msg_header_t *)&message,
    /* option */ MACH_SEND_MSG,
    /* send size */ sizeof(message),
    /* recv size */ 0,
    /* recv_name */ MACH_PORT_NULL,
    /* timeout */ MACH_MSG_TIMEOUT_NONE,
    /* notify port */ MACH_PORT_NULL);

#5 Receiving a reply

Now that the client’s message includes a reply port, it can expect a message back. We’ve already implemented message retrieval in the server program. Client side implementation will be very similar, but with a small extension - a timeout. By default mach_msg blocks the thread until a new message arrives. Using timeout you can specify the maximum wait time, and take action if there’re no new messages before the deadline.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

ReceiveMessage receiveMessage = {0};

mach_msg_return_t ret = mach_msg(
    /* msg */ (mach_msg_header_t *)&receiveMessage,
    /* option */ MACH_RCV_MSG | MACH_RCV_TIMEOUT,
    /* send size */ 0,
    /* recv size */ sizeof(receiveMessage),
    /* recv_name */ recvPort,
    /* timeout in ms */ 1000,
    /* notify port */ MACH_PORT_NULL);

Note that besides for the timeout parameter you also need to use the MACH_RCV_TIMEOUT option to specify what the timeout applies to. In this case it’s a timeout for receiving a message.

Let’s wrap this logic in a message retrieval routine for reusability:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

mach_msg_return_t receive_msg(
    mach_port_name_t recvPort,
    mach_msg_timeout_t timeout) {
  // Message buffer.
  ReceiveMessage receiveMessage = {0};

  mach_msg_return_t ret = mach_msg(
      /* msg */ (mach_msg_header_t *)&receiveMessage,
      /* option */ MACH_RCV_MSG | MACH_RCV_TIMEOUT,
      /* send size */ 0,
      /* recv size */ sizeof(receiveMessage),
      /* recv_name */ recvPort,
      /* timeout */ timeout,
      /* notify port */ MACH_PORT_NULL);

  return ret;
}

And in the main function add a loop to receive messages with a timeout:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

while (ret == MACH_MSG_SUCCESS) {
  ret = receive_msg(replyPort, /* timeout */ 1 * MS_IN_S);
}

if (ret == MACH_RCV_TIMED_OUT) {
  printf("Receive timed out, no more messages from alice yet.\n");
} else if (ret != MACH_MSG_SUCCESS) {
  printf("Failed to receive a message: %#x\n", ret);
  return 1;
}

Server

Now that the client can provide the server with a send right, the server can reply. But before you do that, there’s something important to reiterate about the port semantics in Mach message.

When sending a message the remote port field is the destination, and local port field is the source. When the message is delivered the semantics of local and remote ports do match the recipient’s perspective. The local port of the sender is the remote port of the receiver, and remote port of the sender is the local port of the receiver. Kernel automatically maps those two fields, and not only, ports disposition in the message bits have slightly different meaning too.

When sending a message, values such as MACH_MSG_TYPE_MOVE_SEND or MACH_MSG_TYPE_MOVE_SEND_ONCE indicate how the port rights are transferred.
On the receiving end, the disposition of ports tell what right the receiver has been granted for a given port. There you can use MACH_MSG_TYPE_PORT_* values to distinguish what right you’ve received. Such as the MACH_MSG_TYPE_PORT_SEND or MACH_MSG_TYPE_PORT_SEND_ONCE value.

With this in mind, let’s now teach the server how to send replies.

#1 Response routine

The response routine will be very similar to how you’ve been sending messages so far. This time however, instead of setting the message bits manually you can reuse remote port disposition from the incoming message using MACH_MSGH_BITS_REMOTE_MASK mask :

1
2
3
4
5
6
7

mach_msg_return_t send_reply(mach_port_name_t port, const Message *inMessage) {
  Message response = {0};
  response.header.msgh_bits =
      inMessage->header.msgh_bits &
      MACH_MSGH_BITS_REMOTE_MASK;

  response.header.msgh_remote_port = port;

This is possible because the MACH_MSG_TYPE_PORT_SEND, MACH_MSG_TYPE_PORT_SEND_ONCE values map directly to how rights are transferred when sending messages:

1
2

#define MACH_MSG_TYPE_PORT_SEND         MACH_MSG_TYPE_MOVE_SEND
#define MACH_MSG_TYPE_PORT_SEND_ONCE    MACH_MSG_TYPE_MOVE_SEND_ONCE

So you can apply the remote mask on the bits of the received message to automatically reply to the message without having to distinguish what kind of a send right you’ve received exactly.

Note that this works in scenarios where only one message has to be sent back. If client has given you a send right, and you’d like to send multiple messages back, you’d need to explicitly use MACH_MSG_TYPE_COPY_SEND to send the response and maintain the ownership of the send right.

#2 Sending the response

With the message sending routine ready, let’s extend the message retrival loop and send a message back.

First, verify that the message does indeed include a reply port:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

while (true) {
  // Message buffer.
  ReceiveMessage receiveMessage = {0};

  mach_msg_return_t ret = receive_msg(recvPort, &receiveMessage);
  if (ret != MACH_MSG_SUCCESS) {
    printf("Failed to receive a message: %#x\n", ret);
    continue;
  }

  // Continue if there's no reply port.
  if (receiveMessage.message.header.msgh_remote_port == MACH_PORT_NULL) {
    continue;
  }

  ...
}

If it does, you can add a call to the message send function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

  // Continue if there's no reply port.
  if (receiveMessage.message.header.msgh_remote_port == MACH_PORT_NULL) {
    continue;
  }

  // Send a response.
  ret = send_reply(
      receiveMessage.message.header.msgh_remote_port,
      &receiveMessage.message);

  if (ret != MACH_MSG_SUCCESS) {
    printf("Failed to respond: %#x\n", ret);
  }

Result

Now when you start the server program, and run the client, you should see those messages in the client output:

1
2
3
4
5
6
7
8
9

got response message!
  id      : 4
  bodyS   : Response - Hello Mach!
  bodyI   : 510
got response message!
  id      : 4
  bodyS   : Response - Hello Mach! #2
  bodyI   : 510
Receive timed out, no more messages from alice yet.

The client has sent two messages, the server has responded to both, and then client finished execution as it has reached the timeout for receiving a new message.

Complex messages

So far we’ve said that Mach messages consist of the message header, and the variable-sized body. This is different for complex messages. If the message is complex, its header is followed by a count of type descriptors and the type descriptors themselves, and only then the inline data. Type descriptors in complex messages allow to exchange port rights, out of line data and more. This exchange requires kernel involvement, and so this is why complex messages have different structure. Kernel needs to know where to find the type descriptors.

Complex messages are marked using the the MACH_MSGH_BITS_COMPLEX bit in msgh_bits field. It can be easy to forget about this and it’s not that fun to debug. While preparing the examples I’ve forgotten more than once to set it and then couldn’t figure out for a while why the code wasn’t working.

To get familiar with the concept of complex messages and type descriptors we’ll reimplement the bidirectional communication funcionality, but using port descriptors instead of the local port field. Like previously, let’s start with the client implementation.

Client

Same as in the previous example, the client needs a new port and a receive right to it. Then port descriptor in our message will cary over a send right to the server process.

#1 Message structure

To include a port descriptor you need to send a complex Mach message. The following is the message structure:

1
2
3
4
5

typedef struct {
  mach_msg_header_t header;
  mach_msg_size_t msgh_descriptor_count;
  mach_msg_port_descriptor_t descriptor;
} PortMessage;

PortMessage consists of the message header, the number of type descriptors, a single port descriptor and it has no inline data.

#2 Create a new receive right

Same as in the previous example, client needs to own a receive right to a port so that it can receive a response from the server program.

1
2
3
4
5

mach_port_t replyPort;
if (mach_port_allocate(task, MACH_PORT_RIGHT_RECEIVE, &replyPort) !=
    KERN_SUCCESS) {
  return EXIT_FAILURE;
}

#3 Add a send right

1
2
3
4
5

if (mach_port_insert_right(
        task, replyPort, replyPort, MACH_MSG_TYPE_MAKE_SEND) !=
    KERN_SUCCESS) {
  return EXIT_FAILURE;
}

#4 Message header - message bits

For the kernel to process the type descriptors you need to indicate that the message is complex, otherwise they’re treated as if it was just inline data.

1
2
3
4
5

message.header.msgh_bits = MACH_MSGH_BITS_SET(
    /* remote */ MACH_MSG_TYPE_COPY_SEND,
    /* local */ 0,
    /* voucher */ 0,
    /* other */ MACH_MSGH_BITS_COMPLEX);

#5 Message header - id and size

In this example our client and server can exchange two types of messages:

  • The simple message with inline data.
  • The complex message with a port descriptor.

To ditinguish the two, we will introduce and use message ids:

1
2
3
4
5
6

#define MSG_ID_DEFAULT 1
#define MSG_ID_PORT 2

...

message.header.msgh_id = MSG_ID_PORT;

Last step in the header preparation is the message size:

1

message.header.msgh_size = sizeof(message);

#6 Port descriptor

Passing port descriptors in Mach messages in quite straighforward. First, you need to specify how many descriptors the message includes:

1

message.msgh_descriptor_count = 1;

As you may remember from the message structure definition, we’re using the mach_msg_port_descriptor_t descriptor type:

1
2
3
4
5
6
7

typedef struct{
  mach_port_t                   name;
  mach_msg_size_t               pad1;
  unsigned int                  pad2 : 16;
  mach_msg_type_name_t          disposition : 8;
  mach_msg_descriptor_type_t    type : 8;
} mach_msg_port_descriptor_t;

Crucial field of descriptors is type, each type descriptor has it, as kernel needs to know what kind of a type descriptor it’s dealing with. In this case it’s the port descriptor:

1

message.descriptor.type = MACH_MSG_PORT_DESCRIPTOR;

Now you can fill the port right that the descriptor carries:

1

message.descriptor.name = replyPort;

And at last, define what kind of IPC processing kernel has to perform on the port. In this case, client grants the server a send right to its port:

1

message.descriptor.disposition = MACH_MSG_TYPE_MAKE_SEND;

#7 Send the message

1
2
3
4
5
6
7
8

mach_msg_return_t ret = mach_msg(
    /* msg */ (mach_msg_header_t *)&message,
    /* option */ MACH_SEND_MSG,
    /* send size */ sizeof(message),
    /* recv size */ 0,
    /* recv_name */ MACH_PORT_NULL,
    /* timeout */ MACH_MSG_TIMEOUT_NONE,
    /* notify port */ MACH_PORT_NULL);

#8 Receiving a reply

The routine to receive a message in the client program works the same way as in the previous example. It’s only server that has to receive complex messages.

Server

On the server side there’s a bit more work, since we’re no longer relying on the builtin local port field. Instead, the server has to deal with different message types.

#1 Message types

Let’s have a look at the existing message types and define some helpers to deal with both:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

typedef struct {
  mach_msg_header_t header;
  char bodyStr[32];
  int bodyInt;
} Message;

typedef struct {
  Message message;

  mach_msg_trailer_t trailer;
} ReceiveMessage;

typedef struct {
  mach_msg_header_t header;
  mach_msg_size_t msgh_descriptor_count;
  mach_msg_port_descriptor_t descriptor;
} PortMessage;

typedef struct {
  PortMessage message;

  mach_msg_trailer_t trailer;
} ReceivePortMessage;

typedef union {
  mach_msg_header_t header;

  ReceiveMessage message;
  ReceivePortMessage portMessage;
} ReceiveAnyMessage;

The type you can use when receiving either of the messages is the ReceiveAnyMessage type. Note that it’s a union, so it can store either of the message types. I’ve also added a helper header field to the union. This field can be accessed regardless of which message type the union currently stores since both messages start with the header structure.

#2 Message retrieval

Now let’s extend our routine that receives Mach messages and account for the different message types:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

mach_msg_return_t
receive_msg(mach_port_name_t recvPort, ReceiveAnyMessage *buffer) {
  mach_msg_return_t ret = mach_msg(
      /* msg */ (mach_msg_header_t *)buffer,
      /* option */ MACH_RCV_MSG,
      /* send size */ 0,
      /* recv size */ sizeof(*buffer),
      /* recv_name */ recvPort,
      /* timeout */ MACH_MSG_TIMEOUT_NONE,
      /* notify port */ MACH_PORT_NULL);
  if (ret != MACH_MSG_SUCCESS) {
    return ret;
  }

  if (buffer->header.msgh_id == MSG_ID_DEFAULT) {
    Message *message = &buffer->message.message;

    printf("got default message!\n");
    printf("  id      : %d\n", message->header.msgh_id);
    printf("  bodyS   : %s\n", message->bodyStr);
    printf("  bodyI   : %d\n", message->bodyInt);
  } else if (buffer->header.msgh_id == MSG_ID_PORT) {
    printf(
        "got port message with name: %#x, disposition: %#x!\n",
        buffer->portMessage.message.descriptor.name,
        buffer->portMessage.message.descriptor.disposition);
  } else {
    return RCV_ERROR_INVALID_MESSAGE_ID;
  }

  return MACH_MSG_SUCCESS;
}

We’ve changed the message type to the ReceiveAnyMessage pointer, added checks for the message id, and correspondingly extended the logging logic for their message types.

#3 Reply routine

We also need a way to respond to the messages:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

mach_msg_return_t send_reply(const PortMessage *inMessage) {
  Message response = {0};

  response.header.msgh_bits = MACH_MSGH_BITS_SET(
      /* remote */ inMessage->descriptor.disposition,
      /* local */ 0,
      /* voucher */ 0,
      /* other */ 0);
  response.header.msgh_remote_port = inMessage->descriptor.name;
  response.header.msgh_id = MSG_ID_DEFAULT;
  response.header.msgh_size = sizeof(response);

  strcpy(response.bodyStr, "Response :) ");

  return mach_msg(
      /* msg */ (mach_msg_header_t *)&response,
      /* option */ MACH_SEND_MSG,
      /* send size */ sizeof(response),
      /* recv size */ 0,
      /* recv_name */ MACH_PORT_NULL,
      /* timeout */ MACH_MSG_TIMEOUT_NONE,
      /* notify port */ MACH_PORT_NULL);
}

There are only a few things different from the previous example. First, is the message bits:

1
2
3
4
5

response.header.msgh_bits = MACH_MSGH_BITS_SET(
    /* remote */ inMessage->descriptor.disposition,
    /* local */ 0,
    /* voucher */ 0,
    /* other */ 0);

The disposition field has the same semantics as the disposition in the message bits. On the receiver side, it indicates what port right they were granted. The difference here is that disposition applies to the specific port in the port descriptor, unlike the message bits that include the disposition for remote, local, and voucher ports. So the remote bits mask trick applies specifically to the message bits. Here you can use the MACH_MSGH_BITS_SET macro instead.

And last thing is to use the default message id, as we’re sending a simple message with inline data:

1

response.header.msgh_id = MSG_ID_DEFAULT;

#4 Program loop

In the program loop we’re going to change the messsage buffer type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

while (true) {
  // Message buffer.
  ReceiveAnyMessage receiveMessage = {0};

  mach_msg_return_t ret = receive_msg(recvPort, &receiveMessage);
  if (ret != MACH_MSG_SUCCESS) {
    printf("Failed to receive a message: %#x\n", ret);
    continue;
  }

  ...
}

#5 Reply in loop

And finally server can respond to incoming messages:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

while (true) {
  ...

  // Continue if it's not the port descriptor message.
  if (receiveMessage.header.msgh_id != MSG_ID_PORT) {
    continue;
  }

  ret = send_reply(&receiveMessage.portMessage.message);

  if (ret != MACH_MSG_SUCCESS) {
    printf("Failed to respond: %#x\n", ret);
  }
}

Result

Now when you start the server program, and run the client, you should see those messages in the client output:

1
2
3
4
5

got response message!
  id      : 8
  bodyS   : Response :)
  bodyI   : 0
Receive timed out, no more messages from alice yet.

This time the client has sent one single message, the server has responded to it, and then client finished execution as it has reached the timeout for receiving a new message.

Summary

I hope that you were able to follow along and know now how to establish a bidirectional communication over Mach ports. However, don’t feel discouraged if you didn’t get everything immediately. Make sure to try some things out and you can always come back to fill in the blanks. It’s quite some information, so let’s do a recap.
We’ve implemented a client <-> server communication starting with the reply port approach. We’ve learned there about the send once port right and non-blocking mach_msg calls with a timeout. In server implementation we’ve revisited the duality of local/remote ports and the port rights disposition.
Then there were complex messages and and type descriptors. Using the port descriptor, we’ve reimplemented bidirectional communication without relying on the builtin reply port. Here we’ve also leveraged message ids to deal with heterogeneous message types.

Next, we can start exploring more advanced APIs and use cases of Mach messages. There’s still some more functionalitites left to cover, so stayed tuned for part three!

Full implementation of the examples can be found at GitHub.

Thanks for reading! If you’ve got any questions, comments, or general feedback, you can find all my social links at the bottom of the page.