macOS arm64 runner doesn't allow access to system registers

[wtdcode]

Description

Hello,

Thanks for offering the precious macOS arm runners! However, as suggested in many places, the runners are virtualized with Apple Virtualization Framework and thus run jobs under EL1. This doesn't allow users to access many system registers, including SPRR. Our use case is to use this register to determine the current JIT state, as Apple doesn't offer an API to do so. This works pretty well on bare metal machines as SPRR is accessible.

From the post here, probably Github Action needs to add com.apple.private.hypervisor.vmapple entitlement to the hypervisor to enable the access. I understand this use case might be minor but I would appreciate if Github Action can add this so that we can test and distribute pre-built python wheels.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Image: macos-14-arm64
Version: 20241202.580
Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/20241202.580/images/macos/macos-14-arm64-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20241202.580

Is it regression?

No

Expected behavior

Allow acess to system registers.

Actual behavior

Receive Illegal instruction: 4

Repro steps

We have a code snippet to reproduce this:

#include "stdint.h"
#include "stdio.h"
#include "pthread.h"
static inline uint64_t read_sprr_perm_mrs(void)
{
    uint64_t v;
    __asm__ __volatile__("isb sy\n"
                         "mrs %0, S3_6_c15_c1_5\n"
                         : "=r"(v)::"memory");
    return v;
}
static void print_all() {
    uint64_t v1 = read_sprr_perm_mrs();

    fprintf(stderr, "SPRR=%llx\n", v1);
}

int main() {
     print_all();
    fprintf(stderr, "Enable\n");
    pthread_jit_write_protect_np(1);
    print_all();
    fprintf(stderr, "Disable\n");
    pthread_jit_write_protect_np(0);
    print_all();
    fprintf(stderr, "Enable\n");
    pthread_jit_write_protect_np(1);
    print_all();
    return 0;
}

Run it by:

cc test.c --o ./test
./test

This works well on bare metal machines while crashes on Github Action runners.

[wtdcode]

As a side note, the JIT protection is not enabled at all on the macos arm64 runners. For instance, we can use the following code to confirm:

#include "sys/mman.h"
#include "pthread.h"
#include "stdio.h"
#include "stdlib.h"
int main() {
        uint64_t* p = mmap(NULL, 0x8192, PROT_WRITE | PROT_READ | PROT_EXEC, MAP_JIT | MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
        if (p == MAP_FAILED) {
                perror("allocate");
                return -1;
        }

        pthread_jit_write_protect_np(0);
        p[0] = 0xff;
        fprintf(stderr, "Initial %llx\n", p[0]);
        pthread_jit_write_protect_np(1);
        p[1] = 0xee; // This shall be denied
        fprintf(stderr, "Enable write protection %llx\n", p[1]);
        return 0;
}

On my m2 pro, this gives:

Initial ff
fish: Job 1, './test' terminated by signal SIGBUS (Misaligned address error)

But on Github Action, this gives:

Initial ff
Enable write protection ee

[prasanjitsahoo]

HI @wtdcode , We will look into the issue and keep you posted with updates.

[wtdcode]

Hey guys, can I have some updates? @prasanjitsahoo @erik-bershel

[asdfugil]

no. the hypervisor already has com.apple.private.hypervisor because the hypervisor is in a system process whose executable is in the virtualization framework. (/System/Library/Frameworks/Virtualization.framework/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine). The vmapple kernel simply does not support the sprr and gxf features. The only easy-ish way to run virtualized a kernel that supports SPRR and GXF is to run a PCC research guest on macOS 15.1 and later, but that's a different environment altogether.

TL;DR: This issue can't be fixed for as long as a virtual machine is used, unless Apple changes how the virtual machines work in the future.

[wtdcode]

no. the hypervisor already has com.apple.private.hypervisor because the hypervisor is in a system process whose executable is in the virtualization framework. (/System/Library/Frameworks/Virtualization.framework/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine). The vmapple kernel simply does not support the sprr and gxf features. The only easy-ish way to run virtualized a kernel that supports SPRR and GXF is to run a PCC research guest on macOS 15.1 and later, but that's a different environment altogether.

TL;DR: This issue can't be fixed for as long as a virtual machine is used, unless Apple changes how the virtual machines work in the future.

How do you confirm the entitlement is there?

[asdfugil]

How do you confirm the entitlement is there?

codesign -d --entitlements - /System/Library/Frameworks/Virtualization.framework/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine | grep com.apple.private.hypervisor

[wtdcode]

How do you confirm the entitlement is there?

codesign -d --entitlements - /System/Library/Frameworks/Virtualization.framework/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine | grep com.apple.private.hypervisor

I think the entitlement I'm referring is com.apple.private.hypervisor.vmapple, does com.apple.private.hypervisor imply that?

[asdfugil]

I think the entitlement I'm referring is com.apple.private.hypervisor.vmapple, does com.apple.private.hypervisor imply that?

com.apple.security.hypervisor and com.apple.vm.hypervisor are the most restrictive entitlements
com.apple.private.hypervisor.vmapple is more permissive
com.apple.private.hypervisor is the most permissive