Uncovering what's __hidden#

• There are __hidden# symbols in my crash logs. What do I do?
• Set Enable Bitcode option for your build target to NO.

Looking even online for traces of __hidden# in crash logs reveals that many iOS developers at some point had a similar exchange. This article will recap the App Store distribution, provide some background for the crash logs problem, and cover an alternative solution.

While the workaround of disabling bitcode in itself isn’t necessarily bad, it’s important to understand why it works, as disabling it has drawbacks. Using bitcode, Apple can recompile applications using the latest compiler toolchain to improve application performance and reduce size.
Additionally, tvOS and watchOS require bitcode, so the workaround of disabling it won’t work for those platforms.

The problem

First, let’s recap what the issue is exactly. Here’s an example crash log showcasing it:

The stack trace includes some unfamiliar symbol names starting with __hidden# name, where one would typically expect function names from the code they’ve written. The hidden names result from symbol names obfuscation applied by Xcode when exporting a bitcode enabled build. The obfuscation is used to hide implementation details from Apple and has no impact on the application delivered to user devices.

Distribution recap

Information about distribution, symbol hiding, and crash reports is covered in Apple official documentation in App Distribution¹, Symbol Names in Crash Reports², and Building App to Include Debug Information³. The articles can be a lot to grasp, so those are my related key takeaways:

• Once the application is ready to ship, the first step in distribution is archiving it.
• Archived applications can be exported for different deployment targets, e.g., App Store or Development.
• App Store is the only distribution method that leverages bitcode.
• Exporting for App Store, there’re options not to include bitcode and share symbols with Apple.
• When not sharing symbols, the crash logs received from Apple won’t be symbolicated. Instead, symbolication will need to happen locally. This is applicable regardless of bitcode.
• When using bitcode, debug symbols used for symbolication have to be downloaded from App Store Connect/Xcode.

Cause and solution

The configuration where it most often goes wrong is using bitcode and not sharing symbols with Apple. In that scenario, dSYMs need to be download from Apple, and before they can be used, original symbol names need first to be restored². This is the reason why disabling bitcode is the most common workaround.
When uploading machine code only, dSYMs created during archiving can be used directly for symbolication. So there’s no need to restore symbols as hiding them in the first place is an export option.

Note that there are two methods to retrieve debug symbols - through App Store Connect or directly through Xcode organizer. I recommend the second option as Xcode can automatically restore hidden symbols.

Example

Now I’d like to showcase in practice when this issue can occur and how to resolve it.

I’ve prepared a very simple application with a single Crash me button. Upon the button click, the following code snippet is executed leading to an app crash:

1
2
3
4

  @IBAction func crash_me_action(_ sender: Any) {
    
    os_log("ptr: %d", type: .error, UnsafePointer<Int64>(bitPattern: 123)!.pointee)
  }

Preparing application builds

The earlier referred Apple documentation already shows how to export the application from Xcode, so I’ve used Xcode command-line tools instead. Examples with command-line tools are less common but may come in handy for automation.

First, I’ve archived two application builds, one with bitcode and one without:

1
2
3
4
5
6
7

xcodebuild archive -scheme StagingApp -configuration Release \
  -archivePath StagingApp_2.0.7-bitcode.xcarchive \
  -destination "generic/platform=iOS" ENABLE_BITCODE=YES

xcodebuild archive -scheme StagingApp -configuration Release \
  -archivePath StagingApp_2.0.8-no-bitcode.xcarchive \
  -destination "generic/platform=iOS" ENABLE_BITCODE=NO

Note the ENABLE_BITCODE, which is how I customised whether bitcode should be enabled.

The difference between both archives can be seen already in their root directory:

1
2
3
4
5
6
7
8
9

├── StagingApp_2.0.7-bitcode.xcarchive
│   ├── BCSymbolMaps
│   ├── Info.plist
│   ├── Products
│   └── dSYMs
├── StagingApp_2.0.8-no-bitcode.xcarchive
│   ├── Info.plist
│   ├── Products
│   └── dSYMs

BCSymbolMaps is where the original names of hidden bitcode symbols are stored. Files in that directory are used to recover the original names in debug symbols.

The next step was to export the archive. I did that using the following commands:

1
2
3
4

xcodebuild -exportArchive -archivePath StagingApp_2.0.8-no-bitcode.xcarchive \
  -exportPath export_2.0.8-no-bitcode -exportOptionsPlist export_options.plist
xcodebuild -exportArchive -archivePath StagingApp_2.0.7-no-bitcode.xcarchive \
  -exportPath export_2.0.7-no-bitcode -exportOptionsPlist export_options.plist

The export_options.plist is where export options can be specified. The available options are the same as those available when exporting from Xcode. For this test, I’ve enabled bitcode but didn’t share symbols with Apple to leverage symbol obfuscation. This is the configuration file I’ve used:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>method</key>
    <string>app-store</string>
    <key>uploadBitcode</key>
    <true/>
    <key>uploadSymbols</key>
    <false/>
    <key>teamID</key>
    <string>[team_id]</string>
    <key>provisioningProfiles</key>
    <dict>
      <key>xyz.dmcyk.StagingApp</key>
      <string>[prov_profile_uuid]</string>
    </dict>
</dict>
</plist>

Once exported, I’ve uploaded the application builds using the altool command and App Store Connect API authentication keys⁴ :

1

xcrun altool --upload-app -f StagingApp.ipa --apiKey [api_key_file] --apiIssuer [issuer_id]

Verification using TestFlight

When the application builds have finished processing, I’ve enabled internal testing in TestFlight⁵, and then I used the testing platform to verify the behaviour.

I installed each of the application builds, with and without bitcode, through TestFlight and used the crash button a few times to make sure crash reporting tools correctly picked them up.
Interestingly if I provided additional feedback with the crash report, it was immediately visible in the App Store Connect crashes panel. Still, it took a few hours before they finally showed up in Xcode.

First was the crash log for application without bitcode:

The crash log for this build was immediately symbolicated, as I had the Xcode archive on my Mac alongside the dSYMs.

The bitcode build, on the other hand, wasn’t:

The crash log wasn’t symbolicated because I haven’t yet downloaded the debug symbols from Apple. So to do that, I used the App Store Connect website.

Now that I had dSYMs, I should be able to symbolicate the crash log, right? Nah.

The crash log was now symbolicated but using the hidden symbol names. As I didn’t share my symbols with Apple and manually download dSYMs from App Store Connect, there was one more step required - restoring the original symbol names.

To do that, I used the dsymutil command:

1
2

dsymutil -symbol-map [ARCHIVE_DIR]/StagingApp_2.0.7-bitcode.xcarchive/BCSymbolMaps \
  [dsym_file].dSYM

After having restored symbols, the next symbolicated crash log did now include original function names as expected:

Note that the manual restoration names wouldn’t be necessary if I’d downloaded dSYMs using Xcode. Xcode does it automatically for you when using the organizer:

I did succeed in getting symbolicating the crash log without having to disable bitcode. However, the process is slightly more involved and even more so if you’re using a third-party crash reporting tool.

Conclusion

The article started by identifying a common problem related to bitcode and the app debugging process being symbol obfuscation. Further on, we’ve recapped the App Store distribution process and its options. Lastly, we’ve covered step by step how to reproduce the issue and how to resolve it.

The decision of whether to use bitcode or not remains up to you, but hopefully, after reading the article, the part about crash logs became a tad clearer.

Thanks for reading! If you’ve got any questions, comments, or general feedback, you can find all my social links at the bottom of the page.