How I symbolicated a Apple TV crash report from a bitcode build

Since I released version 1.1 of ISS Live on Wednesday, a number of crash reports have come in via Xcode. Early Saturday morning, I received a very kind email from a user in the Netherlands to let me know that the app crashes a few seconds after launch and dumps him back on the main Apple TV menu. I emailed back and forth with him to do some basic troubleshooting, but it sounded like the trouble was with my code, not with his device.

So I went to my Apple TV, downloaded a fresh copy of my app from the App Store, but was unable to replicate the crash. I tried it in the simulator under a number of regional and bandwidth conditions, but was still unable to replicate the issue. I asked a couple friends to try it on their devices. Nobody else was able to replicate crash, either.

Not good, I thought. On Saturday morning, I made some extra coffee, opened Xcode, and got to work.

I took a look at the crash reports in Xcode. Apple helpfully sends anonymous crash data from users’ devices to developers. These crash reports include bits of machine code that point to memory addresses in the app. In theory, when you open these crash reports in Xcode, it will symbolicate these memory addresses and reveal the part of your code that caused the crash.

But when I opened my crash logs, they weren’t symbolicated. Instead of helpful pointers to my app’s code, I saw a hex string of a memory address, virtually useless for debugging.

How it’s supposed to work

When you upload an app to the App Store, you’re given the option to include app symbols:

Symbols

When you receive crash reports, Xcode uses these app symbols to translate the memory addresses to human-readable identifiers. It does this based on a combination of two things: a UUID and a dSYM file.

An app’s Universal Unique Identifier, or UUID, is assigned to identify a specific build. When an app crashes on a device, the crash report includes this UUID. Xcode matches the UUID in the crash report with the UUID in the local copy of the build.

It then uses a debug symbols (dSYM) file to translate the memory addresses in the crash report to human-readable object names that will make debugging easier.

But in my case, it didn’t.

What went wrong

When I opened my crash logs in Xcode, they didn’t symbolicate. Instead of pointing to the part of my code that caused the crash, it gave me raw memory addresses:

Memory addresses

The crash reports go in reverse-chronological order, with the crash at zero. As you can see, these numbers aren’t very helpful in pinpointing the crash.

I opened the crash report in Sublime Text and discovered that the UUID in the crash report did not match the UUID of the submitted build. So, of course Xcode wouldn’t be able to symbolicate it; it doesn’t have a dSYM file with a matching UUID.

So… wait. Why do my crash reports have a mysterious new UUID in them? Why is that UUID different than the UUID of the app build I submitted?

Why (I think) it went wrong

I think bitcode is the culprit. According to Apple:

Bitcode is an intermediate representation of a compiled program. Apps you upload to iTunes Connect that contain bitcode will be compiled and linked on the store. Including bitcode will allow Apple to re-optimize your app binary in the future without the need to submit a new version of your app to the store.

If my app was recompiled on Apple’s server, it would have been assigned a new UUID, which I suspect is the UUID that showed up in the crash reports. Using bitcode to optimize builds is a great idea, but Xcode 7.2.1 is unable to symbolicate my crash reports, making it impossible to tell for certain what part of my code causes the crash.

Apple continues:

For iOS apps, bitcode is the default, but optional. For watchOS and tvOS apps, bitcode is required.

I was stuck with a bitcode build that didn’t match my dSYM file, a growing number of crash reports for a crash I was unable to replicate.

Crap.

I filed a bug report with Apple and posted to an Apple Developer Forums thread started by an Apple employee (who, ironically, insisted that Xcode specifically would not have this problem). It looked like I wasn’t the only one experiencing this issue. Frustratingly, it also seemed sporadic. Other developers reported seeing symbolicated crash reports for some builds but memory addresses for others.

Meanwhile, a new batch of crash reports came in for my app. The crash appeared to be more widespread than I’d thought, and I still didn’t have a great way to pinpoint the problem. Frustration City.

How I got desperate and symbolicated my crash log

First, I opened up Terminal to match the UUID of my dSYM and build. I wanted to sanity-check myself.

$ dwarfdump -u ISS\ Live.app.dSYM/

That returned:

UUID: 17204124-1C54-9327-7ND3-66B9AD4F5A73 (arm64) ISS Live.app.dSYM/Contents/Resources/DWARF/ISS Live

Great. I had the UUID for my dSYM. Next, I wanted to confirm the UUID of the build. In Xcode:

  1. I opened the Organizer, selected the Archives tab,
  2. right-clicked the 1.1 build and selected “Show in Finder”,
  3. right-clicked the .xarchive file and selected “Show Package Contents”,
  4. navigated to Products > Applications,
  5. and ran dwarfdump again on the ISS Live application.
$ dwarfdump -u ISS\ Live.app/ISS\ Live

UUID: 17204124-1C54-9327-7ND3-66B9AD4F5A73 (arm64) ISS Live.app/ISS Live

Confirmed: both the dSYM and application have matching UUIDs.

Here comes the desperate part.

I opened up the crash log in Sublime Text and navigated to the Binary Images section of the crash file. This is where you can find the UUID of the app that crashed:

Binary Images:
0x10009c000 - 0x1000b3fff ISS Live arm64  <01fa95353176671a4b4c209d2327a424> /var/mobile/Containers/Bundle/Application/6FB8A954-0AA9-4606-853B-73666EC8B083/ISS Live.app/ISS Live

The 01fa95353176671a4b4c209d2327a424 part is the UUID. It’s formatted differently, without dashes and with lowecase letters. I replaced it with 172041241c5493277nd366b9ad4f5a73, saved it, and went back the terminal.

There’s a command line tool called symbolicatecrash that you can use to manually symbolicate a crash report. I followed Christopher Hale’s great blog post to get it set up. First, I set up an alias for the command and the Developer directory:

alias symbolicatecrash='/Applications/Xcode.app/Contents/SharedFrameworks/DTDeviceKitBase.framework/Versions/A/Resources/symbolicatecrash'

export DEVELOPER_DIR='/Applications/Xcode.app/Contents/Developer'

And then I ran symbolicatecrash on my crash report.

symbolicatecrash 2016-02-12_18-32-31.31_-0500-ac90400c13649647ec79e540f1254b2cec5b8fc4.crash

Finally, I ran it on my newly hand-edited crash report. It worked! Instead of memory addresses, helpful data appeared in my terminal window.

...
Thread 0 name:
Thread 0 Crashed:
0  ISS Live  0x00000001000a7438 specialized MainViewController.findCurrentExpedition(NSArray) -> NSObject? (MainViewController.swift:136)
1  ISS Live  0x00000001000a5134 MainViewController.fetchCrewData() -> () (MainViewController.swift:58)
2  ISS Live  0x00000001000a4dac MainViewController.viewDidLoad() -> () (MainViewController.swift:23)
3  ISS Live  0x00000001000a4e28 @objc MainViewController.viewDidLoad() -> () (MainViewController.swift:0)
...

as opposed to something like:

...
Thread 0 name:
Thread 0 Crashed:
0  ISS Live  0x00000001000ff438 0x1000f4000 + 46136
1  ISS Live  0x00000001000fd134 0x1000f4000 + 37172
2  ISS Live  0x00000001000fcdac 0x1000f4000 + 36268
3  ISS Live  0x00000001000fce28 0x1000f4000 + 36392
...

This probably worked because the only version of my app is a tvOS app. When I add builds for iPhone and iPad, I’m not sure this will work. But for now, I made progress, located what I think caused the crash, and submitted a new build. But it would sure be nice to have this work in Xcode.

This whole process was a real pain, and I hope Xcode is able to symbolicate future bitcode builds I submit. But, until they do, this may be a workaround for debugging tvOS crash logs that don’t symbolicate automatically.