Skip to main content

Symbolication

If you are submitting minidump files then you will need to ensure that debug symbols have been uploaded to Backtrace in order to have accurate classification, deduplication, and callstack rendering.

Generating Debug Symbols​

In Visual Studio, your DEBUG build will generate .pdb symbol files. If you wish to generate symbol files in your RELEASE build or in a custom configuration, make sure that "Generate Debug Info" is turned on in your project's Configuration Properties.

C++
  1. In Solution Explorer, select the project.
  2. Select the Properties icon.
  3. In the Configuration list, choose Debug or Release.
  4. In the side pane, choose Linker Debugging, then select options for Generate Debug Info.
  5. For detailed information on project settings for debug configurations in C++, see Project settings for a C++ debug configuration.
  6. Configure options for Generate Program Database Files.
C# and .NET
  1. In Solution Explorer, select the project.
  2. Select the Properties icon.
  3. In the side pane, choose Build.
  4. In the Configuration list, choose Debug or Release.
  5. Select the Advanced button.
  6. In the Debugging information list, choose Full.

Symbol Servers​

To minimize the amount of symbols that users need to upload, Backtrace is configured to pull symbols from the following symbol servers, which are publicly available:

  • Microsoft: Windows OS symbols
  • Mozilla: macOS symbols
  • AMD: ATI (gpu) symbols
  • Electron: Electron framework symbols
  • Unity: Unity game engine symbols
  • Intel
  • NVIDIA
Linux and macOS

Since the symbolication process relies on the name of the application, these symbols will only match with Electron applications named Electron. Reach out to our support team via the in-app chat on the bottom right of your screen for assistance working around this (symbols will need to be adjusted or manually uploaded for renamed ones).

Additional and/or private symbol servers​

Backtrace can be configured to pull from additional public or private symbol servers. For more information, see Connecting to symbol servers.

Symbol Formats and Upload Methods​

Navigate to your project configuration page and click on Symbols to see a record of all things symbol (including upload history, search functionality, symbol access tokens and more). Backtrace recommends uploading archives (a .tar.gz or .zip file containing one or more .sym ,.pdb , ELF or dSYM files) of symbols for every build you expect to see crashes for. These symbol files can be uploaded manually or hooked up into your build system so they are automatically uploaded.

Symbols can be uploaded via the web browser, morgue, curl, and HTTP.

The table above does not include .elf and dSYM files, which you can upload as an archive via Web Browser, curl, and HTTP.

You may also specify a tag query string parameter to any of your symbol uploads in order to group symbols for ease of management. A tag is simply a group of symbols, like a folder on your filesystem. You may want to have different tags for different versions of your application or different platforms. If no tag is specified, symbols are placed into the anon tag.

Regardless of your upload method, Backtrace provides visibility into symbol state. Backtrace supports .pdb + .sym (Breakpad symbol files), ELF + dSYM files and symbol archives (compressed or uncompressed archives containing the files). There are no restrictions on the layout of the files but you must ensure no relative paths are used and that files have the correct basename. For example, debug information for Editor.exe must be in a file called Editor.pdb or Editor.sym.

note

To ensure proper processing, make sure that your symbols files (.pdb .sym, etc) are uploaded before their corresponding .exe and .dll files or included in the same archive.

Uploading Symbols​

Generating Symbol Access Tokens​

You can generate a symbol access token to upload the debug symbols for your project. To generate a symbol access token:

  1. Go to Project Settings > Symbols > Access tokens.
  2. To generate the token, click +.
  3. Copy the token and save it for later.

Breakpad and Socorro​

Backtrace is compatible with existing Breakpad and Socorro integrations. You can upload the .sym files through an HTTP POST or with Breakpad's sym_upload tool. You must provide a token (referring to your symbol access token) and a format query string parameter with the value symbols.

Below is an example of a symbol upload using the sym_upload tool.

sym_upload null_read_av.sym 'https://submit.backtrace.io/{your-subdomain}/format=symbols&token={symbol-access-token}'

To generate a symbol access token, see Generating a Symbol Access Token.

HTTP API​

In order to build automation around symbol upload, such as integration into a build and release process, you'll want to interface directly with the HTTP API provided by Backtrace.

Below is an example of a curl command to submit a symbol archive.

curl --data-binary @symbols.tar  -X POST  -H "Expect: gzip" "https://submit.backtrace.io/{your-subdomain}/format=symbols&token={symbol-access-token}"
note

For large uploads, add the -H "Expect: gzip" flag to the curl command.

To generate a symbol access token, see Generating a Symbol Access Token.

Web Browser​

Navigate to your project configuration page and click on Symbols in order to manage symbols through your web browser. You are able to manually upload .pdb or .sym and compressed archives of .sym, .pdb , ELF or dSYM files directly in your web browser. We recommend uploading a .zip file of all symbols files, rather than individual files. Follow the on-screen instructions for more details.

Troubleshooting​

If you are blocked on uploading symbols from the command line, try uploading directly from your web browser. Instructions are available above.

Invalid Token​

If you receive an "invalid token" error in the response when uploading symbols via HTTP, check to make sure that you're using a symbol access token and not a submission token. To generate a symbol access token, see Generating a Symbol Access Token.

Missing Symbols​

Sometimes, symbols can get missed during the symbol upload process. A list of missing symbols for a dump is available in the web debugger annotations pane. For more information about how to acquire this list, refer to the Web Debugger guide.

After uploading missing symbols, you can reprocess objects to have the dumps with missing symbols reprocessed. Note that the grouping of dumps may change after missing symbols are uploaded. There are a couple of ways to reprocess objects.

  • System Administrators are able to reprocess all objects within a project right from the web browser by opening the menu from the top right of the screen, selecting project settings and navigating to the Reprocess Objects section.
  • System Administrators and Backtrace Support are able to reprocess more specific groups of crashes via the morgue tool.

If you’d like more information or assistance reprocessing objects, reach out to Backtrace Support via the in the in-app chat on the bottom right of your screen.

Locating Missing Symbols​

The first step in locating missing symbol files is to determine the name of the symbol file and its debug identifier. There are two ways to do this via the Debugger UI:

  1. When viewing an error that's missing symbols, Backtrace will add a Missing Symbols branch to the Global annotations section at the lower part of the screen. When you expand an entry number, you can see the path to the corresponding executable, the name of the symbol file, the debug identifier, the exe or module (shown as "path"), and version information:
  1. If an error is missing symbols, you can also see this in the callstack. In an affected frame, you will see a triangular warning symbol along with a memory address where the function signature would normally be. If you mouse over one of these frames, you'll see a yellow warning missing symbols pop-up. The debug identifier and the name of the symbol file are in the heading:

If you have the morgue tool installed, you can also check the status of symbol uploads and obtain lists of missing symbols by using the morgue symbol commands. See https://github.com/backtrace-labs/backtrace-morgue for more information.

You now have the information you need to check your in-house symbol archives for the missing symbol file. How to identify the symbol file depends on what kind of symbol files you're using:

  • .sym files - If you're using Breakpad .sym files, this is quite simple: The first line of Breakpad sym file is the MODULE record which lists the pdb name and debug identifier. For example MODULE windows x86_64 D0489F894E07424AAB5E626FF8C943DD1 advapi32.pdb.

  • .pdb files - (More Likely) If you're using .pdb files, you need to extract the debug identifier from .pdb files using one of the following methods, and find the matching one:

    • dia2dump - From the Microsoft MSDN site.

    • Since .exe and .pdb files are normally stored in the same location, you can use the Windows build tool dumpbin.exe to dump the executable's headers, which includes the matching symbols' name and debug identifier. For example, the following command suggests symbol bcrypt.pdb with id 5C82DF990DA04C46A2B22ABB82D6B66A1.

      dumpbin.exe /headers bcrypt.dll

      Debug Directories

      Time Type       Size      RVA  Pointer
      -------- ------ -------- -------- --------
      584A7C7E cv           23 00021A80    20C80    Format: RSDS, {5C82DF99-0DA0-4C46-A2B2-2ABB82D6B66A}, 1, bcrypt.pdb

      On Windows, the executable, libraries, and PDB files must be uploaded as some unwinding information required for symbolication is contained exclusively in the executable objects.

  • .dsym files - MacOS .dsym files are supported.

  • .elf files - ELF files are supported.

Once you locate and upload missing symbols, make sure to Reprocess Objects, as described in the previous section on missing symbols.

Mismatched Module Name​

In addition to not having uploaded the proper symbol file, another cause for mixups is having a mismatched module name. When viewing missing symbol information, Backtrace will show the name of the module name (executable or dll/library file) that it's expecting that symbol file to match. This is "path" in the missing symbols section of annotations in the Debugger, or shown as "Library" when looking at the tooltip of a frame that's shown as missing symbols.

If the name of the module doesn't match, Backtrace will show the symbol is missing, so make sure your symbol files reflect the correct module name, and if uploading exe/dll files, make sure these have the correct names.

Inaccurate Callstack​

Windows​

On Windows, applications store some unwinding information exclusively in the executable object (.exe or .dll file). For this reason, we advise to include the executable code of your application and library during symbol upload. These files can be uploaded as stand-alone files, but you must ensure that the file base name matches the base name of the .pdb file. For example, the debug information for Editor.exe is expected to be in Editor.pdb. It is important that the name of the executable is Editor.exe in this context to pair with Editor.pdb. If the executable was uploaded as Word.exe, then you are unable to pair with Editor.pdb. It is recommended that a symbol archive is used.

Visual Studio​

If you are using Visual Studio, you'll need to ensure that symbols are being generated correctly. Additional details are available in the Crashpad Visual Studio document.