Breakpad development and server operation/Archive 2006

From Open Source@Seneca

Jump to: navigation, search


Where Do We Go From Here?

(In Addition to our TODO List)

  • Fix the linker error for crashreporter_win.cpp (it should work if you link it against comctl32 and wininet for more info)
  • By building off the existing client, modify the client to display useful information
  • Port the client UI crash reporter to Linux.
  • Integrate client as an extension with the browser (priority low, I haven't seen anyone talking about this yet, but this is how Talkback is implemented in Firefox.


  • Allow for searching crash records.
  • Associate bugzilla bugs with a crash record.
  • Add functionality to divide crashes for different Mozilla products.

Core Breakpad library

  • Breakpad currently does not work on Linux due to non standard symbolserver system

Project Repository

svn co svn://

Project Leader(s)

Sara Minchella (sara)
Dejan Tolj (dtolj)
Dave Bertenshaw (Dbert)

Project Contributor(s)

andrew - thanks for setting up the SVN server and providing us with a login.
bhearsum - thanks for setting up the VNC and SSH.
Richard Chu - Gave us a crash.
Dean Woodside - Gave us a crash.
Mark D'Souza - Gave us a crash.
Melissa Peh - Helped with perl script


  • Figure out what all this output means in minidump
  • Debug Build of Mozilla

Project Details

Testing a crash

To test run a program crash compile with the following command:

cl /Zi /Fetest_app.exe dbghelp.lib

Note that compiling this requires MSVC 8, breakpad does't compile with MSVC 7.1, its a known issue also make sure that you have this in your LIB environment variable, if its not already there:

C:\Program Files\Microsoft Visual Studio 8\VC\lib;
C:\Program Files\Microsoft Visual Studio 8\VC\PlatformSDK\Lib;
C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Lib\;

Running test_app.exe will crash and the HandleException function will generate a dump.dmp minidump file and the compiler will generate test_app.pdb Symbol store

To print the contents of the minidump file:

minidump_dump.exe dump.dmp

To print stack trace:

minidump_stackwalk.exe dump.dmp

Stacktrace will look something like this:

Operating system: Windows NT
                 5.1.2600 Service Pack 2
CPU: x86
    GenuineIntel family 15 model 2 stepping 4
Crash address: 0x0
Thread 0 (crashed)
0  prog1.exe + 0x102e
   eip = 0x0040102e   esp = 0x0012ff3c   ebp = 0x0012ff40   ebx = 0x7c80aa49
   esi = 0x00000002   edi = 0x00000a28   eax = 0x00000000   ecx = 0x00000001
   edx = 0x0041c888   efl = 0x00010202
1  prog1.exe + 0x107e
   eip = 0x0040107f   esp = 0x0012ff48   ebp = 0x0012ff70
2  prog1.exe + 0x153b
   eip = 0x0040153c   esp = 0x0012ff78   ebp = 0x0012ffc0
3  kernel32.dll + 0x16d4e
   eip = 0x7c816d4f   esp = 0x0012ffc8   ebp = 0x0012fff0

There is another utility to dump the line number data from a pdb file to a text-based format that we can use from the minidump processor.

dump_syms.exe test_app.pdb

Unit Testing

This will build unit tests and run them, you should be able to see how many tests have passed.

cd airbag/trunk
make check

We could take the existing breakpad library and help build a sample processor prototype that can accept a report and see how well that stuff works.

We need to figure out the state of the library, is it usable? How does one set it up? What kind of data do we want on the crash side? T=here is no UI, so could we create something in java/php/etc?

Talk to mento, robc, jay on irc and get on the google-breakpad-discuss mailing list. for example, see this discussion on the google-breakpad-discussion for good details.

Project News

Dec. 12, 2006

Collecting crash reports works fine now, crash report list.

Dec. 06, 2006

The html client uploads files successfully and I have changed lusers collect.cgi script to break down the structure of the crash file into MySQL db.

Nov. 29, 2006

Download the modified version of Firefox firefox+breakpad which points to this server list crashreports. The URL that points to the crash server is set in this file: crashreporter.ini. You may also notice a flag under [Settings] in crashreporter.ini labeled "Delete=0" which keeps the minidump file in your mozilla profile/minidump directory. This is useful for testing since you can run moz_stackwalk [minidump] to see if the output on server matches the output on the client.

IMPORTANT: To enable Breakpad crashreporting create a new Environmental variable called MOZ_AIRBAG and set it to 1.

The breakpad build is also located in the lxr win32 trunk. To checkout the latest breakpad build:

cvs -d co mozilla/toolkit/airbag

We are also working on a small client html program which uploads minidump(*.dmp) file submitdmp. This is for testing since you don't need to crash the browser to submit a report.

TODO: We still need to add parameter list to send along with the minidump file, otherwise the collect script will not accept the minidump file.

There is a crashreporter client on windows that sends the crash to the server when Firefox crashes.

TODO: Compile crashreproter fix linker error.



Nov. 24, 2006

Anyone who is interested in contributing to the Breakpad Project, we are looking for ways to crash Firefox so we can test our scripts. Here's how you can help us, we need you to post your crash details on THIS PAGE. Just follow our example, and keep adding to the page. Thanks!

Nov. 22, 2006

Pulled each others brains for a refresher on Perl. Played with trying to get luser's perl scripts running on our own server so we can modify his scripts.

Currently luser's scripts use the information from the minidump and displays them in a html document in a list. Each crash has a link to the stacktrace for when the crash occurred. We are looking to change these scripts to extract more data regrading the crash (symbol information).

Nov. 15, 2006

Installed and configured apache:

This will be used for collecting crash reports.

Nov. 14, 2006

Steps needed to implement Breakpad, contains links to common bugs.

Started working on server side component of Breakpad. luser has some code dealing with this. Breakpad has a crash_report_sender library which sends the minidump file ~60k which is then parsed on the server.

The breakpad processor library provides minidump parsing code

There is a script on the server side called collect_pl.txt which collects the data in a SQLite db. It then lists the results using list_pl.txt.

list_py.txt and collect_py.txt are the reimplementations in python. collect_py.txt has some code for parsing the output of moz_stackwalk and it will use MySQL db.

There is a MySQL db schema for the pyhon implementation of collect_py.txt

Nov. 07, 2006

Ben has setup VNC for us, to connect download the client and log in to
To connect using ssh only use port number 40022

Oct. 30, 2006

Added Breakpad project to the repository.

Oct. 24, 2006

Found a link that explains how to debug mozilla using gdb. Debugging Mozilla on Linux FAQ

This is an unofficial Linux client with server side implementation of a crash report system for SeaMonkey.

Breakpad is currently developed for win32 but future releases will be cross-platform.

Oct. 23, 2006

Downloaded Breakpad code and attempted to build it. If we can manage to build the Breakpad code then we can become familiar with how it works and can begin to develop an interface for it. Breakpad can be downloaded by typing the following:

svn checkout breakpad

however I just used Tortoise (running on a windows system).

  • Once the code is downloaded navigate to the code directory. From here, in a unix shell (I just used Cygwin) type './configure' which runs a configuration script on your computer and configures Breakpad's variables to work with it. The configure script also creates a makefile which is needed to continue the build.
  • Type make to run the make file.
  • Type `make install' to install the programs and any data files and documentation.
  • You can remove the program binaries and object files from the source code directory by typing `make clean'.

Oct. 16, 2006

Researched about the Quality Feedback Agent (Talkback). To understand how Breakpad works we first must have some understanding how Talkback works.
What is Talkback?
Talkback is a small piece of software embedded in the Mozilla suite, Firefox or Thunderbird that sends feedback to Mozilla when the software crashes. Contained in the data sent is a stack trace (where in the code the crashed occured and how it got there), the version and operating system, how long the program ran before crashing, and any data that you fill out. Talkback must be installed and enabled for this to work, you must do a customer install and add the "Quality Feedback Agent" component manually. Statistics about crashes are collected and bug reports are created called "topcrashes", individual incidents can be looked up on Talkback FastFind

I looked at some sites dealing with this:

Oct. 6, 2006

  • Our group has our server setup - now just the task of figuring out how to use and what to do next. Andrew posted a tutorial wiki

Sept. 26, 2006

  • have been introduced to rob, alice and jay
  • read over the above discussion suggested in Project Details - need to inquire about getting a server at school.
  • I've asked bhearsum to setup a VM for you on the Mozilla-Seneca cluster. You'll be able to use this for your install/dev (dave)

Sept. 16, 2006

  • still deciding on this project ( :) ) - going to talk about it a little more with Dave next week
  • news on the licensing of Breakad here.

How to Contribute

We need to add some crash reports to our database. To find out how to do this Click here.

Personal tools
special sections