Breakpad development and server operation/Archive 2006
From Open Source@Seneca
Where Do We Go From Here?
(In Addition to our TODO List)
- Fix the linker error for crashreporter_win.cpp (it should work if you link it against comctl32 and wininet for more info)
- By building off the existing client, modify the client to display useful information
- Port the client UI crash reporter to Linux.
- Integrate client as an extension with the browser (priority low, I haven't seen anyone talking about this yet, but this is how Talkback is implemented in Firefox.
- Allow for searching crash records.
- Associate bugzilla bugs with a crash record.
- Add functionality to divide crashes for different Mozilla products.
Core Breakpad library
- Breakpad currently does not work on Linux due to non standard symbolserver system
andrew - thanks for setting up the SVN server and providing us with a login.
bhearsum - thanks for setting up the VNC and SSH.
Richard Chu - Gave us a crash.
Dean Woodside - Gave us a crash.
Mark D'Souza - Gave us a crash.
Melissa Peh - Helped with perl script
- Figure out what all this output means in minidump
- Debug Build of Mozilla
Testing a crash
To test run a program crash compile test_app.cc with the following command:
cl /Zi /Fetest_app.exe test_app.cc dbghelp.lib
Note that compiling this requires MSVC 8, breakpad does't compile with MSVC 7.1, its a known issue also make sure that you have this in your LIB environment variable, if its not already there:
C:\Program Files\Microsoft Visual Studio 8\VC\lib; C:\Program Files\Microsoft Visual Studio 8\VC\PlatformSDK\Lib; C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Lib\;
Running test_app.exe will crash and the HandleException function will generate a dump.dmp minidump file and the compiler will generate test_app.pdb Symbol store
To print the contents of the minidump file:
To print stack trace:
Stacktrace will look something like this:
Operating system: Windows NT 5.1.2600 Service Pack 2 CPU: x86 GenuineIntel family 15 model 2 stepping 4 Crash reason: EXCEPTION_ACCESS_VIOLATION Crash address: 0x0 Thread 0 (crashed) 0 prog1.exe + 0x102e eip = 0x0040102e esp = 0x0012ff3c ebp = 0x0012ff40 ebx = 0x7c80aa49 esi = 0x00000002 edi = 0x00000a28 eax = 0x00000000 ecx = 0x00000001 edx = 0x0041c888 efl = 0x00010202 1 prog1.exe + 0x107e eip = 0x0040107f esp = 0x0012ff48 ebp = 0x0012ff70 2 prog1.exe + 0x153b eip = 0x0040153c esp = 0x0012ff78 ebp = 0x0012ffc0 3 kernel32.dll + 0x16d4e eip = 0x7c816d4f esp = 0x0012ffc8 ebp = 0x0012fff0
There is another utility to dump the line number data from a pdb file to a text-based format that we can use from the minidump processor.
This will build unit tests and run them, you should be able to see how many tests have passed.
cd airbag/trunk make check
We could take the existing breakpad library and help build a sample processor prototype that can accept a report and see how well that stuff works.
We need to figure out the state of the library, is it usable? How does one set it up? What kind of data do we want on the crash side? T=here is no UI, so could we create something in java/php/etc?
Talk to mento, robc, jay on irc and get on the google-breakpad-discuss mailing list. for example, see this discussion on the google-breakpad-discussion for good details.
Dec. 12, 2006
Collecting crash reports works fine now, crash report list.
Dec. 06, 2006
The html client uploads files successfully and I have changed lusers collect.cgi script to break down the structure of the crash file into MySQL db.
Nov. 29, 2006
Download the modified version of Firefox firefox+breakpad which points to this server list crashreports. The URL that points to the crash server is set in this file: crashreporter.ini. You may also notice a flag under [Settings] in crashreporter.ini labeled "Delete=0" which keeps the minidump file in your mozilla profile/minidump directory. This is useful for testing since you can run moz_stackwalk [minidump] to see if the output on server matches the output on the client.
IMPORTANT: To enable Breakpad crashreporting create a new Environmental variable called MOZ_AIRBAG and set it to 1.
The breakpad build is also located in the lxr win32 trunk. To checkout the latest breakpad build:
cvs -d :pserver:anonymous:firstname.lastname@example.org:/cvsroot co mozilla/toolkit/airbag
We are also working on a small client html program which uploads minidump(*.dmp) file submitdmp. This is for testing since you don't need to crash the browser to submit a report.
TODO: We still need to add parameter list to send along with the minidump file, otherwise the collect script will not accept the minidump file.
There is a crashreporter client on windows that sends the crash to the server when Firefox crashes.
Nov. 24, 2006
Anyone who is interested in contributing to the Breakpad Project, we are looking for ways to crash Firefox so we can test our scripts. Here's how you can help us, we need you to post your crash details on THIS PAGE. Just follow our example, and keep adding to the page. Thanks!
Nov. 22, 2006
Pulled each others brains for a refresher on Perl. Played with trying to get luser's perl scripts running on our own server so we can modify his scripts.
Currently luser's scripts use the information from the minidump and displays them in a html document in a list. Each crash has a link to the stacktrace for when the crash occurred. We are looking to change these scripts to extract more data regrading the crash (symbol information).
Nov. 15, 2006
Installed and configured apache: http://hera.senecac.on.ca:40080/~dejan
This will be used for collecting crash reports.
Nov. 14, 2006
Steps needed to implement Breakpad, contains links to common bugs.
Started working on server side component of Breakpad. luser has some code dealing with this. Breakpad has a crash_report_sender library which sends the minidump file ~60k which is then parsed on the server.
The breakpad processor library provides minidump parsing code moz_stackwalk.cc.
There is a MySQL db schema for the pyhon implementation of collect_py.txt
Nov. 07, 2006
Ben has setup VNC for us, to connect download the client and log in to hera.senecac.on.ca:40443
To connect using ssh only use port number 40022
Oct. 30, 2006
Added Breakpad project to the repository.
Oct. 24, 2006
Found a link that explains how to debug mozilla using gdb. Debugging Mozilla on Linux FAQ
This is an unofficial Linux client with server side implementation of a crash report system for SeaMonkey.
Breakpad is currently developed for win32 but future releases will be cross-platform.
Oct. 23, 2006
Downloaded Breakpad code and attempted to build it. If we can manage to build the Breakpad code then we can become familiar with how it works and can begin to develop an interface for it. Breakpad can be downloaded by typing the following:
svn checkout http://code.google.com/p/google-breakpad/source breakpad
however I just used Tortoise (running on a windows system).
- Once the code is downloaded navigate to the code directory. From here, in a unix shell (I just used Cygwin) type './configure' which runs a configuration script on your computer and configures Breakpad's variables to work with it. The configure script also creates a makefile which is needed to continue the build.
- Type make to run the make file.
- Type `make install' to install the programs and any data files and documentation.
- You can remove the program binaries and object files from the source code directory by typing `make clean'.
Oct. 16, 2006
Researched about the Quality Feedback Agent (Talkback). To understand how Breakpad works we first must have some understanding how Talkback works.
What is Talkback?
Talkback is a small piece of software embedded in the Mozilla suite, Firefox or Thunderbird that sends feedback to Mozilla when the software crashes. Contained in the data sent is a stack trace (where in the code the crashed occured and how it got there), the version and operating system, how long the program ran before crashing, and any data that you fill out. Talkback must be installed and enabled for this to work, you must do a customer install and add the "Quality Feedback Agent" component manually. Statistics about crashes are collected and bug reports are created called "topcrashes", individual incidents can be looked up on Talkback FastFind
I looked at some sites dealing with this:
Oct. 6, 2006
- Our group has our server setup - now just the task of figuring out how to use and what to do next. Andrew posted a tutorial wiki
Sept. 26, 2006
- have been introduced to rob, alice and jay
- read somemore on the current status of the project here: http://groups.google.com/group/google-breakpad-dev and here: http://groups.google.com/group/google-breakpad-discuss
- read over the above discussion suggested in Project Details - need to inquire about getting a server at school.
- I've asked bhearsum to setup a VM for you on the Mozilla-Seneca cluster. You'll be able to use this for your install/dev (dave)
Sept. 16, 2006
- still deciding on this project ( :) ) - going to talk about it a little more with Dave next week
- news on the licensing of Breakad here.
How to Contribute
We need to add some crash reports to our database. To find out how to do this Click here.