Tools for Measuring System and Application Performance • Introduction • GlancePlus • • • • Introduction Glance Motif Glance Character Mode Glance Command-line • Xverbosegc • HPjmeter and –Xeprof • Other Tools • gdb © Copyright 2001 Hewlett-Packard Company Page 1
gdb Debugging Native and Java Code Quick, Sampling-based Profiling with gdb Enabling Remote Support Page 2 © Copyright 2001 Hewlett-Packard Company
Tools for Debugging gdb Introduction • Gnu Debugger • Official name is wdb – Wildebeest Debugger (www.hp.com/go/wdb) • Wildebeest is another name for the Gnu - Connochaetes taurinus • New version of gdb supports unwind of Java 1.3.1 • Interpreter frames • Java run-time compiler compiled frames • Adapters: – i2c – interpreted to compiled – c2i – compiled to interpreted – Java entry frame • Available at: pdlweb.cup.hp.
Tools for Debugging gdb Goals • We will learn how to use gdb with Java 1.3.
Tools for Debugging gdb Summary of Commands Used r [args] set args run program with args as arguments set the arguments for the run command thr thr # info thr print thread number change to thread # list all threads in the process thread apply all bt generate backtrace of all threads or can specify individual threads or range, 1-10 bt c generate back trace of current thread continue quit stop the running process exit the debugger Page 5 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Summary of Commands Used b info break set breakpoint at current location display all breakpoint information x/16i $pc-32 examine 16 words formatted as instructions from current PC minus 32 bytes set redirect-file file specify file to which to send gdb output set redirect-mode append set mode {overwrite | append} set redirect on send output to a file {on|off} set pagination off set state of pagination {on|off} handle signal nostop noprint pass ignore signal Page 6 © 2001 Hewl
Tools for Debugging gdb: Start-up Specify Java Unwind Library Location • gdb support supplied as a shared library • The library supporting Java unwind must be specified • Set the environment variable: GDB_JAVA_UNWINDLIB • For example: export GDB_JAVA_UNWINDLIB=/libjunwind.sl • For this presentation we will put the library in /tmp: export GDB_JAVA_UNWINDLIB=/tmp/libjunwind.
Tools for Debugging gdb Files Required • gdb • Debugger • libjunwind.
Tools for Debugging gdb: Start-up Four Methods for Debugging Java Four methods: 1. Modify application launch script’s java command + Often the easiest approach – uses DEBUG_PROG 2. Modify the /opt/java1.3/bin/.java_wrapper script + Easy to pass command line arguments – Need to have special JDK if many java processes 3. Use built-in DEBUG_PROG environment variable + Easy to set for the process – Hard to pass in large numbers of command line arguments 4.
Tools for Debugging gdb Starting Java from gdb: Modify app Modify Application Page 10 © 2001 Hewlett-Packard Company
gdb Starting Java from gdb: Modify app Tools for Debugging • Easiest method • Example • Change command line in a script such as: exec "$prog” -Xmn500m -ms1024m -mx1024m COM.volano.Mark -port 8000 -count 5000 -rooms 5 • To: export GDB_JAVA_UNWINDLIB=/tmp/libjunwind.sl export DBG_ARGS=“-Xmn500m -ms1024m -mx1024m COM.volano.
Tools for Debugging gdb Starting Java from gdb: Modify wrap Modify Wrapper Page 12 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Starting Java from gdb: Modify wrap The second method to invoke the debugger requires you to modify: /opt/java1.3/bin/.java_wrapper At the end of the file you have the following commands to start Java: # Run.
Tools for Debugging gdb Starting Java from gdb: Modify wrap Modify the file in place by adding the text in red: # Run. if [ -x "$prog" ] then # invoke the VM $args contains arguments which begin with "-", $* contains … # exec $DEBUG_PROG "$prog” “${options[@]}" "$@” export GDB_JAVA_UNWINDLIB=/tmp/libjunwind.
Tools for Debugging gdb Starting Java from gdb: Modify wrap • Problem: /opt/java1.3 directory not write-able • Often hard to modify /opt/java1.3/bin/.java_wrapper • Solution: copy the JDK and modify the copy mkdir /tmp/java1.3 cp -r /opt/java1.3/bin /tmp/java1.3/bin cp -r /opt/java1.3/lib /tmp/java1.3/lib cp -r /opt/java1.3/jre /tmp/java1.3/jre • Modify the .java_wrapper in /tmp/java1.
Tools for Debugging gdb Start Java from the Command Line $ /tmp/java1.3/bin/java HP gdb 3.0 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00. Copyright 1986 - 2001 Free Software Foundation, Inc. Hewlett-Packard Wildebeest 3.0 (based on GDB) is covered by the GNU General Public License. Type "show copying" to see the conditions to change it and/or distribute copies. Type "show warranty" for warranty/support. ..(no debugging symbols found)...
Tools for Debugging gdb Start the Java Process (no program) (gdb) r Starting program: /opt/java1.3/bin/PA_RISC2.0/native_threads/java (no debugging symbols found)...(no debugging symbols found)…( … Program received signal SIGSEGV, Segmentation fault. 0x6f8e69f0 in os_init_2 () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl (gdb) • The SIGSEGV signal is normal part of Java Virtual Machine initialization • Do a back trace • Continue • To not stop or print signals permanently use a ~/.
Tools for Debugging gdb Continue Past the First Signal (gdb) bt #0 0x6f8e69f0 in os_init_2 () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #1 0x6f94721c in create_vm__7ThreadsSFP14JavaVMInitArgs () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #2 0x6f852e94 in JNI_CreateJavaVM () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Starting Java from gdb: DEBUG_PROG Use of DEBUG_PROG Environment Variable Page 19 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Starting Java from gdb: DEBUG_PROG • Built-in mechanism available in .java_wrapper for 1.3.1 • Set DEBUG_PROG in your shell: export DEBUG_PROG=/opt/langtools/bin/gdb export GDB_JAVA_UNWINDLIB=/tmp/libjunwind.sl • Problem using arguments to the java command • You need to remove arguments else you get: /opt/langtools/bin/gdb: unrecognized option ‘Xmn500m’ • Proceed as with modified .java_wrapper method • Type in all of the arguments: (gdb) r -Xmn500m -Xms1024m -Xmx1024m COM.
Tools for Debugging gdb Starting Java from gdb: Attach Attaching to a Java Process Page 21 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Starting Java from gdb: Attach export DEBUG_PROG=/opt/langtools/bin/gdb export GDB_JAVA_UNWINDLIB=/tmp/libjunwind.sl /opt/java1.3/bin/java HP gdb 3.0 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00. Copyright 1986 - 2001 Free Software Foundation, Inc. Hewlett-Packard Wildebeest 3.0 (based on GDB) is covered by the GNU General Public License. Type "show copying" to see the conditions to change it and/or distribute copies. Type "show warranty" for warranty/support. ..
Tools for Debugging gdb Starting Java from gdb: Attach Reading symbols from /usr/lib/libpthread.1...(no debugging symbols found)... done. Reading symbols from /usr/lib/libm.2...(no debugging symbols found)...done. Reading symbols from /usr/lib/libcl.2...(no debugging symbols found)...done. Reading symbols from /usr/lib/libisamstub.1...(no debugging symbols found)... done. Reading symbols from /usr/lib/libCsup.2...(no debugging symbols found)...done. Reading symbols from /usr/lib/libc.2...
Tools for Debugging gdb Starting Java from gdb: Attach Reading symbols from /opt/java1.3/jre/lib/PA_RISC2.0/libzip.sl...(no debugging symbols found)...done. Reading symbols from /opt/java1.3/jre/lib/PA_RISC2.0/libnet.sl...(no debugging symbols found)...done. Reading symbols from /usr/lib/libnss_dns.1...done. Reading symbols from /usr/lib/libnss_nis.1...done. Reading symbols from /usr/lib/libnsl.1...done. Reading symbols from /usr/lib/libxti.2...done.
Tools for Debugging gdb Starting Java from gdb: Attach (gdb) bt #0 0xc01f28d0 in __ksleep () from /usr/lib/libc.2 #1 0xc15c8298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0xc15c7fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0xc40de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0xc41182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0xc4059d30 in JVM_MonitorWait () from /opt/java1.3/jre/lib/PA_RISC2.
Tools for Debugging gdb: Start-up Summary Four methods: 1. Modify application launch script’s java command + Often the easiest approach – uses DEBUG_PROG 2. Modify the /opt/java1.3/bin/.java_wrapper script + Easy to pass command line arguments – Need to have special JDK if many java processes 3. Use built-in DEBUG_PROG environment variable + Easy to set for the process – Hard to pass in large numbers of command line arguments 4.
Tools for Debugging gdb Running an Application Running and Debugging an Application Analysis of Threads in the JVM Process Page 27 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Running an Application • Use VolanoMark as a sample application • Many threads • Easy to run Page 28 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Running an Application - Volano $ runVolano Starting volano client... • Print the DBG_ARGS from the .java_wrapper: -Xmn500m -ms1024m -mx1024m COM.volano.Mark -port 8000 -count 5000 rooms 5 HP gdb 3.0 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00. Copyright 1986 - 2001 Free Software Foundation, Inc. Hewlett-Packard Wildebeest 3.0 (based on GDB) … (gdb) r $DBG_ARGS Starting program: /opt/java1.3/bin/PA_RISC2.0/native_threads/java -Xmn 500m -Xms1024m -Xmx1024m COM.volano.
Tools for Debugging gdb Running an Application - Volano Continuing. (no debugging symbols found)...(no debugging symbols found)… VolanoMark(TM) Benchmark Version 2.1.2 Copyright (C) 1996-1999 Volano LLC. All rights reserved. Creating room number 1 ... 20 connections so far. Creating room number 2 ... 40 connections so far. Creating room number 3 ... 60 connections so far. Creating room number 4 ... 80 connections so far. Creating room number 5 ... 100 connections so far. Running the test ...
Tools for Debugging gdb Stopping the Java Process Control-c Program received signal SIGINT, Interrupt. [Switching to thread 11 (system thread 27173)] 0x6ffad8d0 in __ksleep () from /usr/lib/libc.
Tools for Debugging gdb Where am I? All Threads (gdb) info threads 12 system thread 26778 11 system thread 27173 10 system thread 26775 9 system thread 26774 8 system thread 26773 7 system thread 26772 6 system thread 26771 /usr/lib/libc.2 5 system thread 26770 4 system thread 26769 3 system thread 26768 2 system thread 26767 1 system thread 26762 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 0x6ffad8d0 in __ksleep () from /usr/lib/libc.
Tools for Debugging gdb Where am I? All Threads Let’s look at each of the Java Virtual Machine Threads: 1. Base Thread 2. Thread Helper 3. Timer Thread 4. Reference Handler Thread 5. Finalizer Thread 6. Watcher Thread 7. Suspend Checker Thread 8. Signal Handler Thread 9. Compiler Thread 1 10. Compiler Thread 2 11. User Threads … 12.
Tools for Debugging gdb Where am I? Base Thread, Part 1 (gdb) thr 1 (gdb) bt Thread 1 (system thread 26762): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Where am I? Base Thread, Part 2 Thread 1 (system thread 26762): #9 0xc4b8c in interpreted frame: COM/volano/Mark::Ä {(Ljava/util/Vector;)Z} () #10 0xc4b8c in interpreted frame: COM/volano/Mark::main {([Ljava/lang/String;)V} #11 0x6fe911b8 in Java entry frame () Volano Mark main() #12 0x6f827a48 in JavaCalls::call_helper () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #13 0x6f8e6394 in os::os_exception_wrapper () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Where am I? Thread Helper (gdb) thr 2 (gdb) bt Thread 2 (system thread 26767): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8e8328 in thread_helper () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6fc74db0 in __pthread_create_system () from /usr/lib/libpthread.
Tools for Debugging gdb Where am I? Timer Thread Thread 3 (system thread 26768): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc730c4 in pthread_cond_timedwait () from /usr/lib/libpthread.1 #3 0x6f8d5c64 in Monitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f960924 in VMThread::loop () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f9603a8 in VMThread::run () from /opt/java1.
Tools for Debugging gdb Where am I? Reference Handler, 1 Thread 4 (system thread 26769): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Where am I? Reference Handler, 2 Thread 4 (system thread 26769): #9 0x6fe911b8 in Java entry frame () #10 0x6f827a48 in JavaCalls::call_helper () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #11 0x6f8e6394 in os::os_exception_wrapper () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #12 0x6f826bf4 in JavaCalls::call_virtual () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #13 0x6f867ea8 in thread_entry () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Where am I? Reference Handler, 2 Thread 4 (system thread 26769): #9 0x6fe911b8 in Java entry frame () #10 0x6f827a48 in JavaCalls::call_helper () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #11 0x6f8e6394 in os::os_exception_wrapper () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #12 0x6f826bf4 in JavaCalls::call_virtual () Same Thread from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Where am I? Reference Handler, 2 Thread 4 (system thread 26769): In future stack traces: #9 0x6fe911b8 in Java entry frame () Only show to Java entry frame() #10 0x6f827a48 in JavaCalls::call_helper () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #11 0x6f8e6394 in os::os_exception_wrapper () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #12 0x6f826bf4 in JavaCalls::call_virtual () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Where am I? Finalizer, 1 Thread 5 (system thread 26770): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f859d30 in JVM_MonitorWait () from /opt/java1.
Tools for Debugging gdb Where am I? Finalizer, 2 Thread 5 (system thread 26770): #9 0xc4b68 in interpreted frame: java/lang/ref/Finalizer$FinalizerThread::run {()V} () #10 0x6fe911b8 in Java entry frame () Page 43 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Where am I? Watcher Thread Thread 6 (system thread 26771): #0 0x6ffb1a98 in _nanosleep_sys () from /usr/lib/libc.2 #1 0x6ffbad10 in nanosleep () from /usr/lib/libc.2 #2 0x6f8e7b50 in os_sleep () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #3 0x6f8e7bd4 in os::sleep () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9432e8 in WatcherThread::run () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f8e89b4 in _start () from /opt/java1.
Tools for Debugging gdb Where am I? Suspend Checker Thread 7 (system thread 26772): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8d5944 in Monitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f943790 in SuspendCheckerThread::run () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f8e89b4 in _start () from /opt/java1.
Tools for Debugging gdb Where am I? Signal Handler Thread 8 (system thread 26773): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8d5ad0 in Monitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f8e9bd4 in os::signal_wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f8e2d74 in signal_thread_entry () from /opt/java1.
Tools for Debugging gdb Where am I? Compiler Thread 1 Thread 9 (system thread 26774): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8d5ad0 in Monitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f97cc48 in CompileBroker::compiler_thread_loop () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Where am I? Compiler Thread 2 Thread 10 (system thread 26775): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8d5ad0 in Monitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f97cc48 in CompileBroker::compiler_thread_loop () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Where am I? Volano Thread 1, 1 Thread 11 (system thread 27173): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Where am I? Volano Thread 1, 2 Thread 11 (system thread 27173): #10 0xc4dcc in interpreted frame: java/lang/Thread::run {()V} () #11 0x6fe911b8 in Java entry frame () Page 50 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Where am I? Volano Thread 2, 1 Thread 12 (system thread 26778): #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc77bb0 in pthread_mutex_lock () from /usr/lib/libpthread.1 #2 0x6f8d488c in Mutex::lock () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #3 0x6f91a9d8 in SystemDictionary::find () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f91a530 in SystemDictionary::resolve_instance_class_or_null () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Where am I? Volano Thread 2, 2 Thread 12 (system thread 26778): #7 0x6fc1d6b8 in Java_java_lang_Class_forName0 () from /opt/java1.3/jre/lib/PA_RISC2.0/libjava.
Tools for Debugging gdb Compiled Java Frames • Continue the program and then interrupt with Control-c again (gdb) c • Program runs Control-c Program received signal SIGINT, Interrupt. [Switching to thread 80 (system thread 26912)] 0x6ffaf6a0 in _read_sys () from /usr/lib/libc.2 (gdb) bt #0 0x6ffaf6a0 in _read_sys () from /usr/lib/libc.2 #1 0x6ffbb664 in read () from /usr/lib/libc.2 #2 0x6f866ce0 in JVM_Read () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Compiled Java Frames #4 0x6fbca328 in socketReadOnStack () from /opt/java1.3/jre/lib/PA_RISC2.0/libnet.sl #5 0x6fbca394 in Java_java_net_SocketInputStream_socketRead () from /opt/java1.3/jre/lib/PA_RISC2.0/libnet.
Tools for Debugging HotSpot Compiler Examples Transitions: i2c, c2i and i2native #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f859d30 in JVM_MonitorWait () from /opt/java1.
Tools for Debugging HotSpot Compiler Examples Transitions: i2c, c2i and i2native #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f859d30 in JVM_MonitorWait () from /opt/java1.
Tools for Debugging HotSpot Compiler Examples Transitions: i2c, c2i and i2native #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f859d30 in JVM_MonitorWait () from /opt/java1.
Tools for Debugging HotSpot Compiler Examples Transitions: i2c, c2i and i2native #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f859d30 in JVM_MonitorWait () from /opt/java1.
Tools for Debugging HotSpot Compiler Examples Transitions: i2c, c2i and i2native #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f859d30 in JVM_MonitorWait () from /opt/java1.
Tools for Debugging HotSpot Compiler Examples Transitions: i2c, c2i and i2native #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f859d30 in JVM_MonitorWait () from /opt/java1.
Tools for Debugging gdb Transitions: i2c, c2i and i2native #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f859d30 in JVM_MonitorWait () from /opt/java1.3/jre/lib/PA_RISC2.
Tools for Debugging HotSpot Compiler Examples Transitions: i2c, c2i and i2native #0 0x6ffad8d0 in __ksleep () from /usr/lib/libc.2 #1 0x6fc73298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0x6fc72fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0x6f8de384 in ObjectMonitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #4 0x6f9182ac in ObjectSynchronizer::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0x6f859d30 in JVM_MonitorWait () from /opt/java1.
Tools for Debugging gdb Where am I? Summary • We have seen what’s going on in each of the threads in the JVM • gdb with Java/Native support is a powerful tool • Understand what’s going on in the JVM – What are the native libraries that do the work? – What is the overhead of using a certain Java feature? • Diagnose problems Page 63 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Set a Breakpoint in Compiled Code Set a Breakpoint in Compiled Code Page 64 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Set a Breakpoint in Compiled Code • Breakpoint usage to examine the instructions generated for: • Interpreted code • Compiled code Page 65 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Set a Breakpoint in Compiled Code • Easy, direct method: • If you want to set breakpoint at a specific address: 0x6d4142c4 in compiled frame: java/io/BufferedInputStream::fill {()V} (gdb) b *0x6d4142c4 Page 66 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Set a Breakpoint in Compiled Code • Can also go to the frame to set the breakpoint • Navigate to the correct thread and frame • Set the breakpoint • We will cover the detailed steps subsequently • Subsequent steps: • Continue after setting the breakpoint • Examination of instructions using x command Page 67 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Set a Breakpoint in Compiled Code • Setting a breakpoint in BufferedInputStream::fill – go to the frame: (gdb) thr [Current thread is 11 (system thread 5485)] (gdb) bt #0 0x6ffaf6a0 in _read_sys () from /usr/lib/libc.2 #1 0x6ffbb664 in read () from /usr/lib/libc.2 #2 0x6f866ce0 in JVM_Read () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #3 0x6e219100 in socketReadWork () from /opt/java1.3/jre/lib/PA_RISC2.0/libnet.sl #4 0x6e219328 in socketReadOnStack () from /opt/java1.
Tools for Debugging gdb Set a Breakpoint in Compiled Code #6 0x6e410b58 in compiled frame: java/net/SocketInputStream::socketRead {([BII)I} () • Call to java/net/SocketInputStream::read() has been inlined in fill() #7 0x6e411d04 in compiled frame: java/io/BufferedInputStream::fill {()V} () #8 0x6e408ee8 in compiled frame: java/io/BufferedInputStream::read {()I} () #9 0x6e40c308 in compiled frame: java/io/DataInputStream::readUnsignedShort #10 0x6e4056d8 in compiled frame: java/io/DataInputStream::readUTF
Tools for Debugging gdb Set a Breakpoint in Compiled Code (gdb) up 7 - goes to frame #7 #7 0x6e411d04 in compiled frame: java/io/BufferedInputStream::fill {()V} () (gdb) b - sets a breakpoint at the return PC in BufferedInputStream::fill Breakpoint 1 at 0x6e411d04 (gdb) info break - verify Num Type Disp Enb Address What 1 breakpoint keep y 0x6e411d04 .
Tools for Debugging gdb Continue Execution (gdb) c Continuing.
Tools for Debugging gdb Examine Instructions at Breakpoint (gdb) x/16i $pc-32 - inlined java/net/SocketInputStream::read() code 0x6e411ce4: cmpib,=,n 0,%r9,0x6e411d20 ; if r9 (length) == 0 branch 0x6e411ce8: nop 0x6e411cec: copy %r7,%r26 ; load the argument registers 0x6e411cf0: copy %r5,%r25 0x6e411cf4: copy %r8,%r24 0x6e411cf8: copy %r9,%r23 0x6e411cfc: b,l,n 0x6e410a60,%rp ; java/net/SocketInputStream::socketRead 0x6e411d00: nop 0x6e411d04: ldw -0xa0(%sp),%r22 0x6e411d08: ldw 0x14(%r22),%r21 0x6e411d0c
Tools for Debugging gdb Set a Breakpoint in Compiled Code • Easy to set breakpoint interpreted or compiled code: • Direct: (gdb) b *0x6d4142c4 • Go to the frame to set the breakpoint: (gdb) up # - go to frame # (gdb) b • Demonstrated subsequent steps: • Continue after setting the breakpoint • Examination of instructions using x command Page 73 © 2001 Hewlett-Packard Company
Tools for Debugging gdb SIGBUS when Debugging, 1 • For this example, send a SIGBUS to the Java process identifier • Signal will be caught by the debugger • Can query the state of the Java process • Signal passed to the Java process • Information displayed describing location – Java Stack Trace – Libraries loaded – Information also written to a log file Page 74 © 2001 Hewlett-Packard Company
Tools for Debugging gdb SIGBUS when Debugging, 2 • Send a kill –10 to the Java process from another window Program received signal SIGBUS, Bus error. [Switching to thread 33 (system thread 27266)] 0x6ffb0988 in _sigsetstatemask () from /usr/lib/libc.2 (gdb) bt #0 0x6ffb0988 in _sigsetstatemask () from /usr/lib/libc.2 #1 0x6ffb919c in sigsetjmp () from /usr/lib/libc.2 #2 0x6f866c1c in JVM_Read () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #3 0x6fbc5100 in socketReadWork () from /opt/java1.
Tools for Debugging gdb SIGBUS when Debugging, 3 • Remember method names below to compare with the Java Stack Trace #5 0x6fbc5394 in Java_java_net_SocketInputStream_socketRead () from /opt/java1.3/jre/lib/PA_RISC2.0/libnet.
Tools for Debugging gdb SIGBUS when Debugging, 4 [Switching to thread 12 (system thread 27222)] 0x6ffb2390 in sched_yield () from /usr/lib/libc.2 (gdb) c Continuing. Program received signal SIGBUS, Bus error. [Switching to thread 43 (system thread 27284)] 0x6ffaf6a0 in _read_sys () from /usr/lib/libc.2 (gdb) c Continuing. Unexpected Signal : 10 occurred at PC=0x6f94ef74 Function name= copy_array__14typeArrayKlassFP12arrayOopDesciT1N22P6Thread Library= /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb SIGBUS when Debugging, 5 • Here is the Java Stack Trace that you can compare with the previous Native Current Java thread: "Receiver-17" prio=6 tid=0x002c6238 nid=43 lwp_id=27284 runnable [0x667de000..0x667de478] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked <0x6bc276b0> (a java.io.BufferedInputStream) at java.
Tools for Debugging gdb SIGBUS when Debugging, 6 Dynamic libraries: /opt/java1.3/bin/PA_RISC2.0/native_threads/java text:0x00001000-0x000066cc data:0x00007000-0x00007324 /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl text:0x6f400000-0x6fbbd000 data:0x6fde7000-0x6fe9c000 /usr/lib/libpthread.1 text:0x6fc6b000-0x6fc81000 data:0x6fc68000-0x6fc6b000 /usr/lib/libm.2 text:0x6fc82000-0x6fcae000 data:0x6fc81000-0x6fc82000 /usr/lib/libcl.
Tools for Debugging gdb SIGBUS when Debugging, 7 Dynamic libraries (cont): /usr/lib/libc.2 text:0x6febb000-0x6ffe6000 data:0x6fea4000-0x6febb000 /usr/lib/libdld.2 text:0x6fe9d000-0x6fea0000 data:0x6fe9c000-0x6fe9d000 /opt/java1.3/jre/lib/PA_RISC2.0/native_threads/libhpi.sl text:0x6fc57000-0x6fc68000 data:0x6fc56000-0x6fc57000 /opt/java1.3/jre/lib/PA_RISC2.0/libverify.sl text:0x6fc35000-0x6fc45000 data:0x6fc34000-0x6fc35000 /opt/java1.3/jre/lib/PA_RISC2.0/libjava.
Tools for Debugging gdb SIGBUS when Debugging, 8 Dynamic libraries (cont): /usr/lib/libnss_dns.1 text:0x6d21c000-0x6d220000 data:0x6d21b000-0x6d21c000 /usr/lib/libnss_nis.1 text:0x6d215000-0x6d21b000 data:0x6d214000-0x6d215000 /usr/lib/libnsl.1 text:0x6c704000-0x6c77f000 data:0x6c6ef000-0x6c704000 /usr/lib/libxti.2 text:0x6c6da000-0x6c6ef000 data:0x6d20d000-0x6d212000 .
Tools for Debugging gdb SIGBUS when Debugging, 9 Local Time = Mon Oct 8 20:09:10 2001 Elapsed Time = 6927 # # HotSpot Virtual Machine Error : 10 # Please report this error to HP customer support. # # Java VM: Java HotSpot(TM) Server VM (1.3.1 1.3.1.01-release-01081613:34-PA_RISC2.0 PA2.0 mixed mode) # # An error report file has been saved as hs_err_pid19019.log. # Please refer to the file for further information.
Tools for Debugging gdb SIGBUS when Debugging, Summary • gdb stack traces (Java and Native) • Enable you to distinguish problems that are in: – Interpreted code – Runtime compiled code – Native code • Give additional information about the native code executed – Which instruction was being executed • Permit you to look at the other threads in the process Page 83 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Debugging a Core File Debugging a Core File Page 84 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Debugging a Core File, 1 • For remote analysis • Need to reproduce environment – Correct versions of all libraries • Easy to look at the core file where generated • All of the libraries are already in place • Remote problem analysis • Use gdb to generate a preliminary report – Stack traces can be used to compare with known problems • In the example that follows: • Core file generated by the 1.3.1.
Tools for Debugging gdb Debugging a Core File, 2 $ export GDB_JAVA_UNWINDLIB=/tmp/libjunwind.sl $ gdb /opt/java1.3/bin/PA_RISC2.0/native_threads/java --core=core • Need to: • Specify correct path to executable – Might be installed elsewhere – Could be a 1.0 machine (PA_RISC in the path above) • Have correct libraries (including system libraries) on the system – Can use gdb’s GDB_SHLIB_PATH environment variable HP gdb 3.0 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00.
Tools for Debugging gdb Debugging a Core File, 3 #0 0xc01f4000 in kill () from /usr/lib/libc.2 (gdb) bt #0 0xc01f4000 in kill () from /usr/lib/libc.2 #1 0xc0191358 in raise () from /usr/lib/libc.2 #2 0xc01d198c in abort_C () from /usr/lib/libc.2 #3 0xc01d19e4 in abort () from /usr/lib/libc.2 #4 0xc40eacd0 in os::abort () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #5 0xc40e4598 in os::handle_unexpected_exception () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Debugging a Core File, 4 (gdb) thr 10 - Example to show that you can look at other threads [Switching to thread 10 (system thread 28979)] #0 0xc01f28d0 in __ksleep () from /usr/lib/libc.2 (gdb) bt #0 0xc01f28d0 in __ksleep () from /usr/lib/libc.2 #1 0xc15c8298 in _lwp_cond_timedwait () from /usr/lib/libpthread.1 #2 0xc15c7fd4 in pthread_cond_wait () from /usr/lib/libpthread.1 #3 0xc40d5ad0 in Monitor::wait () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Debugging a Core File, 5 #7 0xc414e6dc in typeArrayKlass::allocate () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #8 0xc40e0540 in oopFactory::new_typeArray () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.sl #9 0xc4304c48 in OptoRuntime::new_typeArray_C () from /opt/java1.3/jre/lib/PA_RISC2.0/server/libjvm.
Tools for Debugging gdb Debugging a Core File, 6 • Use following command to view all thread stacks: (gdb) set pagination off (gdb) thread apply all bt • Remote analysis • Direct the output to a file using: (gdb) set redirect-file (gdb) set redirect on (gdb) thread apply all bt • To return the display to the debugger: (gdb) set redirect off Page 90 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Debugging a Core File, Summary • Java/Native core file analysis • Powerful tool for problem diagnosis • Enables quick remote diagnosis of problems Page 91 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Native + Java Support: Summary gdb Native + Java Support Page 92 © 2001 Hewlett-Packard Company
Tools for Debugging gdb Native + Java Support: Summary • Powerful tool for problem diagnosis • Debugging problems – Correctness – To answer question – What’s going on in the JVM? – Hung processes – Curious phenomena that occur during execution • ENABLED: Fast remote diagnosis of problems • Redirect output to a file • gdb is another diagnostic tool that complements – Xverbosegc – Xeprof/HPjmeter Page 93 © 2001 Hewlett-Packard Company
