How do I use a Debugger on the HPC Machines?



  • An updated discussion November 5, 2008. This discussion is meant to give users an idea how to use debuggers and how to apply them in parallel, but it is by no means complete. One on-line tutorial is Norm Matloff's Debugging Tutorial. Of course, you can find many other on-line tutorials, e.g., gdb for C and C++. If you have more questions on how to use debuggers on the NC State blade center, please contact me (gary_howell@ncsu.edu).

  • What Does a Debugger Do?

    One way to debug Fortran or C code is to write print statements and recompile and rerun. For instance, if you have just changed a bit of code and want to make sure that the new code executes as you think, you might print variables to see if the code modifies them in the way you predict. Or if having added or changed a subroutine, you find that the code fails to execute correctly, you might put print statements at the start of the subroutine to verify that variables are passed correctly.

    Using a debugger allows you to accomplish these tasks without repeatedly recompiling. So if you've had to change hundreds of lines of code without good test cases for each few lines, and want to monitor the code behavior line by line, perhaps comparing to a known test case, using a debugger can be helpful. Learning to use a debugger may be useful either for your own future projects or in aiding colleagues.

    Stepping through programs. A debugger allows you to step through a Fortran or C program. At each step the program listing is displayed and before going on, you can check current values of program variables. Before starting program execution under the debugger, the user specifies one or more break points. On command, the program runs till the first break point. The user can then go on step by step or can set a new break point and ask the program to continue execution to the next break. All the debuggers described below can be used in this fashion.

    Examining core files. When code execution fails, a core file is created, typically called "core" or "core.jobnumber". The core file is in binary format so is not viewable with an editor. Assuming the code has been compiled with the -g flag, debuggers can allow you to examine a core file to see what subroutine crashed and at what line, what program called that routine (and so on through the whole stack). Also the user can print out values of program variables at each level of the stack. dbx on the IBM shared memory machine works well for examining core files. On the blade center, Totalview allows examination of core files; I have not succeeded in examining fortran core files using pgdbg or gdb, but suspect it may be possible.

  • Compiling Code So You Can Use a Debugger

    The program should be compiled with the -g flag, constructing a symbol table that allows a line by line stepping through the source code. Also turn off the -O2 optimizations and all other optimizations. Compiler optimizations are quite a nice set of tricks, but they usually work by rearranging the order of operations, so they make it hard for the debugger to correlate program lines with code execution.

  • What Debuggers are Available?

    On the Linux Blade Center, the Portland Group C and Fortran compilers work with the gdb and pgdbg debuggers. Totalview works with Intel as well as Portland group compiled codes, with gnu codes. Below we describe a method of using any of the gdb debuggers in parallel. It is also possible to use the pgdbg and Totalview debuggers in parallel. On the shared memory IBM machines, the IBM supplied dbx and pdbx debuggers work well with IBM xlf and xlc compilers. dbx is a good serial debugger and pdbx works well in parallel. dbx works well with core files.

  • Debuggers on the Linux BladeCenter

    On the Linux blade center, the gdb, pgdbg, and Totalview debuggers are available.
  • The GDB Debugger
  • The pgdbg Debugger
  • The Totalview Debugger

  • The GDB Debugger

    GDB is a classic open source program developed by Richard Stallman. The GUI based interface is called ddd. By making small modifications to code, you can debug parallel MPI jobs. If you learn gdb (or ddd) you can use them on almost any linux based system. gdb works well with codes compiled with gcc, g++ or gfortran. In the past, it also worked well with PGI compilers, but I have not verified that recently.

    >info gdb

    gives a complete and fairly easy to follow set of instructions. If X11 forwarding works so that you can pop a GUI,

    >ddd

    brings up a ddd session that includes a "help" button.



    For debugging purposes, compile with the -g flag and no optimization (optimizing can confuse things by rearranging code execution order). For example,

    >gfortran foo.f -g -o foo

    compiles foo.f to produce the executable file foo, where the -g preserves the symbol table in such a way that the debugger can step through the source code, listing the current code line. Typically at run time, one sets a break point, lets the code execute to that point, then steps it through a suspect section of code, observing variables to see where they go astray.

    >gdb ./foo

    starts a gdb session attached to the executable foo. Similarly

    >ddd ./foo

    brings up a ddd GUI based version of gdb, which lets you do more with a mouse, but which also has a window which allows the commands given here to work.

    Suppose that the know the code's problem is in SUBROUTINE FOOSUB. At the prompt one can enter,

    gdb>break foosub_

    or

    gdb>b foosub_

    Then entering

    gdb>run

    or

    gdb>r

    will run the code till it enters SUBROUTINE FOOSUB.

    gdb>n

    will step through the code to the next executable line. (Actually I've often found that the code misses the break point at foosub_ the first time and has to run again). 'n' (short for 'next') steps through an executable a line at a time, stepping past a subroutine or function call in one step. To step into a subroutine, use

    gdb>s

    (short for 'step'). If ivar is a variable inside foosub

    gdb> print ivar

    or

    p ivar

    will display the current value of ivar. Suppose that A is a two dimensional matrix

    gdb> print a(2,3)@5

    would print a(2,3) and a total of five adjacent elements from memory, which in Fortran storage is the consecutive entries from a column, but a peculiarity of gdb is that this notation only works in the main program. Inside subroutines, fortran arrays are stored as a vector starting with position 0, stored by columns. So if A has leading dimension lda and A is being used to store a matrix of m rows and n columns

    gdb>p a(0)@m

    would print the first column of A.

    gdb>p a(2*lda)@m

    would print the second column of A.

    gdb does not seem to have a good way to print a section of a Fortran matrix row (in C matrix rows are stored consecutively, so gdb would easily display a matrix row). So a Fortran row would have to be displayed one print statement at a time (where in pgdbg you could use matlab notation to print a matrix row).

    Once you're stepping through foosub, and want to leap to a breakpoint at line 1142, you can set a new breakpoint.

    gdb>break 1142

    and jump to it by

    gdb> cont

    (provided your code would execute this line). One way to tell where to put the next breakpoint is by opening another xterm with an edit session of the source code. Find the line number you want (in vi, you would park the cursor on the line you want and ascertain its line number by typing :.= ), say 1311, then

    gdb> break 1311

    would put a break at that line. In ddd, the file you are editing is diplayed as part of a split screen in which you can scroll up and down, so the spare xterm is not quite as necessary, though the spare xterm may still be convenient.

    gdb> l 1311

    lists lines around 1311 in the command line window.

    dbg> quit

    Of course, this is just a start on how to use a debugger, but you can get a hint that using the debugger can save time on recompiling just to put in print statements.
  • The pgdbg Debugger The pgdbg debugger uses most of the conventional dbg debugger commands. For some on-line documentation, see The Portland group user guide The sample session for gdb will also work for pgdbg, where the session is initiated by

    >pgdbg ./foo

    Displaying a slice of a matrix is a bit easier. While the gdb notation still works, the easier column slice

    pgdbg> print a(2:6,3)

    and row slice

    pgdbg> print a(2,3:5)

    notations are also available. Pgdbg has man pages. Help is available from within the debugging sessions by typing

    pgdbg> help



  • The Totalview Debugger We have a license for the Totalview debugger. It works well with Intel ifc compiled codes. A Totalview tutorial is available at Totalview Tutorial .

  • Debuggers on the IBM p590 To start the dbx debugger, produce an executable foo.exe by compiling it with the IBM Fortran or C compilers with the -g flag.

    >xlf90 -o foo.exe -g foo.f

    You can start a debug session by

    >dbx ./foo.exe

    Breakpoints are set by the name and line number of the file containing them.

    (dbx) stop at "foo.f":1169

    This will set a break at line 1169 of foo.f.

    The syntax

    (dbx) print a(1,2)

    is valid, but there seems to be no way to show a slice of a matrix. Another problem can be that though scalar variables print quickly, there can be a long delay in printing elements of a matrix.

    (dbx) cont

    continues execution of the code to the next breakpoint.

    One virtue of the dbx debugger is convenience of examining core files.

    Suppose that a -g compiled code foo runs and dumps a core

    > foo

    Segmentation fault - core dumped

    To investigate the error,

    >dbx foo

    Dbx reports the line where the dump occurred. You can examine the stack (what program called the subroutine and what program called that routine, and so on), and can print variables on each level of the stack.

    (dbx) up

    (dbx) down

    move up and down the stack respectively.

    (dbx) quit

    exits dbx.

    The p590 has a long man page for dbx which includes example sessions.



  • Parallel Debuggers on the Linux BladeCenter

    The pgdbg, Totalview, and gdb debuggers should each work in parallel. The description here is of using the gdb (or ddd debugger) used on a VCL node.

    MPI (Message Passing Interface) code are typically used for distributed memory codes, i.e., each processor has its own memory and communicates to other processors by sending and receiving messages. For purposes of debugging with gdb, we've implemented a shared memory version of MPI. Here the messages are passed within a single shared memory node. It's possible to start more processes than the number of physical cores.

    The mpirun call starts up a process. When an MPI_Init call is encountered, that root process starts up the requested number of new processes. By putting a pause after the MPI_Init, we can identify the new processes and attach each new process to a gdb session. Then we can step through each of the processes individually.

    Here's an example. From one of the 32 bit login nodes,

    source /home/gwhowell/mpiches/mpich-1.2.7p1/gnu32/ch_shmem/gnu32sh.csh

    This sets up the mpif77, mpicc, mpicxx commands to use the gnu32 library. Compile a simple MPI code, e.g., the monte.f code from the MPI short course. An excerpt is as follows. 

          real*8 ans(10), ans2(10)
      real*8 startim, entim, sum, sindex
c
c   function
      integer string_len
      iflag = 1
c
      call MPI_INIT(ierr)
      do while (iflag.eq.1)
      end do
c
      call MPI_COMM_SIZE(MPI_COMM_WORLD, p, ierr)
      call MPI_COMM_RANK(MPI_COMM_WORLD, my_rank, ierr)
*     print*,' I am ', my_rank 
c  
      if (my_rank.eq.0) then
cc         print*,'input random seed'

    Notice the "do while" line just after the MPI_INIT. This line stalls the code indefinitely (taking 100% of the cycles on some core). For this reason it's a good idea to debug on a vcl node (go to vcl.ncsu.edu and get an HPC linux image) as opposed to login01 or login02.
    Typing 

    >mpirun -np 2 ./monte

    starts up the code. Doing "ps -ef | grep monte" 
     
[gwhowell@login02 ~]$ ps -ef | grep monte
gwhowell 25532 25386  0 14:39 pts/64   00:00:00 /bin/sh /home/gwhowell/mpiches/mpich-1.2.7p1/gnu32/ch_shmem/bin/mpirun -np 2 ./monte
gwhowell 25560 25532  6 14:39 pts/64   00:01:42 /home/gwhowell/ppmpi_f/chap03/./monte
gwhowell 25561 25560 10 14:39 pts/64   00:02:40 /home/gwhowell/ppmpi_f/chap03/./monte

    shows that 2 monte jobs have started up. In another window, 
    gdb 
gdb> attach 25560
gdb>  set iflag = 0

    and in yet another window 
    gdb
gdb> attach 25561
gdb> set iflag = 0

    bring up gdb sessions attached to these two processes. Setting iflag as 0 in each session pulls the process out of infinite loop. Repeatedly typing "n" will step through a process, and we now have two parallel gdb debug sessions attached to the two processes. A similar approach is outlined in Parallel debugging. (The gdb attach process syntax given there does not quite work on the blade center). The link shows syntax for stalling C codes while the debugger is attached.

    You may prefer to start up one or more of the gdb sessions with "ddd". This has a GUI interface which can be useful. For example, when I tried 

    gdb>p my_rank

    gdb did not know of any such local variable. By clicking the "display" button in ddd, I was able to find a local variable called my_rank__ and print that.

    Totalview debugging can be accomplished in much the same way as outlined here (but so far works only for fortran and C codes, not for C++). For the totalview debugger, try 

    source /home/gwhowell/mpiches/mpich-1.2.7p1/intel32/ch_shmem/tv32.csh