Debugging MXNet GDB with Python

Compiled the code using

make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda-9.0 USE_CUDNN=1 DEBUG=1

Updated python dependency
cd python
pip install -e .

Trying to debug specific part of function in C header file
Hence, ran
gdb python
(gdb) start tests/python/unittest/test_random.py
(gdb) b src/operator/random/sampler.h:100
No source file named src/operator/random/sampler.h.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (src/operator/random/sampler.h:100) pending.

Then, I continue, hoping that it would hit the breakpoint, it just doesn’t.
(gdb) c

Code executes and terminates (without halting at the breakpoint)

Where am I going wrong?

@kellen @lebeg I remember you mentioned something about debugging not being possible in MXNet and possible work-around, can you shed some lights here? Thanks!

@larroy might have some ideas

Please compile with CMake and use -DCMAKE_BUILD_TYPE=Debug

The rest of what you are doing is correct. It will hit a breakpoint with gdb.

You can also try cgdb.

2 Likes

Python has a debugger , which is available as a module called pdb . It supports setting conditional breakpoints , stepping through the source code one line at a time, stack inspection, and more.

import pdb
msg = "this is a test"
pdb.set_trace()
print(msg)

Insert pdb.set_trace() anywhere and it will function as a breakpoint . When you execute the script by python test.py, you will in the debug mode.

Thanks for this reply. pdb.set_trace() can be added as you correctly mentioned in *.py (python files)
But my original question had to do with *.cc (c files) Hence adding breakpoint using python wouldn’t go into C files. So although your commands are correct, it is not relevant to this question.

Thanks nonetheless. Pedro’s suggestion helped me (build with cmake)
These days I build with cmake using Pedro’s build scripts

incubator-mxnet$ cp cmake/cmake_options.yml .

Edit the cmake_options.yml file so that CMAKE_BUILD_TYPE: "Debug"

./dev_menu.py build

Hi @larroy, Is it necessary to compile with CMake? what if I only compiled with make comand(based on config.mk) and set DEBUG = 1 during the compilation.

@huangh12 It’s not necessary to build with cmake. But latest mxnet master has migrated from make to cmake. cmake seems to be chosen over make everywhere these days. Hence advised.