Debugging a hung ruby process with GDB

GDB is a great xNix debugger than is hugely useful into tracking down why a ruby process is hung. Below is a very simple how-to.

1. Install GDB

This is different between operating system. On Ubuntu, it is as simple as:

sudo apt-get install gdb

2. Find the hung process id

It will typically be the far left number associated with process you are looking for.

ps aux | grep ruby

3. Point GDB at that process

Once started you will see a lot of text scroll past.

gdb -p <process-id>

4. Print the backtrace

This is the part where you find what you are looking for.

bt

While this may look a little cryptic, it is a C backtrace and will tell you exactly where things are seizing. In this example it was a long running profiling tool that locked things up.

	#0 0x007611e0 in rb_stack_trace (result=0xbfd493ac, max_depth=64) at perftools.c:87
	#1 0x007613af in CpuProfiler::prof_handler (sig=27, signal_ucontext=0xbfd4958c, cpu_profiler=0x76b4e0) at src/profiler.cc:285
	#2 0x007622da in ProfileHandler::SignalHandler (sig=27, sinfo=0xbfd4950c, ucontext=0xbfd4958c) at src/profile-handler.cc:450
	#3 <signal handler called>
	#4 0x08055b78 in catch_timer ()
	#5 <signal handler called>
	#6 0x0018a606 in memcpy () from /lib/tls/i686/nosegneg/libc.so.6
	#7 0x11aebb00 in ?? ()
	#8 0x080573fa in stack_extend ()
	#9 0x0df18c24 in ?? ()

view raw gistfile1.txt hosted with ❤ by GitHub

5. Go forth and fix your code

Now you just need to quit out of GDB; thank the world for its existance and go fix your code.

quit

The Exception to global ruby variables

Global variables are often the over looked cousin of ruby variables. If you have never used them before, here is a quick breakdown

Today I took a closer look as to how they are used. I have seen a lot of ruby code that uses "$!" to refer to a caught exception. Simple example:

	begin
	raise "boom"
	rescue
	logger.error($!)
	end

view raw gistfile1.rb hosted with ❤ by GitHub

Well, today, as I was trying to write a style guide for OtherInbox, I started to get really concerned about the thread safety of such code. Since global variables are accessible by any thread, there appeared to be a pretty clear thread-safety issue with using $! to refer to an exception. I was ready to scrub the codebase and rewrite everything to look like this:

	begin
	raise "boom"
	rescue StandardError => e
	logger.error(e)
	end

view raw gistfile1.rb hosted with ❤ by GitHub

Prior to jumping off the deep end, I discussed this with the Capital Thought team. After a rather lengthy discussion about the proper way we should all be handling exceptions (which I hope to share in the future), I wrote a quick test to see if there was a thread safety issue or not.

	threads = []
	threads << Thread.new do
	begin
	raise "boom"
	rescue
	puts $!
	sleep(1)
	puts $!
	end
	end
	threads << Thread.new do
	begin
	raise "pow"
	rescue
	puts $!
	end
	end
	threads.each { |t| t.join }

view raw gistfile1.rb hosted with ❤ by GitHub

If you run this script, it will output "boom, pow, boom". This clearly shows that although "$" defines a global variable in the case of $ERROR_INFO ($!), it is has a local scope.