Debugging a hung ruby process with GDB

GDB is a great xNix debugger than is hugely useful into tracking down why a ruby process is hung. Below is a very simple how-to.

1. Install GDB

This is different between operating system. On Ubuntu, it is as simple as:

sudo apt-get install gdb

2. Find the hung process id

It will typically be the far left number associated with process you are looking for.

ps aux | grep ruby

3. Point GDB at that process

Once started you will see a lot of text scroll past.

gdb -p <process-id>

4. Print the backtrace

This is the part where you find what you are looking for.

bt

While this may look a little cryptic, it is a C backtrace and will tell you exactly where things are seizing. In this example it was a long running profiling tool that locked things up.

	#0 0x007611e0 in rb_stack_trace (result=0xbfd493ac, max_depth=64) at perftools.c:87
	#1 0x007613af in CpuProfiler::prof_handler (sig=27, signal_ucontext=0xbfd4958c, cpu_profiler=0x76b4e0) at src/profiler.cc:285
	#2 0x007622da in ProfileHandler::SignalHandler (sig=27, sinfo=0xbfd4950c, ucontext=0xbfd4958c) at src/profile-handler.cc:450
	#3 <signal handler called>
	#4 0x08055b78 in catch_timer ()
	#5 <signal handler called>
	#6 0x0018a606 in memcpy () from /lib/tls/i686/nosegneg/libc.so.6
	#7 0x11aebb00 in ?? ()
	#8 0x080573fa in stack_extend ()
	#9 0x0df18c24 in ?? ()

view raw gistfile1.txt hosted with ❤ by GitHub

5. Go forth and fix your code

Now you just need to quit out of GDB; thank the world for its existance and go fix your code.

quit

The Exception to global ruby variables

Global variables are often the over looked cousin of ruby variables. If you have never used them before, here is a quick breakdown

Today I took a closer look as to how they are used. I have seen a lot of ruby code that uses "$!" to refer to a caught exception. Simple example:

	begin
	raise "boom"
	rescue
	logger.error($!)
	end

view raw gistfile1.rb hosted with ❤ by GitHub

Well, today, as I was trying to write a style guide for OtherInbox, I started to get really concerned about the thread safety of such code. Since global variables are accessible by any thread, there appeared to be a pretty clear thread-safety issue with using $! to refer to an exception. I was ready to scrub the codebase and rewrite everything to look like this:

	begin
	raise "boom"
	rescue StandardError => e
	logger.error(e)
	end

view raw gistfile1.rb hosted with ❤ by GitHub

Prior to jumping off the deep end, I discussed this with the Capital Thought team. After a rather lengthy discussion about the proper way we should all be handling exceptions (which I hope to share in the future), I wrote a quick test to see if there was a thread safety issue or not.

	threads = []
	threads << Thread.new do
	begin
	raise "boom"
	rescue
	puts $!
	sleep(1)
	puts $!
	end
	end
	threads << Thread.new do
	begin
	raise "pow"
	rescue
	puts $!
	end
	end
	threads.each { |t| t.join }

view raw gistfile1.rb hosted with ❤ by GitHub

If you run this script, it will output "boom, pow, boom". This clearly shows that although "$" defines a global variable in the case of $ERROR_INFO ($!), it is has a local scope.

Why I don't test

Three words and one contraction and the entire development community probably hates me. Well, almost the entire community (I hope to find a few developers that share the my sentiment).

I work for a (not-so-much-anymore) early-stage startup and we have more abandoned ideas than successful ones. This is indicative of many companies. You have to throw a lot of spaghetti against the wall to see what sticks. This is rarely the environment for BDD, TDD, TAFT or perfectionists. This is where you write a couple of functional tests and release your code into the wild. The words "Alpha" & "Beta" are dear to my heart.

Startups are the entrepreneurial spirit at its best. Even if you are smart, motivated and hard working, this environment is largely a numbers game. The more ideas you actualize, the more are given a chance to take hold. The 80/20 rule is in full effect and great code doesn't matter if no one uses it.

As a developer, ask yourself, "How many cool side projects never made it past the first couple days of coding?". I guarantee all of you have at least one and most of you have a lot more. Now, ask yourself, "What is the main reason you didn't get further in your project?". I also guarntee that the most common reason is getting bogged down in the minutia (like testing). If you would have taken the quickest path to throwing up something that worked most of the time, you could have discovered if your idea was successful.

Not testing isn't as crazy as it sounds. If an idea sticks, then you need to build out a test suite as you handle edge cases and add further functionality. On the other hand, odds are, what you are coding right now isn't going to be around 6 months from now. So, don't waste your time.

I once had to write an IMAP client coded to rfc spec in less than a week. Their is no way I could have done this while writing tests the whole way through. The requirements changed constantly and I would have had to mock an entire IMAP server first. (Fortunately, this idea gained in popularity and now has great test coverage.)

Code is a means to an end and not an end in itself. Tests support the code we write, proving it works well. If your tests don't allow you to code faster then they are failing (if you work for a startup). Here, we write code to prove ideas work. This is why Ruby on Rails has become so popular; it's ability to rapidly prototype (aka quickly birth a concept).

Their are many environments where this attitude doesn't work and wouldn't even make sense (contractors, opensource contributions, large companies, proven models, etc, etc), but TAFT is wrong. Their are no hard and fast rules, but if their is one, I would suggest it is this: T.O.S. (Test On Success).

A Rhythm to the Madness

OtherInbox, the company I work for, has been in development for two years now. In that time, the team of developers has varied in size from one to six people. We have explored many avenues of email management; turned on a dime to address emerging technological trends and met some very tight deadlines. As we have been doing this, we have also been constantly defining and refining our teams rhythm.

The first step in doing this was to define a block of time in which we would measure our progress and deliver new features. For us, two weeks was the sweet spot. It is long enough to solve complex problems, but short enough to get rapid feedback from users. We refer to these two week chunks of time as "sprints".

With that defined many questions arose:

• What do we need to do to be prepared for a sprint?


• When should we start our sprint?


• How can we make sure we will reach our sprint goals?


• What does it mean for a sprint to be successful?


• When should we deliver these new features to the website?


• How do we incorporate user feedback into this rhythm?

To address these questions we plotted what we should be thinking about on a recurring calendar.

The interesting thing that we discovered was that to effectively manage all these concerns and keep an open pipeline of progress; we had to be focused on four sprints in any given two week timespan. For instance, lets look at "Sprint 3", that begins development on September 23rd (9/23). We started planning for that sprint back on 9/7 in a product planning meeting (Sprint 3 - Ideas) where we listed out the ideas we would like to develop and determined which ones were worth investigating further. The next day (9/8) we take a week to draw up, refine and negotiate between our Marketing, Product and Development Departments what these features will look like. From there (9/15), the Development Department takes 4 days (Sprint 3 - Design & Estimation) to translate these specs in to tickets and to estimate how complex each ticket is. At the beginning of this ticketing process(9/15), the Development Department will also talk to our Customer Service Advocate and incorporate our user's feedback into the tickets available for Sprint 3. On Monday & Tuesday (9/21 - 22) we will organize the tickets created based on priority. This all leads us up to Wednesday morning (9/23), the developers now have an organized list of defined, designed and estimated tickets to work on.

That is just the first half of our rhythm. On October 6th Sprint 3 will end for our developers and acceptance testing and deployment concerns begin. Sprint 3 will spend the next five days being tested (10/7 - 10/11) at alpha.otherinbox.com (go check it out!). Then, finally, this sprint, that began on 9/7 with group of people huddled around a table throwing around ideas will be deployed to our production (my.otherinbox.com) on October 12 and be available to the public.

While this might seem like a long cycle, its important to note that development never stops. For that two week period that we were actually writing the code for Sprint 3 we also:

• QA'ed Sprint 2


• Spec'ed Sprint 4


• Deployed Sprint 2


• Demo'ed Sprint 2


• Designed & Estimated Sprint 4


• Gathered User feedback for Sprint 4


• Started Dreaming & Spec'ing Sprint 5

You may wonder why we start our sprints on Wednesdays. The answer is simple: we wanted to deploy on Mondays. This gives us the entire week to address any issues that may arise in production that were not caught in QA. Nobody likes working weekends (and that often happens with late week deploys).

On that note, I will leave you with what I think the most successful aspect of our development cycle is: flexibility. Don't have a schedule that is set it stone or you will leave no room for improvement.