The Four Hundred Sixty Five Million Dollar Software Bug

I love the story of a really good bug, and this one is a $465M whopper:

"While processing 212 small retail orders that Knight had received from its customers,
SMARS routed millions of orders into the market over a 45-minute period, and obtained over 4
million executions in 154 stocks for more than 397 million shares. By the time that Knight
stopped sending the orders, Knight had assumed a net long position in 80 stocks of approximately
$3.5 billion and a net short position in 74 stocks of approximately $3.15 billion"

http://www.sec.gov/litigation/admin/2013/34-70694.pdf 

MAXUINT emails?

I mean, I was on vacation for a couple of weeks, but I couldn't have received *that* many emails!

Worst error message ever?

The error message that Outlook gave me must be one of the worst ones I've ever seen.

Errors have been detected in the file c:\Documents and Settings\outlook.ost. Quit Outlook and all mail-enabled applications, and then use the Inbox repair tool (Scanpst.exe) to diagnose and repair the errors in the file. For more information about the Inbox repair tool, see Help.

If they know what to do, and have created a tool to fix this, then why do they bother the user with the painful details?

software bug in military robot injures 14

I like collecting stories of horrendous software bugs, but this one is a bit scarier than most:

March of the terminators: But what happens when robot warriors turn their guns on us? | Mail Online: "Another 14 are seriously injured. The report will later blame the bloodbath on a 'software glitch'."

F22 International Dateline Bug

Six $125M airplanes almost crash because someone forgot that airplanes will sometimes cross the international dateline.

DailyTech - Lockheed's F-22 Raptor Gets Zapped by International Date Line

Famos F-16 inversion bug

"Apparently a problem in the navigation software inverted the [f-16] aircraft whenever it crosssed the equator. Luckily it was caught early in simulation testing and promptly fixed. In the July issue, J.N. Frisina at Singer-Kearfott wrote to Mr. Neumann, 'concerned that readers might have mistakenly believed there was a bug in the flight software, which was of course not the case.'

Mr. Frisina wrote: 'In the current search for reliable software, the F16 Navigation software is an example of the high degree of reliability and quality that can be obtained with the application of proper design verification and testing methodologies. All primary misison functions were software correct.'"

old school linkage

comp sci 101 practices realy do work

They teach all kinds of maxims in your first programming class: no magic numbers, no gotos, no globals, etc.

It's still amazing how many people still violate these maxims though. Today I saw further evidence supporting my favorite of these maxims: no magic numbers.

consider the call to a wait function when reading an accelerometer:
TimerHW_busyWait(wait_200_usecs);

It would have been only slightly less clear, but by no means unintelligible to call:
TimerHW_busyWait(200);

And clarity could have been fixed if the function were named differently:
TimerHW_busyWaitUsec(200);

But two months after I originally wrote this code, and several slight revisions by another developer later, we found that .08% of accelerometer samples were getting corrupted because we needed to wait a little longer than the 200us the accelerometer datasheet recommends.

This is when the compiler and the no magic numbers maxim helped me avoid a stupid error. You see, I changed the #define to be:
#define wait_225_usecs 225

Unbeknownst to me, at another place in the same file someone else was sampling the accelerometer using the wait_200_usecs constant. Had they both been magic numbers, I might have only fixed the error in once place.

software bug killed Mars Global Surveyor

I'm always interested to see what brings down these hundred million dollar babies.

In this case two bad addresses brought down a satellite that cost about $220 million to build and launch.

NASA Decides That A Software Error Doomed The Mars Global Surveyor Spacecraft | SpaceRef - Your Space Reference: "This software tried to synch up two flight processors. Two addresses were incorrect - two memory addresses were over written. As the geometry evolved, we drove the [solar] arrays against a hard stop and the spacecraft went into safe mode. The radiator for the battery pointed at the sun, the temperature went up, and battery failed. But this should be treated as preliminary."

Code inspection would have been faster!

So, I've read the white papers, I've done a 7 week intensive PSP training course, I believe what the experts say about code reviews being faster to find bugs than conventional debugging.

But this morning I spent 3 hours chasing a bug that would likely have been caught in code inspection: I was using the wrong loop index name. dang.

It would have been caught as a compiler error if I were in C++ (do to delcaring the loop index in the for loop) but I'm doing C on a little 8051 with 16kB of code space and 1280 bytes of RAM (yes, bytes).

Another excuse (but an excuse nonetheless): I was having debugger problems, to the tune of semi random crashes of the IDE, necessitating hard power cycle of board and debugger, with a little vitamin reboot on the PC thrown in for good measure. Since the tools have been rock solid up till now, and they're relatively new to me I didn't want to wallpaper over problems, so I ruled out bad hardware, bad USB cables, out of date IDE/debugger software and firmware. Maybe because I was using 1154 of my 1280 RAM bytes? Commenting large blocks of code helped in a similar way that I feel like my car runs better after an oil change.

In the end printf debugging made me actually look at what the hell the code was doing. Why didn't I dash off a hard copy of the code and read it in the first place?