Performance Anomalies Can Be A Sign of Bigger Problems...

I have a great story to tell you...I get a call from one of my clients,
they are a big real estate management and development company
with many large Oracle databases and various Unix and Microsoft
systems, a large SAN, and they run the network on the high-end
Cisco stuff. Typical architecture for a large enterprise.

These guys have a lot of vendor consultants on site helping them out
with new Oracle Apps, Unix systems, etc. The problem, I'm told by one
of their Senior Vice Presidents, is intermittent performance problems that affect the network and the Oracle Apps.

Hmmm...probing him further for more details did not give me much. He did explain to me that none of their Oracle or Unix consultants saw any problems with anything, the system administrators didn't see any problems on the systems, but the network guys were seeing intermittent network utilization problems coming from the application servers - basically they're monitoring the network with SNMP and every now and then they get alarms stating high utilization of the network. They give this info to the consultants yet they find no problems.

Their VP asks me to do my own performance assessment and I did. I arrive on site and begin speaking with the network team. They basically tell me that two particular application servers intermittently consume a large amount of bandwidth. This has been going on and off now for many months and no one knows why.

So, I take a look at these two application servers. They happen to be Solaris Unix systems. I ran some netstat commands and saw that there were some input/output errors on these systems. I then proceeded to run some process status commands, I use both the ATT and UCB versions of ps when I want to get various process info. The weird thing here is that the UCB version of ps listed some additional processes that the ATT version didn't show - this is not normal. The process tree should be the same. So I download lsof (a powerful Unix utility that didn't ship on many earlier Solaris distributions), and I found some hidden processes that the running ps commands didn't show. Hmmm. This is looking bad. I trace the processes to some hidden files and found that the system was hacked. Not only was the system hacked, it was hacked by two different groups and over a year ago!

The first hackers installed what is known as a root kit. The root kit basically installs hacked versions of many system utility programs to keep system administrators in the dark about the running hacker programs. Basically my clients systems were used as bots to run denial of service attacks against other nodes on the internet. During these attacks, the bandwidth would be consumed and the performance problems would occur. When the system administrators and consultants looked at the problem using the system utilities, they did not see the actual running programs as the root kit hid it from them.

Looking at the security of the system, they were originally breached through an Apache vulnerability. When I suggested they upgrade to a more secure version of Apache, their development team stated that would violate their Oracle support agreement! OMG!!!

Anyway, another group of hackers were also using the systems based on the time stamps I saw on their programs. These guys used a cool tool called stunnel (it was renamed and hidden in this case) which allowed them remote access into there system via IRC servers. I found the embedded irc servers and cryptic login information being used by stunnel. I was curious and logged into the irc server with the login details I uncovered and low and behold, the hacker was online and boy did I catch him off guard. I had enough info on the hacker and gave a lot of forensic data to my FBI contacts and the hacker ended up being prosecuted and convicted and had to pay restitution to my client. Nice ending to a twisted scene.

So the next time you hear you may be having a recurring performance problem, take a deeper look into the situation, you may be surprised what you find out... ;)

-boni bruno