Tuesday, October 27, 2009

How to prepare for Denial of Service attacks against E-commerce sites

[This is from a response I sent to someone on a mailing list earlier today]

The first thing you should know: unless you are Google or Amazon or some entity of that size and have money to burn, you really should not rely 100% on an on-premise solution against DOS attacks; let your bandwidth or hosting providers be the first defense against it.

You can (and should) have your own solution, but without the protection beginning before your perimeter, the attack will block users from ever reaching you and thus become a successful attack.

Here is why:
The general principle of a DOS attack against an e-commerce site is to send a flood of HTTP requests. Most other types of DOS attack against some known problems with various TCP stacks have been fixed a while ago (or can be handled by various on-premise solutions)

TopLayer, Tipping Point, Arbor or SourceFire (and others) makes excellent intrusion prevention systems (IPS) that can block vanilla DOS attacks launched by 1 or 10 computers. But they are not effective against distributed denial of service attacks that go after your bandwidth.

If 20,000 computers hit your website at the same time, your bandwidth is going to be saturated (unless you have a grow-on-demand pipe).

For example, I just measured the front page for Amazon.com. Images, stylesheet and everything else combines for about 77.6 KB. Let's say (for this argument's sake) that Amazon's bandwidth is 100 MBps. So that pipe can serve roughly 13,195 page requests per second for that 77.6K page before becoming 100% saturated (100 x 1024 kbps / 77.6 kb/page). I am not even counting the webserver's CPU/Memory utilization, since that is inside the perimeter.

It won't matter what IPS they have on-premise; if the pipe is full, legitimate requests are going to be denied or delayed, resulting in a successful Denial of Service attack. This is exactly what happened to Amazon, Yahoo, E-Trade, CNN and some others in Feb 2000.

The bad guys do have thousands of machines with spyware installed at their disposal for this. Some groups reportedly have millions. It is a fact that they rent them out by the hour in blocks of hundreds or thousands for as little as $200 for 10,000 bots. They are mostly used to send out spam, but it is just as easy to launch a Distributed DOS attack.

So a belts and suspenders approach would be:
- Have a good IPS--you need that any way
- If using Linux servers, look at Netfilter so you can tar-pit the attacks
- If you are in a co-lo, talk to your bandwidth providers (you should have more than 1) about DDOS protection.
- If you are on a hosted server, pick a vendor like RackSpace that provides DoS Mitigation.

I know Cable & Wireless, ATT and Verizon all offer DDOS mitigation. They route away the bad packets away from you, and even the RBN does not have enough bots to saturate those bandwidths.

Tuesday, October 13, 2009

Wal-Mart breach in 2005-6: a lesson on things not to do

In 2006, a group of hackers targeted some Wal-Mart developers and stole source code to the Point-of-Sale (POS) system. Wired is reporting that the stolen source code ended up being sent to a server in Minsk, Belarus, in the former Soviet Union.
The Wal-Mart intrusion began unraveling on Nov. 5, 2006, when the company’s IT security group was brought in to investigate the server crash.

Wal-Mart has thousands of servers nationwide, and any one of them crashing would ordinarily be a routine event. But this one raised a red flag. Someone had installed L0phtcrack, a password-cracking tool, onto the system, which crashed the server when the intruder tried to launch the program.

Investigators found that the tool had been installed remotely by someone using a generic network administrator account. The intruder had reached the machine through a VPN account assigned to a former Wal-Mart worker in Canada, which administrators had failed to close after the worker left the company. The day the server crashed, the intruder had been connected to Wal-Mart’s network for about seven hours, originating from an IP address in Minsk, the documents show.

The security team disabled the compromised VPN account, but the intruder, who should have realized the jig was up, came back in through another account belonging to a different Canadian employee. When that VPN account was closed, the intruder grabbed yet a third account while Wal-Mart workers were still scrambling to get a fix on the scope of the breach.

When Wal-Mart reviewed its VPN logs, it found that the activity had begun at least as early as June 2005, according to memos written by Wal-Mart employees during the initial stage of the investigation. The company’s server logs recorded only unsuccessful log-in attempts, not successful ones, frustrating a detailed analysis.

Wired is also reporting that Wal-Mart had 4 years worth of unencrypted customer and credit card data at the time, but it was not breached. So they did not have to disclose it until now.
Wal-Mart had a number of security vulnerabilities at the time of the attack, according to internal security assessments seen by Wired.com, and acknowledged as genuine by Wal-Mart. For example, at least four years’ worth of customer purchasing data, including names, card numbers and expiration dates, were housed on company networks in unencrypted form. Wal-Mart says it was in the process of dramatically improving the security of its transaction data, and in 2006 began encrypting the credit card numbers and other customer information, and making other important security changes.

Wal-Mart's external IT/PCI auditor, CyberTrust, found some astonishing breach of common-sense security (this blog abhors the term "Security Best Practices"):
The assessment lasted six days, during which CyberTrust found numerous problems. Each of the five stores, for example, housed complete backup copies of transaction logs on network-connected UNIX servers, which included at least four years’ worth of unencrypted credit card numbers, cardholder names and expiration dates from purchases at the stores.

The auditors also discovered that servers, transaction processing systems, and other network-connected devices handling sensitive information used the same usernames and passwords across every Wal-Mart store nationwide. In some cases, the passwords could be easily guessed. A hacker or malicious insider who compromised a point-of-sale controller or in-store card processor at one store, could “access the same device at every Wal-Mart store nationwide,” CyberTrust wrote.

And ofcourse, the intrusion could be traced back to the VPN account of a system administrator who had left the company but his account was not shut down (the report does not implicate the employee)

Wal-Mart now claims that they have identified every single finding and are now PCI compliant. Fat lot of good being PCI-compliant did Hannaford.

These companies either forget, or do not understand (we suspect over strenuous objection of their security people) that being PCI complaint is only the lowest common denominator--they can, and should, do much more.

Thursday, October 8, 2009

The Thursday Maxim of Security

A gem from Dr. Roger Johnston at Argonne National Lab.

Thursday Maxim: Organizations and security managers will tend to automatically invoke irrational or fanciful reasons for claiming that they are immune to any postulated or demonstrated attack.

Comments: So named because if the attack or vulnerability was demonstrated on a Tuesday, it won’t be viewed as applicable on Thursday. Our favorite example of this maxim is when we made a video showing how to use GPS spoofing to hijack a truck that uses GPS tracking. In that video, the GPS antenna was shown attached to the side of the truck so that it could be easily seen on the video. After viewing the video, one security manager said it was all very interesting, but not relevant for their operations because their trucks had the antenna on the roof.

Wednesday, October 7, 2009

Ignore email archiving / public record laws at your own peril

The saga of the deleted emails continues, with now the AG getting involved.

Prior to that, the Secretary of State stated that he was unhappy with the City Hall's continued failure to comply:
Galvin expressed frustration Tuesday over what he described as the city's failure to fully cooperate with investigators. He told the Globe he was considering taking further action against the Menino administration. Under state law, he could turn the case over to Coakley for possible prosecution.

Now state Attorney General Martha Coakley (who is running for Ted Kennedy's senate seat BTW) states that she is "involved."
Coakley said in a statement issued this afternoon that Secretary of State William F. Galvin's office has been working to ensure that public records are preserved and "to determine whether there have been any violations of the public records law by City officials."

"We are now involved in that review," Coakley said.

She said her office would continue to work with Galvin's office going forward in the effort to find mayoral aide Michael Kineavy's e-mails and "we remain prepared to conduct a full investigation and take all necessary steps to guarantee the preservation of evidence and full compliance with the law."

Apart from jokes about the Chicken being 'involved' and the Pig being 'committed' in the making of Ham and Eggs, what does that word mean, anyway?

I personally know that there are Encase-certified digital forensics experts working for Coakley's office. Since City Hall is 'cooperating', it would be trivial to have the AGs office take a look at the hard disk instead of paying an outside consultant, right?

Marketing, Uncertainty and Doubt: Information Security and Cloud Computing

What is the minimum security due diligence that a company needs to do before putting it's data in the cloud?

Since 2007, Amazon has been telling us they are ".. working with a public accounting firm to ... attain certifications such as SAS70 Type II"  but these have not happened in 2+ years.

On one side of the cloud security issue we have the marketing people, who hype up the existing security and gloss over the non-existing. On the other side we have security services vendors, who hawk their wares by hyping up the lack of security. And there are also Chicken Littles who are running around crying that the sky is falling.


The truth is, there is a class of data for every cloud out there, and there is also someone who will suffer a data breach because they did not secure it properly.


Can you put the New York Times on a cloud server? Of course, provided certain basic security measures are taken. After all, the Times is designed to be accessible to people (forget the stupid PayWall experiment they tried a few years ago)


On the other hand, you should not leave your customer's credit card data on Amazon EC2--they specifically suggest you don't do that.


Another problem is, people are still not sure what "cloud" is. I saw a cartoon recently: "I fell down the stairs and something white is sticking out of my arms, and it hurts like hell. Is it swine flu?"

Most cloud security questions feel like that to me, so I have been accused of ranting in a presentation I did in September. Enjoy.



Tuesday, October 6, 2009

Tangled web woven at Boston City Hall

The saga continues at Boston City Hall. Readers of this blog will remember that in response to public record requests, it came out that the Boston Mayor's right-hand man was deleting emails in a way that they were not getting backed up. So the Secretary of State got involved and ordered the City Hall to change the practice and also to retrieve the emails.

Today it came out that our international man of mystery actually complained in April 2009 that his computer was running too slow and as a result, received a new computer. But gosh darn, he plum forgot! And he still does not remember getting a new computer.
City corporation counsel William F. Sinnott said in an interview yesterday that he had been relying on what Kineavy had told him and that Kineavy, the mayor’s chief policy aide and key political strategist, still does not remember getting a new computer.
Fortunately for people who like sunshine on their government affairs, and possibly unfortunately for Mr. Kineavy, that computer's hard disk was not wiped clean and reissued to another user--it was just sitting in another room. Now it has gone to the forensics firm hired by the City, and presumably the emails (or their remnants from temp files) will be recovered.

Ironically, I bet that this particular PC was not recycled because the user was a powerful man, and IT suspected/feared that he will ask for some old file from the hard disk that was not accurately transferred to the new PC.

Now to the cost of recovery. The most well-known commercial software used for digital forensics, Encase (there are others), will suck out anything relevant form that disk in a few hours and nicely categorize them in emails, word documents, etc.

One might even call the work technologically trivial. If StoneTurn group is really asking for 250K for a single hard disk examination, they are either smoking weed, or abusing a single-source, no-bid contract. I know many highly reputable forensics consultants who will do this for under $10,000, probably for as low as $5,000.


There are some good lessons here.
  • Mr. Kineavy is every information security officer's dream. That man knows how to protect against information leakage
  • Mr. Kineavy is every compliance officer's nightmare. That man is costing the City Hall time, money and prestige
  • Not having a good decommissioning policy is hurting Mr. Kiveavy but may help make the City Hall become compliant with the public records law (or at least get away with a slap on the wrist and the hundreds of thousands of dollars in forensics expense)
Bottom line: the wheels of justice grinds slowly, but once caught in its maw, there is often no escape