Exclusive IT staff at the world's largest securities transaction clearing house are facing a rough few days after a Reg reader was inadvertently deluged with emails leaking session IDs, transfers, and account details for executives at big-name customers.
The Depository Trust & Clearing Corporation (DTCC) handles the vast bulk of stock and securities transactions for the US, and last year moved more than $1.7 quadrillion in deals through its servers. When a financial asset is traded, the firm handles the paperwork at each end, and works with some of the largest banks and financial institutions in the world.
So a Reg reader, who wishes to remain anonymous, was a little surprised on Monday when he started receiving diagnostic emails from DTCC at a furious rate.
These log files detailed corporate network activity, such as Windows and Unix logon events and mail server warnings. The alerts revealed sensitive session IDs; email addresses for executives at the Bank of America, Barclays, and Deutsche Bank; IP network addresses; user rankings that identified admin accounts; the time stamps of transactions and logins; and more.
Our man was at home with the flu watching Lord of the Rings on loop in an effort to get to sleep (a technique he reported was eventually successful) when his iPad started to ring constantly with new email alerts.
Thinking the issue was a bug, he turned off the fondleslab, but when checking his email account a little later he noticed thousands of emails coming in from DTCC. In total he received more than 20,000 messages, some so long that Gmail had automatically cut them off.
The reader, a mature university student but with a background in ISP networking, initially thought it was a mailbomb attack, but after he checked the emails he was astonished to find they contained machine logs from DTCC servers. He checked out a handful, and found they contained a host of information that would be very useful for miscreants with a talent for social engineering or network penetration.
He first tried emailing DTCC to tell them about the problem, but got an email bounce back. Next he tried an email address of a banker found in one of the files, but the recipient misunderstood the message completely and simply emailed back asking to be removed from the reader's mailing list.
"I got a bit twitchy when I saw lots of different bankers logging in: I'm studying Internet Crime, so I've been doing a lot of researching on, well, internet crimes," he told El Reg.
As the torrent of emails continued, the reader's feelings moved from concern to annoyance, since the spam was masking personal messages that could be important and was eating into his data plan at a frightening rate via his Gmail-linked Android phone. So he contacted El Reg about the problem to see what could be done.
To its credit, DTCC did respond to the issue quite quickly. Its press flack was at her child's birthday party, but alerted the company to the issue and the email flood has now ended.
"These messages were inadvertently sent out as a result of human error. We have confirmed that this was an isolated incident and that no other individuals received this or similar information," the spokeswoman said in a statement.
"We have identified the situation quickly and taken steps to protect our client's information. We are also working with the individual who received these messages to resolve any remaining issues."
The spokeswoman couldn’t confirm the exact problem, but from the content of the emails the fault appears to lie in the configuration of an IBM QRadar Security Intelligence Platform (ironically). QRadar can send a snapshot of network activity to an admin's email, and it appears that the reader's address was put in by accident.
If so, this raises some intriguing questions – not least why this kind of data was being sent to a Google webmail address in the first place and why no one noticed the misdirected network traffic capture. DTCC is conducting an investigation into what went wrong and how to prevent it happening in future, but the reader has some ideas of his own.
"I know one thing, from when I worked at an ISP, if this leak happened there we'd have done a complete strip down: new user IDs, passwords (for systems and software), new host names, new IP ranges, new everything! Anything and everything that went outside the core IT staff would be pulled and set up as new, especially if it went to some random person's email," he suggested.