While it's hard to understand how thousands of investors around the world could have been duped by fund manager Bernard Madoff, it is even more difficult to figure how it happened to people down the hall.

While lawmakers blame regulators for failing to pick up on the fraud, it's clear from court documents that dozens of opportunities were missed to catch it, even by those who were closest to the action. More than one dozen individuals or entities listed in a 162-page court document filed Wednesday had an address listed at Madoff's firm, Bernard Madoff Investment Securities.

Full Text: The Madoff Victim List

Interactive: Mapping Madoff's Victims

For some of these individuals, the address was purely administrative. But not all. One, Noel Levine, had an office on the very floor the fraud was alleged to have been carried out. Levine's real estate investment company, Troon Management, has an address on the 17th floor of 885 Third Avenue in Manhattan. Bernard Madoff Investment Securities occupied the 17th through 19th floors of that building, and the investment advisory business (alleged Ponzi operation) Madoff himself ran was run on the 17th floor. Levine could not be reached for comment.

Levine and his wife, New York socialite Harriette Levine, are listed a half-dozen times as Madoff account holders, either through her interior design firm, their family foundation or Troon Management.

Levine is not as universally famous as some other big victims of the alleged swindle, but he is believed to have lost a substantial fortune. He and his wife travel in New York society and live in a posh Park Avenue apartment building that is also home to Oscar de la Renta. He used to run a company that made bathroom accessories before registering Troon Management in the early 1990s. He is also on the board of a 3-year-old-start up bank in Greenwich, Conn.

Swindler's List (as it's called by Los Angeles-based Jewish Journal newspaper), is loaded with more than 13,000 names, including celebrities Kevin Bacon and John Malkovich, wealthy businessmen like Carl Shapiro and Larry Silverstein, and philanthropists like the late Avery Fisher and others.

In a hearing earlier this week, members of the Senate Banking Committee lambasted representatives from the Securities and Exchange Commission for failing to detect the fraud even after repeated warnings from a Boston-area investigator, Harry Markopolos, who first raised alarms about Madoff with the SEC in 2000.

The Madoff affair has become a lightning rod for those who want to dramatically overhaul the way financial companies are regulated. Madoff's firm was a registered broker dealer and one of the biggest over-the-counter market makers on Wall Street, yet Madoff's investment advisory business wasn't registered with the SEC until 2006 and was never examined as an adviser.

The mystery is how Madoff, who prosecutors say confessed to the Ponzi in December, could have pulled it off undetected for so long. Markopolos claims Madoff's scheme was a widely held suspicion on Wall Street.

Many of Madoff's own family members worked at the firm, including his two sons, his brother and his niece, all of whom are listed as having had accounts there. Madoff, prosecutors say, told police in December he acted alone.

Another firm, Cohmad Securities, is listed as having an office on the 18th floor of the building, the same space occupied by the Madoff broker-dealer operations. Cohmad is a combination of the last names of Maurice Cohn and Bernie Madoff. Cohn, whose firm employed the son-in-law of Carl Shapiro, one of Madoff's oldest friends who lost hundreds of millions in the scheme, is also said to have been an early Madoff investor. The company declined to comment for this story.

Hines Interests Limited, a real estate investment, development and management company, has offices in the building, though not on the corresponding floor. It is listed as a Madoff account holder, as is its eastern regional executive vice president, Kenneth Hubbard.

Many of Madoff's earliest investors are believed to have substantial ties either as family or as friends from the old neighborhood in Queens, where Madoff and his wife grew up. A heavy concentration of the U.S. investors come from predominantly Jewish social circles in the New York and Palm Beach, Fla., areas.

Levine's sisters appear to have been victims of the alleged fraud as well. In a death notice last May, Levine is listed as a brother of Myra Levine Davis of Palm Beach. She appears on the list of Madoff clients that was filed this week. Another sister, Norma Shapiro, is also named in the death notice as well as on the list of Madoff clients.

The one connection that has yet to be made is whether Norma Shapiro, also of Palm Beach, is related to Carl Shapiro. A Shapiro spokesman had not confirmed or disputed it by deadline.