Around the time the Port Authority gained control of the building in 2006, Condé Nast executives began to discuss the future of their magazine empire. Condé’s lease at 4 Times Square was set to expire in 2019, and its broker, Mary Ann Tighe of CB Richard Ellis, worked with [Chairman Si] Newhouse to develop options...It's hard to imagine that Condé Nast would have left Manhattan--after all, don't some of their biggest magazines rely on close relationships with the fashion industry?
For its role in remaking Times Square, Condé Nast had been rewarded with tax breaks, giving it an annual rent of about $40 per square foot. In looking for a similar deal, Newhouse’s team even pushed into outer boroughs, touring Atlantic Yards in Brooklyn, addresses in Long Island City, and locations along the Jersey City waterfront. One morning in the fall of 2009, [Port Authority Executive Director Christopher] Ward got the call he’d been waiting for: Si Newhouse told him he wanted to tour ground zero. Months later, outlines of a deal emerged.
Note that broker Tighe has a longtime relationship with Forest City Ratner and bought a piece of the Nets (though I'm not sure she still has it). So either she was just doing Bruce Ratner a courtesy or someone thought the AY office space--with extra subsidies?--could have been spectacularly affordable.