Before it became synonymous with Lean Cuisine and French-bread pizza, the frozen-foods monolith was just a little restaurant company that could.

666 Fifth Avenue was just another bulky and unremarkable skyscraper taking up space in midtown Manhattan. Besides Isamu Noguchi’s rippled waterfall sculpture in the lobby, the 41-story building’s only other notable feature was its penthouse. But in 1958, a year after the building’s completion, that penthouse transformed into Top of the Sixes, a restaurant famous for its stellar views. Along with the Rainbow Room, which sat 65 stories above the city’s streets (Windows on the World wouldn’t open at the top of the World Trade Center for nearly another 20 years), Top of the Sixes’ dramatic perch was a riveting novelty.

The restaurant’s menu wasn’t quite as arresting: Its dishes offered satisfying but unremarkable American stalwarts with a touch of European flair. In 1973, according to a menu spotted on eBay, diners could have ordered tenderloin of beef stroganoff, crabmeat à la Newburg, and individual baked Alaska. A circa-1960s edition of the menu offered guests the option of beginning their meal with a $1.25 martini and ending it with a Pink Squirrel nightcap.

The only element of surprise at Top of the Sixes came from its ownership. The restaurant was not the work of some enterprising real estate mogul or hospitality maven. Instead, it was owned by Stouffer’s, the same company that filled America’s freezer aisles with Lean Cuisine and those orange-red boxes of French-bread pizza and spaghetti and meatballs. But as unlikely as it seems, Stouffer’s rich restaurant past actually formed the foundation for its frozen-food empire.

Although Stouffer’s has been owned by Nestlé since 1973, its roots are humble. They reach back to 1914, when Abraham Stouffer and his father, James, launched a creamery and dairy stand in northeastern Ohio. Eight years later, Abraham and his wife, Lena Mahala, started serving fresh buttermilk in the Cleveland Arcade, one of the country’s early indoor shopping malls, and soon elevated their stand into a coffee shop where they sold countless slabs of Lena Mahala’s beloved Dutch apple pie.

In 1924, the couple expanded into a full-fledged restaurant called Stouffer Lunch, where they whipped up sandwiches for 25 cents or less. A few years later, under the guidance of the Stouffers’ entrepreneurial sons Vernon and Gordon, the company began to open restaurants in cities including Detroit, Pittsburgh, and New York, attracting diners with crowd-pleasing dishes like baked pork chops and hot butterscotch sundaes.

In its prime, says Dr. Richard Klein, a professor of urban studies at Cleveland State University, Stouffer’s was a good, medium-priced sit-down restaurant, “the kind of place you could go with your grandmother or take a date to,” he recalls. “It was hopping.”

At Cleveland’s Shaker Square location, the chandelier-laden dining room was perfect for milestone birthday fetes, while its Tack Room, a separate space with a casual pub ambience, attracted a younger crowd. At Top of the Town downtown, the atmosphere was just as convivial, with diners ordering sauerkraut balls and listening to live WJW radio broadcasts and a pianist. Client meetings were sweetened by city views.

Stouffer’s took a swankier turn in the mid-1950s, when it unveiled its collection of “Top of” restaurants. Each one crowned a skyscraper: Top of the Rock was perched in Chicago’s newly unveiled Prudential Building, while Top of the Flame occupied the 26th floor of Detroit’s Michigan Consolidated Gas Building. Top of the Hub, a revolving restaurant in Boston’s then-new Prudential Center, is still in operation today. From a paddlewheel theme in Milwaukee to Mediterranean garden vibes in Atlanta, the distinctive decor of each Top of iteration was just like the menu: pleasing and straightforward.

But Stouffer’s New York outpost was easily its most storied. Bob Lape, the broadcast journalist and food critic, vividly remembers visiting the restaurant in 1965 with a group of Westinghouse Broadcasting executives. They were joined by media mogul Merv Griffin and Arthur Treacher, a British actor and the namesake of the now largely defunct fish-and-chips chain.

“We were already boggled by dining on the 39th floor of anything, let alone the company we kept,” Lape recalls. “It was fun and funny, and I’d never experienced the meticulous level of service paid to our table.”

The venerable restaurateur George Lang was working there as a consultant at the time, and when food accidentally landed on Lape’s suit, “Mr. Lang almost tore it off my back, vowing to have it dry-cleaned and returned the next day,” Lape recalls. “I demurred.”

Top of the Sixes was indeed popular, claiming to have served its 10 millionth meal in 1973, but it “never attained food fame,” Lape points out. “It was, to put it kindly, a tourist magnet, but not a trap because you got what you expected—glitz and sizzle. I always wondered whether the frozen-food ownership lineage impacted public perception.”

It’s worth noting that the Stouffer’s name wasn’t a deterrent to more worldly locals; in his 2007 memoir, The Wolf of Wall Street, the infamous former stockbroker Jordan Belfort described Top of the Sixes as a place “where Masters of the Universe could get blitzed on martinis and exchange war stories.” The opening scene of the 2013 film version shows Belfort and his friends doing just that.

New Yorkers are quick to reminisce about dining experiences of yore, but their tales are usually linked to more-famous establishments, like the Rainbow Room or the Four Seasons—places synonymous with consistent quality, glamour, and theatrics. Top of the Sixes never went much beyond special-occasion dinner territory, so when it shuttered in 1996, David Stout wrote, in a New York Times article entitled “No More Tables for Two at the Top of the Sixes,” that it “closed with no more ceremony than attends the clearing of a table after a meal…. It was not clear…whether Top of the Sixes was the victim of changing tastes, real estate ups and downs, or both.”

666 Fifth’s penthouse subsequently became home to the Grand Havana Room, a luxe cigar lounge filled with humidors, mahogany, and financial titans and celebrities ensconced in leather armchairs. If Top of the Sixes reflected the wholesome Midwestern values that shaped Stouffer’s from day one, its clubby successor suggested that New York had plunged into brasher, more exclusive territory.

By the time Top of the Sixes closed, Stouffer’s hospitality operations had all but ceased to function. In 1960, the company had started to expand its hospitality operations by introducing the now-defunct Stouffer Hotels & Resorts, beginning with the Anacapri Inn in Fort Lauderdale and growing to include known properties like the Mayflower Hotel in Washington, D.C. By 1990, the Stouffer’s name was attached to 40 largely upscale resorts and hotels and 68 restaurants. But two years later, the company began selling off all of its properties to better focus on what most people think of when they think of Stouffer’s: frozen iterations of baked ziti, chicken-and-broccoli pasta bake, and turkey tetrazzini.

The company’s frozen-food arm reaches back to 1946, when Stouffer’s responded to customer demand for easy-to-reheat dishes by unrolling the 227 Club, a separate business devoted to frozen takeout meals. It eventually led to those ubiquitous boxes, which were a supermarket fixture by the early 1960s. Initially, there was some synergy between Stouffer’s restaurants and its frozen entrees: According to a company representative, its macaroni and cheese, spinach soufflé, corn soufflé, stuffed peppers, and Salisbury steak were all influenced by dishes served in the original Stouffer’s restaurant.

If anything unites Stouffer’s restaurants and packaged goods, it’s smart timing: The company’s move into frozen foods dovetailed with the 1950s appliance craze, while the 1981 creation of Lean Cuisine—its line of under-300-calorie entrees—came in the midst of the aerobics boom and cannily targeted the burgeoning subset of health-conscious diners with meals like Mediterranean-style chicken and vegetables over angel hair pasta.

Today, the frozen-food aisle is a testament to the success of Lean Cuisine, with a vast arsenal of brands whose entrees all promise lower fat, sodium, and calories. But regardless of its upgraded packaging and options, the original Stouffer’s is still there, even if most shoppers know nothing of the company’s restaurant origins. Stouffer’s may have long ceased to champion a night out on the town, but it continues to do what it always did: make people’s lives just a little easier with the promise of homey, unfussy cooking.