BRAUNAU, Austria (AP) -- Living space in Braunau is scarce, but an imposing Renaissance-era building stands empty in this post-card pretty Austrian town because of the sinister shadow cast by a former tenant: Adolf Hitler.

With its thick walls, huge arched doorway and deep-set windows, the 500-year-old house near the town square would normally be prime property. Because Hitler was born here, it has become a huge headache for town fathers forced into deciding what to do with a landmark so intimately linked to evil.

The building was most recently used as a workshop for the mentally handicapped, which some saw as atonement for the murders of tens of thousands of disabled people by the Nazi regime. But that tenant moved out last year for more modern quarters.

The departure reignited debate on what to do with the house that burst from the town hall chamber into the public domain last week after the mayor declared that he preferred creating apartments over turning the building into an anti-Nazi memorial.

"We are already stigmatized," Johannes Waidbacher told the Austrian daily Der Standard. "We, as the town of Braunau, are not ready to assume responsibility for the outbreak of World War II."

That sparked a storm of criticism, with Waidbacher accused of trying to bury memories of the Nazi past.

The comments were particularly ill-received due to the fact that Braunau's town council only withdrew honorary citizenship from Hitler last year, 78 years after the Nazi dictator was given the accolade -- as did nearly a dozen other towns and cities after checking their archives.

Stung by the criticism, Waidbacher has since stepped back, saying he can conceive of "all possible uses" for the building.

On Thursday, Waidbacher expressed surprise at the vehement reaction his comments caused, saying he did not mean to make light of the significance of the house. "Our town has definitely done its homework as far as its past is concerned," he told The Associated Press.

Nonetheless, concerns about the building's fate continue to reverberate on the ancient cobble-stoned streets of this town of 16,600.

One major fear: The house could fill up with Hitler worshippers if converted into living space.

"These are certainly people we don't want here," said town council member Harry Buchmayr, noting that most visitors are not normal tourists but neo-Nazis stopping to pay homage to Hitler, even though he spent only the first few months of his life in the building.

And it's unclear who else might want to take up residence in the house.

"I wouldn't want to live there," said 19-year-old Susanne Duerr, as she paused from pushing her baby carriage to gaze at the yellow stucco building. "I think I would have a bad conscience."

Other townsfolk old enough to remember the Fuehrer echo that sentiment. Georg Hoedl, 88, recalls Hitler as the man who dragged depression-era Austria and Germany out of the kind of abject poverty that forced him to go begging. But he also is aware of the evil Hitler spawned.

"There should be something else inside, something cultural. But apartments -- I'm not for that," he said

Wife Erika, 73, says that bearing the burden of the house's legacy "wouldn't be pleasant for the tenants -- once they moved in they would be asked about this all the time."

Austria's Interior Ministry has rented the house since 1972 from the owner, a woman in her 60s who refuses to be identified publicly. The ministry has been careful to sublet only to tenants with no history of admiring Hitler. Asked about the debate, Interior Ministry spokeswoman Sonja Jell said the ministry remained "particularly sensitive" about the future uses of the building considering its legacy.

The owner refused a request by Braunau officials to let the city mount a sign on the house warning of the evils of the Nazi past. But an inscription on a chunk of granite on public property near the building calls out to passersby: "Never again fascism, never again war."

The building still has the initials MB in the iron grillwork above the massive wooden doorway. It stands for Martin Bormann, Hitler's private secretary, who bought the house shortly before World War II with thoughts of turning it into a shrine to the dictator.

The house is one of the few remaining structures directly linked to Hitler.

A house in nearby Leonding where he spent some teenage years is now used to store coffins for the town cemetery. At that graveyard, the tombstone marking the grave of Adolf Hitler's parents, a place of pilgrimage for neo-Nazis, was removed last year at the request of a descendant. A school Hitler attended in Fischlham, also near Braunau, displays a plaque condemning his crimes against humanity.

The underground bunker in Berlin where Hitler committed suicide on April 30, 1945, was demolished after the war. It was left vacant until the East German government built an apartment complex around the site in the late 1980s. The apartments, which are still occupied, overlook the German capital's monument to victims of the Holocaust.

Ultimately, it's the owner who will decide the Branau building's fate. She's known to be opposed to turning it into a Holocaust memorial, meaning there's still a chance it could be converted into apartments.

That's a nightmare scenario for Buchmayr, a member of Austria's Socialist Party that has done much over the past four decades to sensitize citizens to their country's Nazi past.

"You can't simply wish it away," Buchmayr said of the house. "Unfortunately we have it here.

"Hitler was born here."