Human rights and refugee advocates have expressed scepticism over the prime minister of Sri Lanka's claim that asylum seekers who fled his country in a bid to reach Australia would not be punished if they returned home, citing recent allegations of torture in the South Asian nation.

Prime Minister Ranil Wickremesinghe, who is in Canberra on an official visit, urged asylum seekers from his country seeking residence in Australia to "come back".

"All is forgiven," he said. "They are welcome to return to Sri Lanka and we won't prosecute them.

"We will help them. But remember, they broke the law in ... attempting to come to Australia."

However not everyone is convinced by Mr Wickremesinghe's assurances, with advocates citing concern over ongoing human rights abuses.

Aran Mylvaganam, of the Tamil Refugee Council, works closely with asylum seekers who have been granted temporary settlement in Melbourne as they wait for their refugee claims to be processed.

"It is definitely not safe for Tamils to go back," he said.

"Just three weeks ago, the United Nations Special Rapporteur on Torture claimed that torture is still being used in Sri Lanka."

One of his clients is a Sri Lankan man who says he worked with the Tamil Tigers - considered a terrorist organisation by the United States and the European Union - in an unarmed, civilian capacity for 10 years, up until the end of the war in 2009.

The man, who spoke to SBS News on the condition of anonymity, said he was twice abducted and tortured by Sri Lankan intelligence agents.

He is living in temporary accommodation in Melbourne as he waits for his political asylum claim to be processed.

"I think the Sri Lankan Prime Minister's speech is absurd," the man said. "Even militants who were rehabilitated [and went on to] live normal lives were arrested."

Mr Mylvaganam said his client would be more vulnerable to government retribution because he has given evidence to the United Nations about human rights violations he saw in the final days of the war.

He said he fears he would be arrested again if he returned to Sri Lanka.

"I felt unsafe ... there was no guarantee for my safety."

"These asylum seekers who have given witness statements to the United Nations, who have spoken to various organisations in Australia about what they experienced in Sri Lanka, are at risk if they return to Sri Lanka," he said.

Hundreds of Sri Lankans, mostly Tamils, fled to Australia in the wake of a decades-long civil war, which killed tens of thousands of civilians.

In 2009, the Sri Lankan army declared victory over the Tamil separatist. The UN has accused both sides of gross human rights violations.

Immigration Department figures from December last year show there were 86 Sri Lankans being held in detention centres in Australia, but the government would not say how many are on Manus Island and Nauru.

There are currently 1,241 people subject to regional processing on the two islands. A spokeswoman for the Immigration Minister said only that the largest groups were Iranian, Sri Lankan and Afghani.

The Australian director of Human Rights Watch, Elaine Pearson, told SBS asylum seekers have good reason to be sceptical of the prime minister's promise.

"You have to bear in mind the past track record of the previous government. The fact that people know that failed asylum seekers were returned to Sri Lanka, they were picked up, they were detained by security forces, they were tortured, in some cases sexually," Ms Pearson said.

"People are going to need a lot of convincing. They want to see the perpetrators of those abuses brought to justice.

"I think to convince Sri Lankans to return to Sri Lanka, we need to see more than just sweet talk from the Sri Lankan prime minister."

(SBS News)