An Oxfam comissioned report from Swansea University says failed asylum seekers are being forced to choose between destitution here or persecution at home.
It claims some develop mental health problems and others are pushed into the sex trade.
But the report has been criticised by thinktank Migration Watch UK who say the it is misleading and risks reducing support for genuine asylum seekers.
The report, published by the Centre for Migration Policy Research at Swansea, said when asylum seekers were refused permission to stay in the UK and choose to appeal they opted to go underground rather than apply for financial support and advice.
The UK Border Agency says it provides financial support while claims are dealt with and no asylum seeker need be destitute.
Chris Johnes, head of Oxfam Cymru, said:
“These are people who have made heartbreaking decisions to leave their families and flee their homes.
“They end up living as ghosts on the streets of Britain because of government policy and decision making that strips them of their rights and dignity.
“The current system is designed to make people feel as low as possible and sends out a message that those who are refused asylum are not even worthy of our compassion.”
Criticising the report Sir Andrew Green, Migration Watch UK, said charities needed to start giving their support to current asylum laws;
“Asylum seekers are supported by the taxpayer right through the asylum process, including the appeal stage, and if they fail they can then be further supported while they make arrangements to return home,” he said.
“Failure to remove those who, after a long and expensive process, have no further right to be here can only bring the law itself into disrepute.”