Home Secretary David Blunkett denied police in Bradford were "institutionally racist" and said the community could not blame outsiders for the riots.
"There is no doubt whatsoever that the initial confrontation was caused by those who intervened from outside," he told Sky News's Sunday with Adam Boulton programme.
Their actions in Bradford were "the catalyst for what happened later", he said.
"But it is also clear that what took place in the Manningham district was sheer mindless violence and therefore people acting in a totally anti-social and thuggish fashion, rather than some inherent cause through disadvantage that we need to address."
However the Home Secretary said the message to those from outside Bradford who had got involved was: "Please keep out of other people's back yards.
"With the best will in the world for those who are anti-fascist - this is not a game we are in, this is serious politics."
Mr Blunkett said it was "ridiculous" to blame police attitudes to the Asian community for the disturbances.
"I don't think that last night's riots and violence and destruction of a community that was putting itself back together has anything to do with institutional racism whatsoever.
"It is about people who are prepared to resort to violence and self-destruction in a way that can take us absolutely nowhere."
The rioters had to take responsibility for their actions and should stop looking for "scapegoats".
But he added: "If there are lessons to be learned, including in terms of the police, then we must learn them."
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules hereComments are closed on this article