Four permanent secretaries on Tuesday refused to take responsibility for the controversial demolition of houses in Nairobi.