The Conservative Party's former chair says now is not the time to "bury our heads in the sand".