Forecasters were wrong about an immediate recession but right that we would be worse off outside the EU