We propose to improve BERT model calibration via on-manifold smoothing and off-manifold smoothing.