Solved my problem. It was due to some negative width or height.
For example if input box was like (x0, y0, x1, y1). Here if y1-y0 or x1-x0 was negative then LayoutLM will throw error.
On GPU error will be, CUDA error: device-side assert triggered.
On CPU error wiil be, Index error: index out of range in self