You are indeed not supposed to drag. Clicking above/below the thumb should move by a page. Holding it down should repeat moving pages. That is what scollbars did on X11 when this was implemented. Dragging is simply interpreted as holding down in this context.
Of course, it should not freeze either. I does not on my current version. 10.0.x is reaching end-of-life, moving to 10.2.x. If it bothers you, please try the latest development release.