Both of these are used to stop runaway trains. However, Catch sidings are much more effective than sand humps and can be used for higher speeds.

In a sand hump, the rails are covered by sand. When a train runs into the sand, the sand absorbs some energy from the train, slowing it down. The train derails at this point, but since the sand has already slowed down the train, it is a more controlled derailment. Thus the sand hump stops the train with minimal damage to it.
...
more...

However, there are limits to the sand hump's usage. As the speed of the incoming train increases, a sand hump's ability to slow it down decreases. Also, remember that a sand hump derails the train. So, if the sand doesn't slow down the train enough, the derailment will not be 'controlled', and there will be massive damage.

For this reason, sand humps are typically used just outside stations. These are intended to stop trains parked in the yard from rolling out onto the main tracks. The speeds involved are quite low, probably 5-15 kmph, so they are cheap and effective.

A catch siding is usually a track constructed on a steep upward slope. As a runaway train starts climbing up the gradient, the weight of train starts pulling it back down the slope, eventually bringing it to a stop. There is no derailment in this case and the train is brought to a complete stop by its own weight.

Catch sidings are almost always more effective than sand humps and can even be used for moderately high speeds. Since there is no derailment involved, they are also safer.

However, a catch siding typically requires a lot of space and a large upward gradient. So, it is not an effective solution on flat terrain and is mainly useful in hilly areas.