This is what 'eroneously everyone fancies' about solving the delay.
halts at every stations are specific and passengers manage boarding/getting down the train. And the time of halt is more or less around 2-3-5 minutes at minor stations and 10-15 minutes at major stations. If you add upto all halt time, that too will not measure upto 1-2 hours. And if ,as per your fanciful idea, halts are reduced, and time made up, many passengers may miss the train or not able to get down the train, resulting into ticketless travel.
If...
more... your suggestion is implemented, it can make normal express trains , which have 10-20-30 halts , as 'Aam aadmi ka Duronto'.
And if driver tries to speed up the train to cover the timeloss, the 'safety factor' comes into play, and a dangerous precedence will be eshtablished.