Yes. You need to get used to the idea that you can enjoy life by going relatively slowly. 4 mph is a comfortable walking pace, so someone on the bank can keep up with you. The speed is set to minimize damage to canal banks, and disturbance to the whole host of wildlife that the canal is home to. And lets face it with the speed we all live our lives these days it has to be good just to slow down.
And slower past moored boats!
Yes, if you are on a moored boat, however carefully moored it will move as a boat goes past, and if, as you will no doubt experience, someone goes past too fast the boat will rock alarmingly, so tickover past moored boats.
It is likely that if someone is shouting at you that you are moving too fast its because you are – just get into the rhythm of it and slow down.