Yes, it is very dangerous. You should put a great PFD (life jacket) designed for a four-year-old. If your kayak were to get swamped or pulled under something your child would go with it if they were tied to the kayak. It would be better to have your child bobbing down the river and catch him then to have them pulled under something with the kayak. I have seen people do this also with their pet also, another dangerous practice.
Additionally, make sure you wear a PFD too - it is an excellent choice for a child, and puts you in a position to save them if you should be in a threatening situation.