This doesn't need to be the case. Children as young as third grade can start learning how to write code by playing games. CargoBot is a free iPad app that is a lot of fun. In playing the game, students work with a series of commands that program a robot arm to move crates. Scratch a website that uses a programming language that makes it easy to create your own interactive stories, animations, games, music, and art.
Third graders are old enough to start using these tools to develop their skills. I have included two videos about programming. The first, I couldn't get to embed, but it stresses the importance of programming and can be found here. The other, features a ten year old boy who has published several apps on the iTunes store.