Why are villians in most espionage novels are of the rank: colonel?

Colonels are the right level on the hierarchy. They have enough power for their villainous activities to actually make a difference and they are told enough about the plans to be of use to the other side. But they also do not have too much power. If they had as much power as someone at the top, the 'hero' of the novel would have no chance against them which wouldn't make much of a plot.