It was the creation of the Isthmus of Panama that closed off ocean currents between the Atlantic and Pacific which was the start of this "Ice Age", prior to that there wasn't any permanent ice caps (not including Snow Ball Earth ~600 million years ago).
Without getting into the snowball earth debate, there are several glaciations that precede the current (Quaternary) one:
Those times not marked with a blue bar are when the earth didn't have ice caps at all.