Citing from my translation of the Rif, on Megillah 24b:
Mishna:That it has to say this, I would note, means that it is reacting to people actually doing this. Making it round is merely a mistake, while the other actions are heresy, because they are based on literal interpretations of Scripture. But it would seem clear that some people back then at least had the idea. Finding such tefillin shel rosh would not even be so surprising then.כהן שיש בידו מומין לא ישא את כפיוA kohen who has on his hand blemishes should not raise his hands {for birchat kohanim}.
רבי יהודה אומר אף מי שהיו ידיו צבועות איסטיס וקוצה לא ישא את כפיו מפני שהעם מסתכלין בו
האומר איני עובר לפני התיבה בצבועים אף בלבנים לא יעבור בסנדל איני עובר אף יחף לא יעבור העושה תפלתו עגולה סכנה ואין בה מצוה נתנה על מצחו ועל פס ידו הרי זה דרך מינות
Rabbi Yehuda says: Even one whose hands are dyed with wood or madder roots should not lift up his hands, because the people stare at them.
If one says: I will not go before the lectern {as a shliach tzibbur} with dyed {clothing}, even with white clothing he may not go up. {I will not go} with a sandal, he may not go up even barefoot. If one makes his tefillin {shel rosh} round, it is a danger and there is no mitzvah. If he places it {the tefillin} on his forehead or on the palm of his hand, this is the way of heresy.
But, he asks in the article, what then to make of these pictures of tefillin shel yad which are circular?
He notes that, in truth, this custom of oval tefillin shel yad is already brought down in the Mordechai as a pushed off minhag, a minhag dachui. His words are based on Menachot 35a:
They taught: Square tefillin are a halacha leMoshe miSinai.And upon this writes the Mordechai:
Rav Pappa said: In their sewing and in their diagonal. {But in a footnote Sperber brings down the opinion of some Rishonim that the primary requirement is in their sewing.}
Let us say this {Mishna} supports him: If one makes his tefillin {shel rosh} round, it is a danger and there is no mitzvah.
[Rashi explains that when he passes by a doorway and hits his head upon the lintel, the tefillin should not be driven into his head.]
Rav Pappa said: The Mishna refers to where it is like an amgoza.
[where the bottom is also rounded and thus sharp, but if it is flat, there is nothing to worry about. Note that though the tefillin shel yad in question are oval, they do indeed have a flat base.]
They taught: Square tefillin are a halacha leMoshe miSinai. This would seem to refer to the actual body of the houses {batim} {not the base}, whether of the shel yad or of the shel rosh, to the exclusion of those who make the shel yad in a circular form {that is, the oval form mentioned above}, with only the base underneath being square. And just the opposite! The brayta speaks about the actual body of the batim, which are the essence of tefillin. And Rava {in our girsa of gemara it is Rav Pappa} comes to add about the sewing and the diagonal, and not to subtract.And his words are brought down in the poskim, including the Magen Avraham, the Olat haTamid, and the Mishna Berurah.
Thus we see from his words that there were those who had the custom of making the batim of the shel yad oval and the base square. And it is against this custom that the Mordechai is complaining. Despite this, it appears that in certain places the long-standing practice was to make the tefillin shel yad oval.
Here is the article -- the above was a very rough translation and summary. Click on the scans to see it bigger. And get the set of sefarim. A really good read, if you are into this kind of stuff.