Why was Hashem furious with Aharon?

Summary: After all, the narrative in Ki Tisa, especially as explained by the midrash, would seem to entirely excuse Aharon. Why, then, was Hashem furious with him?

Post: In parshat Ekev, perek 9, we encounter the following pasuk and Rashi:

20. And with Aaron, the Lord was very furious, to destroy him; so I prayed also for Aaron at that time.		כ. וּבְאַהֲרֹן הִתְאַנַּף ה מְאֹד לְהַשְׁמִידוֹ וָאֶתְפַּלֵּל גַּם בְּעַד אַהֲרֹן בָּעֵת הַהִוא:
		ובאהרן התאנף ה': לפי ששמע לכם:

The surrounding context is the chet ha'egel. The rather strong implication with this is that Aharon did something wrong, even if it was just listening to them.

The Taz points out a seeming contradiction in this. After citing the pasuk and Rashi, he writes:

"This excludes the idea that he did not sin in the egel. And there is to be confounded, for behold it was necessary for him to build the altar before him {ויבן מזבח לפניו} because he understood {ויבן} from the one slaughtered {מזבוח} before him, for they killed Chur, for not listening to them, as Rashi explains in parashat Ki Tisa. And if so, his intent was leshem Shamayim. And at the least, it should be counted like one who performed an aveira lishma, like that incident with Yael with Sisera, as is stated in Nazir (23b). And there is not in my hand to answer except in this manner: that Aharon should have prayed that Hashem turn their hearts, and he did not do so. Rather, he listened to them, to do it, by force of the fear because they had killed Chur. And with this is resolved why there was destruction of sons {as mentioned in the next Rashi}, for it is written, 'and sons shall not be put to death for the fathers'; and that which we say in the gemara, that when they seize upon the actions of their fathers in their hands, is not relevant here regarding the sons of Aharon. But for me it works out well, in that they did seize upon the act of Aharon, for they did not pray as well, just like Aharon."

The pasuk and Rashi in Ki Tisa are in Shemot perek 32:

5. When Aaron saw [this], he built an altar in front of it, and Aaron proclaimed and said: "Tomorrow shall be a festival to the Lord."		ה. וַיַּרְא אַהֲרֹן וַיִּבֶן מִזְבֵּחַ לְפָנָיו וַיִּקְרָא אַהֲרֹן וַיֹּאמַר חַג לַה' מָחָר:
		וירא אהרן: שהיה בו רוח חיים, שנאמר (תהלים קו כ) בתבנית שור אוכל עשב, וראה שהצליח מעשה שטן, ולא היה לו פה לדחותם לגמרי:
		ויבן מזבח: לדחותם:
		ויאמר חג לה' מחר: ולא היום, שמא יבא משה קודם שיעבדוהו, זהו פשוטו. ומדרשו בויקרא רבה (ה /י/, ג) דברים הרבה ראה אהרן, ראה חור בן אחותו, שהיה מוכיחם והרגוהו, וזהו ויבן מזבח לפניו - ויבן מזבוח לפניו. ועוד ראה ואמר מוטב שיתלה בי הסירחון ולא בהם, ועוד ראה ואמר אם הם בונים אותו המזבח, זה מביא צרור וזה מביא אבן, ונמצאת מלאכתן נעשית בבת אחת, מתוך שאני בונה אותו ומתעצל במלאכתי, בין כך ובין כך משה בא:
		חג לה': בלבו היה לשמים, בטוח היה שיבא משה, ויעבדו את המקום:

Perhaps, since Rashi gives both peshat and derash there, they are meant to stand apart and not work together. And if so, the Rashi here could be operating on the level of the peshat. Even so, the midrash itself would need to somehow answer to the challenge posed by this explicit pasuk.

Besides what the Taz offered, above, perhaps one could say that despite all these cheshbonos of Aharon, Hashem was still angry. Who says that these cheshbonos were right? He said the sirchon should be with me and not with them, and maybe this worked too well, and indeed this was the case. He sinned on behalf of klal Yisrael, but it was still a sin. Perhaps he could have confronted them and been more successful than Chur; or he should have died trying.