Why would the shivtei Kah try to kill their brother?

Summary: After all, isn't this a violation of lo tirtzach? And didn't the avos and, by extension, the shevatim, keep all 613 mitzvos?

Post: See this post by Rabbi Yaakov Haber who addresses this question, in another form. If they were great men, why did they do this? Based on the Seforno, because they thought that Yosef would be Yaakov's choice and that they would become the chaff.

This is also addressed in midrash, if we want to adopt a more "maximalist" perspective on the avos keeping the Torah. For example, this pasuk:

17. And the man said, "They have traveled away from here, for I overheard them say, 'Let us go to Dothan.' " So Joseph went after his brothers, and he found them in Dothan.

יז. וַיֹּאמֶר הָאִישׁ נָסְעוּ מִזֶּה כִּי שָׁמַעְתִּי אֹמְרִים נֵלְכָה דֹּתָיְנָה וַיֵּלֶךְ יוֹסֵף אַחַר אֶחָיו וַיִּמְצָאֵם בְּדֹתָן:

where Rashi offers the following explanation:

‘Let us go to Dothan.’: Heb. נֵלְכָה דֹתָינָה, to seek regarding you legal pretexts (נִכְלֵי דָתוֹת), by which they could put you to death. According to its simple meaning, however, it is a place-name, and a Biblical verse never loses its simple sense.

נלכה דתינה: לבקש לך נכלי דתות שימיתוך בהם. ולפי פשוטו שם מקום הוא, ואין מקרא יוצא מדי פשוטו:

Thus, perhaps their actions were done in an halachic manner.

If I might suggest a justification of the brothers' actions, I would suggest as follows. Obviously, they knew of the pesukim in Devarim 18 which state:

20. But the prophet who intentionally speaks a word in My name, which I did not command him to speak, or who speaks in the name of other gods, that prophet shall die.		כ. אַךְ הַנָּבִיא אֲשֶׁר יָזִיד לְדַבֵּר דָּבָר בִּשְׁמִי אֵת אֲשֶׁר לֹא צִוִּיתִיו לְדַבֵּר וַאֲשֶׁר יְדַבֵּר בְּשֵׁם אֱ־לֹהִים אֲחֵרִים וּמֵת הַנָּבִיא הַהוּא:
21. Now if you say to yourself, "How will we know the word that the Lord did not speak?"		כא. וְכִי תֹאמַר בִּלְבָבֶךָ אֵיכָה נֵדַע אֶת הַדָּבָר אֲשֶׁר לֹא דִבְּרוֹ יְ־הֹוָ־ה:
22. If the prophet speaks in the name of the Lord, and the thing does not occur and does not come about, that is the thing the Lord did not speak. The prophet has spoken it wantonly; you shall not be afraid of him.		כב. אֲשֶׁר יְדַבֵּר הַנָּבִיא בְּשֵׁם יְ־הֹוָ־ה וְלֹא יִהְיֶה הַדָּבָר וְלֹא יָבוֹא הוּא הַדָּבָר אֲשֶׁר לֹא דִבְּרוֹ יְ־הֹוָ־ה בְּזָדוֹן דִּבְּרוֹ הַנָּבִיא לֹא תָגוּר מִמֶּנּוּ:

Elsewhere, a navi is lumped together with a cholem chalomot, of prophetic dreams, in this regard. Now, they knew that Yosef was offering prophetic dreams, and they believed these prophecies to be false. Thus, looking to parashat Vayeishev:

5. And Joseph dreamed a dream and told his brothers, and they continued to hate him.		ה. וַיַּחֲלֹם יוֹסֵף חֲלוֹם וַיַּגֵּד לְאֶחָיו וַיּוֹסִפוּ עוֹד שְׂנֹא אֹתוֹ:
6. And he said to them, "Listen now to this dream, which I have dreamed:		ו. וַיֹּאמֶר אֲלֵיהֶם שִׁמְעוּ נָא הַחֲלוֹם הַזֶּה אֲשֶׁר חָלָמְתִּי:
7. Behold, we were binding sheaves in the midst of the field, and behold, my sheaf arose and also stood upright, and behold, your sheaves encircled [it] and prostrated themselves to my sheaf."		ז. וְהִנֵּה אֲנַחְנוּ מְאַלְּמִים אֲלֻמִּים בְּתוֹךְ הַשָּׂדֶה וְהִנֵּה קָמָה אֲלֻמָּתִי וְגַם נִצָּבָה וְהִנֵּה תְסֻבֶּינָה אֲלֻמֹּתֵיכֶם וַתִּשְׁתַּחֲוֶיןָ לַאֲלֻמָּתִי:

The brothers' reaction is one of disbelief:

8. So his brothers said to him, "Will you reign over us, or will you govern us?" And they continued further to hate him on account of his dreams and on account of his words.

ח. וַיֹּאמְרוּ לוֹ אֶחָיו הֲמָלֹךְ תִּמְלֹךְ עָלֵינוּ אִם מָשׁוֹל תִּמְשֹׁל בָּנוּ וַיּוֹסִפוּ עוֹד שְׂנֹא אֹתוֹ עַל חֲלֹמֹתָיו וְעַל דְּבָרָיו:

They didn't think he would ever rule over them. Thus, he was uttering false prophetic dreams. Thus, regarding his dreams and his words relating the false dreams. In terms of the second dream:

9. And he again dreamed another dream, and he related it to his brothers, and he said, "Behold, I have dreamed another dream, and behold, the sun, the moon, and eleven stars were prostrating themselves to me."		ט. וַיַּחֲלֹם עוֹד חֲלוֹם אַחֵר וַיְסַפֵּר אֹתוֹ לְאֶחָיו וַיֹּאמֶר הִנֵּה חָלַמְתִּי חֲלוֹם עוֹד וְהִנֵּה הַשֶּׁמֶשׁ וְהַיָּרֵחַ וְאַחַד עָשָׂר כּוֹכָבִים מִשְׁתַּחֲוִים לִי:
10. And he told [it] to his father and to his brothers, and his father rebuked him and said to him, "What is this dream that you have dreamed? Will we come I, your mother, and your brothers to prostrate ourselves to you to the ground?"		י. וַיְסַפֵּר אֶל אָבִיו וְאֶל אֶחָיו וַיִּגְעַר בּוֹ אָבִיו וַיֹּאמֶר לוֹ מָה הַחֲלוֹם הַזֶּה אֲשֶׁר חָלָמְתָּ הֲבוֹא נָבוֹא אֲנִי וְאִמְּךָ וְאַחֶיךָ לְהִשְׁתַּחֲוֹת לְךָ אָרְצָה:

This prophetic dream couldn't come true, for Yosef's mother, Rachel, had already passed away! Indeed, the best proof is when they decide to kill Yosef:

19. So they said one to the other, "Behold, that dreamer is coming.		יט. וַיֹּאמְרוּ אִישׁ אֶל אָחִיו הִנֵּה בַּעַל הַחֲלֹמוֹת הַלָּזֶה בָּא:
20. So now, let us kill him, and we will cast him into one of the pits, and we will say, 'A wild beast devoured him,' and we will see what will become of his dreams."		כ. וְעַתָּה לְכוּ וְנַהַרְגֵהוּ וְנַשְׁלִכֵהוּ בְּאַחַד הַבֹּרוֹת וְאָמַרְנוּ חַיָּה רָעָה אֲכָלָתְהוּ וְנִרְאֶה מַה יִּהְיוּ חֲלֹמֹתָיו:

They know that they are going to kill him. Therefore, neither of his dreams could come to fruition.  Therefore, he is a navi sheker who is deserving of death.

(This in contrast to Rashi / midrash that וְנִרְאֶה מַה יִּהְיוּ חֲלֹמֹתָיו is a statement by Hashem, rather than by the brothers.)

Their decision to kill Yosef is thus not an aveira, chas veShalom. It is a mitzvah, and is carrying out the Torah's instructions!