Thursday's failure to win support for his "Plan B" tax outline hitting just 400,000 families earning $1 million or more isn't likely to cost House Speaker John Boehner his leadership position, but some insiders and foes are already working to push three conservative members to challenge the Ohio lawmaker for the top job.

Outside conservative groups and some advisors to House members want either Majority Leader Eric Cantor, House Majority Whip Kevin McCarthy or Jim Jordan, the chairman of the House Republican Study Committee, to run for the job.

But House leadership aides said that a challenge now is unlikely, especially as the speaker continues to look for a deal to solve the oncoming fiscal cliff crash. On reason: With so many factions in the House GOP, being speaker isn't an easy job and Boehner has done the best he can to push the Republican agenda, they add. "Being speaker of this Congress is no easy task to be sure," said a leadership aide.

Sign Up for the Paul Bedard newsletter!

Please enter your email address below to begin receiving the Paul Bedard newsletter.

You must enter a valid email address in the field above!

Thank you for signing up for the Paul Bedard newsletter! You should receive your first newsletter very soon.

We're sorry, there was an error processing your newsletter signup. Please click here to visit our Newsletter Signup Center to register for this newsletter.

What's more Cantor and McCarthy are on Boehner's leadership team and worked hard to push Plan B, pulled from the House floor Thursday night when it appeared that it would not win enough GOP support, especially from anti-tax conservative critics. And Boehner aides--who hear unfounded mutiny rumors almost daily--said there is no serious challenge to the speaker right now.

Only outside groups and liberal media outlets are publicly talking about their wishes that somebody would challenge Boehner. One of the most outspoken is American Majority Action, and their spokesman Ron Meyer. "We like Jordan for speaker," he told Secrets. His group has even been orchestrating a Twitter campaign to remove Boehner, #FireBoehner.

A longtime GOP leadership advisor said "we're very unhappy with Boehner" and his push for taxes on those earning $1 million or more. But, he added, a leadership coup is "a bad bet."