Rep. Charlie Rangel, D-N.Y., on Tuesday will face the toughest challenge of his career over the seat he has held for the past 41 years.

The results of the New York Democratic primary could spell the end of Rangel’s storied and controversial career in the House of Representatives. While Rangel, 82, campaigned hard in the remaining weeks of the race, he fought questions of whether past ethics issues and a three-month absence due to a back injury impede his ability to serve in Congress.

State Sen. Adriano Espaillat, 57, poses the biggest threat to Rangel’s bid for a 22nd term with a campaign mantra of “it is time for a change.”

If he wins the primary, Espaillat would likely have a shot at becoming the first Dominican-American member of Congress in history. His campaign also notes this potential milestone in the newly redrawn Harlem district that now holds a majority of Hispanic voters.

But it’s possible the vote will ultimately split several ways as four candidates vie to replace Rangel, potentially handing Rangel a win.

The second most competitive challenger is Clyde Williams, 50, a former national political director for the Democratic National Committee. Williams also received boosts after The New York Times and The New York Daily News endorsed him. The incumbent later called the Times endorsement “ridiculous.”

Rangel also tried to cast his campaign as a way to enact President Obama’s agenda — but the White House declined to link itself to the race.

Asked last month if Obama would endorse Rangel, White House Press Secretary Jay Carney told reporters he would “have to get back to you on that.” He didn’t.

It’s not the first time Obama has declined to back Rangel. Shortly before the 2010 censure vote against Rangel over ethics allegations, Obama said Rangel was “somebody who’s at the end of his career.” He went on to say, “I’m sure that what he wants is to be able to end his career with dignity.”

Rangel made history in 2007 when he became the first African-American chairman of the tax-writing House Ways and Means Committee. But he relinquished the gavel three years later after a series of ethics allegations put him under pressure to resign.

READ MORE: FOX NEWS