If an overwhelming majority of the people in an ideological group say that some person is not in that group, odds are that the overwhelming majority is correct. For instance, an overwhelming majority of Christians, in spite of their inter-denominational disputes with each other, would agree that members of other faith groups are not Christians. Even if some member of another faith, say a Hindu, were to say that he was a Christian (without changing any of his Hindu beliefs) he would not be one on his own say-so, but would instead be regarded as someone who is either lying or deluded.
There are cases that are not so clear cut, but an ideological group can be defined by the shared ideology. If one does not fit in, within the natural flexible boundaries of that ideology, then one does not fit in. There can be people who vary one way or another within ideological groups, but there comes a time when someone is actually outside the group. Libertarianism has purists, reformers, radicals, left-libertarians, right-libertarians, anarchists, moderate libertarians, but no socialists or fascists.
But in spite of it being obvious that there are those who are not libertarian, anyone can claim to be one. And sometimes those who oppose libertarianism choose to believe the claims in spite of the evidence otherwise.
Alan Greenspan has not been considered to be a libertarian by libertarians for at least two decades, possibly longer. No action taken during his tenure as Chairman of the Federal Reserve has been one that libertarians would approve of. Yet the fact that he was at one point in the past a member of Ayn Rand's inner circle and he blames the economic downturn on his allegedly free-market actions has given statists great cause to trumpet him not only as a libertarian but as one who has admitted that libertarian ideas do not work. Which libertarian ideas? He hasn't demonstrated any in years, unless the conspiracy theory is true.
He brought the economy of the United States to the Second Great Depression. He was once in Rand's inner circle. He claimed the downturn was the result of his libertarian beliefs. In spite of all the overwhelming evidence against him still being a libertarian, and in spite of the overwhelming majority of libertarians who say he is not one, he is considered one by the statists.
The objective is not to have an honest debate, but to attack libertarians through the failings of Greenspan. If the debate where honest the statist would have to admit Greenspan is not a libertarian.
Then there's Milton Friedman. He wasn't even an advocate of the free market in the first place, starting as a Keynesian before becoming a Monetarist. He was part of the Chicago School of economics, which is not a free market school. Yet many times when libertarians speak in defense of the free market, some statist will challenge libertarians to defend what Friedman did in Chile. Good or bad, his actions with regards to Pinochet cannot reflect for good or ill on libertarianism in any way.
Yet because Monetarism is less interventionist than Keynesianism, and in spite of the overwhelming evidence against it being free market, and in spite of the overwhelming majority of libertarians who say it is not, it is considered so by the statists.
Under Pinochet, people were not allowed to refuse to sell their labor. There's nothing libertarian about that. But the comparison is not made for the intent of honest debate but to shut down honest debate.
It seems that those who advocate most strongly for increased government power are the least willing to honestly engage libertarianism qua libertarianism, arguing against Alan Greenspan and MIlton Friedman instead.