There's an estimated 5000-8000 white supremacists in the US. The largest party - National Alliance - claims 1500 members.
Antifa is a movement more than a group, and as such is much more diffuse and undefined. Those estimate range from 4000-175000. A good proxy is probably socialist party membership - the PSL has a membership of approx. 30,000.
But the nature of both groups means that membership tends to be very fluid (groups dissolve and reform, members belong to multiple groups or move from group to group.) 5000-8000 and 30,000 are hard numbers, but should be considered representative of order-of-magnitude estimates only (thus indicating that antifa and white supremacist numbers are roughly equivalent).
In truth, no one really knows.