some would argue that eating honey is not immoral

Sure.

The point of going vegetarian or vegan is to aim at a reduction of animal suffering and environmental footprint. Not to starve by choice. Agriculture is still better than more agriculture needed to consume meat.
What’s the problem exactly?