Allergies to cauliflower are extremely rare but have been documented. If one is allergic to cauliflower, they may also be allergic to other members of the Brassicaceae family such as broccoli, brussels sprouts, and cabbage. Symptoms of a cauliflower allergy include: Facial and hand swelling.