Calcium oxalate is considered toxic due to its potential to cause irritation and injury when ingested or handled. The crystals can lodge in tissues, leading to mechanical irritation. Ingesting foods high in calcium oxalate can contribute to the formation of kidney stones, which are painful and can lead to further health complications. Additionally, some plants that contain calcium oxalate, such as dieffenbachia, can cause skin irritation upon contact.