Several factors can influence binding affinity, including the chemical structure of the ligand, the conformation of the target protein, and environmental conditions like pH and temperature. Additionally, mutations in the target protein can alter affinity, which can lead to differences in susceptibility to toxins.