While annatto is typically safe, certain individuals may experience allergic reactions or food intolerances. These reactions can range from mild symptoms such as skin rashes to more severe symptoms like difficulty breathing. Such cases are rare, but they highlight the importance of monitoring food additives in sensitive populations.