Yes, like all mammals, mice can taste the capsaicin and it’ll burn.

On a side note i have pet parrots and they have no receptors to register the capsaicin so those fuckers will sit there eating raw jalapeño and little red Thai peppers like it’s candy, they love the seeds. Then they come over later and give me spicy kisses