There are many pests that can harm your chestnuts and set back your orchard significantly. Deer are certainly a problem, as are voles, insects and other exotic pathogens.

Penn State has a page devoted to pests.