Yes. The potential for environmental impact is based on the amount of nitrogen (N) and phosphorus (P) deposited or applied per acre versus the N and P being utilized per acre by the pasture plants. A few horses on a small land base can produce more nutrients than can be utilized by the pasture.
Do not locate your manure compost or storage pile near ponds, streams, or your wellhead. Your horse farm (large or small) should be following best management practices to prevent any environmental impacts.