The scope of your question is more than we can respond to in a concise answer. There are so many elements you should consider. The first thing we would suggest is that you contact the Extension horse specialist in your state. That person would be employed by the state's land-grant university. This person can help you find reputable sellers and help you sort through and make decisions on the kind of horse, size, age, etc. Based on your comments, we would suggest a horse over 10 years old that is very broke and extremely quiet. There are some very good ponies out there, so don't let the stereotype of "ponies are mean" force you away from a pony.

The following are fact sheets concerning buying and evaluating horses. They will answer the majority of your questions.
ASC-118 Horse Judging Manual Animal Science

Locating and Buying the Right Horse

Evaluating Skeletal Structure of Horses

The following Web sites have general horse information.
University of Tennessee

University of Kentucky

North Carolina State University

Texas A&M University

Oklahoma State University