You do not indicate what variety of fig you are growing and whether it is planted in the ground or in a pot which is moved indoors during the winter. It is possible that the loss of early fruit is due to cold injury. The other possibility is that the variety you have selected requires pollination and what pollination that occurs is insufficient to set and mature the fruit.
You may want to refer to the Alabama Cooperative Extension System publication: Fig Production Guide.
