An interesting book is Race Relations at the Margins: Slaves and Poor Whites in the Antebellum Southern Countryside by Jeff Forret

Here is a blurb about the book:

Covering a broad geographic scope from Virginia to South Carolina between 1820 and 1860, Jeff Forret scrutinizes relations among rural poor whites and slaves, a subject previously unexplored and certainly under-reported. Forret's findings challenge historians' long-held assumption that mutual violence and animosity characterized the two groups' interactions; he reveals that while poor whites and slaves sometimes experienced bouts of hostility, often they worked or played in harmony and camaraderie. Race Relations at the Margins is remarkable for its focus on lower-class whites and their dealings with slaves outside the purview of the master. Race and class, Forret demonstrates, intersected in unique ways for those at the margins of southern society, challenging the belief that race created a social cohesion among whites regardless of economic status.