SVMs operate by finding the optimal hyperplane that separates data points of different classes by the largest possible margin. This hyperplane is determined based on the "support vectors," which are the data points closest to the decision boundary. SVMs can also handle non-linear relationships by using kernel functions to transform the data into a higher-dimensional space where a linear separation is possible.