Estimation of GAMs involves finding the smooth functions that best fit the data while balancing the trade-off between fit and smoothness. This is typically done using iterative algorithms such as backfitting. The parameters are estimated by minimizing a penalized likelihood function, which includes both the likelihood of the data given the model and a penalty term for the smoothness of the functions.