Model suggestion alternative to categorical regression I am developing a model framework in the context of online marketing: the dependent variable is purchase, the independent variables includes a mix of continuous (e.g., impressions, ad spend) and categorical variables (e.g., day of the week, channel).
One of the categorical variable is creative which has about 20 different designs. The goal of the model is to measure which design is most effective in terms of driving the purchase.
I know a standard procedure would be using a mixed variable regression which convert the creative variable into dummies.  The regression fit was not satisfying given the p value and R square. My suspect is the data does not meet the regression requirement's of normal distribution and the relationship is non-linear.
I have considered other modeling techniques such as SVM and RF. The challenge with those models are that in order to get an effect for each level in the creative variable, I would have to convert the creative variable into dummy variables. I am not sure if this is a plausible approach.
If anyone can point me to some model framework that would be great. Any suggestions would be greatly appreciated.
Thanks,
Yao

First, have you prepared some visualizations of your data? Plotting the data points should always be the first step, this helps to gain insight into your data. A trivial pair-plot (in R e.g. pairs()) could be a first step.
Next some comments on your approach with a mixed effect model: You say that you expect the relationship to be nonlinear, so I am wondering whether you have tried some nonlinear basis functions in your linear model? Your data visualizations might give you some clues as to what might work.
And if your errors are indeed not normal, there are generalized linear models (in R e.g. the glm function, glmm for mixed effects, or glmnet for regularization) that could help with this issue. Some GAM regression might be beneficial, too.
If you want to leave the realm of linear models for sure, there are many possibilities to consider next, e.g. some nonlinear fitting with nls() or some Bayesian approaches with e.g. rstan.
Once you tried all those and other model-driven approaches and you are not yet satisfied, you could explore those data-driven models like xgboost or random forest, as you have already mentioned. Deep neural networks also fall into this category. Given enough data, they are often very good at predicting but don't give much insight.
You mentioned concerns w.r.t. the one-hot encoding for SVM or RF, but I don't see any problems here, this is a plausible approach.
