4% increase sounds good! I usually try Dropout first and then weight decay. But I don’t have an idea which one would work for your use case.
What kind of features do you have?
Are you normalizing the features?