L1 regularization for feature selection in neural net In statistics, a lasso regression do some feature selection (or reduce the dimensionality of the problem). This is a very efficient technique as both the prediction and the feature selection use the same loss function. It is a "one step" optimization (as opposed to "two steps" where a preselection is done using a feature selection technique and a prediction is done then).
I would like to apply to NN a similar idea to lasso, if my first layer contains activation functions with L1 regularizations, I should be able to look at the weights and select the features that are the most relevant.
The general idea is to reduce the size of my input in production, if I train with 5000 features, I would like to use only the most relevant (max 50 features) in production (and achieve this selection using my neural network as it is linked to my prediction).
Any thought? Or recommendations on how to achieve this?

What you are describing is called pruning and there is many papers and tools (in PyTorch or TensorFlow) that will do that for you. Removing the parameters by hand would be a bad idea and could easily lead to breaking your model. There are many ways of doing it because it is a non-trivial problem. Notice that if you remove a parameter from the model, then same as with removing parameters from regression with $L_1$ penalty, you are affecting the whole model and all the other parameters, since they were estimated assuming that the parameter you are removing plays her role in the model. Saying it differently, the parameters are not independent of each other, so after removing any of the parameters you should re-train the model and have no guarantees it would work equally well as before.
