MichaelMMeskhi:

For example, I learn some weights and then given a new dataset, say I pass the input through the weights without activations, those are my new learned examples based on the learned representation of my network?


The activations will be the output of each layer, so unless I misunderstand the sentence, you won’t be able to pass the input through the model without creating activations.
If you are trying to create new “features”, you could use the penultimate activations and try to train a new classifier on them. This would be similar to the fine-tuning approach, where you would freeze all but the last layers and only train the last layer.
However, if you store these activations separately you are also free to use other ML models such as SVMs, RandomForest  etc.



 MichaelMMeskhi:

my goal is to let the neural network perform feature selection/engineering


I’m a bit confused about this sentence, as feature selection is used in the some ML approaches to select input features. E.g. a random forest classifier can be used to remove highly correlated or noisy features based on the entropy etc. of the splits.
However, it seems you would like to use feature selection or engineering “inside the model”?
Could you explain your use case a bit?