Dear DL researchers: how do you design your neural networks?
Genuine question,
how do you take some architectural decisions like the size of the neural network and the whole set of hyperparameters.
I get that there's brute forcing and hyperparameter search (which sometimes, really, it's a LOT), or some notes in literature regarding the choice of activations or loss based on context, but how would one really target some specific design choices when starting to explore efficiently, especially in terms of number of layers and latent space dimensions.
I appreciate your time, will take every tip into account