Hyperparameter
Think of a hyperparameter like the settings on a kitchen oven before you start baking; you choose the temperature and time yourself, rather than the oven figuring them out while the food cooks.
A configuration variable that dictates the structure or behavior of a machine learning model, which must be manually defined or tuned before the training process begins, as it cannot be derived from the training data.
A non-learnable parameter whose value is set prior to the commencement of the learning process to govern the model's architecture or the optimization algorithm's dynamics, thereby defining the hypothesis space and the objective function's traversal strategy.
evolution
- 1950 · historyOrigins of Parameterization
Alan Turing's 'Computing Machinery and Intelligence' introduces the concept of machine learning parameters, distinguishing between fixed settings and learned weights.
- 1986 · historyBackpropagation Formalization
Rumelhart, Hinton, and Williams popularize backpropagation, explicitly separating the architecture's structural settings from the weights optimized during training.
- 2011 · historyAutomated Hyperparameter Optimization
Bergstra et al. introduce Random Search as a more efficient alternative to Grid Search for navigating high-dimensional hyperparameter spaces.
- 2017 · historyBayesian Optimization Adoption
The rise of AutoML frameworks like Hyperopt and Optuna standardizes the use of Bayesian optimization for tuning complex deep learning architectures.