调参¶
Note
利用回调函数，我们可以很方便地实现诸如EarlyStopping、模型自动保存之类的功能。
使用TensorBoard可以让我们更加直观地了解训练过程。
sklearn中的RandomizedSearchCV也可以用在tensorflow的调参中。
回调函数¶
回调函数默认在每个epoch之后被调用。
from tensorflow import keras
import utils
# 获得模型和数据
model = keras.models.load_model("my_california_housing_model")
(X_train, y_train), (X_val, y_val), (X_test, y_test) = utils.load_california_housing()
# 每个epoch后都会保存模型
checkpoint_cb = keras.callbacks.ModelCheckpoint("my_callback_housing")
# 在训练处定义回调函数
history = model.fit(X_train, y_train, epochs=3,
callbacks=[checkpoint_cb])
Epoch 1/3
363/363 [==============================] - 1s 656us/step - loss: 0.3341
INFO:tensorflow:Assets written to: my_callback_housing/assets
Epoch 2/3
363/363 [==============================] - 0s 640us/step - loss: 0.3284
INFO:tensorflow:Assets written to: my_callback_housing/assets
Epoch 3/3
363/363 [==============================] - 0s 641us/step - loss: 0.3274
INFO:tensorflow:Assets written to: my_callback_housing/assets
# save_best_only=True: 只保存在验证集中表现最好的模型
checkpoint_cb = keras.callbacks.ModelCheckpoint("my_callback_housing", save_best_only=True)
# EearlyStopping: patient个epoch之后指标还没有改进则停止训练，默认指标为`val_loss`
# restore_best_weights=True: 训练停止后保存指标最佳的epoch的参数
early_stopping_cb = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)
TensorBoard¶
# 记录信息的地址
run_logdir = "tensorboard_log"
# TensorBoard回调函数，需指定记录信息的地址
tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)
history = model.fit(X_train, y_train, epochs=5,
validation_data=(X_val, y_val),
callbacks=[tensorboard_cb])
Epoch 1/5
363/363 [==============================] - 0s 1ms/step - loss: 0.3175 - val_loss: 0.3151
Epoch 2/5
363/363 [==============================] - 0s 792us/step - loss: 0.3135 - val_loss: 0.3323
Epoch 3/5
363/363 [==============================] - 0s 798us/step - loss: 0.3180 - val_loss: 0.3134
Epoch 4/5
363/363 [==============================] - 0s 799us/step - loss: 0.3169 - val_loss: 0.3199
Epoch 5/5
363/363 [==============================] - 0s 799us/step - loss: 0.3103 - val_loss: 0.3031
运行以下语句后, 就可以在 http://localhost:6006/ 中查看TensorBoard了
tensorboard --logdir=./tensorboard_log --port=6006
调参¶
1.创建一个函数，参数为需要调的超参数，返回相应的模型¶
def build_model(n_hidden=1, n_neurons=30, learning_rate=3e-3, input_shape=[8]):
model = keras.models.Sequential()
model.add(keras.layers.InputLayer(input_shape=input_shape))
# n_hidden和n_neurons在此时起作用
for layer in range(n_hidden):
model.add(keras.layers.Dense(n_neurons, activation="relu"))
model.add(keras.layers.Dense(1))
# 学习率
optimizer = keras.optimizers.SGD(learning_rate=learning_rate)
model.compile(loss="mse", optimizer=optimizer)
return model
# wrapper之后就可以使用sklearn类型的api啦
keras_reg = keras.wrappers.scikit_learn.KerasRegressor(build_model)
2.定义RandomizedSearchCV对象并训练¶
from scipy.stats import reciprocal
from sklearn.model_selection import RandomizedSearchCV
# 候选超参数
param_distribs = {
"n_hidden": [1, 2],
"n_neurons": [10, 20, 30],
"learning_rate": reciprocal(3e-4, 3e-2).rvs(5).tolist(),
}
# 对参数采样2次，3-折交叉验证，找参数需训练2*3=6次，最后再在最佳参数上全量训练一次
rnd_search_cv = RandomizedSearchCV(keras_reg, param_distribs, n_iter=2, cv=3, verbose=0)
# 训练
rnd_search_cv.fit(X_train, y_train, epochs=3,
validation_data=(X_val, y_val),
callbacks=[keras.callbacks.EarlyStopping(patience=2)])
Epoch 1/3
242/242 [==============================] - 0s 1ms/step - loss: 0.8797 - val_loss: 0.5377
Epoch 2/3
242/242 [==============================] - 0s 778us/step - loss: 0.4876 - val_loss: 0.4484
Epoch 3/3
242/242 [==============================] - 0s 734us/step - loss: 0.4327 - val_loss: 0.4137
121/121 [==============================] - 0s 430us/step - loss: 0.3995
Epoch 1/3
242/242 [==============================] - 0s 986us/step - loss: 1.0059 - val_loss: 0.5978
Epoch 2/3
242/242 [==============================] - 0s 759us/step - loss: 0.5425 - val_loss: 0.5288
Epoch 3/3
242/242 [==============================] - 0s 729us/step - loss: 0.4734 - val_loss: 0.4738
121/121 [==============================] - 0s 426us/step - loss: 0.4922
Epoch 1/3
242/242 [==============================] - 0s 1ms/step - loss: 0.8062 - val_loss: 0.5525
Epoch 2/3
242/242 [==============================] - 0s 732us/step - loss: 0.5010 - val_loss: 0.4674
Epoch 3/3
242/242 [==============================] - 0s 747us/step - loss: 0.4572 - val_loss: 0.4449
121/121 [==============================] - 0s 435us/step - loss: 0.4346
Epoch 1/3
242/242 [==============================] - 0s 970us/step - loss: 0.8407 - val_loss: 0.4824
Epoch 2/3
242/242 [==============================] - 0s 765us/step - loss: 0.4548 - val_loss: 0.4436
Epoch 3/3
242/242 [==============================] - 0s 748us/step - loss: 0.4368 - val_loss: 0.4355
121/121 [==============================] - 0s 433us/step - loss: 0.4140
Epoch 1/3
242/242 [==============================] - 0s 1ms/step - loss: 1.1115 - val_loss: 0.6556
Epoch 2/3
242/242 [==============================] - 0s 740us/step - loss: 0.5744 - val_loss: 0.5627
Epoch 3/3
242/242 [==============================] - 0s 740us/step - loss: 0.5023 - val_loss: 0.5028
121/121 [==============================] - 0s 439us/step - loss: 0.5117
Epoch 1/3
242/242 [==============================] - 0s 1ms/step - loss: 0.7560 - val_loss: 0.5341
Epoch 2/3
242/242 [==============================] - 0s 760us/step - loss: 0.5063 - val_loss: 0.4830
Epoch 3/3
242/242 [==============================] - 0s 748us/step - loss: 0.4593 - val_loss: 0.4382
121/121 [==============================] - 0s 435us/step - loss: 0.4241
Epoch 1/3
363/363 [==============================] - 1s 870us/step - loss: 0.7060 - val_loss: 0.5281
Epoch 2/3
363/363 [==============================] - 0s 660us/step - loss: 0.4717 - val_loss: 0.4481
Epoch 3/3
363/363 [==============================] - 0s 672us/step - loss: 0.5252 - val_loss: 0.4190
RandomizedSearchCV(cv=3,
estimator=<keras.wrappers.scikit_learn.KerasRegressor object at 0x7ff612374580>,
n_iter=2,
param_distributions={'learning_rate': [0.01977815536482554,
0.005777126678456366,
0.00509614216000481,
0.0007066002665498963,
0.013545852545882942],
'n_hidden': [1, 2],
'n_neurons': [10, 20, 30]})
3.获取最佳参数和最佳模型¶
# 最佳参数
rnd_search_cv.best_params_
{'n_neurons': 20, 'n_hidden': 2, 'learning_rate': 0.013545852545882942}
# 最佳模型
model = rnd_search_cv.best_estimator_.model
model.evaluate(X_test, y_test)
162/162 [==============================] - 0s 433us/step - loss: 0.4789
0.4788932502269745